skip to navigation
skip to content

link.parallel 1.1

Parallel computing framework

link.parallel is a database agnostic query system.

See documentation for more informations.

Installation

pip install link.parallel

Features

  • parallel loops interface with IPython and multiprocessing drivers
  • Map/Reduce middleware

Examples

Create your map/reduce functions:

from b3j0f.task import register_task


@register_task('mymap')
def mymap(mapper, item):
    if item['i'] < 5:
        mapper.emit('i_lt_5', item)

    elif item['i'] > 5:
        mapper.emit('i_gt_5', item)

    else:
        mapper.emit('i_eq_5', item)


@register_task('myreduce')
def myreduce(key, values):
    return (key, len(values))

Get input data and pass it to the middleware:

from link.middleware.core import Middleware

mapreduce = Middleware.get_middleware_by_uri(
    'mapreduce+ipython:///test/classify?mapcb=mymap&reducecb=myreduce'
)

items = # load items
result = dict(mapreduce(items))

print(result)

Donating

 
File Type Py Version Uploaded on Size
link.parallel-1.1-py2.py3-none-any.whl (md5) Python Wheel 3.4 2016-09-15 12KB
link.parallel-1.1.tar.gz (md5) Source 2016-09-15 6KB