skip to navigation
skip to content

parmap 1.4.0

map and starmap implementations passing additional arguments and parallelizing if possible

This small python module implements two functions: map and starmap.

What does parmap offer?

  • Provide an easy to use syntax for both map and starmap.
  • Parallelize transparently whenever possible.
  • Handle multiple (positional -for now-) arguments as needed.

Installation:

pip install parmap

Usage:

Here are some examples with some unparallelized code parallelized with parmap:

import parmap
# You want to do:
y = [myfunction(x, argument1, argument2) for x in mylist]
# In parallel:
y = parmap.map(myfunction, mylist, argument1, argument2)

# You want to do:
z = [myfunction(x, y, argument1, argument2) for (x,y) in mylist]
# In parallel:
z = parmap.starmap(myfunction, mylist, argument1, argument2)

# You want to do:
listx = [1, 2, 3, 4, 5, 6]
listy = [2, 3, 4, 5, 6, 7]
param = 3.14
param2 = 42
listz = []
for (x, y) in zip(listx, listy):
    listz.append(myfunction(x, y, param1, param2))
# In parallel:
listz = parmap.starmap(myfunction, zip(listx, listy), param1, param2)

map (and starmap on python 3.3) already exist. Why reinvent the wheel?

Please correct me if I am wrong, but from my point of view, existing functions have some usability limitations:

  • The built-in python function map [1] is not able to parallelize.
  • multiprocessing.Pool().starmap [2] is only available in python-3.3 and later versions.
  • multiprocessing.Pool().map [3] does not allow any additional argument to the mapped function.
  • multiprocessing.Pool().starmap allows passing multiple arguments, but in order to pass a constant argument to the mapped function you will need to convert it to an iterator using itertools.repeat(your_parameter) [4]

parmap aims to overcome this limitations in the simplest possible way.

Additional features in parmap:

  • Create a pool for parallel computation automatically if possible.
  • parmap.map(..., ..., parallel=False) # disables parallelization
  • parmap.map(..., ..., chunksize=3) # size of chunks (see multiprocessing.Pool().map)
  • parmap.map(..., ..., pool=multiprocessing.Pool()) # use an existing pool, in this case parmap will not close the pool.

To do:

Pull requests and suggestions are welcome.

  • Pass keyword arguments to functions?

Acknowledgments:

The original idea for this implementation was given by J.F. Sebastian. I just provided an alternative answer implementing it in a package.

Known works using parmap

  • Davide Gerosa, Michael Kesden, “PRECESSION. Dynamics of spinning black-hole binaries with python.” arXiv:1605.01067, 2016
 
File Type Py Version Uploaded on Size
parmap-1.4.0-py2.py3-none-any.whl (md5) Python Wheel py2.py3 2017-07-19 7KB
parmap-1.4.0.tar.gz (md5, pgp) Source 2017-07-19 18KB