Skip to main content

A minimalist tool to run multiple parallel tasks in Python

Project description

This is a minimalist tool to run multiple parallel tasks in python.

Most programming languages have full support for threads but often require a lot of overhead work even for the simplest tasks. This package aims to provide an easy way to parallelize these tasks with very little effort.

Getting Started

First install the package.

pip install mparallel

In your python modules, just import it and use it as follows:

import time
import mparallel

def some_expensive_or_waiting_task(some_param):
  # ...
  time.sleep(2)
  return some_param

def my_method():
  runner = mparallel.Runner()
  for i in range(10):
    runner.add_task(some_expensive_or_waiting_task, i)
  print runner.results()

You can see the tasks are being run in parallel from the previous code because even though they are being started in order (0..9), the final output will likely appear in a different order. Also, the total waiting time will be less than 20 seconds, which is the time it would take to serially run them.

Project details


Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page