Utility which provides the ability to distribute a task across a fixed number of processes, for better utilization of multiprocessing
Project description
Disttask is a utility which provides the ability to distribute a task across a fixed number of processes, for better utilization of multiprocessing.
Use it with existing single-threaded/process tools and scripts to take full advantage of your computer’s resources.
Usage: ./disttask [cmd] [concurrent tasks] [argset]Use a %s in [cmd] where you want the args to go. use %d for the pipe number.
The application runs at most “concurrent task” # of processes ( good to match total number of processors available to this number - 1 ). It captures stdout and stderr to ensure that any output is not intertwined between the applications.
Each command should specify a “%s” to where each argument will go (one argument from @argset per application). “%d” is also available as the pipe number, but may not be very useful.
Example:
./disttask “echo %d %s” 3 “this” “is” “some” “text” “blah” “whooptie” “Doo”0 this1 is2 some0 text1 blah2 whooptie0 Doo
Another Example:
Run pyflakes, using 10 simultanious processes, on all python files in subdirectories (requires shopt -s globstar. Notice the backticks, not single-quotes.)
./disttask “pyflakes %s” 10 echo **/*.py
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.