Skip to main content

Distribute commands to many nodes

Project description

cooperate is a shell command that execute commands in a cooperative manner, by distributing them to many nodes.

It requires Python >= 3.3 and asyncio.

For example:

cooperate --local -- echo FOO

Will execute the job “echo FOO” locally.

This one:

cooperate --ssh me@my.node -- echo FOO

Is barelly equivalent to:

ssh me@my.node echo FOO

You can declare as many nodes as you want. For example:

cooperate --local --ssh me@my.node --ssh me@my.second.node -- echo FOO

Is equivalent to:

echo FOO
ssh me@my.node echo FOO
ssh me@my.second.node echo FOO

You can also declare many jobs at once. For example:

cooperate --local --command "echo FOO" --command "echo BAR"

Is equivalent to:

echo FOO
echo BAR

Installation

pip install cooperate

Nodes

Commands can be distribued thru these kind of nodes:

  • –local execute locally

  • –ssh execute thru ssh

  • –docker execute in a local docker container

  • –lxc execute in a local lxc container

These options can be repeated as often as needed.

Modes

By default, it spawns every commands to every nodes.

The -m, –mode allow to configure the desired mode.

The all mode executes all commands in all nodes:

cooperate --local --ssh me@my.node --ssh me@my.second.node \
    --command="echo FOO" --command="echo BAR"

Is equivalent to:

echo FOO
echo BAR
ssh me@my.second.node echo FOO
ssh me@my.second.node echo BAR

The distribute mode share the commands out among all the nodes:

cooperate --local --ssh me@my.node --ssh me@my.second.node \
    --command="echo FOO" --command="echo BAR" --mode=distribute

Is equivalent to:

echo FOO
ssh me@my.second.node echo BAR

Concurrency

By default, it executes all jobs simultaneously.

The -b, –batch option allows to execute on only a specify number of jobs at a time. Both percentages and finite numbers are supported:

cooperate --local --concurrence 1 \
    --command="echo FOO" --command="echo BAR" --command="echo BAZ"

cooperate --local --concurrence 33% \
    --command="echo FOO" --command="echo BAR" --command="echo BAZ"

The concurrency system maintains a window of running jobs. When a job returns then it starts a remnant job and so on.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cooperate-0.3.1.tar.gz (5.5 kB view hashes)

Uploaded Source

Built Distribution

cooperate-0.3.1-py3-none-any.whl (9.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page