Skip to main content

Python library for dataflow programming with Amazon SWF

Project description

https://badge.fury.io/py/simpleflow.png https://travis-ci.org/botify-labs/simpleflow.png?branch=master https://pypip.in/d/simpleflow/badge.png

Simple Flow is a Python library that provides abstractions to write programs in the distributed dataflow paradigm. It relies on futures to describe the dependencies between tasks. It coordinates the execution of distributed tasks with Amazon SWF.

A Future object models the asynchronous execution of a computation that may end.

It tries to mimics the interface of the Python concurrent.futures library.

Features

  • Provides a Future abstraction to define dependencies between tasks.

  • Define asynchronous tasks from callables.

  • Handle workflows with Amazon SWF.

  • Implement replay behavior like the Amazon Flow framework.

  • Handle retry of tasks that failed.

  • Automatically register decorated tasks.

  • Handle the completion of a decision with more than 100 tasks.

  • Provides a local executor to check a workflow without Amazon SWF (see simpleflow --local command).

Quickstart

Let’s take a simple example that computes the result of (x + 1) * 2.

We need to declare the functions as activities to make them available:

from simpleflow import activity

@activity.with_attributes(task_list='quickstart')
def increment(x):
    return x + 1

@activity.with_attributes(task_list='quickstart')
def double(x):
    return x * 2

And then define the workflow itself in a example.py file:

from simpleflow import Workflow

class SimpleComputation(Workflow):
    def run(self, x):
        y = self.submit(increment, x)
        z = self.submit(double, y)
        return z.result

Now check that the workflow works locally:

$ simpleflow --local -w example.SimpleComputation -i example/input.json

The file example/input.json contains the input passed to the workflow. It should have the format:

{"args": [1],
 "kwargs": {}
}

which is equivalent to:

{"kwargs": {"x": 1}}

Documentation

Full documentation is available at https://simpleflow.readthedocs.org/.

Requirements

  • Python >= 2.6 or >= 3.3

License

MIT licensed. See the bundled LICENSE file for more details.

Changelog

0.1.0 (2014-02-19)

  • First release.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

simpleflow-0.1.0.tar.gz (5.6 kB view hashes)

Uploaded Source

Built Distribution

simpleflow-0.1.0-py2.py3-none-any.whl (6.4 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page