Skip to main content

Python library for dataflow programming with Amazon SWF

Project description

Simpleflow
==========

<p class=badges>
[![Pypi Status](https://badge.fury.io/py/simpleflow.png)](http://badge.fury.io/py/simpleflow) [![Build Status](https://travis-ci.org/botify-labs/simpleflow.svg?branch=master)](https://travis-ci.org/botify-labs/simpleflow)
</p>

Simpleflow is a Python library that provides abstractions to write programs in
the [distributed dataflow paradigm](https://en.wikipedia.org/wiki/Distributed_data_flow).
It coordinates the execution of distributed tasks with [Amazon SWF](https://aws.amazon.com/swf/).

It relies on *futures* to describe the dependencies between tasks. A `Future` object
models the asynchronous execution of a computation that may end. It tries to mimic
the interface of the Python [concurrent.futures](http://docs.python.org/3/library/concurrent.futures) library.


Features
--------

- Provides a `Future` abstraction to define dependencies between tasks.
- Define asynchronous tasks from callables.
- Handle workflows with Amazon SWF.
- Implement replay behavior like the Amazon Flow framework.
- Handle retry of tasks that failed.
- Automatically register decorated tasks.
- Encodes/decodes large fields to S3 objects transparently (aka "jumbo fields").
- Handle the completion of a decision with more than 100 tasks.
- Provides a local executor to check a workflow without Amazon SWF (see
`simpleflow --local` command).
- Provides decider and activity worker process for execution with Amazon SWF.
- Ships with the `simpleflow` command. `simpleflow --help` for more information
about the commands it supports.

You can read more in the **Features** section of the documentation.


Overview
--------

Please read and even run the `demo` script to have a quick glance of
`simpleflow` commands. To run the `demo` you will need to start decider
and activity worker processes.

Start a decider with::

$ simpleflow decider.start --domain TestDomain --task-list test examples.basic.BasicWorkflow

Start an activity worker with::

$ simpleflow worker.start --domain TestDomain --task-list quickstart

Then execute `./extras/demo`.


More informations
-----------------

Read the main documentation at https://botify-labs.github.io/simpleflow/.


Changelog
=========

0.20.2
------

- add meta to metrology, upload stats only at the end of a task (#324)

0.20.1
------

- inject context into python subprocess (#322)

0.20.0
------

- Feature/318/simpleflow download binary (#321)

0.19.2
------

- Slow jumbo cache fixes (#320)

0.19.1
------



0.19.0
------

- Kubernetes integration (#313)

0.18.15
-------

- Bugfix: propagate signal (#312)
- Enhancement: inherit tag list (#314)
- * blank SWF decision execution context when needed
* rename ambiguous "execution_context" to "run_context" (#310)

0.18.14
-------

- Update the link to the documentation (#306)
- Fork on each decision task to protect against memory leaks (#200) (#308)
- Don't truncate too long fields, raise instead (closes #307) (#309)

0.18.13
-------

- Fix diskcache OperationalError (#303)

0.18.12
-------

- Enhancement: save waiting_signals in the execution context (#300)
- Mark flaky tests as expected to fail (#301)

0.18.11
-------

- Allow Workflow instances in Group (#299)

0.18.10
-------

- Don't raise when ThrottlingException occur on RecordActivityTaskHeartbeat endpoint (#297)

0.18.9
------

- Fix activity.rerun not working with class based tasks (#289)
- Add a new option (and parameter) --kill-children (#292)

0.18.8
------

- Move workers cleanup/start outside SIGCHLD handler (#290)

0.18.7
------

- Fix MANIFEST.in so README.md is included in final package

0.18.6
------

- Add a new timeout parameter (#286)

0.18.5
------

- Documentation overhaul (#284)
- Add a release script (closes #179) (#287)

0.18.4
------

- Improve process stopping (#283)

0.18.3
------

- Enhancement/276/improve execute python (#280)

0.18.2
------

- Bugfix: task failed details (#281)
- Add sets support to json_dumps (#275)

0.18.1
------

- Add back get_workflow_history


0.18.0
------

- Implement "jumbo" fields (#265)

0.17.0
------

- Enhancement/272/implement workflow cancelation (#273)
- Bugfix: 270: signals improvements (#271)
- Enhancement: timer: get_event_details (#269)
- Append "/" to get_step_path_prefix (#268)
- Enhancement/misc (#266)
- Repair reruns successful child workflows (#191)

0.16.0
------

- Feature: timers (#258)

0.15.7
------

- Kill worker on UnknownResourceFault's during a heartbeat (#88) (#263)
- Sort keys by default in json_dumps (#264)

0.15.6
------

- Fix step attribute propagation (#261)
- Enhancement: get_event_details (#235)

0.15.5
------

- Enhancement: distinguish raises_on_failure between tasks and groups (#255)
- Add time constants
- Relax activity.with_attributes timeouts types

0.15.4
------

- Enhancement: add canvas option break_on_failure (#253)
- Compute task_id from ActivityTask if has get_task_id method (#237)
- Another case of wrong task list (#234)

0.15.3
------

- make raises_on_failure=True on step activities (#249)
- SWF: support for non-Python tasks (#219)
- Fix get_step_path_prefix
- Make MarkerTask's idempotents

0.15.2
------

- mark when a step is scheduled before it's executed (#243)

0.15.1
------

- Enhancement: better activity type reason (#238)
- Fix workers not catching errors during dispatch() step (#246)
- Fix canvas.Chain send_result regression (#247)

0.15.0
------

- Feature: steps (#221)
- Make activity task result optional (#225)
- Use details in addition to name to find markers (#227)
- Logging: add exception information (#163)
- swf/actors: support 'Message' key (#224)
- Implement markers (#216) (#217)
- Add retry on swf.process.Poller.poll and fail (#208)

0.14.2
------

- propagate_attribute: skip signal objects (#215)
- Local executor: check add_activity_task (#215)

0.14.1
------

- Don't send exception up if raises_on_failure is false (#213)
- Fix UnicodeDecodeError on windows machine (#211)
- Try to use less memory (#209)
- Standalone mode: use created task list for children activities (#207)

0.14.0
------

- Fix workers not stopping in case they start during a shutdown (#205)
- Add support for SWF signals (#188)
- Improvements on canvas.Group (#204)

0.13.4
------

- Implement metrology on SWF and local workflows (#186)

0.13.3
------

- Try..except pass for NoSuchProcess (#182)

0.13.2
------

- Add optional canvas (#193)
- Reorganize tests/ directory (#198)
- Relax DeciderPoller task list check (#201)
- Implement priorities on SWF tasks (#199)

0.13.1
------

- Fix SWF executor not accepting ActivityTask's in submit() method (#196)

0.13.0
------

- Implement child workflow (#74)
- Don't schedule idempotent tasks multiple times (#107)
- Child workflow ID: use parent's id to generate

0.12.7
------

- Control SWF processes identity via environment (#184)

0.12.6
------

- Replace `execution` object with a more flexible `get_execution_method()` (#177)
- Fix README_SWF.rst format (#175)
- Fix CONTRIBUTING.rst format
- docs/conf.py: remove relative import

0.12.5
------

- Executor: expose workflow execution (#172)

0.12.4
------

- Avoid returning too big responses to RespondDecisionTaskCompleted endpoint (#166)
- Worker: remove useless monitor_child (#168)

0.12.3
------

- Add max_parallel option in Group (#164)

0.12.2
------

- Make the dynamic dispatcher more flexible (#161)
- Fix README.rst format (#160)
- Tiny command-line usability fixes (#158)

0.12.1
------

- Don't override passed "default" in json_dumps() (#155)
- Expose activity context (#156)

0.12.0
------

- Improve process management (#142)

0.11.17
-------

- Don't reap children in the back of multiprocessing (#141)
- Don't force to pass a workflow to activity workers (#133)
- Don't override the task list if not standalone (#139)
- Split FuncGroup submit (#146)
- CI: Test on python 3 (#144)
- Decider: use workflow's task list if unset (#148)

0.11.16
-------

- Refactor: cleanups and many python 3 compatibility issues fixed (#135)
- Introduce AggregationException to inspect exceptions inside canvas.Group/Chain (#92)
- Improve heartbeating, now enabled by default on activity workers (#136)

0.11.15
-------

- Fix tag_list declaration in case no tag is associated with the workflow
- Fix listing workflow tasks not handling "scheduled" (not started) tasks correctly
- Fix CSV formatter outputing an extra "None" at the end of the output
- Fix 'simpleflow activity.rerun' resolving the bad function name if not the last event

0.11.14
-------

- Various little fixes around process management, heartbeat, logging (#110)

0.11.13
-------

- Add ability to provide a 'run ID' with 'simpleflow standalone --repair'

0.11.12
-------

- Fix --tags argument for simpleflow standalone (#114)
- Improve tests and add integration tests (#116)
- Add 'simpleflow activity.rerun' command (#117)

0.11.11
-------

- Fix a circular import on simpleflow.swf.executor

0.11.10
-------

- Fix previous_history initialization (#106)
- Improve WorkflowExecutionQueryset default date values (#111)

0.11.9
------

- Add a --repair option to simpleflow standalone (#100)

0.11.8
------

- Retry boto.swf connection to avoid frequent errors when using IAM roles (#99)

0.11.7
------

Same as 0.11.6 but the 0.11.6 on pypi is broken (pushed something similar to 0.11.5 by mistake)

0.11.6
------

- Add `issubclass_` method (#96)
- Avoid duplicate logs if root logger has an handler (#97)
- Allow passing SWF domain via the SWF_DOMAIN environment variable (#98)

0.11.5
------

- Don't mask activity cancel exception (#84)
- Propagate all decision response attributes up to Executor.replay() (#76, #94)

0.11.4
------

- ISO dates in workflow history #91
- Fix potential infinite retry loop #90

0.11.3
------

- Fix replay hooks introduced in 0.11.2 (#86)
- Remove python3 compatibility from README (which was not working for a long time)

0.11.2
------

- Add new workflow hooks (#79)

0.11.1
------

- Fix logging when an exception occurs

0.11.0
------

- Merge `swf` package into simplefow for easier maintenance.


0.10.4 and below
----------------

Sorry changes were not documented for simpleflow <= 0.10.x.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

simpleflow-0.20.2.tar.gz (134.6 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page