SQS Task Master for unwieldy python data processing by Upserve
Project description
SQS Task Master is one of many packages for distributed asynchronous task execution in python. This package is different in that it is designed to work with uncooperative tasks. It is designed for CPU intensive tasks which will not work in a python cooperative threading model. It allows for tasks that may take upto 12 hours (the limit for SQS). It also provides for applying a hard timeout for a task, less than 12 hours. The implementation relies on using Singals to achieve these behaviors. Retries and failures are handled using SQS infrastructure. It is designed to work with standard SQS queues which guarantee at least once delivery.
Development
Git clone the respository:
git clone git@github.com:upserve/sqstaskmaster.git
Pip install the development dependencies in a virtual environment:
pip install -e .[dev]
Run unit tests:
python -m unittest -v
Run flake8:
flake8 .
Run black:
black .
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for sqstaskmaster-0.0.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d4ed0a0f24619a246cf33d1815ed1bef454a2f618abe28386ea7fb873ffa2569 |
|
| MD5 | df5828a5f3241f6e43ad8f55582e7ca3 |
|
| BLAKE2b-256 | f500fa59f8d3d34b3afdec2b532d894834339f1440fd9540384c7f4e520e418a |
