Skip to main content

Priority queue with task categorisation support

Project description

Priority queue with task categorisation support

Overview

taskq is a heap based priority queue that adds support to partition tasks into categories for selective removal.

The main driver for the project was work/task queueing where jobs need to be distributed to workers with different capabilities and non-uniform processing times (for calculating efficient prefetch).

This library is not thread-safe and is not intended as a replacement for python’s queue.Queue class.

Getting Started

Install using pip:

pip install taskq

Basic FIFO usage:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1')
>>> q.push('task2')
>>> q.pop()
'task1'
>>> q.pop()
'task2'

Task priorities (complex types that define __cmp__ are also supported):

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1', 2)
>>> q.push('task2', 1)
>>> q.pop()
'task2'
>>> q.pop()
'task1'

Multi-pop:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1')
>>> q.push('task2')
>>> q.push('task3')
>>> q.pop(2)
['task1', 'task2']

Task categories:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1', category='foo')
>>> q.push('task2', category='bar')
>>> q.push('task3', category='foo')
>>> q.pop(2, categories=['dog', 'foo'])
['task1', 'task2']

Category ratios:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1', category='foo')
>>> q.push('task2', category='bar')
>>> q.push('task3', category='foo')
>>> q.pop(2, categories=['bar', 'foo'], ratios=[1, 0.5])
['task1', 'task2', 'task3']

The above example is useful when prefetching/distributing tasks that have non-uniform processing durations. If the category ratios represent average duration in seconds, the count can be used to pull approximately count seconds worth of the higest priority tasks from the queue.

See source documentation for other feature examples.

Issues

Source code for taskq is hosted on GitHub. Any bug reports or feature requests can be made using GitHub’s issues system.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

taskq-0.1.tar.gz (16.6 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page