A simple, flexible way to parallelize processing in Python.
Project description
Consumers is a simple, flexible way to parallelize processing in Python.
Examples
Distributing work accross separate consumer instances.
from consumers import Consumer, Queue
class SumConsumer(Consumer):
def initialize(self):
self.sum = 0
def process(self, num):
self.sum += num
def shutdown(self):
print('Sum', self.sum)
with Queue(SumConsumer) as queue:
for i in range(5):
queue.put(i)
A system with two virtual CPUs will output two results.
Sum 4 Sum 6
Orchestration
Orchestrating multiple types of consumers to achieve a single solution.
import logging
from consumers import Consumer, Queue
logging.basicConfig(level=logging.INFO)
class SquareConsumer(Consumer):
def initialize(self, sum_queue):
self.sum_queue = sum_queue
def process(self, num):
square = num * num
self.logger.info('Square of %d is %d', num, square)
self.sum_queue.put(square)
class SumConsumer(Consumer):
def initialize(self):
self.sum = 0
def process(self, num):
self.logger.info('Processing %s', num)
self.sum += num
def shutdown(self):
self.logger.info('Sum %d', self.sum)
sum_queue = Queue(SumConsumer, quantity=1)
square_queue = Queue(SquareConsumer(sum_queue), queues=[sum_queue])
with square_queue:
for i in range(5):
square_queue.put(i)
A system with two virtual CPUs will output log entries for two instances of SquareConsumer and one instance of SumConsumer.
INFO:SquareConsumer-2:Square of 1 is 1 INFO:SumConsumer-1:Processing 0 INFO:SumConsumer-1:Processing 1 INFO:SquareConsumer-1:Square of 2 is 4 INFO:SquareConsumer-2:Square of 3 is 9 INFO:SumConsumer-1:Processing 4 INFO:SumConsumer-1:Processing 9 INFO:SquareConsumer-1:Square of 4 is 16 INFO:SumConsumer-1:Processing 16 INFO:SumConsumer-1:Sum 30
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
consumers-0.2.0.tar.gz
(3.7 kB
view hashes)
Built Distribution
Close
Hashes for consumers-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82ab51836241d5b77d6b8b50d6b286ac199b45ca3ef0ffc3e308483653460b68 |
|
MD5 | a4b0e23abd84826e10ca7ae5f6da4756 |
|
BLAKE2b-256 | 6694de0bfe54bddb142cfeb9aaaa65b9e297c1ab5adc3a24f784e2aefe4a8b9f |