Streamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions.
Project description
Snapstream
An easy to use, extensible data-flow model, providing sensible defaults to produce and consume to and from kafka, serialize/deserialize messages, and cache large amounts of data.
Installation
pip install snapstream
Usage
In the example below, snap decorates the handle function, binding the iterable range(5) to it:
from snapstream import snap, stream
@snap(range(5))
def handle(msg):
print('Greetings', msg)
stream()
➜ python main.py
Greetings 0
Greetings 1
Greetings 2
Greetings 3
Greetings 4
Features
snapstream.Topic: consume from topic (iterable), produce to topic (callable), uses confluent-kafka by defaultsnapstream.Cache: persist data, uses rocksdictsnapstream.Conf: configuration singleton class, manages threads for all streamssnapstream.stream: start streamssnapstream.snap: bind streams (iterable) and sinks (callable) to handler functions
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
snapstream-0.0.0.dev2.tar.gz
(6.6 kB
view hashes)
Built Distribution
Close
Hashes for snapstream-0.0.0.dev2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f31db9e90d5f76d87ec06a15a4ae96b73afca25157a0bda9de0551b05828beb7 |
|
| MD5 | d09e5542a1b220ea334d10c41bb4fc86 |
|
| BLAKE2b-256 | cd6f6459a56e22be05b23032fd136c50f05431e3875fdf2f928fdc6b1f7681c9 |
