Streamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions.
Project description
Snapstream
A tiny data-flow model with a user-friendly interface that provides sensible defaults for Kafka integration, message serialization/deserialization, and data caching.
Installation
pip install snapstream
Usage
We snap
iterables to user functions, and process them in parallel when we call stream
:
We pass the callable print
to print out the return value. Multiple iterables and sinks can be passed.
from snapstream import snap, stream
@snap(range(5), sink=[print])
def handler(msg):
return f'Hello {msg}'
stream()
Hello 0
Hello 1
Hello 2
Hello 3
Hello 4
To try it out for yourself, spin up a local kafka broker with docker-compose.yml, using localhost:29091
to connect:
docker compose up broker -d
Features
snapstream.snap
: bind streams (iterables) and sinks (callables) to user defined handler functionssnapstream.stream
: start streamingsnapstream.Topic
: consume from (iterable) and produce to (callable) kafka using confluent-kafkasnapstream.Cache
: store data to disk using rocksdictsnapstream.Conf
: set global kafka configuration (can be overridden per topic)snapstream.codecs.AvroCodec
: serialize and deserialize avro messagessnapstream.codecs.JsonCodec
: serialize and deserialize json messages
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for snapstream-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a987bc1b2e542cc8315c62b96e655ac428dfd403cd64709be7e97eef341dccea |
|
MD5 | c86f83083accd8b38e70ca5439e2a05b |
|
BLAKE2b-256 | daf788069802f6e462ca9b6f5309099b3dc1c04054dceb219ef8e07f57b9d6e7 |