Streamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions.
Project description
Snapstream
Snapstream provides a data-flow model to simplify development of stateful streaming applications.
Installation
pip install snapstream
Usage
We snap
iterables to user functions, and process them in parallel when we call stream
:
We pass the callable print
to print out the return value. Multiple iterables and sinks can be passed.
from snapstream import snap, stream
@snap(range(5), sink=[print])
def handler(msg):
yield f'Hello {msg}'
stream()
Hello 0
Hello 1
Hello 2
Hello 3
Hello 4
To try it out for yourself, spin up a local kafka broker with docker-compose.yml, using localhost:29091
to connect:
docker compose up broker -d
Use the cli tool to inspect Topic/Cache:
snapstream topic emoji --offset -2
>>> timestamp: 2023-04-28T17:31:51.775000+00:00
>>> offset: 0
>>> key:
🏆
Features
snapstream.snap
: bind streams (iterables) and sinks (callables) to user defined handler functionssnapstream.stream
: start streamingsnapstream.Topic
: consume from (iterable), and produce to (callable) kafka using confluent-kafkasnapstream.Cache
: store data to disk using rocksdictsnapstream.Conf
: set global kafka configuration (can be overridden per topic)snapstream.codecs.AvroCodec
: serialize and deserialize avro messagessnapstream.codecs.JsonCodec
: serialize and deserialize json messages
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for snapstream-0.0.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d23897b1434d9563408209b29fd3839e17095f87aa9febce4fa4551a3a8bf635 |
|
MD5 | b4dab7995ed097e3eba5ec88f97ce466 |
|
BLAKE2b-256 | 1ee267a2724afe43ecd0f26e386053049a4e590f6dc9d3828b5f6c38d42d7b65 |