Data aggregation pipeline for running real-time predictive models
Project description
What is Blurr?
Blurr transforms structured, streaming raw data into features for model training and prediction using a high-level expressive YAML-based language called the Data Transform Configuration (DTC).
Blurr vs. stream/batch processors
The DTC is a data transform definition for structured data. The DTC encapsulates the business logic of data transforms and Blurr orchestrates the execution of data transforms.
Blurr is processor-agnostic, so DTCs can be run by event processors such as Spark.
- Blurr is to Spark as Hibernate is to databases
- Blurr can be used on Spark in the same way as SparkSQL
- Blurr is WORAIS (Write Once, Run on Any Infrastructure Stack)
Because real world infrastructure is extremely diverse, Blurr is designed to run on virtually any infrastructure stack that runs Python 3.6+.
Give us feedback on the metaphors and help improve Blurr!
The future of MLOps
We believe in a world where everyone is a data engineer. Or a data scientist. Or an ML engineer. The lines are blurred (cough). Just like development and operations became DevOps over time
--- Blurr authors
Blurr is a collection of components built for MLOps, the DTC is one of them. DTC ⊆ Blurr
We see a future where MLOps means teams putting together various technologies to suit their needs. For production ML applications, the speed of experimentation and iterations is the difference between success and failure. The DTC helps teams iterate on features faster. The vision for Blurr is to build MLOps components to help ML teams experiment at high speed.
Table of contents
Coming up with features is difficult, time-consuming, requires expert knowledge. 'Applied machine learning' is basically feature engineering
--- Andrew Ng
DTC at a glance
Raw data like this
{ "user_id": "09C1", "session_id": "915D", "country" : "US", "event_id": "game_start" }
{ "user_id": "09C1", "session_id": "915D", "country" : "US", "event_id": "game_end", "won": 1 }
{ "user_id": "09C1", "session_id": "915D", "country" : "US", "event_id": "game_start" }
{ "user_id": "09C1", "session_id": "915D", "country" : "US", "event_id": "game_end", "won": 1 }
{ "user_id": "B6FA", "session_id": "D043", "country" : "US", "event_id": "game_start" }
{ "user_id": "B6FA", "session_id": "D043", "country" : "US", "event_id": "game_end", "won": 1 }
{ "user_id": "09C1", "session_id": "T8KA", "country" : "UK", "event_id": "game_start" }
{ "user_id": "09C1", "session_id": "T8KA", "country" : "UK", "event_id": "game_end", "won": 1 }
turns into
| session_id | user_id | games_played | games_won |
|---|---|---|---|
| 915D | 09C1 | 2 | 2 |
| D043 | B6FA | 1 | 1 |
| T8KA | 09C1 | 1 | 1 |
using this DTC
Type: Blurr:Transform:Streaming
Version: '2018-03-01'
Name : sessions
Stores:
- Type: Blurr:Store:MemoryStore
Name: hello_world_store
Identity: source.user_id
Time: parser.parse(source.timestamp)
Aggregates:
- Type: Blurr:Aggregate:BlockAggregate
Name: session_stats
Store: hello_world_store
Split: source.session_id != session_stats.session_id
Fields:
- Name: session_id
Type: string
Value: source.session_id
- Name: games_played
Type: integer
When: source.event_id == 'game_start'
Value: session_stats.games_played + 1
- Name: games_won
Type: integer
When: source.event_id == 'game_end' and source.won == '1'
Value: session_stats.games_won + 1
Tutorial and Docs
Streaming DTC Tutorial | Window DTC Tutorial
Preparing data for specific use cases using Blurr
Dynamic in-game offers (Offer AI) | Frequently Bought Together
Use Blurr
We interact with Blurr using a Command Line Interface (CLI). Blurr is installed via pip:
$ pip install blurr
Transform data
$ blurr transform \
--streaming-dtc ./dtcs/sessionize-dtc.yml \
--window-dtc ./dtcs/windowing-dtc.yml \
--source file://path
Contribute to Blurr
Welcome to the Blurr community! We are so glad that you share our passion for making data management and machine learning accessible to everyone.
Please create a new issue to begin a discussion. Alternatively, feel free to pick up an existing issue!
Please sign the Contributor License Agreement before raising a pull request.
Data Science 'Joel Test'
Inspired by the (old school) Joel Test to rate software teams, here's our version for data science teams. What's your score? We'd love to know!
- Data pipelines are versioned and reproducible
- Pipelines (re)build in one step
- Deploying to production needs minimal engineering help
- Successful ML is a long game. You play it like it is
- Kaizen. Experimentation and iterations are a way of life
Roadmap
Blurr is currently in developer preview. Stay in touch!: Star this project or email hello@blurr.ai
Local transformations onlySupport for custom functions and other python libraries in the DTC- Spark runner
- S3-S3 data transformations
- Add DynamoDB as a Store
- Features server
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for blurr_dev-0.699-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | ee0652151413495aa5797b4decb331609a1b94a59665b5ddf2101af21541cf6f |
|
| MD5 | 38debc84a0d69152fc76f0ace21d78dd |
|
| BLAKE2b-256 | 858131557b04453e76eefa7e00ca3aea446bbe5b7ccc2762f70e295e35d940e7 |
