Skip to main content

Data aggregation pipeline for running real-time predictive models

Project description

Blurr

We believe in a world where everyone is a data engineer. Or a data scientist. Or an ML engineer. The lines are blurred (cough). Just like development and operations became DevOps over time

--- Blurr authors

CircleCI Documentation Status Coverage Status PyPI version Binder

What is Blurr?

Blurr transforms raw data into features for model training and prediction using a high-level expressive YAML-based language called the Data Transform Configuration (DTC).

For production ML applications, experimentation and iteration speed is important. Working directly with raw data provides the most flexibility. Blurr allows product teams to iterate quickly during ML dev and provides a self-service way to take experiments to production.

How is Blurr different from Spark/Kafka?

Blurr is a data transform definition. (Technically - the DTC contains the definition, or descriptor)

Blurr is processor-agnostic, so DTCs can be run by stream processors like Spark or Kafka. Because real world infrastructure is extremely diverse, Blurr is designed to run on virtually any infrastructure stack.

Table of contents

Coming up with features is difficult, time-consuming, requires expert knowledge. 'Applied machine learning' is basically feature engineering

--- Andrew Ng

DTC at a glance

Raw data like this

{ "user_id": "09C1", "session_id": "915D", "country" : "US", "event_id": "game_start" }
{ "user_id": "09C1", "session_id": "915D", "country" : "US", "event_id": "game_end", "won": 1 }
{ "user_id": "09C1", "session_id": "915D", "country" : "US", "event_id": "game_start" }
{ "user_id": "09C1", "session_id": "915D", "country" : "US", "event_id": "game_end", "won": 1 }
{ "user_id": "B6FA", "session_id": "D043", "country" : "US", "event_id": "game_start" }
{ "user_id": "B6FA", "session_id": "D043", "country" : "US", "event_id": "game_end", "won": 1 }
{ "user_id": "09C1", "session_id": "T8KA", "country" : "UK", "event_id": "game_start" }
{ "user_id": "09C1", "session_id": "T8KA", "country" : "UK", "event_id": "game_end", "won": 1 }

turns into

session_id user_id games_played games_won
915D 09C1 2 2
D043 B6FA 1 1
T8KA 09C1 1 1

using this DTC

Type: Blurr:Transform:Streaming
Version: '2018-03-01'
Name : sessions

Stores:
   - Type: Blurr:Store:MemoryStore
     Name: hello_world_store

Identity: source.user_id

Time: parser.parse(source.timestamp)

Aggregates:

 - Type: Blurr:Aggregate:BlockAggregate
   Name: session_stats
   Store: hello_world_store

   Split: source.session_id != session_stats.session_id

   Fields:

     - Name: session_id
       Type: string
       Value: source.session_id

     - Name: games_played
       Type: integer
       When: source.event_id == 'game_start'
       Value: session_stats.games_played + 1

     - Name: games_won
       Type: integer
       When: source.event_id == 'game_end' and source.won == '1'
       Value: session_stats.games_won + 1

Tutorial and Docs

Read the docs

Streaming DTC Tutorial | Window DTC Tutorial

Preparing data for specific use cases using Blurr

Dynamic in-game offers (Offer AI) | Frequently Bought Together

Use Blurr

We interact with Blurr using a Command Line Interface (CLI). Blurr is installed via pip:

$ pip install blurr

Transform data

$ blurr transform \
     --streaming-dtc ./dtcs/sessionize-dtc.yml \
     --window-dtc ./dtcs/windowing-dtc.yml \
     --source file://path

CLI documentation

Contribute to Blurr

Welcome to the Blurr community! We are so glad that you share our passion for making data management and machine learning accessible to everyone.

Please create a new issue to begin a discussion. Alternatively, feel free to pick up an existing issue!

Please sign the Contributor License Agreement before raising a pull request.

Data Science 'Joel Test'

Inspired by the (old school) Joel Test to rate software teams, here's our version for data science teams. What's your score? We'd love to know!

  1. Data pipelines are versioned and reproducible
  2. Pipelines (re)build in one step
  3. Deploying to production needs minimal engineering help
  4. Successful ML is a long game. You play it like it is
  5. Kaizen. Experimentation and iterations are a way of life

Stay in touch! Star this project or email hello@blurr.ai

Roadmap

Blurr is all about enabling machine learning and AI teams to run faster.

Developer Preview 0: Local transformations only

Developer Preview 1: S3-S3 data transformations

Developer Preview 2: Add DynamoDB as a Store + Features server for ML production use

Ingestion connectors to Kafka and Spark

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blurr-dev-0.673.tar.gz (30.7 kB view hashes)

Uploaded Source

Built Distribution

blurr_dev-0.673-py3-none-any.whl (43.1 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page