daft 0.4.7
pip install daft
Released:
Distributed Dataframes for Multimodal Data
Navigation
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Unverified details
These details have not been verified by PyPIProject links
Meta
- Author: Eventual Inc
- Maintainer: Sammy Sidhu
- Requires: Python >=3.9
-
Provides-Extra:
all
,aws
,azure
,deltalake
,gcp
,hudi
,iceberg
,lance
,numpy
,pandas
,ray
,sql
,unity
,viz
Project description
Website • Docs • Installation • Daft Quickstart • Community and Support
Daft: Unified Engine for Data Analytics, Engineering & ML/AI
Daft is a distributed query engine for large-scale data processing using Python or SQL, implemented in Rust.
Familiar interactive API: Lazy Python Dataframe for rapid and interactive iteration, or SQL for analytical queries
Focus on the what: Powerful Query Optimizer that rewrites queries to be as efficient as possible
Data Catalog integrations: Full integration with data catalogs such as Apache Iceberg
Rich multimodal type-system: Supports multimodal types such as Images, URLs, Tensors and more
Seamless Interchange: Built on the Apache Arrow In-Memory Format
Built for the cloud: Record-setting I/O performance for integrations with S3 cloud storage
Table of Contents
About Daft
Daft was designed with the following principles in mind:
Any Data: Beyond the usual strings/numbers/dates, Daft columns can also hold complex or nested multimodal data such as Images, Embeddings and Python objects efficiently with it’s Arrow based memory representation. Ingestion and basic transformations of multimodal data is extremely easy and performant in Daft.
Interactive Computing: Daft is built for the interactive developer experience through notebooks or REPLs - intelligent caching/query optimizations accelerates your experimentation and data exploration.
Distributed Computing: Some workloads can quickly outgrow your local laptop’s computational resources - Daft integrates natively with Ray for running dataframes on large clusters of machines with thousands of CPUs/GPUs.
Getting Started
Installation
Install Daft with pip install getdaft.
For more advanced installations (e.g. installing from source or with extra dependencies such as Ray and AWS utilities), please see our Installation Guide
Quickstart
Check out our quickstart!
In this example, we load images from an AWS S3 bucket’s URLs and resize each image in the dataframe:
import daft
# Load a dataframe from filepaths in an S3 bucket
df = daft.from_glob_path("s3://daft-public-data/laion-sample-images/*")
# 1. Download column of image URLs as a column of bytes
# 2. Decode the column of bytes into a column of images
df = df.with_column("image", df["path"].url.download().image.decode())
# Resize each image into 32x32
df = df.with_column("resized", df["image"].image.resize(32, 32))
df.show(3)
Benchmarks
To see the full benchmarks, detailed setup, and logs, check out our benchmarking page.
More Resources
Daft Quickstart - learn more about Daft’s full range of capabilities including dataloading from URLs, joins, user-defined functions (UDF), groupby, aggregations and more.
User Guide - take a deep-dive into each topic within Daft
API Reference - API reference for public classes/functions of Daft
Contributing
To start contributing to Daft, please read CONTRIBUTING.md
Here’s a list of good first issues to get yourself warmed up with Daft. Comment in the issue to pick it up, and feel free to ask any questions!
Telemetry
To help improve Daft, we collect non-identifiable data via our own analytics as well as Scarf (https://scarf.sh).
To disable this behavior, set the following environment variables: - DAFT_ANALYTICS_ENABLED=0 - SCARF_NO_ANALYTICS=true or DO_NOT_TRACK=true
The data that we collect is:
Non-identifiable: Events are keyed by a session ID which is generated on import of Daft
Metadata-only: We do not collect any of our users’ proprietary code or data
For development only: We do not buy or sell any user data
Please see our documentation for more details.
Related Projects
Dataframe |
Query Optimizer |
Multimodal |
Distributed |
Arrow Backed |
Vectorized Execution Engine |
Out-of-core |
---|---|---|---|---|---|---|
Daft |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
No |
Python object |
No |
optional >= 2.0 |
Some(Numpy) |
No |
|
Yes |
Python object |
No |
Yes |
Yes |
Yes |
|
Eagar |
Python object |
Yes |
No |
Some(Pandas) |
Yes |
|
Yes |
No |
Yes |
Pandas UDF/IO |
Pandas UDF |
Yes |
|
No |
Python object |
Yes |
No |
Some(Pandas) |
Yes |
Check out our dataframe comparison page for more details!
License
Daft has an Apache 2.0 license - please see the LICENSE file.
Project details
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Unverified details
These details have not been verified by PyPIProject links
Meta
- Author: Eventual Inc
- Maintainer: Sammy Sidhu
- Requires: Python >=3.9
-
Provides-Extra:
all
,aws
,azure
,deltalake
,gcp
,hudi
,iceberg
,lance
,numpy
,pandas
,ray
,sql
,unity
,viz
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Uploaded
CPython 3.9+
manylinux: glibc 2.28+ x86-64
Uploaded
CPython 3.9+
manylinux: glibc 2.28+ ARM64
Uploaded
CPython 3.9+
macOS 11.0+ ARM64
Uploaded
CPython 3.9+
macOS 10.12+ x86-64
File details
Details for the file daft-0.4.7.tar.gz
.
File metadata
- Download URL: daft-0.4.7.tar.gz
- Upload date:
- Size: 4.7 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68c455189ef94448f2030a830fbc8eea00e04ed84ee51f0a29876bcd6d127ba6 |
|
MD5 | bab91b9571b240cb6ac926a8491851fb |
|
BLAKE2b-256 | 9a37a9d7f252745f006da33b4fdc1f7af34f0ddd98dd638c2d86dd118d39d888 |
Provenance
The following attestation bundles were made for daft-0.4.7.tar.gz
:
Publisher:
publish-pypi.yml
on Eventual-Inc/Daft
-
Statement:
- Statement type:
https://in-toto.io/Statement/v1
- Predicate type:
https://docs.pypi.org/attestations/publish/v1
- Subject name:
daft-0.4.7.tar.gz
- Subject digest:
68c455189ef94448f2030a830fbc8eea00e04ed84ee51f0a29876bcd6d127ba6
- Sigstore transparency entry: 178921532
- Sigstore integration time:
- Permalink:
Eventual-Inc/Daft@c7df611b99d0fba7f3330b8829b97d328549da40
- Branch / Tag:
refs/tags/v0.4.7
- Owner: https://github.com/Eventual-Inc
- Access:
public
- Token Issuer:
https://token.actions.githubusercontent.com
- Runner Environment:
github-hosted
- Publication workflow:
publish-pypi.yml@c7df611b99d0fba7f3330b8829b97d328549da40
- Trigger Event:
push
- Statement type:
File details
Details for the file daft-0.4.7-cp39-abi3-win_amd64.whl
.
File metadata
- Download URL: daft-0.4.7-cp39-abi3-win_amd64.whl
- Upload date:
- Size: 34.1 MB
- Tags: CPython 3.9+, Windows x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2145571d8e61a3b97cae805619402b7f4e6a9ae2e81473250e4370bd4a1b79d2 |
|
MD5 | 1f4a635ae9b44653ef21a978a01c37be |
|
BLAKE2b-256 | bd4834704358a7d598eca2ccd0e93ad369be917d67b590196cd2c25f0dc6fa31 |
Provenance
The following attestation bundles were made for daft-0.4.7-cp39-abi3-win_amd64.whl
:
Publisher:
publish-pypi.yml
on Eventual-Inc/Daft
-
Statement:
- Statement type:
https://in-toto.io/Statement/v1
- Predicate type:
https://docs.pypi.org/attestations/publish/v1
- Subject name:
daft-0.4.7-cp39-abi3-win_amd64.whl
- Subject digest:
2145571d8e61a3b97cae805619402b7f4e6a9ae2e81473250e4370bd4a1b79d2
- Sigstore transparency entry: 178921627
- Sigstore integration time:
- Permalink:
Eventual-Inc/Daft@c7df611b99d0fba7f3330b8829b97d328549da40
- Branch / Tag:
refs/tags/v0.4.7
- Owner: https://github.com/Eventual-Inc
- Access:
public
- Token Issuer:
https://token.actions.githubusercontent.com
- Runner Environment:
github-hosted
- Publication workflow:
publish-pypi.yml@c7df611b99d0fba7f3330b8829b97d328549da40
- Trigger Event:
push
- Statement type:
File details
Details for the file daft-0.4.7-cp39-abi3-manylinux_2_28_x86_64.whl
.
File metadata
- Download URL: daft-0.4.7-cp39-abi3-manylinux_2_28_x86_64.whl
- Upload date:
- Size: 37.1 MB
- Tags: CPython 3.9+, manylinux: glibc 2.28+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 46c7b0ef8a057b67d0b3ac9774b4443e1459388a4526ed96ba117e203e4c07b8 |
|
MD5 | 39c0400c15567f209d24bde06889a3d6 |
|
BLAKE2b-256 | e23fbfac86e8c92982dccdb213c068fd1638400fe36ca32c7ffb84232ee410ad |
Provenance
The following attestation bundles were made for daft-0.4.7-cp39-abi3-manylinux_2_28_x86_64.whl
:
Publisher:
publish-pypi.yml
on Eventual-Inc/Daft
-
Statement:
- Statement type:
https://in-toto.io/Statement/v1
- Predicate type:
https://docs.pypi.org/attestations/publish/v1
- Subject name:
daft-0.4.7-cp39-abi3-manylinux_2_28_x86_64.whl
- Subject digest:
46c7b0ef8a057b67d0b3ac9774b4443e1459388a4526ed96ba117e203e4c07b8
- Sigstore transparency entry: 178921666
- Sigstore integration time:
- Permalink:
Eventual-Inc/Daft@c7df611b99d0fba7f3330b8829b97d328549da40
- Branch / Tag:
refs/tags/v0.4.7
- Owner: https://github.com/Eventual-Inc
- Access:
public
- Token Issuer:
https://token.actions.githubusercontent.com
- Runner Environment:
github-hosted
- Publication workflow:
publish-pypi.yml@c7df611b99d0fba7f3330b8829b97d328549da40
- Trigger Event:
push
- Statement type:
File details
Details for the file daft-0.4.7-cp39-abi3-manylinux_2_28_aarch64.whl
.
File metadata
- Download URL: daft-0.4.7-cp39-abi3-manylinux_2_28_aarch64.whl
- Upload date:
- Size: 35.4 MB
- Tags: CPython 3.9+, manylinux: glibc 2.28+ ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | a375b75453d63896ec18c2b72ec954661fe34955bedf175695f22851247927f3 |
|
MD5 | a2d1b6567c3a666383ca1fff1155392b |
|
BLAKE2b-256 | 2a6d1ee6353caa9326f99a79f77c485ebb888bcfff526111166b36d51adf40ab |
Provenance
The following attestation bundles were made for daft-0.4.7-cp39-abi3-manylinux_2_28_aarch64.whl
:
Publisher:
publish-pypi.yml
on Eventual-Inc/Daft
-
Statement:
- Statement type:
https://in-toto.io/Statement/v1
- Predicate type:
https://docs.pypi.org/attestations/publish/v1
- Subject name:
daft-0.4.7-cp39-abi3-manylinux_2_28_aarch64.whl
- Subject digest:
a375b75453d63896ec18c2b72ec954661fe34955bedf175695f22851247927f3
- Sigstore transparency entry: 178921617
- Sigstore integration time:
- Permalink:
Eventual-Inc/Daft@c7df611b99d0fba7f3330b8829b97d328549da40
- Branch / Tag:
refs/tags/v0.4.7
- Owner: https://github.com/Eventual-Inc
- Access:
public
- Token Issuer:
https://token.actions.githubusercontent.com
- Runner Environment:
github-hosted
- Publication workflow:
publish-pypi.yml@c7df611b99d0fba7f3330b8829b97d328549da40
- Trigger Event:
push
- Statement type:
File details
Details for the file daft-0.4.7-cp39-abi3-macosx_11_0_arm64.whl
.
File metadata
- Download URL: daft-0.4.7-cp39-abi3-macosx_11_0_arm64.whl
- Upload date:
- Size: 32.1 MB
- Tags: CPython 3.9+, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b841dd217357cf5241d98fc54f8fde65c0f7e21a290f36b8e3c83f4b8893c489 |
|
MD5 | a88c792640815f9383d6d11c28309045 |
|
BLAKE2b-256 | f6d6bb2ed2fc336dc1845f2a18372e2ec3f1d8e75dea0af403a7582c2311c683 |
Provenance
The following attestation bundles were made for daft-0.4.7-cp39-abi3-macosx_11_0_arm64.whl
:
Publisher:
publish-pypi.yml
on Eventual-Inc/Daft
-
Statement:
- Statement type:
https://in-toto.io/Statement/v1
- Predicate type:
https://docs.pypi.org/attestations/publish/v1
- Subject name:
daft-0.4.7-cp39-abi3-macosx_11_0_arm64.whl
- Subject digest:
b841dd217357cf5241d98fc54f8fde65c0f7e21a290f36b8e3c83f4b8893c489
- Sigstore transparency entry: 178921657
- Sigstore integration time:
- Permalink:
Eventual-Inc/Daft@c7df611b99d0fba7f3330b8829b97d328549da40
- Branch / Tag:
refs/tags/v0.4.7
- Owner: https://github.com/Eventual-Inc
- Access:
public
- Token Issuer:
https://token.actions.githubusercontent.com
- Runner Environment:
github-hosted
- Publication workflow:
publish-pypi.yml@c7df611b99d0fba7f3330b8829b97d328549da40
- Trigger Event:
push
- Statement type:
File details
Details for the file daft-0.4.7-cp39-abi3-macosx_10_12_x86_64.whl
.
File metadata
- Download URL: daft-0.4.7-cp39-abi3-macosx_10_12_x86_64.whl
- Upload date:
- Size: 34.7 MB
- Tags: CPython 3.9+, macOS 10.12+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | d31fc8cf6a830cde6a981ec44e26e90d5e4ef73570b615d50d555947cdeb06da |
|
MD5 | cc7bd033b614bd457f9fcb5e201d7011 |
|
BLAKE2b-256 | 26b1e7c508da2992f34885c5110fe033ae8703f5b425a82094d02e88da53eb62 |
Provenance
The following attestation bundles were made for daft-0.4.7-cp39-abi3-macosx_10_12_x86_64.whl
:
Publisher:
publish-pypi.yml
on Eventual-Inc/Daft
-
Statement:
- Statement type:
https://in-toto.io/Statement/v1
- Predicate type:
https://docs.pypi.org/attestations/publish/v1
- Subject name:
daft-0.4.7-cp39-abi3-macosx_10_12_x86_64.whl
- Subject digest:
d31fc8cf6a830cde6a981ec44e26e90d5e4ef73570b615d50d555947cdeb06da
- Sigstore transparency entry: 178921584
- Sigstore integration time:
- Permalink:
Eventual-Inc/Daft@c7df611b99d0fba7f3330b8829b97d328549da40
- Branch / Tag:
refs/tags/v0.4.7
- Owner: https://github.com/Eventual-Inc
- Access:
public
- Token Issuer:
https://token.actions.githubusercontent.com
- Runner Environment:
github-hosted
- Publication workflow:
publish-pypi.yml@c7df611b99d0fba7f3330b8829b97d328549da40
- Trigger Event:
push
- Statement type: