HuggingFace library to process and filter large amounts of webdata
Project description
datatrove
Installation
pip install -e ".[dev]"
Install pre-commit code style hooks:
pre-commit install
Run the tests:
pytest -n 4 --max-worker-restart=0 --dist=loadfile tests
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
datatrove-0.0.1.dev0.tar.gz
(2.6 kB
view hashes)
Built Distribution
Close
Hashes for datatrove-0.0.1.dev0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ee4652f936197d3ad3e6a831827dfb788fd3b1d2cad31434caec39743f5cf63 |
|
MD5 | cd4c7e52de3e2b8159c09ffedd84e499 |
|
BLAKE2b-256 | d39c9edcdf7a95fdbe4de6db47282159edcfd70e40a3e7b5a200fbd534e0e79e |