Skip to main content

Lightweight piece tokenization library

Project description

🍴 cutlery

This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:

Tokenizer Binding Example model
BPE sentencepiece
Byte BPE Native RoBERTa/GPT-2
Unigram sentencepiece XLM-RoBERTa
Wordpiece Native BERT

⚠️ Warning: experimental package

This package is experimental and it is likely that the APIs will change in incompatible ways.

⏳ Install

Cutlery is availble through PyPI:

pip install cutlery

🚀 Quickstart

The best way to get started with cutlery is through the curated-transformers library. curated-transformers also provides functionality to load tokenization models from Huggingface Hub.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cutlery-0.0.5.tar.gz (2.2 MB view hashes)

Uploaded Source

Built Distributions

cutlery-0.0.5-cp311-cp311-win_amd64.whl (590.4 kB view hashes)

Uploaded CPython 3.11 Windows x86-64

cutlery-0.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (683.1 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (658.6 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp311-cp311-macosx_11_0_arm64.whl (658.8 kB view hashes)

Uploaded CPython 3.11 macOS 11.0+ ARM64

cutlery-0.0.5-cp311-cp311-macosx_10_9_x86_64.whl (688.5 kB view hashes)

Uploaded CPython 3.11 macOS 10.9+ x86-64

cutlery-0.0.5-cp310-cp310-win_amd64.whl (592.2 kB view hashes)

Uploaded CPython 3.10 Windows x86-64

cutlery-0.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (683.2 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (658.1 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp310-cp310-macosx_11_0_arm64.whl (661.9 kB view hashes)

Uploaded CPython 3.10 macOS 11.0+ ARM64

cutlery-0.0.5-cp310-cp310-macosx_10_9_x86_64.whl (691.6 kB view hashes)

Uploaded CPython 3.10 macOS 10.9+ x86-64

cutlery-0.0.5-cp39-cp39-win_amd64.whl (594.2 kB view hashes)

Uploaded CPython 3.9 Windows x86-64

cutlery-0.0.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (686.1 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (661.6 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp39-cp39-macosx_11_0_arm64.whl (661.9 kB view hashes)

Uploaded CPython 3.9 macOS 11.0+ ARM64

cutlery-0.0.5-cp39-cp39-macosx_10_9_x86_64.whl (691.8 kB view hashes)

Uploaded CPython 3.9 macOS 10.9+ x86-64

cutlery-0.0.5-cp38-cp38-win_amd64.whl (594.4 kB view hashes)

Uploaded CPython 3.8 Windows x86-64

cutlery-0.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (687.1 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (663.1 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp38-cp38-macosx_11_0_arm64.whl (659.4 kB view hashes)

Uploaded CPython 3.8 macOS 11.0+ ARM64

cutlery-0.0.5-cp38-cp38-macosx_10_9_x86_64.whl (688.8 kB view hashes)

Uploaded CPython 3.8 macOS 10.9+ x86-64

cutlery-0.0.5-cp37-cp37m-win_amd64.whl (592.9 kB view hashes)

Uploaded CPython 3.7m Windows x86-64

cutlery-0.0.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (684.1 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (660.8 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp37-cp37m-macosx_10_9_x86_64.whl (688.3 kB view hashes)

Uploaded CPython 3.7m macOS 10.9+ x86-64

cutlery-0.0.5-cp36-cp36m-win_amd64.whl (644.6 kB view hashes)

Uploaded CPython 3.6m Windows x86-64

cutlery-0.0.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (684.2 kB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (660.8 kB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARM64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page