This repository contains code to run faster sentence-transformers using tools like quantization, ONNX and pruning.
Project description
Fast Sentence Transformers
This repository contains code to run faster sentence-transformers using tools like quantization and ONNX. Just run your model much faster, while a lot of memory. There is not much to it!
Install
pip install fast-sentence-transformers
Or for GPU support.
pip install fast-sentence-transformers[gpu]
Quickstart
from fast_sentence_transformers import FastSentenceTransformer as SentenceTransformer
# use any sentence-transformer
encoder = SentenceTransformer("all-MiniLM-L6-v2", device="cpu", quantize=True)
encoder.encode("Hello hello, hey, hello hello")
encoder.encode(["Life is too short to eat bad food!"] * 2)
Benchmark
Indicative benchmark for CPU usage with smallest and largest model on sentence-transformers. Note, ONNX doesn't have GPU support for quantization yet.
| model | Type | default | ONNX | ONNX+quantized | ONNX+GPU |
|---|---|---|---|---|---|
| paraphrase-albert-small-v2 | memory | 1x | 1x | 1x | 1x |
| speed | 1x | 2x | 5x | 20x | |
| paraphrase-multilingual-mpnet-base-v2 | memory | 1x | 1x | 4x | 4x |
| speed | 1x | 2x | 5x | 20x |
Shout-Out
This package heavily leans on sentence-transformers and txtai.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for fast_sentence_transformers-0.4.1.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d4d963b5495cb070701e67517f89c7d9a0f11179281e1d9c4f4dcafd230ada0f |
|
| MD5 | ed2babe98827b0796c98476ec9a9defe |
|
| BLAKE2b-256 | 7bc3d8911a83ea89808bacb066f778db78ef60cd02363ed2e31fd93ffc5f7b1b |
Close
Hashes for fast_sentence_transformers-0.4.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 83d47d623c2ad5b5d2bf7d5fffcfeb32a994df82be71414fd60c0e849026a94b |
|
| MD5 | 0c7535f2aba353588d05024a69ad8a2c |
|
| BLAKE2b-256 | 836919293e3d201ce4f46ea56559fac1f76d81484e5d7fc75739a7585e2e2098 |
