A fast(not yet :) bleu score calculator
Project description
bleuscore
bleuscore
is a fast BLEU score calculator written in rust.
Installation
The python package has been published to pypi, so we can install it directly with many ways:
-
pip
pip install bleuscore
-
poetry
poetry add bleuscore
-
uv
uv pip install bleuscore
Quick Start
The usage is exactly same with huggingface evaluate:
- import evaluate
+ import bleuscore
predictions = ["hello there general kenobi", "foo bar foobar"]
references = [
["hello there general kenobi", "hello there !"],
["foo bar foobar"]
]
- bleu = evaluate.load("bleu")
- results = bleu.compute(predictions=predictions, references=references)
+ results = bleuscore.compute(predictions=predictions, references=references)
print(results)
# {'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0,
# 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}
Benchmark
TLDR: We got more than 10x speedup when the corpus size beyond 100K
We use the demo data shown in quick start to do this simple benchmark. You can check the benchmark/simple for the benchmark source code.
rs_bleuscore
: bleuscore python librarylocal_hf_bleu
: huggingface evaluate bleu algorithm in localsacre_bleu
: sacrebleu- Note that we got different result with sacrebleu in the simple demo data and all the rests have same result
hf_evaluate
: huggingface evaluate bleu algorithm with evaluate package
The N
is used to enlarge the predictions/references size by simply duplication the demo data as shown before.
We can see that as N
increase, the bleuscore gets better performance.
You can navigate benchmark for more benchmark details.
N=100
hyhyperfine --warmup 5 --runs 10 \
"python simple/rs_bleuscore.py 100" \
"python simple/local_hf_bleu.py 100" \
"python simple/sacre_bleu.py 100" \
"python simple/hf_evaluate.py 100"
Benchmark 1: python simple/rs_bleuscore.py 100
Time (mean ± σ): 19.0 ms ± 2.6 ms [User: 17.8 ms, System: 5.3 ms]
Range (min … max): 14.8 ms … 23.2 ms 10 runs
Benchmark 2: python simple/local_hf_bleu.py 100
Time (mean ± σ): 21.5 ms ± 2.2 ms [User: 19.0 ms, System: 2.5 ms]
Range (min … max): 16.8 ms … 24.1 ms 10 runs
Benchmark 3: python simple/sacre_bleu.py 100
Time (mean ± σ): 45.9 ms ± 2.2 ms [User: 38.7 ms, System: 7.1 ms]
Range (min … max): 43.5 ms … 50.9 ms 10 runs
Benchmark 4: python simple/hf_evaluate.py 100
Time (mean ± σ): 4.504 s ± 0.429 s [User: 0.762 s, System: 0.823 s]
Range (min … max): 4.163 s … 5.446 s 10 runs
Summary
python simple/rs_bleuscore.py 100 ran
1.13 ± 0.20 times faster than python simple/local_hf_bleu.py 100
2.42 ± 0.35 times faster than python simple/sacre_bleu.py 100
237.68 ± 39.88 times faster than python simple/hf_evaluate.py 100
N = 1K ~ 1M
Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
---|---|---|---|---|
python simple/rs_bleuscore.py 1000 |
20.3 ± 1.3 | 18.2 | 21.4 | 1.00 |
python simple/local_hf_bleu.py 1000 |
45.8 ± 1.2 | 44.2 | 47.5 | 2.26 ± 0.16 |
python simple/rs_bleuscore.py 10000 |
37.8 ± 1.5 | 35.9 | 39.5 | 1.87 ± 0.14 |
python simple/local_hf_bleu.py 10000 |
295.0 ± 5.9 | 288.6 | 304.2 | 14.55 ± 0.98 |
python simple/rs_bleuscore.py 100000 |
219.6 ± 3.3 | 215.3 | 224.0 | 10.83 ± 0.72 |
python simple/local_hf_bleu.py 100000 |
2781.4 ± 42.2 | 2723.1 | 2833.0 | 137.13 ± 9.10 |
python simple/rs_bleuscore.py 1000000 |
2048.8 ± 31.4 | 2013.2 | 2090.3 | 101.01 ± 6.71 |
python simple/local_hf_bleu.py 1000000 |
28285.3 ± 100.9 | 28182.1 | 28396.1 | 1394.51 ± 90.21 |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for bleuscore-0.1.2-cp38-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 72232c43828437c20c6b785db8e75f3cfa5b4ac84e6e17a8d2e58a77e6f25832 |
|
MD5 | 49f1df227e3220b304c729f24946efa4 |
|
BLAKE2b-256 | bb17f116deb0c70860d042fd7e96920d969e4da9618e5787f1332652cbe8a23f |
Hashes for bleuscore-0.1.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a027b416b53751b27deeba26016ab25821f7c3655d70a5df73f7d5ddd5cd106d |
|
MD5 | 7b511e3a30d8eda1baef9f3fabe8408e |
|
BLAKE2b-256 | b411d0dba773fdd9ae0ad27cb0a5ea9a2f5a53d91cb28386614c542ed019d6ab |
Hashes for bleuscore-0.1.2-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69389abc853fe1e9b15c2b482a546b0cff208acf594c852ac51918893a281916 |
|
MD5 | 678a982a1a9fd65c3a2bcc1fea66d2cd |
|
BLAKE2b-256 | d9312f7fb17e97e91ea96a85c4c72a8db45d4223fd6a17a3496b09d7c9a12348 |
Hashes for bleuscore-0.1.2-cp38-abi3-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd25c9799b00f5c445503584db07c5cb86cb1985c5ae9f31229a4d1841a1da25 |
|
MD5 | 6824e1c92db45876331b94e4e39c881b |
|
BLAKE2b-256 | b88b2544d46bda7f44d0c32990e1dcf10b20a1aa0f7e820af3b300808e4cb5f6 |
Hashes for bleuscore-0.1.2-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97100a4214dc4f854297fd04334e063824049a6ec48a4de0219b993fdb86abf2 |
|
MD5 | a5142bc2ea25a4996df3d90661150185 |
|
BLAKE2b-256 | 678d2b6e560af6d9d492e3694dbaf015eba9cce6dc06888040eed98dc180038d |