Perplexity filter for documents and bulc HTML and WARC boilerplate removal.
Project description
Pyplexity
This package provides a simple interface to apply perplexity filters to any document. Furthermore, it provides a WARC and HTML bulk processor, with distributed capabilities.
Usage example
Process a folder containing a dataset using a trigrams model.
poetry build
pip3 install dist/pyplexity-0.1.31-py3-none-any.whl
pyplexity bulk-perplexity --perpl-model ../../clueweb-b13-rawtext2/trigrams_bnc.st --perpl-limit 8000.0 \
--trigrams --base-dir ./cleaned_webkb --output-dir ./perpl_filtered_webkb
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyplexity-0.1.33.tar.gz
(12.2 kB
view hashes)
Built Distribution
Close
Hashes for pyplexity-0.1.33-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de4d580fd8fc98009644baa864d7fa4588c679fa2cd37718ec5da25380310c49 |
|
MD5 | d2c3d74a53b15229861328bbb2faeb4f |
|
BLAKE2b-256 | 148cc6c0b98059059f63151d7784095ee4616f11c59a9b0747d69d840d86062a |