Perplexity filter for documents and bulc HTML and WARC boilerplate removal.

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for manueldeprada from gravatar.com manueldeprada Avatar for MarcosFP97 from gravatar.com MarcosFP97

Unverified details

These details have not been verified by PyPI

View statistics for this project via Libraries.io, or by using our public dataset on Google BigQuery

Meta

Author: Manuel de Prada Corral

Requires: Python >=3.6.1, <4.0.0

Classifiers

Project description

Pyplexity

This package provides a simple interface to apply perplexity filters to any document. Furthermore, it provides a WARC and HTML bulk processor, with distributed capabilities.

Usage example

Process a folder containing a dataset using a trigrams model.

poetry build
pip3 install dist/pyplexity-0.1.31-py3-none-any.whl
pyplexity bulk-perplexity --perpl-model ../../clueweb-b13-rawtext2/trigrams_bnc.st --perpl-limit 8000.0 \ 
    --trigrams --base-dir ./cleaned_webkb --output-dir ./perpl_filtered_webkb

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for manueldeprada from gravatar.com manueldeprada Avatar for MarcosFP97 from gravatar.com MarcosFP97

Unverified details

These details have not been verified by PyPI

View statistics for this project via Libraries.io, or by using our public dataset on Google BigQuery

Meta

Author: Manuel de Prada Corral

Requires: Python >=3.6.1, <4.0.0

Classifiers

Release history Release notifications | RSS feed

0.2.12

0.2.11

0.2.10

0.2.9

0.2.8

0.2.7

0.2.6

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

0.1.37

0.1.36

0.1.35

0.1.33

This version

0.1.32

0.1.31

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyplexity-0.1.32.tar.gz (12.2 kB view hashes)

Uploaded Source

Built Distribution

pyplexity-0.1.32-py3-none-any.whl (9.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page