Simple tokenizers: n-grams and chargrams splitting, white space splitting, or splitting using configurable REGEX expression, or detection into context tokenization. Based on Span and Token objects from the tokenspan package.
Project description
Tokenization for language processing
This package contains some basic tools allowing to cut a string in sub-parts (cf. Wikipedia), called Token
.
iamtokenizing
classes allow basic tokenization of text, such as
- word splitting, n-gram splitting, (using
NGrams
class) - char-gram splitting of arbitrary size (using
CharGrams
class).
NGrams
also accepts any REGular EXpression (REGEX) to match pattern that will serve as splitting string. The class RegexDetector
also allows to extract the REGEX pattern as token. In addition, ContextDetector
allow to split text on some REGEX, and to detect inside these splits an other REGEX, keeping some organisation (called context) of the text between the two detection and splitting scales.
Installation
- The documentation is available on https://nlp.frama.io/iamtokenizing/
- The PyPi package is available on https://pypi.org/project/iamtokenizing/
- The official repository is on https://framagit.org/nlp/iamtokenizing
From Python Package Index (PIP)
Simply run
pip install iamtokenizing
is sufficient.
From the repository
The official repository is on https://framagit.org/nlp/iamtokenizing
Once the repository has been downloaded (or cloned), one can install this package using pip
:
git clone https://framagit.org/nlp/iamtokenizing.git
cd iamtokenizing/
pip install .
Once installed, one can run some tests using
cd tests/
python3 -m unittest -v
(verbosity -v
is an option).
Basic examples
Basic examples can be found in the documentation.
Versions
- Versions before 0.4 only present the
Token
andTokens
classes. They have been splitted after in three classes, namedSpan
,Token
andTokens
. Importantly, the methodsToken.append
andToken.remove
no longer exist in the next version. They have been replaced byToken.append_range
,Token.append_ranges
,Token.remove_range
andToken.remove_ranges
. - Version 0.4 add the class
Span
toToken
andTokens
.Span
handles the sub-parts splitting of a given string, whereasToken
andTokens
now consumesSpan
objects and handle the attributes of theToken
. - From version 0.5, one has split the basic tools
Span
,Token
andTokens
from theiamtokenizing
package (see https://pypi.org/project/iamtokenizing/). Only the advanced tokenizer are now present in the packageiamtokenizing
, which depends on the packagetokenspan
. The objectsSpan
,Token
andTokens
can be called as before from the newly deployed packagetokenspan
, available on https://pypi.org/project/tokenspan/.
About us
Package developped for Natural Language Processing at IAM : Unité d'Informatique et d'Archivistique Médicale, Service d'Informatique Médicale, Pôle de Santé Publique, Centre Hospitalo-Universitaire (CHU) de Bordeaux, France.
You are kindly encouraged to flag any trouble, and to propose ameliorations and/or suggestions to the authors, via issue or merge requests.
Last version : August 6, 2021
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for iamtokenizing-0.5.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26e6b6051f59296f585cec518d4a71d3ef94716512912e344a55befb790e7b10 |
|
MD5 | f9d61321c88d9af7921f9131883b659f |
|
BLAKE2b-256 | 8ad120c0566473f1238fba8d1551ca0d24c7d8dc7181864027bdb53d2e59dbc1 |