syntok 1.4.4

Command-line

After installing the package, two command-line usages will be available, python -m syntok.segmenter and python -m syntok.tokenizer. Each takes [UTF-8 encoded] plain-text files (or STDIN until EOF (CTRL-D)) as input and transforms that into newline-separated sentences or space-separated tokens, respectively. You can control Python3’s file open encoding by configuring the environment variable PYTHONIOENCODING to your needs (e.g. export PYTHONIOENCODING="utf-16-be"). The tokenizer produces single-space separated tokens for each input line. The segmenter produces line-segmented sentences for each input file (or after STDIN closes).

syntok.tokenizer

This module provides the Tokenizer class to tokenize input text into words and symbols (value Tokens), prefixed with (possibly empty) spacing strings, while recording their offset positions. The Tokenizer comes with utility static functions, to join hyphenated words across line-breaks, and to reproduce the original string from a sequence of tokens. The Tokenizer considers camelCase words as individual tokens (here: camel and Case) and by default considers underscores and Unicode hyphens inside words as spacing characters (not Token values). It does not split numeric tokens (without letters) if they contain symbols (e.g. maintaining “2018-11-11”, “12:30:21”, “1_000_000”, “1,000.00”, or “1..3” all as single tokens) Finally, as it splits English negation contractions (such as “don’t”) into their root and “not” (here: do and not), it can be configured to refrain from replacing this special “n’t” token with “not”, and instead emit the actual “n’t” value.

To track the spacing and offset of tokens, the module contains the Token class, which is a str wrapper class where the token value itself is available from the value property and adding a spacing and a offset property that will hold the spacing prefix and the offset position of the token, respectively.

Basic example:

from syntok.tokenizer import Tokenizer

document = open('README.rst').read()
tok = Tokenizer()  # optional: keep "n't" contractions and "-", "_" inside words as tokens

for token in tok.tokenize(document):
    print(repr(token))

syntok.segmenter

This module provides several functions to segment documents into iterators over paragraphs, sentences, and tokens (functions analyze and process) or simply sentences and tokens (functions split and segment). The analytic segmenter can even keep track of the original offset of each token in the document while processing (but does not join hyphen-separated words across line-breaks). All segmenter functions accept arbitrary Token streams as input (typically as generated by the Tokenizer.tokenize method). Due to how syntok.tokenizer.Token objects “work”, it is possible to establish the exact sentence content (with the original spacing between the tokens). The pre-processing functions and paragraph-based segmentation splits paragraphs, i.e., chunks of text separated by at least two consecutive linebreaks (\\r?\\n).

Basic example:

import syntok.segmenter as segmenter

document = open('README.rst').read()

# choose the segmentation function you need/prefer

for paragraph in segmenter.process(document):
    for sentence in paragraph:
        for token in sentence:
            # roughly reproduce the input,
            # except for hyphenated word-breaks
            # and replacing "n't" contractions with "not",
            # separating tokens by single spaces
            print(token.value, end=' ')
        print()  # print one sentence per line
    print()  # separate paragraphs with newlines

for paragraph in segmenter.analyze(document):
    for sentence in paragraph:
        for token in sentence:
            # exactly reproduce the input
            # and do not remove "imperfections"
            print(token.spacing, token.value, sep='', end='')
    print("\n")  # reinsert paragraph separators