tokenizer-hub

Some features may not work without JavaScript. Please try enabling it if you encounter problems.

Yoctol Natural Language Tokenizer

These details have not been verified by PyPI

Project description

# tokenizer-hub

yoctol 乂卍oO煞氣ㄟtokenizerOo卍乂

Tokenizers have the same interface of Jieba:

`python from tokenizer_hub import XXX_tokenizer tokenizer = XXX_tokenizer() tokenizer.lcut('我来到北京清华大学') > ['我', '来到', '北京', '清华大学'] `