skip to navigation
skip to content

neologdn 0.2.1

Japanese text normalizer for mecab-neologd

neologdn

neologdn is a Japanese text normalizer for mecab-neologd.

The normalization is based on the neologd’s rules: https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja

Contributions are welcome!

NOTE: Installing this module requires C++11 compiler.

Installation

$ pip install neologdn

Usage

import neologdn
neologdn.normalize("ハンカクカナ")
# => 'ハンカクカナ'
neologdn.normalize("全角記号!?@#")
# => '全角記号!?@#'
neologdn.normalize("全角記号例外「・」")
# => '全角記号例外「・」'
neologdn.normalize("長音短縮ウェーーーーイ")
# => '長音短縮ウェーイ'
neologdn.normalize("チルダ削除ウェ~∼∾〜〰~イ")
# => 'チルダ削除ウェイ'
neologdn.normalize("いろんなハイフン˗֊‐‑‒–⁃⁻₋−")
# => 'いろんなハイフン-'
neologdn.normalize("   PRML  副 読 本   ")
# => 'PRML副読本'
neologdn.normalize(" Natural Language Processing ")
# => 'Natural Language Processing'
neologdn.normalize("かわいいいいいいいいい", repeat=6)
# => 'かわいいいいいい'

Benchmark

# Sample code from
# https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja#python-written-by-hideaki-t--overlast
import normalize_neologd

%timeit normalize(normalize_neologd.normalize_neologd)
# => 1 loop, best of 3: 18.3 s per loop


import neologdn
%timeit normalize(neologdn.normalize)
# => 1 loop, best of 3: 9.05 s per loop

neologdn is about x2 faster than sample code.

details are described as the below notebook: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

License

Apache Software License.

CHANGES

0.2.1 (2017-01-23)

  • Fix bug (Check if a previous character of daku-ten character is in maps) (Many thanks unnonouno)

0.2 (2016-04-12)

  • Add lengthened expression (repeating character) threshold

0.1.2 (2016-03-29)

  • Fix installation bug

0.1.1.1 (2016-03-19)

  • Support Windows
  • Explicitly specify to -std=c++11 in build (Many thanks id774)

0.1.1 (2015-10-10)

Initial release.

 
File Type Py Version Uploaded on Size
neologdn-0.2.1.tar.gz (md5) Source 2017-01-23 44KB