Skip to main content

tokenizers for Whoosh designed for Japanese language

Project description

About

Tokenizers for Whoosh full text search library designed for Japanese language. This package conteins two Tokenizers.

  • IgoTokenizer

  • TinySegmenterTokenizer

  • MeCabTokenizer

How To Use

IgoTokenizer:

import igo.Tagger
import whooshjp
from whooshjp.IgoTokenizer import IgoTokenizer

tk = IgoTokenizer(igo.Tagger.Tagger('ipadic'))
scm = Schema(title=TEXT(stored=True, analyzer=tk), path=ID(unique=True,stored=True), content=TEXT(analyzer=tk))

TinySegmenterTokenizer:

import tinysegmenter
import whooshjp
from whooshjp.TinySegmenterTokenizer import TinySegmenterTokenizer

tk = TinySegmenterTokenizer(tinysegmenter.TinySegmenter())
scm = Schema(title=TEXT(stored=True, analyzer=tk), path=ID(unique=True,stored=True), content=TEXT(analyzer=tk))

Changelog for Japanese Tokenizers for Whoosh

2011-02-19 – 0.1
  • first release.

2011-02-21 – 0.2
  • add TinySegmenterTokenizer

  • change module name

2011-02-24 – 0.3
  • add FeatureFilter

2011-02-27 – 0.4
  • add MeCabTokenizer

  • add a mode for don’t pickle igo tagger to minimize index.

2011-04-17 – 0.5
  • correct char offsets

2011-04-17 – 0.6
  • correct char offsets(TinySegmenterTokenizer)

2012-04-14 – 0.7
  • rename package(WhooshJapaneseTokenizer to whooshjp)

  • no longer import sub modules automatically

  • Python3 compatibility(3.2, 3.3)

  • Drop Python2.5 support

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

whoosh-igo-0.7.tar.gz (8.0 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page