delune 0.3.2.1

Connecting to Delune Resources

import delune

dln = delune.connect ("/home/delune")

As result, delune check anf create directories.

/home/delune/delune/config
/home/delune/delune/collections

Creating New Collection

col = dln.create ("mycol", ["mycol"], 1)
col.save ()

As result, collection created like this.

/home/delune/delune/config/mycol : JSON file contains configure options
/home/delune/delune/collections/mycol

If you use multiple disks for increasing speed or capacity of collection.

First of all mount your disks to /home/delune/delune/collections,

/home/delune/delune/collections/hdd0
/home/delune/delune/collections/hdd1

Then create collection.

col = dln.create ("mycol", ["hdd0/mycol", "hdd1/mycol"], 1)
col.save ()

As a result, collection will be created like this.

/home/delune/delune/collections/hdd0/mycol
/home/delune/delune/collections/hdd1/mycol

Your segment filess of collection will be created these directories randomly (with considering free space of disks).

Configuring Collection

There’re 2 way for configuring tour collections.

First, use col.config dictionalry.

col = dln.create ("mycol", ["mycol"], version = 1)
col.config

>> {
     'name': 'mycol',
     'data_dir': ["mycol"],
     "version": 1,

     'analyzer': {
       "max_terms": 3000,
       "stem_level": 1,
       "strip_html": 0,
       "make_lower_case": 1,
       "ngram": 1,
       "biword": 0,
       "stopwords_case_sensitive": 1,
       "ngram_no_space": 0,
       "contains_alpha_only": 0,
       "stopwords": [],
       "endwords": [],
     },
     'indexer': {
       'optimize': 1,
       'force_merge': 0,
       'max_memory': 10000000,
       'max_segments': 10,
       'lazy_merge': (0.3, 0.5),
     },
     'searcher': {
       'max_result': 2000,
       'num_query_cache': 1000
     }
   }

You just change values as you want.

Another way is set options when creating collection.

col = dln.create (
  "mycol",
  ["mycol"],
  version = 1,
  max_terms = 5000,
  strip_html = 1,
  force_merge = 1,
  max_result = 10000
)

For more detail for analyzer, indexer and searcher options, see Low Level API section.

Adding Dcouments To Collection

with col.documents as D:
  for code, title in my_codes:
    d = D.new (code) # code is used as document ID
    d.content ([code, title])
    d.field ("code", code, delune.STRING)
    d.field ("default", title, delune.TEXT)
    D.add (d)
  D.commit ()

It is important to understand, above operation actually dosen’t make any change to your collection. It just saves your documents at:

/home/delune/delune/collections/mycol/.que/

If you commit multiple time, que files will be created as you commit.

Adding Dcouments Without ID

d = D.new ()

Note that in this case you canmoy update/modify your documents.

Deleting Dcouments From Collection

If your document has ID,

with col.documents as D:
  for code, title in my_codes:
    D.delete (code)
  D.commit ()

Else,

with col.documents as D:
  D.qdelete ("milk")
  D.commit ()

It will be deleted all documents contain ‘milk’.

Indexing

If you run delune indexer, these saved documents will be automatically indexed. Or you can index mannually,

delune index -v /home/delune

Searching

dln = delune.connect ("/home/delune")
col = dln.load ("mycol")
with col.documents as D:
  D.search ("violin")

search() spec is:

D.search (
  q,
  offset = 0,
  limit = 10,
  sort = "", # INT field name
  snippet = 30, # number of terms for snippet
  partial = "", # specify index or key of a content
  nthdoc = 0, # specify index of contents
  lang = "un",
  analyze = 1, # query terms are already analyzed, set to 0
  data = 1, # whether or not return content part
  qlimit = 1 # whether or not apply limitation for number of searched documents by max_result
)

Truncating Documents

col.documents.truncate ("mycol")
col.documents.commit ()

Drop Collection

col.drop (include_data = True)

Implemented Stemmers

Except English stemmer, all stemmers can be obtained at IR Multilingual Resources at UniNE.

• ar: Arabic

• de: German

• en: English

• es: Spanish

• fi: Finnish

• fr: French

• hu: Hungarian

• it: Italian

• pt: Portuguese

• sv: Swedish

Bi-Gram Index

If ngram is set to True, these languages will be indexed with bi-gram.

• cn: Chinese

• ja: Japanese

• ko: Korean

Also note that if word contains only alphabet, will be used English stemmer.

Tri-Gram Index

The other languages will be used English stemmer if all spell is Alphabet. And if ngram is set to True, will be indexed with tri-gram if word has multibytes.

Methods Spec

• analyzer.index (document, lang)

• analyzer.freq (document, lang)

• analyzer.stem (document, lang)

• analyzer.count_stopwords (document, lang)

Set Returning Content

DeLune serialize returning contents by pickle, so you can set any objects pickle serializable.

document.content ({"userid": "hansroh", "preference": {"notification": "email", ...}})

or

document.content ([32768, "This is smaple ..."])

For saving multiple contents,

document.content ({"userid": "hansroh", "preference": {"notification": "email", ...}})
document.content ([32768, "This is smaple ..."])

You can select one of these by query time using nthdoc=0 or 1 parameter.

Snippet Generating Field

This field should be unicode/utf8 encoded bytes.

document.snippet ("This is sample...")

Searchable Fields

document also recieve searchable fields:

document.field (name, value, ftype = delune.TEXT, lang = "un", encoding = None)

document.field ("default", "violin sonata in c k.301", delune.TEXT, "en")
document.field ("composer", "wolfgang amadeus mozart", delune.TEXT, "en")
document.field ("lastname", "mozart", delune.STRING)
document.field ("birth", 1756, delune.INT16)
document.field ("genre", "01011111", delune.BIT8)
document.field ("home", "50.665629/8.048906", delune.COORD6)

• name: if ‘default’, this field will be searched by simple string, or use ‘name:query_text’

• value: unicode/utf8 encode text, or should give encoding arg.

• ftype: see below

• encoding: give like ‘iso8859-1’ if value is not unicode/utf8

• lang: language code for standard_analyzer, “un” (unknown) is default

Avalible Field types are:

• TEXT: analyzable full-text, result-not-sortable

• TERM: analyzable full-text but position data will not be indexed as result can’t search phrase, result-not-sortable

• STRING: exactly string match like nation codes, result-not-sortable

• LIST: comma seperated STRING, result-not-sortable

• FNUM: foramted number, value should be int or float and format parameter required, format is “digit.digit” that number of digit interger part with zero leading, and number of float part length. It make possible to search range efficiently.

• COORDn, n=4,6,8 decimal precision: comma seperated string ‘latitude,longititude’, latitude and longititude sould be float type range -90 ~ 90, -180 ~ 180. n is precision of coordinates. n=4 is 10m radius precision, 6 is 1m and 8 is 10cm. result-sortable

• BITn, n=8,16,24,32,40,48,56,64: bitwise operation, bit makred string required by n, result-sortable

• INTn, n=8,16,24,32,40,48,56,64: range, int required, result-sortable

Note1: You make sure COORD, INT and BIT fields are at every documents even they havn’t got a value, because these types are depend on document indexed sequence ID. If they have’t a value, please set value to None NOT omit fields.

Note2: FNUM 100.12345 with format=”5.3” is interanlly converted into “00100.123” and negative value will be -00100.123 and MAKE SURE your values are within -99999.999 and 99999.999.

Repeat add_document as you need and close indexer.

for ...:
  document = delune.document ()
  ...
  indexer.add_document (document)
  indexer.close ()

If searchers using this collection runs with another process or thread, searcher automatically reloaded within a few seconds for applying changed index.

Query Syntax

• violin composer:mozart birth:1700~1800

  search ‘violin’ in default field, ‘mozart’ in composer field and search range between 1700, 1800 in birth field

• violin allcomposer:wolfgang mozart

  search ‘violin’ in default field and any terms after allcomposer will be searched in composer field

• violin -sonata birth2:1700~1800

  birth2 is between ‘1700’ and ‘1800’

• violin -sonata birth:~1800

  not contain sonata in default field

• violin -composer:mozart

  not contain mozart in composer field

• violin or piano genre:00001101/all

  matched all 5, 6 and 8th bits are 1. also /any or /none is available

• violin or ((piano composer:mozart) genre:00001101/any)

  support unlimited priority ‘()’ and ‘or’ operators

• (violin or ((allcomposer:mozart wolfgang) -amadeus)) sonata (genre:00001101/none home:50.6656,8.0489~10000)

  search home location coordinate (50.6656, 8.0489) within 10 Km

• “violin sonata” genre:00001101/none home:50.6656/8.0489~10

  search exaclt phrase “violin sonata”

• “violin^3 piano” -composer:”ludwig van beethoven”

  search loose phrase “violin sonata” within 3 terms