skip to navigation
skip to content

cubicweb-ner 0.1.0

Cube for named entities source and recognition (NER).

Cube for named entities source and recognition (NER).

This cube provides:

- the notion of NerSource (i.e. Named Entities Source), e.g. dbpedia or dbpedia-en (for
Dbpedia in english).

- the notion of NerEntry, which is a token/word/entry that could be recognized.
Basically it requires a "label" and a "cwuri", but an "unormalize_label"
could be given for quicker match, a "weight" for disambiguation or
a "lang" for sorting. It should be related to a NerSource.

- the notion of NerProcess, which is an entity type that stores the parameters
for a Named Entities Recognition: a "name", an "host" (appid or url of a sparql endpoint),
a request (RQL or SPARQL, with the "token" key for substitution), a type ('rql' or 'sparql')
for now, and a lang (for sorting).

Basically a lexic could be defined (NerSource), that contains entries (NerEntry).
Thus processes (NerProcess) could be defined in other applications to retrieve these entries
in some content.


Creation of the instance:

* Create an instance using: cubicweb-ctl create ner <name-of-instance>

* Create the instance's database using: cubicweb-ctl db-create <name-of-instance>

Creating entities

For creating a NerSource (in a cw shell):

session.create_entity('NerSource', name=<name of="" the="" source="">)


session.create_entity('NerSource', name=u"dbpedia-38-en")

For creating a simple NerEntry (in a cw shell):

session.create_entity('NerEntry', label=<label of="" the="" entry="">, cwuri=<uri of="" the="" entry="">)


session.create_entity('NerEntry', label=u"Barack Obama", cwuri=u""

or :

session.create_entity('NerEntry', label=u"Barack Obama", cwuri=u"",
ner_source=123, unormalize_label=u"barack obama", lang=u'fr', weight=1)

For creating a NerProcess, probably in another application (in a cw shell):

session.create_entity('NerProcess', name=<name of="" the="" process="">, host=<name url="" of="" the="" host="">,
type=<rql or="" sparql="">, request=<rql or="" sparql="" query="" with="" %(token)s="">)


session.create_entity('NerProcess', name=u'dbpedia38-en', host=u'ner',
type=u'rql', lang=u'en',
request=u'Any U WHERE X label %(token)s, X cwuri U, '
'X ner_source NS, NS name "dbpedia38-en"')

or :

session.create_entity('NerProcess', name=u'dbpedia-sparql', host=u''
type=u'sparql', lang=u'en',
request=u'''SELECT ?uri
?uri rdfs:label "%(w)s"@en .
?uri rdf:type ?type
FILTER(?type IN (dbpedia-owl:Agent, dbpedia-owl:Event,


A command "NerImportDbpedia" exists to import the labels from a dbpedia dump:

* Download the 'labels_en.nt' from Dbpedia (e.g.
in the Dataset "Titles". WARNING ! You should download the NT file.

* Decompress the file

* Use the command:

cubicweb-ctl ner-import-dbpedia <instance name=""> labels_en.nt --name=<name of="" the="" source="">

where <name of="" the="" source=""> could be "dbpedia38-en" for example.


The "INamedEntitiesContentAbstract" adapter could be use to imply that an etype
has a content where Named Entities Recognition could be applied.  
File Type Py Version Uploaded on Size
cubicweb-ner-0.1.0.tar.gz (md5) Source 2014-03-14 19KB