Skip to main content

On disk key/value store based on levelDB backend

Project description

http://dl.dropbox.com/u/2497327/baneer.png

Key-Value store written in Python and based on levelDB, allows high performance on-disk bulk read/write.

Allows async, multithreaded and/or remote access to a multi-leveldb backend.

Relying on the zeromq network library and msgpack serialization format, it is made to be portable between languages and platforms.

See Documentation for more details

Depends on

  • zmq-3.X

  • pyzmq (built with zmq-3.X)

  • leveldb

  • plyvel

Installation

The easy way:

pip install elevator

The hacker way:

$ pip install fabric
$ fab build
$ pip install -r requirements.txt
$ python setup.py install

Usage

When elevator is installed, you can then launch the server using the elevator executable. Note that a –daemon option is disposable, and allows you to run elevator server as a daemon, storing it’s pid in .pid file in /tmp.

See config/elevator.conf for an example of Elevator configuration.

Example:

$ elevator --help
usage: elevator -h

Elevator command line manager

optional arguments:
    -h, --help       show this help message and exit
    -d, --daemon      Launch elevator as a daemon
    -c, --config      Path to elevator server config file, eventually
    -t, --transport   Transfert protocol (tcp | ipc)
    -b, --bind        Ip to bind server to
    -p, --port        Port the server should listen on
    -w, --workers     How many workers should be spawned (Threads with concurrent access to all the db store)
    -P, --paranoid    If option is set, Elevator will shutdown and log on first unhandled exception

Clients

Python: py-elevator

Go: go-elevator (Early early, so early version)

Clojure : Coming soon

C : Coming soon

Thanks

Thanks to srinikom for its leveldb-server which was a very good base to start from.

n.n.n / 2013-02-01

  • Remove: legacy setup_loggers function

  • fix #123: exposing a database object

  • ref #123: Renamed DatabasesHandler to DatabaseStore

  • Fix: elevator benchmarks

  • Fix: supervisor test should remove their tests files

  • Add: tests for backend atm

  • Add: backend supervisor tests + fixes

  • update: enhance backend majordome management

  • fix #125: backend does not instantiate it’s own DatabasesHandler anymore

  • Fix: elevator tests fakers now uses a clear files/dirs pattern

  • Update: more obvious DatabaseHandler args names

  • Add: benchmarks using hurdles and pyelevator

  • update #120 : Auto re-mount unmounted database on new requests

  • Fix : backend properly tears down workers

  • fix #120, fix #91: Implement Majordom watcher thread

  • Update #120: set databases last access marker

  • Update 120: move ocd worker to backend module

  • Update #121: implement last activity action on workers

  • Update #121: Documented worker

  • Update #121: Workers poll to reduce cpu usage + backend refactoring

  • Update #121: use an internal message protocol between supervisor and workers

  • Update: Moved the backend elements in their own module

  • Fix #122: workers now set their processing state

  • Refactor: moved loggers init in their own log module

  • Update #121: fixed workers stop action

  • Update #121: Added constants to normalize interaction with workers

  • Add #121: basic workers supervisor implementation, implies a lot of refactoring

  • Update: rename server poller

  • Update: use ROUTER/DEALER terminology and rename workerpool and proxy to backend and frontend

  • Update: renamed conf module to args

0.4b / 2013-01-28

  • Fix: Refactor api tests

  • Fix #119: Range and Slice now support include_key, and include_value params

  • Remove: max cache management + Add: Lru cache and bloom filters

0.4a / 2013-01-22

  • Add : Implement PING command

  • Add : Cli module

  • Add : Debian packaging files

  • Update: Use plyvel leveldb backend

  • Update: Use plyvel bloom filter in read operations

  • Update: Add experimental command line doc

  • Update: Set fabfile as a module

  • Update: Documentation to fit with plyvel

  • Update #114: Run MGet against db snapshot

  • Update : working cmdline

  • Fix #114: Enhance MGET perfs by acting on a min/max keys range slice

  • Fix #113: handle MGET arguments in command line

  • Many other little updates and fixes, see logs

0.4 / 2012-10-22

  • Add: restore theme

  • Add : Base sphinx documentation

  • Update : new License MIT

  • Fix #86: IOError when bad config file supplied as cmdline argument

  • Fix #95: Elevator will start and log errors even though some databases are corrupted

  • Fix : log-level debug messages format

  • Fix : travis, tests, requirements

0.3d / 2012-10-19

  • Add : Request error for invalid request messages

  • Update #91: Mount default at server startup

  • Update #91: Mount/Unmount command + auto-mount on connect

  • Update #91: add a ticker class, which executes a function every x seconds

  • Update #30, Update #99: Compress Responses on demande (request.meta[‘compression’])

  • Update #88, Update #99: now responses comes in two parts header+content

  • Update #88: Fix MGet, Range, Slice return values types to suite with new responses format

  • Update #88: Refactored Request/Responses format

  • Update : Refactored DatabasesHandler internal storage

  • Update : Few refactoring on loggers handling

  • Update : Refactored DBConnect no more db_uid to provide in request

  • Fix #97: provide mono-letters args

  • Fix #89: Log requests/responses on log-level=DEBUG

  • Fix #87: Refactored logging

  • Fix #100: Non blocking workers, graceful shutdown == PERFORMANCES

  • Fix #98: Activity logging on both file and stdout

  • Fix #101: fixed ipc handling

  • Fix : api tests for compatibility with new Req/Resp

  • Fix : refactored tests for new Range/Slice behavior when size == 1

  • Fix : Mount/Unmount passed args

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Elevator-0.5.tar.gz (34.5 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page