Skip to main content

LeakDB is a very simple and fast key value store for Python

Project description

Why ?

For the fun o/

Overview

LeakDB is a very simple and fast key value store for Python.

All data is stored in memory and the persistence is defined by the user. A max queue size can be defined for a auto-flush.

API

>>> from leakdb import PersistentQueueStorage
>>> leak = PersistentQueueStorage(filename='/tmp/foobar.db')

# set the value of a key
>>> leak.set('bar', {'foo': 'bar'})
>>> leak.set('foo', 2, key_prefix='bar_')

# increment a key
>>> leak.incr(key='bar_foo', delta=5)
 7

>>> leak.incr(key='foobar', initial_value=1000)
 1000

# looks up multiple keys
>>> leak.get_multi(keys=['bar', 'foobar'])
 {u'foobar': 1000, u'bar': {u'foo': u'bar'}}

# ensure changes are sent to disk
>>> print leak
 /tmp/foobar.db 12288 bytes :: 3 items in queue :: 3 items in storage memory

>>> leak.flush(force=True)
 /tmp/foobar.db 12338 bytes :: 0 items in queue :: 3 items in storage memory

>>> leak.close()

STORAGE

  • DefaultStorage :: The default storage, all API operations are implemented set set_multi incr decr get_multi delete

  • QueueStorage :: Use the DefaultStorage with a queue. You can override the QueueStorage.worker_process method and make what you want when the flush method is called.

from leakdb import QueueStorage

class MyQueueStorage(QueueStorage):

    def worker_process(self, item):
        """ Default action execute by each worker.
            Must return a True statement to remove the item,
            otherwise the worker put the item into the queue.
        """
        logger.info('process item :: {}'.format(item))
        return True
  • PersistentStorage :: Use the DefaultStorage, otherwise each operation is stored through the shelve module.

  • PersistentQueueStorage :: Use the QueueStorage and the PersistentStorage.

# see also the API part
from leakdb import PersistentQueueStorage

storage = PersistentQueueStorage(filename="/tmp/foobar.db",  maxsize=1, workers=1)
# the queue is auto-flush, each operations check the queue size
storage.set('foo', 1)

TODO

  • finish the transport layer through zeroMQ

  • cleanup the code

  • improves the unittests

  • write a CLI

  • benchmark each storage

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

LeakDB-0.2.tar.gz (5.2 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page