Skip to main content

An extension to MongoEngine for memoization and document-aware caching.

Project description

© 2014-2015 Alice Bevan-McGregor and contributors.

https://github.com/marrow/cache

Latest Version Downloads per Week Release Build Status Release Test Coverage Package Requirements Code Climate Github Issues

1. What is Marrow Cache?

Marrow Cache is a light-weight transparent caching system for memoizing functions and MongoEngine Document model methods. It is fully tested and highly focused to this task. Primary features include:

  • “Memoize” the result of arbitrary function calls.

  • Organize cached values into “prefixes”.

  • Intelligently cache the result of Document method calls, with the cached value bound to the primary key of the document; optionally also keying on other fields.

A TTL index in MongoDB will automatically cull expired values once a minute. If overwhelmed, it won’t be able to do them all in one pass. Incremental garbage collection is automatically accounted for by validating the expiry time on any potential cache hit. If invalid, the record will be explicitly deleted and a new one generated.

Be aware of MongoDB’s power of 2 sized allocations.

2. Installation

Installing marrow.cache is easy, just execute the following in a terminal:

pip install marrow.cache

Note: We strongly recommend always using a container, virtualization, or sandboxing environment of some kind when developing using Python; installing things system-wide is yucky (for a variety of reasons) nine times out of ten. We prefer light-weight virtualenv, others prefer solutions as robust as Vagrant.

If you add marrow.cache to the install_requires argument of the call to setup() in your applicaiton’s setup.py file, Marrow Cache will be automatically installed and made available when your own application or library is installed. We recommend using “less than” version numbers to ensure there are no unintentional side-effects when updating. Use marrow.cache<1.1 to get all bugfixes for the current release, and marrow.cache<2.0 to get bugfixes and feature updates while ensuring that large breaking changes are not installed.

Remember to build the indexes. Executing Cache.ensure_indexes() in a shell after first deployment will do so. If you forget and begin to cache data, refer to the MongoEngine documentation on building indexes in the background.

2.1. Development Version

Development Build Status Development Test Coverage Package Requirements

Development takes place on GitHub in the marrow.cache project. Issue tracking, documentation, and downloads are provided there.

Installing the current development version requires Git, a distributed source code management system. If you have Git you can run the following to download and link the development version into your Python runtime:

git clone https://github.com/marrow/cache.git
(cd cache; python setup.py develop)

You can then upgrade to the latest version at any time:

(cd cache; git pull; python setup.py develop)

If you would like to make changes and contribute them back to the project, fork the GitHub project, make your changes, and submit a pull request. This process is beyond the scope of this documentation; for more information see GitHub’s documentation.

3. Functional Interface

Given you have a function that is expensive to execute you can use the Marrow Cache functional interface to automatically preserve the result for more rapid recall on subsequent calls.

There are some important notes regarding behaviour:

  • Arguments to the “generation” function are hashed after being passed through pprint; collisions may occur. This can be alleviated by ensuring reasonable __repr__ implementations or participation in the pretty-print protocol.

  • The returned (and hence cached) values must be encodeable as a DynamicField, i.e. it must map to a BSON type. Some transformations may occur; subclasses of dict will return as an instance of the subclass on cache miss, only to be returned as an actual dict instance on cache hit. See the examples for an approach that works around this.

The most basic approach is a function that takes arguments, does something to them, and returns a result:

from marrow.cache import Cache

@Cache.memoize(minutes=1)
def multiply(x, y):
    return x * y

The memoize decorator takes the same named arguments as timedelta, and defaults to a one week period. The full argument specification is as follows:

Cache.memoize(
        prefix = None,  # defaults to callable's qualified name
        reference = None,  # ObjectId or saved Document instance, optional
        expires = datetime.utcnow,  # you can override the expiry time point of reference

        # timedelta values used against the expiry time during generation
        weeks = 0,  # actually defaults to 1, but not if anything else is defined
        days = 0,
        hours = 0,
        minutes = 0,
        seconds = 0,

        refresh = False  # automatically re-calculate and update the expiry time
    )

In our example, a call such as print(multiply(2, 4)) will generate a MongoDB record like the following:

{
    _id: {
            p: '__main__.multiply',
            r: None,
            hash: '... hash of arguments ...'
        },
    v: 8,
    e: now() + timedelta(minutes=1)
}

If attempting to cache the result of an unreachable function (i.e. most closures) you must supply a prefix.

The original decorated function is available (to bypass caching) using the __func__ attribute.

3.1. Cache Control

The decorated function is given an attribute that when dereferenced becomes a QuerySet mapping to the cached values relevant to that callable. It can be further queried, cleared, etc.

4. Object-Oriented Interface

There is a second decorator that is method-aware. It takes the same arguments as the memoize decorator, but only as positional parameters. It has a simple definition:

Cache.method(*attributes, **kw)

Positional arguments may be strings referring to attributes pulled from the first argument passed to the callable. Presumably this will be a self or cls refernece. These may be nested using dot-notation, with attributes tried first, then array dereferencing. (Numerical values will be array dereferenced regardless.)

For example, to make the value cached automatically dependant on the x attribute of the instance:

from marrow.schema import Container, Attribute

class Multiply(Container):
    x = Attribute()

    @Cache.method('x', minutes=1)
    def do(self, y):
        return self.x * y

If the first argument (self, etc.) is a saved Document instance, pk will be automatically included in the dependant attribute list.

5. Version History

Version 1.0

Version 1.0.1

  • Timezone issue correction. Now correctly handles when timezone-awareness is enabled in MongoEngine/pymongo.

Version 1.0.2

  • Automatic prefix naming. Automatic prefixes are now available on Python versions < 3.3.

6. License

Marrow Cache has been released under the MIT Open Source license.

6.1. The MIT License

Copyright © 2014-2015 Alice Bevan-McGregor and contributors.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

marrow.cache-1.0.3.tar.gz (18.3 kB view hashes)

Uploaded Source

Built Distributions

marrow.cache-1.0.3-py3.4.egg (17.7 kB view hashes)

Uploaded Source

marrow.cache-1.0.3-py3.3.egg (17.8 kB view hashes)

Uploaded Source

marrow.cache-1.0.3-py3.2.egg (17.6 kB view hashes)

Uploaded Source

marrow.cache-1.0.3-py2.py3-none-any.whl (16.1 kB view hashes)

Uploaded Python 2 Python 3

marrow.cache-1.0.3-py2.7.egg (17.3 kB view hashes)

Uploaded Source

marrow.cache-1.0.3-py2.6.egg (17.3 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page