httplib2 caching for requests
Project description
CacheControl is a port of the caching algorithms in httplib2 for use with requests session object.
It was written because httplib2’s better support for caching is often mitigated by its lack of threadsafety. The same is true of requests in terms of caching.
Usage
There are two ways to use CacheControl. The first is by using the CacheControl wrapper object.
import requests from cachecontrol import CacheControl sess = requests.session() cached_sess = CacheControl(sess) response = cached_sess.get('http://google.com')
If the URL contains any caching based headers, it will cache the result in a simple dictionary.
The second way to use CacheControl is by the CacheControlAdapter. Requests in v1.x introduced the concept of a Transport Adapter. This allows you to configure different behaviors based on the endpoint you are using. For more info check the the docs.
Here is how you can use the adapter:
import requests from cachecontrol import CacheControlAdapter sess = requests.session() sess.mount('http://', CacheControlAdapter()) response = sess.get('http://google.com')
Cache Implementations
CacheControl comes with a simple, threadsafe, in memory dictionary like cache. Implementing a cache interface is simply a matter of inheriting from the BaseCache or implementing the necessary methods.
To make it clear how simple this is, here is the base class:
class BaseCache(object): def get(self, key): raise NotImplemented() def set(self, key, value): raise NotImplemented() def delete(self, key): raise NotImplemented()
See? Simple.
Design
The CacheControl object’s main task is to wrap the GET call of the session object. The caching takes place by examining the request to see if it should try to ue the cache. For example, if the request includes a ‘no-cache’ or ‘max-age=0’ Cache-Control header, it will not try to cache the request. If there is an cached value and its value has been deemed fresh, the it will return the cached response.
If the request cannot be cached, the actual request is peformed. At this point we then analyze the response and see if we should add it to the cache. For example, if the request contains a ‘max-age=3600’ in the ‘Cache-Control’ header, it will cache the response before returning it to the caller.
Tests
The tests are all in cachecontrol/tests and is runnable by py.test.
TODO
Support the Vary header (only match when all headers are the same)
Disclaimers
CacheControl is brand new and maybe totally broken. I have some tests and it is a pretty direct port of httplib2 caching, which I’ve found to be very reliable. With that in mind, it hasn’t been used in a production environment just yet. If you check it out and find bugs, let me know.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.