cache_requests 4.0.0

Decorated Methods

method.ex

sets the default expiration (seconds) for new cache entries.

method.redis

creates the connection to the redis or redislite database. By default this is a redislite connection. However, a redis connection can be dropped in for easy scalability.

cache_requests.Session

• ex is shared between request methods. They can be accessed by Session.cache.ex or Session.get.ex, where get is the requests.get method

• By default requests that return and error will not be cached. This can be overridden by overriding the Session.cache.set_cache_cb to return False. The callback takes the response object as an argument

from cache_requests import Session

requests = Session()

requests.cache.set_cache_db = lambda _:False

• By default only autonomous methods are cached (get, head, options). Each method can be setup to be cached using the Session.cache config option.

These methods are accessed through the Session objects Session.cache.[method name]. They can be overridden with the Session.cache.all setting.

For example

from cache_requests import Session

requests = Session()

requests.cache.delete = True

# cached, only called once.
requests.delete('http://google.com')
requests.delete('http://google.com')

requests.cache.delete = True

# not cached, called twice.
requests.delete('http://google.com')
requests.delete('http://google.com')

# cache ALL methods
requests.cache.all = True

# don't cache any methods
requests.cache.all = False

# Use individual method cache options.
requests.cache.all = None

Function Level Config

Cache Busting

Use keyword bust_cache=True in a memoized function to force reevaluation.

Conditionally Set Cache

Use keyword set_cache to provide a callback. The callback takes the results of function as an argument and must return a bool. Alternatively, True and False can be used.

Development: 3rd Party APIs

Scenario:

Working on a project that uses a 3rd party API or service.

Things you want:

• A cache that persists between sessions and is lightning fast.

• Ability to rapidly explore the API and it’s parameters.

• Ability to inspect and debug response content.

• Ability to focus on progress.

• Perfect transition to a production environment.

Things you don’t want:

• Dependency on network and server stability for development.

• Spamming the API. Especially APIs with limits.

• Responses that change in non-meaningful ways.

• Burning energy with copypasta or fake data to run piece of your program.

• Slow. Responses.

Make a request one time. Cache the results for the rest of your work session.

import os

if os.environ.get('ENV') == 'DEVELOP':
    from cache_requests import Session

    request = Session(ex=60 * 60 )  # Set expiration, 60 min
else:
    import requests

# strange, complicated request you might make
headers = {"accept-encoding": "gzip, deflate, sdch", "accept-language": "en-US,en;q=0.8"}
payload = dict(sourceid="chrome-instant", ion="1", espv="2", ie="UTF-8", client="ubuntu",
               q="hash%20a%20dictionary%20python")
response = requests.get('http://google.com/search', headers=headers, params=payload)

# spam to prove a point
response = requests.get('http://google.com/search', headers=headers, params=payload)
response = requests.get('http://google.com/search', headers=headers, params=payload)
response = requests.get('http://google.com/search', headers=headers, params=payload)
response = requests.get('http://google.com/search', headers=headers, params=payload)
response = requests.get('http://google.com/search', headers=headers, params=payload)
response = requests.get('http://google.com/search', headers=headers, params=payload)
response = requests.get('http://google.com/search', headers=headers, params=payload)

# tweak your query, we're exploring here
payload = dict(sourceid="chrome-instant", ion="1", espv="2", ie="UTF-8", client="ubuntu",
               q="hash%20a%20dictionary%20python2")
# do you see what changed? the caching tool did.
response = requests.get('http://google.com/search', headers=headers, params=payload)
response = requests.get('http://google.com/search', headers=headers, params=payload)
response = requests.get('http://google.com/search', headers=headers, params=payload)

Production: Web Scraping

Automatically expire old content.

• How often? After a day? A week? A Month? etc. 100% of this logic is built in with the Session.cache.ex setting.

• Effectively it can manage all of the time-based rotation.

• Perfect if you theres more data then what your API caps allow.

One line of code to use a redis full database.

• Try redislite; it can handle quite a bit. The redislite api used by this module is 1:1 with the redis package. Just replace the connection parameter/config value.

• redis is a drop in:

    connection  = redis.StrictRedis(host='localhost', port=6379, db=0)
    requests = Session(connection=connection)

* Everything else just works.  There's no magic required.

from cache_requests import Session

connection  = redis.StrictRedis(host='localhost', port=6379, db=0)
ex = 7 * 24 * 60 * 60 # 1 week

requests = Session(ex=ex, connection=connection)

for i in range(1000)
    payload = dict(q=i)
    response = requests.get('http://google.com/search', params=payload)
    print(response.text)

Usage: memoize

from cache_requests import Memoize

@Memoize(ex=15 * 60)  # 15 min, default, 60 min
def amazing_but_expensive_function(*args, **kwargs)
    print("You're going to like this")