zc.zodbwsgi 1.2.0

Basic usage

Let’s look at some examples.

First we define an demonstration “application” that we can pass to our factory:

import transaction, ZODB.POSException
from sys import stdout

class demo_app:
    def __init__(self, default):
        pass
    def __call__(self, environ, start_response):
        start_response('200 OK', [('content-type', 'text/html')])
        root = environ['zodb.connection'].root()
        path = environ['PATH_INFO']
        if path == '/inc':
            root['x'] = root.get('x', 0) + 1
            if 'transaction.manager' in environ:
                environ['transaction.manager'].get().note('path: %r' % path)
            else:
                transaction.commit() # We have to commit our own!
        elif path == '/conflict':
            print >>stdout, 'Conflict!'
            raise ZODB.POSException.ConflictError
        elif path == "/tm":
            tm = environ["transaction.manager"]
            return ["thread tm: " + str(tm is transaction.manager)]
        return [repr(root)]

Now, we’ll define our application factory using a paste deployment configuration:

[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>

Here, for demonstration purposes, we used an in-memory demo storage.

Now, we’ll create an application with paste:

>>> import paste.deploy, os
>>> app = paste.deploy.loadapp('config:'+os.path.abspath('paste.ini'))

The resulting applications has a database attribute (mainly for testing) with the created database. Being newly initialized, the database is empty:

>>> conn = app.database.open()
>>> conn.root()
{}

Let’s do an “increment” request.

>>> import webtest
>>> testapp = webtest.TestApp(app)
>>> testapp.get('/inc')
<200 OK text/html body="{'x': 1}">

Now, if we look at the database, we see that there’s now data in the root object:

>>> conn.sync()
>>> conn.root()
{'x': 1}

Database initialization

We can supply a database initialization function using the initializer option. Let’s define an initialization function:

import transaction

def initialize_demo_db(db):
    conn = db.open()
    conn.root()['x'] = 100
    transaction.commit()
    conn.close()

and update our paste configuration to use it:

[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>

initializer = zc.zodbwsgi.tests:initialize_demo_db

Now, when we use the application, we see the impact of the initializer:

>>> app = paste.deploy.loadapp('config:'+os.path.abspath('paste.ini'))
>>> testapp = webtest.TestApp(app)
>>> testapp.get('/inc')
<200 OK text/html body="{'x': 101}">

Disabling transaction management

Sometimes, you may not want the middleware to control transactions. You might do this if your application used multiple databases, including non-ZODB databases [1]. You can suppress transaction management by supplying a value of “false” for the transaction_management option:

[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>

initializer = zc.zodbwsgi.tests:initialize_demo_db
transaction_management = false

Suppressing request retry

By default, zc.zodbwsgi adds repoze.retry middleware to retry requests when there are conflict errors:

>>> import ZODB.POSException
>>> app = paste.deploy.loadapp('config:'+os.path.abspath('paste.ini'))
>>> testapp = webtest.TestApp(app)
>>> try: testapp.get('/conflict')
... except ZODB.POSException.ConflictError: pass
... else: print 'oops'
Conflict!
Conflict!
Conflict!
Conflict!

Here we can see that the request was retried 3 times.

We can suppress this by supplying a value of “0” for the retry option:

[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>

retry = 0

Now, if we run the app, the request won’t be retried:

>>> app = paste.deploy.loadapp('config:'+os.path.abspath('paste.ini'))
>>> testapp = webtest.TestApp(app)
>>> try: testapp.get('/conflict')
... except ZODB.POSException.ConflictError: pass
... else: print 'oops'
Conflict!

Using non-thread-aware (non thread-local) transaction managers

By default, the middleware uses a thread-aware transaction manager:

[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>
initializer = zc.zodbwsgi.tests:initialize_demo_db

This can be controlled via the thread_transaction_manager key:

[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>
initializer = zc.zodbwsgi.tests:initialize_demo_db
thread_transaction_manager = false

demostorage_manage_header

Providing an value for this options enables hooks that allow one to push/pop the underlying demostorage.

[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>

key = connection
transaction_key = manager
demostorage_manage_header = X-FOO

If the push or pop header is provided, the middleware returns a response immediately without sending it to the end of the pipeline.

>>> testapp.get('/', {}, headers={'X-FOO': 'push'}).body
'Demostorage pushed\n'

>>> testapp.get('/inc')
<200 OK text/html body="{'x': 2}">

>>> testapp.get('/', {}, {'X-FOO': 'pop'}).body
'Demostorage popped\n'

>>> testapp.get('/')
<200 OK text/html body="{'x': 1}">

If you have access to the middleware object, you can accomplish the same thing by calling the push and pop methods, which also return the database. This is useful when you’re running the server in the test process and have Python access:

>>> db = app.application.push()

Note that app is a repoze.retry, so we have to use .application to get the wsgi app.

>>> with db.transaction() as conn:
...     conn.root.x = 41

>>> testapp.get('/inc')
<200 OK text/html body="{'x': 42}">

>>> db = app.application.pop()
>>> with db.transaction() as conn:
...     print conn.root.x
1

>>> testapp.get('/')
<200 OK text/html body="{'x': 1}">

This also works with multiple dbs.

class demo_app:
    def __init__(self, default):
        pass
    def __call__(self, environ, start_response):
        start_response('200 OK', [('content-type', 'text/html')])
        path = environ['PATH_INFO']
        root_one = environ['connection'].get_connection('one').root()
        root_two = environ['connection'].get_connection('two').root()
        if path == '/inc':
            root_one['x'] = root_one.get('x', 0) + 1
            root_two['y'] = root_two.get('y', 0) + 1
            environ['manager'].get().note('path: %r' % path)

        data = {'one': root_one,
                'two': root_two}

        return [repr(data)]

[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb one>
     <demostorage>
     </demostorage>
   </zodb>
   <zodb two>
     <demostorage>
     </demostorage>
   </zodb>

key = connection
transaction_key = manager
demostorage_manage_header = X-FOO

If the storage of any of the databases is not a demostorage, an error is returned.

[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb one>
     <demostorage>
     </demostorage>
   </zodb>
   <zodb two>
     <filestorage>
       path /tmp/Data.fs
     </filestorage>
   </zodb>

key = connection
transaction_key = manager
demostorage_manage_header = foo

Limiting the number of connections

If you’re using a threaded server, one that dedicates a thread to each active request, you can limit the number of simultaneous database connections by specifying the number with the max_connections option.

(This only works for threaded servers because it uses threaded semaphores. In the future, support for other locking mechanisms, such as gevent Semaphores, may be added. In the mean time, if you’re inclined to monkey patch, you can replace zc.zodbwsgi.Semaphore with an alternative semaphore implementation, like gevent’s.)

Escaping connection and transaction management

Normally, having connections and transactions managed for you is convenient. Sometimes, however, you want to take over transaction management yourself.

If you close environ['zodb.connection'], then it won’t be closed by zc.zodbwsgi, nor will zc.zodbwsgi commit or abort the transaction it started. If you’re using max_connections, closing environ['zodb.connection'] will make the connection available for other requests immediately, rather than waiting for your request to complete.

Dealing with the occasional long-running requests

Database connections can be pretty expensive resources, especially if they have large database caches. For this reason, when using large caches, it’s common to limit the number of application threads, to limit the number of connections used. If your application is compute bound, you generally want to use one application thread per process and a process per processor on the host machine.

If your application itself makes network requests (e.g calling external service APIs), so it’s network/server bound rather than compute bound, you should increase the number of application threads and decrease the size of the connection caches to compensate.

If your application is mostly compute bound, but sometimes calls external services, you can take a hybrid approach:

• Increase the number of application threads.

• Set max_connections to 1.

• In the parts of your application that make external service calls:

  • Close environ['zodb.connection'], committing first, if necessary.

  • Make your service calls.

  • Open and close ZODB connections yourself when you need to use the database.

    If you’re using ZEO or relstorage, you might want to create separate database clients for use in these calls, configured with smaller caches.

1.2.0 (2015-03-08)

• Record a request summary in transaction meta-data

• Added dependencies to reflect imports.

1.1.0 (2014-04-22)

• Provide Python push and pop methods for use when testing and when running the server in the test process.

1.0.0 (2013-09-15)

• Added support for occasional long-running requests:

  • You can limit the number of database connections with max_connections.

  • You can take over connection and transaction management to release connections while blocking (typically when calling external services).

• Add an option to use a thread-aware transaction manager, and make it the default.

0.3.0 (2013-03-07)

• Using the demostorage hook now returns a response immediately without processing the rest of the pipeline. Makes use of this feature less confusing.

0.2.1 (2013-03-06)

• Fix reference to a file that was renamed.

0.2.0 (2013-03-06)

• Add hooks to manage (push/pop) underlying demostorage based on headers.

• Refactor filter to use instance attributes instead of a closure.

0.1.0 (2010-02-16)

Initial release