Skip to main content

A context manager to do rollbacks automatically

Project description

Purpose

Sometimes we need to perform a series of operations:

op[0], op[1], ... op[N]

These operations may allocate files, locks, connections, and op[K] may depend on result of op[K-1]. Each operation creates a context. When the operations are done, we want to destroy those contexts. Sometimes it’s not feasible to use standard Python Context Manager Protocol, because the number of the resource involved in a transaction can be a variable. There is no way to use a variable number of the with statement, and contextlib.nested is being deprecated. Sometimes it’s just too verbose to create standard Python Context Manager for each context.

This library implements a concise Context Manager and proposes an idiom for rollback.

How It Works

RollbackContext is a context manager to be used in a with statement. Once instantiated, it creates a undo stack. It allows the programme to register undo function after each successful operation. Each new undo function is push to the top of the stack. When the the operations are successful and execution flow leaves the with statement, it tries to pop the undo functions and run them. In all, it runs the following operation series:

op[0], op[1], ... op[N], DONE, -op[N], ... , -op[1], -op[0]

If there is an exception from op[i], it aborts the execution of the op series and starts to run undos registered so far. If there is an exception from the undos, it ignores the exception temporary and continues to run the rest of the undos. At last, it re-raises the earliest exception it sees. This is because latter exceptions may be caused by an earlier exception, so the most helpful exception for diagnosing the problem is the earliest one. Meanwhile, it should execute all the undos to destroy all the contexts as much as possible.

How to Use

  1. Run as superuser, easy_install rollbackcontext or pip install rollbackcontext.

  2. In Python, from rollbackcontext import RollbackContext.

  3. Use it in a with statement as following:

with RollbackContext() as rollback:
  1. Write code for the operations, after each operation success, register a reverse operation by calling the push method.

with RollbackContext() as rollback:
    op0()
    rollbcak.push(op0Reverse)  # op0Reverse is a callable.
    op1(args, ...)
    rollbcak.push(op1Reverse, argsForOp1Reverse, ...)

The push method accepts callable and the optional arguments that would be passed to the callable. In the above case, RollbackContext runs op0(), then op1(args, ...), then op1Reverse(argsForOp1Reverse, ...), and at last op0Reverse().

Examples

A most simple example may be the following:

from sys import stdout

from rollbackcontext import RollbackContext

with RollbackContext() as rollback:
    print "Op 0"
    rollback.push(lambda: stdout.write("Undo 0\n"))
    print "Op 1"
    rollback.push(lambda: stdout.write("Undo 1\n"))
    print "Op 2"
    rollback.push(lambda: stdout.write("Undo 2\n"))
# Prints the following
# Op 0
# Op 1
# Op 2
# Undo 2
# Undo 1
# Undo 0

You can refer to unit test code to find many more examples.

Here are some examples from simplified production code:

def vm_lifecycle(...):
    ''' The task is to create the VM, perform some tests and
    destroy the VM and related resources. '''
    with RollbackContext() as rollback:
        templates_create('testTemplate', ...)
        rollback.push(template_delete, 'testTemplate')

        vms_create('testVM', ...)
        rollback.push(vm_delete, 'testVM')

        vm_start('testVM', ...)
        rollback.push(vm_stop, 'testVM')

        # Do whatever with the VM

Another one:

def prepare(...):
    ''' The task is to detect if a NFS export could be mounted or not. '''
    with RollbackContext() as rollback:
        mnt_point = tempfile.mkdtemp(dir='/tmp')
        rollback.push(os.rmdir, mnt_point)

        mount_cmd = ["mount", ..., mnt_point]
        try:
            run_command(mount_cmd, 30)  # Wait for 30 seconds
        except TimeoutError:
            return False
        umount_cmd = ["umount", "-f", mnt_point]
        rollback.push(run_command, umount_cmd)

        # Do whatever with the mounted filesystem
    return True

Yet another one:

def probe_user(self):
    ''' The task is to start a libvirt domain and detect the user id of the
    VM process. '''
    user = None
    with RollbackContext() as rollback:
        conn = libvirt.open('qemu:///system')
        rollback.push(conn.close)
        dom = conn.defineXML('...')
        rollback.push(dom.undefine)
        dom.create()
        rollback.push(dom.destroy)
        with open('/var/run/libvirt/qemu/%s.pid' % self.vm_name) as f:
            pidStr = f.read()
        p = psutil.Process(int(pidStr))
        user = p.username
    return user

The above code comes from project kimchi, a HTML5 based management tool for KVM.

More Helpful Features

Cancel All Rollbacks

Most of the time we need to run all the undos, but sometimes we want to cancel the undos if all operations are successful. In this case, call the commitAll method to cancel all the undos as following:

with RollbackContext() as rollback:
    print 'Op 0'
    rollback.push(op0Reverse)
    print 'Op 1'
    rollback.push(op1Reverse)
    rollback.commitAll()

If the with statement ends successfully, commitAll() cancels all undo, so that the contexts created in the with statement will be left untouched for future use. If there is exception from the with statement, commitAll() will not be run, so it still runs all the undo functions. Sounds like “start transaction”, “commit transaction” and “automatic rollback” in a database stored procedure, isn’t it?

Cancel a Particular Rollback

Sometimes we want to cancel a particular undo if all operations are successful. In this case, call the setAutoCommit method of the object returned from the push method.

with RollbackContext() as rollback:
   print 'Op 0'
   rollback.push(op0Reverse).setAutoCommit()
   print 'Op 1'
   rollback.push(op1Reverse)

If any exception would be raised within the with statement, op1Reverse() and op2Reverse() would be run. If the with statement was successful, only op1Reverse() would be run.

Register Undo Function to the Bottom of the Stack

Normally the push method adds the undo function to the top of the undo stack. In case you want to insert undo function to the bottom of the undo stack, use the pushBottom method.

from sys import stdout


with RollbackContext() as rollback:
    rollback.pushBottom(lambda: stdout.write("0\n"))
    rollback.pushBottom(lambda: stdout.write("1\n"))
    rollback.pushBottom(lambda: stdout.write("2\n"))
# Should print
# 0
# 1
# 2

Anti-pattern Examples

Unfortunately, C programmers can not enjoy the delight from our RollbackContext, they have to detect error code of each operation and use goto out0, goto out1, and so on, to simulate our RollbackContext manually. The following function comes from Linux kernel source code:

static int __init init_nfs_fs(void)
{
       int err;

       err = register_pernet_subsys(&nfs_net_ops);
       if (err < 0)
               goto out9;

       err = nfs_fscache_register();
       if (err < 0)
               goto out8;

       err = nfsiod_start();
       if (err)
               goto out7;

       err = nfs_fs_proc_init();
       if (err)
               goto out6;

       err = nfs_init_nfspagecache();
       if (err)
               goto out5;

       err = nfs_init_inodecache();
       if (err)
               goto out4;

       err = nfs_init_readpagecache();
       if (err)
               goto out3;

       err = nfs_init_writepagecache();
       if (err)
               goto out2;

       err = nfs_init_directcache();
       if (err)
               goto out1;

#ifdef CONFIG_PROC_FS
       rpc_proc_register(&init_net, &nfs_rpcstat);
#endif
       if ((err = register_nfs_fs()) != 0)
               goto out0;

       return 0;
out0:
#ifdef CONFIG_PROC_FS
       rpc_proc_unregister(&init_net, "nfs");
#endif
       nfs_destroy_directcache();
out1:
       nfs_destroy_writepagecache();
out2:
       nfs_destroy_readpagecache();
out3:
       nfs_destroy_inodecache();
out4:
       nfs_destroy_nfspagecache();
out5:
       nfs_fs_proc_exit();
out6:
       nfsiod_stop();
out7:
       nfs_fscache_unregister();
out8:
       unregister_pernet_subsys(&nfs_net_ops);
out9:
       return err;
}

If this function was to be written in Python (of course it never would), without RollbackContext, you have to write as the following:

def init_nfs_fs():
    with op0() as context0:  # Suppose you wrap opX into context managers
        with op1() as context1:
            # ...
            for c in [contextN, ... , context1, context0]:
               c.cancelDestroy()

Or the following:

def init_nfs_fs():
   try:
       op0()
       try:
           op1()
           # ...
       except Exception:
           op1Reverse()
           raise
   except Exception:
       op0Reverse()
       raise

With the help of RollbackContext, we can re-structure it as the following:

def init_nfs_fs():
    with RollbackContext() as rollback:
        op0()  # Suppose op0() raises exception when it fails
        rollback.push(op0Reverse)
        op1()
        rollback.push(op1Reverse)
        # ...
        rollback.commitAll()

It’s cleaner. Whenever you find yourself dealing with similar case in Python, nesting try...except...finally or with blocks, you might want to have a try on RollbackContext.

For more anti-pattern examples, you can just git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git, and git grep 'goto out5', git grep 'goto out6' and more. Currently the worst case is bfin_lq035q1_probe function in drivers/video/bfin-lq035q1-fb.c, it goto out10. Think of when a developer wants to add a new operation and cleanup code into the existing operation series, he has to manually change all the X in goto outX and outX:. 有多痛苦,你们感受一下 ;-).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rollbackcontext-0.1-2.tar.gz (7.3 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page