decorator Better living through Python with decorators </pre><?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" /> <title>The decorator module</title> <meta name="author" content="Michele Simionato" /> <style type="text/css"> .highlight { background: #f8f8f8; } .highlight .c { color: #408080; font-style: italic } /* Comment */ .highlight .err { border: 1px solid #FF0000 } /* Error */ .highlight .k { color: #008000; font-weight: bold } /* Keyword */ .highlight .o { color: #666666 } /* Operator */ .highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */ .highlight .cp { color: #BC7A00 } /* Comment.Preproc */ .highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */ .highlight .cs { color: #408080; font-style: italic } /* Comment.Special */ .highlight .gd { color: #A00000 } /* Generic.Deleted */ .highlight .ge { font-style: italic } /* Generic.Emph */ .highlight .gr { color: #FF0000 } /* Generic.Error */ .highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */ .highlight .gi { color: #00A000 } /* Generic.Inserted */ .highlight .go { color: #808080 } /* Generic.Output */ .highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */ .highlight .gs { font-weight: bold } /* Generic.Strong */ .highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ .highlight .gt { color: #0040D0 } /* Generic.Traceback */ .highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */ .highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */ .highlight .kp { color: #008000 } /* Keyword.Pseudo */ .highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */ .highlight .kt { color: #008000; font-weight: bold } /* Keyword.Type */ .highlight .m { color: #666666 } /* Literal.Number */ .highlight .s { color: #BA2121 } /* Literal.String */ .highlight .na { color: #7D9029 } /* Name.Attribute */ .highlight .nb { color: #008000 } /* Name.Builtin */ .highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */ .highlight .no { color: #880000 } /* Name.Constant */ .highlight .nd { color: #AA22FF } /* Name.Decorator */ .highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */ .highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */ .highlight .nf { color: #0000FF } /* Name.Function */ .highlight .nl { color: #A0A000 } /* Name.Label */ .highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */ .highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */ .highlight .nv { color: #19177C } /* Name.Variable */ .highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */ .highlight .w { color: #bbbbbb } /* Text.Whitespace */ .highlight .mf { color: #666666 } /* Literal.Number.Float */ .highlight .mh { color: #666666 } /* Literal.Number.Hex */ .highlight .mi { color: #666666 } /* Literal.Number.Integer */ .highlight .mo { color: #666666 } /* Literal.Number.Oct */ .highlight .sb { color: #BA2121 } /* Literal.String.Backtick */ .highlight .sc { color: #BA2121 } /* Literal.String.Char */ .highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */ .highlight .s2 { color: #BA2121 } /* Literal.String.Double */ .highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */ .highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */ .highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */ .highlight .sx { color: #008000 } /* Literal.String.Other */ .highlight .sr { color: #BB6688 } /* Literal.String.Regex */ .highlight .s1 { color: #BA2121 } /* Literal.String.Single */ .highlight .ss { color: #19177C } /* Literal.String.Symbol */ .highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */ .highlight .vc { color: #19177C } /* Name.Variable.Class */ .highlight .vg { color: #19177C } /* Name.Variable.Global */ .highlight .vi { color: #19177C } /* Name.Variable.Instance */ .highlight .il { color: #666666 } /* Literal.Number.Integer.Long */ </style> </head> <body> <div class="document" id="the-decorator-module"> <h1 class="title">The <tt class="docutils literal">decorator</tt> module</h1> <table class="docinfo" frame="void" rules="none"> <col class="docinfo-name" /> <col class="docinfo-content" /> <tbody valign="top"> <tr><th class="docinfo-name">Author:</th> <td>Michele Simionato</td></tr> <tr class="field"><th class="docinfo-name">E-mail:</th><td class="field-body"><a class="reference external" href="mailto:michele.simionato@gmail.com">michele.simionato@gmail.com</a></td> </tr> <tr><th class="docinfo-name">Version:</th> <td>3.1.2 (2009-08-25)</td></tr> <tr class="field"><th class="docinfo-name">Requires:</th><td class="field-body">Python 2.4+</td> </tr> <tr class="field"><th class="docinfo-name">Download page:</th><td class="field-body"><a class="reference external" href="http://pypi.python.org/pypi/decorator/3.1.1">http://pypi.python.org/pypi/decorator/3.1.2</a></td> </tr> <tr class="field"><th class="docinfo-name">Installation:</th><td class="field-body"><tt class="docutils literal">easy_install decorator</tt></td> </tr> <tr class="field"><th class="docinfo-name">License:</th><td class="field-body">BSD license</td> </tr> </tbody> </table> <div class="contents topic" id="contents"> Contents <ul class="simple"> <li><a class="reference internal" href="#introduction" id="id3">Introduction</a></li> <li><a class="reference internal" href="#definitions" id="id4">Definitions</a></li> <li><a class="reference internal" href="#statement-of-the-problem" id="id5">Statement of the problem</a></li> <li><a class="reference internal" href="#the-solution" id="id6">The solution</a></li> <li><a class="reference internal" href="#a-trace-decorator" id="id7">A <tt class="docutils literal">trace</tt> decorator</a></li> <li><a class="reference internal" href="#decorator-is-a-decorator" id="id8"><tt class="docutils literal">decorator</tt> is a decorator</a></li> <li><a class="reference internal" href="#blocking" id="id9"><tt class="docutils literal">blocking</tt></a></li> <li><a class="reference internal" href="#async" id="id10"><tt class="docutils literal">async</tt></a></li> <li><a class="reference internal" href="#the-functionmaker-class" id="id11">The <tt class="docutils literal">FunctionMaker</tt> class</a></li> <li><a class="reference internal" href="#getting-the-source-code" id="id12">Getting the source code</a></li> <li><a class="reference internal" href="#dealing-with-third-party-decorators" id="id13">Dealing with third party decorators</a></li> <li><a class="reference internal" href="#caveats-and-limitations" id="id14">Caveats and limitations</a></li> <li><a class="reference internal" href="#compatibility-notes" id="id15">Compatibility notes</a></li> <li><a class="reference internal" href="#licence" id="id16">LICENCE</a></li> </ul> </div> <div class="section" id="introduction"> <h1><a class="toc-backref" href="#id3">Introduction</a></h1> Python decorators are an interesting example of why syntactic sugar matters. In principle, their introduction in Python 2.4 changed nothing, since they do not provide any new functionality which was not already present in the language. In practice, their introduction has significantly changed the way we structure our programs in Python. I believe the change is for the best, and that decorators are a great idea since: <ul class="simple"> <li>decorators help reducing boilerplate code;</li> <li>decorators help separation of concerns;</li> <li>decorators enhance readability and maintenability;</li> <li>decorators are explicit.</li> </ul> Still, as of now, writing custom decorators correctly requires some experience and it is not as easy as it could be. For instance, typical implementations of decorators involve nested functions, and we all know that flat is better than nested. The aim of the <tt class="docutils literal">decorator</tt> module it to simplify the usage of decorators for the average programmer, and to popularize decorators by showing various non-trivial examples. Of course, as all techniques, decorators can be abused (I have seen that) and you should not try to solve every problem with a decorator, just because you can. You may find the source code for all the examples discussed here in the <tt class="docutils literal">documentation.py</tt> file, which contains this documentation in the form of doctests. </div> <div class="section" id="definitions"> <h1><a class="toc-backref" href="#id4">Definitions</a></h1> Technically speaking, any Python object which can be called with one argument can be used as a decorator. However, this definition is somewhat too large to be really useful. It is more convenient to split the generic class of decorators in two subclasses: <ul class="simple"> <li>signature-preserving decorators, i.e. callable objects taking a function as input and returning a function with the same signature as output;</li> <li>signature-changing decorators, i.e. decorators that change the signature of their input function, or decorators returning non-callable objects.</li> </ul> Signature-changing decorators have their use: for instance the builtin classes <tt class="docutils literal">staticmethod</tt> and <tt class="docutils literal">classmethod</tt> are in this group, since they take functions and return descriptor objects which are not functions, nor callables. However, signature-preserving decorators are more common and easier to reason about; in particular signature-preserving decorators can be composed together whereas other decorators in general cannot. Writing signature-preserving decorators from scratch is not that obvious, especially if one wants to define proper decorators that can accept functions with any signature. A simple example will clarify the issue. </div> <div class="section" id="statement-of-the-problem"> <h1><a class="toc-backref" href="#id5">Statement of the problem</a></h1> A very common use case for decorators is the memoization of functions. A <tt class="docutils literal">memoize</tt> decorator works by caching the result of the function call in a dictionary, so that the next time the function is called with the same input parameters the result is retrieved from the cache and not recomputed. There are many implementations of <tt class="docutils literal">memoize</tt> in <a class="reference external" href="http://www.python.org/moin/PythonDecoratorLibrary">http://www.python.org/moin/PythonDecoratorLibrary</a>, but they do not preserve the signature. A simple implementation for Python 2.5 could be the following (notice that in general it is impossible to memoize correctly something that depends on non-hashable arguments): <div class="codeblock python"> <div class="highlight"><pre>def memoize25(func): func.cache = {} def memoize(*args, **kw): if kw: # frozenset is used to ensure hashability key = args, frozenset(kw.iteritems()) else: key = args cache = func.cache if key in cache: return cache[key] else: cache[key] = result = func(*args, **kw) return result return functools.update_wrapper(memoize, func) </pre></div> </div> Here we used the <a class="reference external" href="http://www.python.org/doc/2.5.2/lib/module-functools.html">functools.update_wrapper</a> utility, which has been added in Python 2.5 expressly to simplify the definition of decorators (in older versions of Python you need to copy the function attributes <tt class="docutils literal">__name__</tt>, <tt class="docutils literal">__doc__</tt>, <tt class="docutils literal">__module__</tt> and <tt class="docutils literal">__dict__</tt> from the original function to the decorated function by hand). The implementation above works in the sense that the decorator can accept functions with generic signatures; unfortunately this implementation does not define a signature-preserving decorator, since in general <tt class="docutils literal">memoize25</tt> returns a function with a different signature from the original function. Consider for instance the following case: <div class="codeblock python"> <div class="highlight"><pre>>>> @memoize25 ... def f1(x): ... time.sleep(1) # simulate some long computation ... return x </pre></div> </div> Here the original function takes a single argument named <tt class="docutils literal">x</tt>, but the decorated function takes any number of arguments and keyword arguments: <div class="codeblock python"> <div class="highlight"><pre>>>> from inspect import getargspec >>> print getargspec(f1) ([], 'args', 'kw', None) </pre></div> </div> This means that introspection tools such as pydoc will give wrong informations about the signature of <tt class="docutils literal">f1</tt>. This is pretty bad: pydoc will tell you that the function accepts a generic signature <tt class="docutils literal">*args</tt>, <tt class="docutils literal">**kw</tt>, but when you try to call the function with more than an argument, you will get an error: <div class="codeblock python"> <div class="highlight"><pre>>>> f1(0, 1) Traceback (most recent call last): ... TypeError: f1() takes exactly 1 argument (2 given) </pre></div> </div> </div> <div class="section" id="the-solution"> <h1><a class="toc-backref" href="#id6">The solution</a></h1> The solution is to provide a generic factory of generators, which hides the complexity of making signature-preserving decorators from the application programmer. The <tt class="docutils literal">decorator</tt> function in the <tt class="docutils literal">decorator</tt> module is such a factory: <div class="codeblock python"> <div class="highlight"><pre>>>> from decorator import decorator </pre></div> </div> <tt class="docutils literal">decorator</tt> takes two arguments, a caller function describing the functionality of the decorator and a function to be decorated; it returns the decorated function. The caller function must have signature <tt class="docutils literal">(f, *args, **kw)</tt> and it must call the original function <tt class="docutils literal">f</tt> with arguments <tt class="docutils literal">args</tt> and <tt class="docutils literal">kw</tt>, implementing the wanted capability, i.e. memoization in this case: <div class="codeblock python"> <div class="highlight"><pre>def _memoize(func, *args, **kw): if kw: # frozenset is used to ensure hashability key = args, frozenset(kw.iteritems()) else: key = args cache = func.cache # attributed added by memoize if key in cache: return cache[key] else: cache[key] = result = func(*args, **kw) return result </pre></div> </div> At this point you can define your decorator as follows: <div class="codeblock python"> <div class="highlight"><pre>def memoize(f): f.cache = {} return decorator(_memoize, f) </pre></div> </div> The difference with respect to the Python 2.5 approach, which is based on nested functions, is that the decorator module forces you to lift the inner function at the outer level (flat is better than nested). Moreover, you are forced to pass explicitly the function you want to decorate to the caller function. Here is a test of usage: <div class="codeblock python"> <div class="highlight"><pre>>>> @memoize ... def heavy_computation(): ... time.sleep(2) ... return "done" >>> print heavy_computation() # the first time it will take 2 seconds done >>> print heavy_computation() # the second time it will be instantaneous done </pre></div> </div> The signature of <tt class="docutils literal">heavy_computation</tt> is the one you would expect: <div class="codeblock python"> <div class="highlight"><pre>>>> print getargspec(heavy_computation) ([], None, None, None) </pre></div> </div> </div> <div class="section" id="a-trace-decorator"> <h1><a class="toc-backref" href="#id7">A <tt class="docutils literal">trace</tt> decorator</a></h1> As an additional example, here is how you can define a trivial <tt class="docutils literal">trace</tt> decorator, which prints a message everytime the traced function is called: <div class="codeblock python"> <div class="highlight"><pre>def _trace(f, *args, **kw): print "calling %s with args %s, %s" % (f.__name__, args, kw) return f(*args, **kw) </pre></div> </div> <div class="codeblock python"> <div class="highlight"><pre>def trace(f): return decorator(_trace, f) </pre></div> </div> Here is an example of usage: <div class="codeblock python"> <div class="highlight"><pre>>>> @trace ... def f1(x): ... pass </pre></div> </div> It is immediate to verify that <tt class="docutils literal">f1</tt> works <div class="codeblock python"> <div class="highlight"><pre>>>> f1(0) calling f1 with args (0,), {} </pre></div> </div> and it that it has the correct signature: <div class="codeblock python"> <div class="highlight"><pre>>>> print getargspec(f1) (['x'], None, None, None) </pre></div> </div> The same decorator works with functions of any signature: <div class="codeblock python"> <div class="highlight"><pre>>>> @trace ... def f(x, y=1, z=2, *args, **kw): ... pass >>> f(0, 3) calling f with args (0, 3, 2), {} >>> print getargspec(f) (['x', 'y', 'z'], 'args', 'kw', (1, 2)) </pre></div> </div> That includes even functions with exotic signatures like the following: <div class="codeblock python"> <div class="highlight"><pre>>>> @trace ... def exotic_signature((x, y)=(1,2)): return x+y >>> print getargspec(exotic_signature) ([['x', 'y']], None, None, ((1, 2),)) >>> exotic_signature() calling exotic_signature with args ((1, 2),), {} 3 </pre></div> </div> Notice that the support for exotic signatures has been deprecated in Python 2.6 and removed in Python 3.0. </div> <div class="section" id="decorator-is-a-decorator"> <h1><a class="toc-backref" href="#id8"><tt class="docutils literal">decorator</tt> is a decorator</a></h1> It may be annoying to write a caller function (like the <tt class="docutils literal">_trace</tt> function above) and then a trivial wrapper (<tt class="docutils literal">def trace(f): return decorator(_trace, f)</tt>) every time. For this reason, the <tt class="docutils literal">decorator</tt> module provides an easy shortcut to convert the caller function into a signature-preserving decorator: you can just call <tt class="docutils literal">decorator</tt> with a single argument. In our example you can just write <tt class="docutils literal">trace = decorator(_trace)</tt>. The <tt class="docutils literal">decorator</tt> function can also be used as a signature-changing decorator, just as <tt class="docutils literal">classmethod</tt> and <tt class="docutils literal">staticmethod</tt>. However, <tt class="docutils literal">classmethod</tt> and <tt class="docutils literal">staticmethod</tt> return generic objects which are not callable, while <tt class="docutils literal">decorator</tt> returns signature-preserving decorators, i.e. functions of a single argument. For instance, you can write directly <div class="codeblock python"> <div class="highlight"><pre>>>> @decorator ... def trace(f, *args, **kw): ... print "calling %s with args %s, %s" % (f.func_name, args, kw) ... return f(*args, **kw) </pre></div> </div> and now <tt class="docutils literal">trace</tt> will be a decorator. Actually <tt class="docutils literal">trace</tt> is a <tt class="docutils literal">partial</tt> object which can be used as a decorator: <div class="codeblock python"> <div class="highlight"><pre>>>> trace <function trace at 0x...> </pre></div> </div> Here is an example of usage: <div class="codeblock python"> <div class="highlight"><pre>>>> @trace ... def func(): pass >>> func() calling func with args (), {} </pre></div> </div> If you are using an old Python version (Python 2.4) the <tt class="docutils literal">decorator</tt> module provides a poor man replacement for <tt class="docutils literal">functools.partial</tt>. There is also an easy way to create one-parameter factories of decorators, based on the following <tt class="docutils literal">decorator_factory</tt> utility: <div class="codeblock python"> <div class="highlight"><pre>def decorator_factory(decfac): # partial is functools.partial "decorator_factory(decfac) returns a one-parameter family of decorators" return partial(lambda df, param: decorator(partial(df, param)), decfac) </pre></div> </div> <tt class="docutils literal">decorator_factory</tt> converts a function with signature <tt class="docutils literal">(param, func, *args, **kw)</tt> into a one-parameter family of decorators. Suppose for instance you want to generated different tracing generator, with different tracing messages. Here is how to do it: <div class="codeblock python"> <div class="highlight"><pre>>>> @decorator_factory ... def trace_factory(message_template, f, *args, **kw): ... name = f.func_name ... print message_template % locals() ... return f(*args, **kw) </pre></div> </div> <div class="codeblock python"> <div class="highlight"><pre>>>> trace_factory <functools.partial object at 0x...> >>> trace = trace_factory('Calling %(name)s with args %(args)s ' ... 'and keywords %(kw)s') </pre></div> </div> In this example the parameter (<tt class="docutils literal">message_template</tt>) is just a string, but in general it can be a tuple, a dictionary, or a generic object, so there is no real restriction (for instance, if you want to define a two-parameter family of decorators just use a tuple with two arguments as parameter). Here is an example of usage: <div class="codeblock python"> <div class="highlight"><pre>>>> @trace ... def func(): pass >>> func() Calling func with args () and keywords {} </pre></div> </div> </div> <div class="section" id="blocking"> <h1><a class="toc-backref" href="#id9"><tt class="docutils literal">blocking</tt></a></h1> Sometimes one has to deal with blocking resources, such as <tt class="docutils literal">stdin</tt>, and sometimes it is best to have back a "busy" message than to block everything. This behavior can be implemented with a suitable family of decorators, where the parameter is the busy message: <div class="codeblock python"> <div class="highlight"><pre>@decorator_factory def blocking(not_avail, f, *args, **kw): if not hasattr(f, "thread"): # no thread running def set_result(): f.result = f(*args, **kw) f.thread = threading.Thread(None, set_result) f.thread.start() return not_avail elif f.thread.isAlive(): return not_avail else: # the thread is ended, return the stored result del f.thread return f.result </pre></div> </div> Functions decorated with <tt class="docutils literal">blocking</tt> will return a busy message if the resource is unavailable, and the intended result if the resource is available. For instance: <div class="codeblock python"> <div class="highlight"><pre>>>> @blocking("Please wait ...") ... def read_data(): ... time.sleep(3) # simulate a blocking resource ... return "some data" >>> print read_data() # data is not available yet Please wait ... >>> time.sleep(1) >>> print read_data() # data is not available yet Please wait ... >>> time.sleep(1) >>> print read_data() # data is not available yet Please wait ... >>> time.sleep(1.1) # after 3.1 seconds, data is available >>> print read_data() some data </pre></div> </div> </div> <div class="section" id="async"> <h1><a class="toc-backref" href="#id10"><tt class="docutils literal">async</tt></a></h1> We have just seen an examples of a simple decorator factory, implemented as a function returning a decorator. For more complex situations, it is more convenient to implement decorator factories as classes returning callable objects that can be used as signature-preserving decorators. The suggested pattern to do that is to introduce a helper method <tt class="docutils literal">call(self, func, *args, **kw)</tt> and to call it in the <tt class="docutils literal">__call__(self, func)</tt> method. As an example, here I show a decorator which is able to convert a blocking function into an asynchronous function. The function, when called, is executed in a separate thread. Moreover, it is possible to set three callbacks <tt class="docutils literal">on_success</tt>, <tt class="docutils literal">on_failure</tt> and <tt class="docutils literal">on_closing</tt>, to specify how to manage the function call. The implementation is the following: <div class="codeblock python"> <div class="highlight"><pre>def on_success(result): # default implementation "Called on the result of the function" return result </pre></div> </div> <div class="codeblock python"> <div class="highlight"><pre>def on_failure(exc_info): # default implementation "Called if the function fails" pass </pre></div> </div> <div class="codeblock python"> <div class="highlight"><pre>def on_closing(): # default implementation "Called at the end, both in case of success and failure" pass </pre></div> </div> <div class="codeblock python"> <div class="highlight"><pre>class Async(object): """ A decorator converting blocking functions into asynchronous functions, by using threads or processes. Examples: async_with_threads = Async(threading.Thread) async_with_processes = Async(multiprocessing.Process) """ def __init__(self, threadfactory): self.threadfactory = threadfactory def __call__(self, func, on_success=on_success, on_failure=on_failure, on_closing=on_closing): # every decorated function has its own independent thread counter func.counter = itertools.count(1) func.on_success = on_success func.on_failure = on_failure func.on_closing = on_closing return decorator(self.call, func) def call(self, func, *args, **kw): def func_wrapper(): try: result = func(*args, **kw) except: func.on_failure(sys.exc_info()) else: return func.on_success(result) finally: func.on_closing() name = '%s-%s' % (func.__name__, func.counter.next()) thread = self.threadfactory(None, func_wrapper, name) thread.start() return thread </pre></div> </div> The decorated function returns the current execution thread, which can be stored and checked later, for instance to verify that the thread <tt class="docutils literal">.isAlive()</tt>. Here is an example of usage. Suppose one wants to write some data to an external resource which can be accessed by a single user at once (for instance a printer). Then the access to the writing function must be locked. Here is a minimalistic example: <div class="codeblock python"> <div class="highlight"><pre>>>> async = Async(threading.Thread) >>> datalist = [] # for simplicity the written data are stored into a list. >>> @async ... def write(data): ... # append data to the datalist by locking ... with threading.Lock(): ... time.sleep(1) # emulate some long running operation ... datalist.append(data) ... # other operations not requiring a lock here </pre></div> </div> Each call to <tt class="docutils literal">write</tt> will create a new writer thread, but there will be no synchronization problems since <tt class="docutils literal">write</tt> is locked. <pre class="doctest-block"> >>> write("data1") <Thread(write-1, started...)> </pre> <pre class="doctest-block"> >>> time.sleep(.1) # wait a bit, so we are sure data2 is written after data1 </pre> <pre class="doctest-block"> >>> write("data2") <Thread(write-2, started...)> </pre> <pre class="doctest-block"> >>> time.sleep(2) # wait for the writers to complete </pre> <pre class="doctest-block"> >>> print datalist ['data1', 'data2'] </pre> </div> <div class="section" id="the-functionmaker-class"> <h1><a class="toc-backref" href="#id11">The <tt class="docutils literal">FunctionMaker</tt> class</a></h1> You may wonder about how the functionality of the <tt class="docutils literal">decorator</tt> module is implemented. The basic building block is a <tt class="docutils literal">FunctionMaker</tt> class which is able to generate on the fly functions with a given name and signature from a function template passed as a string. Generally speaking, you should not need to resort to <tt class="docutils literal">FunctionMaker</tt> when writing ordinary decorators, but it is handy in some circumstances. You will see an example shortly, in the implementation of a cool decorator utility (<tt class="docutils literal">decorator_apply</tt>). <tt class="docutils literal">FunctionMaker</tt> provides a <tt class="docutils literal">.create</tt> classmethod which takes as input the name, signature, and body of the function we want to generate as well as the execution environment were the function is generated by <tt class="docutils literal">exec</tt>. Here is an example: <div class="codeblock python"> <div class="highlight"><pre>>>> def f(*args, **kw): # a function with a generic signature ... print args, kw >>> f1 = FunctionMaker.create('f1(a, b)', 'f(a, b)', dict(f=f)) >>> f1(1,2) (1, 2) {} </pre></div> </div> It is important to notice that the function body is interpolated before being executed, so be careful with the <tt class="docutils literal">%</tt> sign! <tt class="docutils literal">FunctionMaker.create</tt> also accepts keyword arguments and such arguments are attached to the resulting function. This is useful if you want to set some function attributes, for instance the docstring <tt class="docutils literal">__doc__</tt>. For debugging/introspection purposes it may be useful to see the source code of the generated function; to do that, just pass the flag <tt class="docutils literal">addsource=True</tt> and a <tt class="docutils literal">__source__</tt> attribute will be added to the generated function: <div class="codeblock python"> <div class="highlight"><pre>>>> f1 = FunctionMaker.create( ... 'f1(a, b)', 'f(a, b)', dict(f=f), addsource=True) >>> print f1.__source__ def f1(a, b): f(a, b) <BLANKLINE> </pre></div> </div> <tt class="docutils literal">FunctionMaker.create</tt> can take as first argument a string, as in the examples before, or a function. This is the most common usage, since typically you want to decorate a pre-existing function. A framework author may want to use directly <tt class="docutils literal">FunctionMaker.create</tt> instead of <tt class="docutils literal">decorator</tt>, since it gives you direct access to the body of the generated function. For instance, suppose you want to instrument the <tt class="docutils literal">__init__</tt> methods of a set of classes, by preserving their signature (such use case is not made up; this is done in SQAlchemy and in other frameworks). When the first argument of <tt class="docutils literal">FunctionMaker.create</tt> is a function, a <tt class="docutils literal">FunctionMaker</tt> object is instantiated internally, with attributes <tt class="docutils literal">args</tt>, <tt class="docutils literal">varargs</tt>, <tt class="docutils literal">keywords</tt> and <tt class="docutils literal">defaults</tt> which are the the return values of the standard library function <tt class="docutils literal">inspect.getargspec</tt>. For each argument in the <tt class="docutils literal">args</tt> (which is a list of strings containing the names of the mandatory arguments) an attribute <tt class="docutils literal">arg0</tt>, <tt class="docutils literal">arg1</tt>, ..., <tt class="docutils literal">argN</tt> is also generated. Finally, there is a <tt class="docutils literal">signature</tt> attribute, a string with the signature of the original function. Notice that while I do not have plans to change or remove the functionality provided in the <tt class="docutils literal">FunctionMaker</tt> class, I do not guarantee that it will stay unchanged forever. For instance, right now I am using the traditional string interpolation syntax for function templates, but Python 2.6 and Python 3.0 provide a newer interpolation syntax and I may use the new syntax in the future. On the other hand, the functionality provided by <tt class="docutils literal">decorator</tt> has been there from version 0.1 and it is guaranteed to stay there forever. </div> <div class="section" id="getting-the-source-code"> <h1><a class="toc-backref" href="#id12">Getting the source code</a></h1> Internally <tt class="docutils literal">FunctionMaker.create</tt> uses <tt class="docutils literal">exec</tt> to generate the decorated function. Therefore <tt class="docutils literal">inspect.getsource</tt> will not work for decorated functions. That means that the usual '??' trick in IPython will give you the (right on the spot) message <tt class="docutils literal">Dynamically generated function. No source code available</tt>. In the past I have considered this acceptable, since <tt class="docutils literal">inspect.getsource</tt> does not really work even with regular decorators. In that case <tt class="docutils literal">inspect.getsource</tt> gives you the wrapper source code which is probably not what you want: <div class="codeblock python"> <div class="highlight"><pre>def identity_dec(func): def wrapper(*args, **kw): return func(*args, **kw) return wrapper </pre></div> </div> <div class="codeblock python"> <div class="highlight"><pre>@identity_dec def example(): pass >>> print inspect.getsource(example) def wrapper(*args, **kw): return func(*args, **kw) <BLANKLINE> </pre></div> </div> (see bug report <a class="reference external" href="http://bugs.python.org/issue1764286">1764286</a> for an explanation of what is happening). Unfortunately the bug is still there, even in Python 2.6 and 3.0. There is however a workaround. The decorator module adds an attribute <tt class="docutils literal">.undecorated</tt> to the decorated function, containing a reference to the original function. The easy way to get the source code is to call <tt class="docutils literal">inspect.getsource</tt> on the undecorated function: <div class="codeblock python"> <div class="highlight"><pre>>>> print inspect.getsource(factorial.undecorated) @tail_recursive def factorial(n, acc=1): "The good old factorial" if n == 0: return acc return factorial(n-1, n*acc) <BLANKLINE> </pre></div> </div> </div> <div class="section" id="dealing-with-third-party-decorators"> <h1><a class="toc-backref" href="#id13">Dealing with third party decorators</a></h1> Sometimes you find on the net some cool decorator that you would like to include in your code. However, more often than not the cool decorator is not signature-preserving. Therefore you may want an easy way to upgrade third party decorators to signature-preserving decorators without having to rewrite them in terms of <tt class="docutils literal">decorator</tt>. You can use a <tt class="docutils literal">FunctionMaker</tt> to implement that functionality as follows: <div class="codeblock python"> <div class="highlight"><pre>def decorator_apply(dec, func): """ Decorate a function by preserving the signature even if dec is not a signature-preserving decorator. """ return FunctionMaker.create( func, 'return decorated(%(signature)s)', dict(decorated=dec(func)), undecorated=func) </pre></div> </div> <tt class="docutils literal">decorator_apply</tt> sets the attribute <tt class="docutils literal">.undecorated</tt> of the generated function to the original function, so that you can get the right source code. Notice that I am not providing this functionality in the <tt class="docutils literal">decorator</tt> module directly since I think it is best to rewrite the decorator rather than adding an additional level of indirection. However, practicality beats purity, so you can add <tt class="docutils literal">decorator_apply</tt> to your toolbox and use it if you need to. In order to give an example of usage of <tt class="docutils literal">decorator_apply</tt>, I will show a pretty slick decorator that converts a tail-recursive function in an iterative function. I have shamelessly stolen the basic idea from Kay Schluehr's recipe in the Python Cookbook, <a class="reference external" href="http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496691">http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496691</a>. <div class="codeblock python"> <div class="highlight"><pre>class TailRecursive(object): """ tail_recursive decorator based on Kay Schluehr's recipe http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496691 with improvements by me and George Sakkis. """ def __init__(self, func): self.func = func self.firstcall = True self.CONTINUE = object() # sentinel def __call__(self, *args, **kwd): CONTINUE = self.CONTINUE if self.firstcall: func = self.func self.firstcall = False try: while True: result = func(*args, **kwd) if result is CONTINUE: # update arguments args, kwd = self.argskwd else: # last call return result finally: self.firstcall = True else: # return the arguments of the tail call self.argskwd = args, kwd return CONTINUE </pre></div> </div> Here the decorator is implemented as a class returning callable objects. <div class="codeblock python"> <div class="highlight"><pre>def tail_recursive(func): return decorator_apply(TailRecursive, func) </pre></div> </div> Here is how you apply the upgraded decorator to the good old factorial: <div class="codeblock python"> <div class="highlight"><pre>@tail_recursive def factorial(n, acc=1): "The good old factorial" if n == 0: return acc return factorial(n-1, n*acc) </pre></div> </div> <div class="codeblock python"> <div class="highlight"><pre>>>> print factorial(4) 24 </pre></div> </div> This decorator is pretty impressive, and should give you some food for your mind ;) Notice that there is no recursion limit now, and you can easily compute <tt class="docutils literal">factorial(1001)</tt> or larger without filling the stack frame. Notice also that the decorator will not work on functions which are not tail recursive, such as the following <div class="codeblock python"> <div class="highlight"><pre>def fact(n): # this is not tail-recursive if n == 0: return 1 return n * fact(n-1) </pre></div> </div> (reminder: a function is tail recursive if it either returns a value without making a recursive call, or returns directly the result of a recursive call). </div> <div class="section" id="caveats-and-limitations"> <h1><a class="toc-backref" href="#id14">Caveats and limitations</a></h1> The first thing you should be aware of, it the fact that decorators have a performance penalty. The worse case is shown by the following example: <pre class="literal-block"> $ cat performance.sh python -m timeit -s " from decorator import decorator @decorator def do_nothing(func, *args, **kw): return func(*args, **kw) @do_nothing def f(): pass " "f()" python -m timeit -s " def f(): pass " "f()" </pre> On my MacBook, using the <tt class="docutils literal">do_nothing</tt> decorator instead of the plain function is more than three times slower: <pre class="literal-block"> $ bash performance.sh 1000000 loops, best of 3: 0.995 usec per loop 1000000 loops, best of 3: 0.273 usec per loop </pre> It should be noted that a real life function would probably do something more useful than <tt class="docutils literal">f</tt> here, and therefore in real life the performance penalty could be completely negligible. As always, the only way to know if there is a penalty in your specific use case is to measure it. You should be aware that decorators will make your tracebacks longer and more difficult to understand. Consider this example: <div class="codeblock python"> <div class="highlight"><pre>>>> @trace ... def f(): ... 1/0 </pre></div> </div> Calling <tt class="docutils literal">f()</tt> will give you a <tt class="docutils literal">ZeroDivisionError</tt>, but since the function is decorated the traceback will be longer: <div class="codeblock python"> <div class="highlight"><pre>>>> f() Traceback (most recent call last): ... File "<string>", line 2, in f File "<doctest __main__[18]>", line 4, in trace return f(*args, **kw) File "<doctest __main__[47]>", line 3, in f 1/0 ZeroDivisionError: integer division or modulo by zero </pre></div> </div> You see here the inner call to the decorator <tt class="docutils literal">trace</tt>, which calls <tt class="docutils literal">f(*args, **kw)</tt>, and a reference to <tt class="docutils literal">File "<string>", line 2, in f</tt>. This latter reference is due to the fact that internally the decorator module uses <tt class="docutils literal">exec</tt> to generate the decorated function. Notice that <tt class="docutils literal">exec</tt> is not responsibile for the performance penalty, since is the called only once at function decoration time, and not every time the decorated function is called. At present, there is no clean way to avoid <tt class="docutils literal">exec</tt>. A clean solution would require to change the CPython implementation of functions and add an hook to make it possible to change their signature directly. That could happen in future versions of Python (see PEP <a class="reference external" href="http://www.python.org/dev/peps/pep-0362">362</a>) and then the decorator module would become obsolete. However, at present, even in Python 3.1 it is impossible to change the function signature directly, therefore the <tt class="docutils literal">decorator</tt> module is still useful. Actually, this is one of the main reasons why I keep maintaining the module and releasing new versions. In the present implementation, decorators generated by <tt class="docutils literal">decorator</tt> can only be used on user-defined Python functions or methods, not on generic callable objects, nor on built-in functions, due to limitations of the <tt class="docutils literal">inspect</tt> module in the standard library. Moreover, notice that you can decorate a method, but only before if becomes a bound or unbound method, i.e. inside the class. Here is an example of valid decoration: <div class="codeblock python"> <div class="highlight"><pre>>>> class C(object): ... @trace ... def meth(self): ... pass </pre></div> </div> Here is an example of invalid decoration, when the decorator in called too late: <div class="codeblock python"> <div class="highlight"><pre>>>> class C(object): ... def meth(self): ... pass ... >>> trace(C.meth) Traceback (most recent call last): ... TypeError: You are decorating a non function: <unbound method C.meth> </pre></div> </div> The solution is to extract the inner function from the unbound method: <div class="codeblock python"> <div class="highlight"><pre>>>> trace(C.meth.im_func) <function meth at 0x...> </pre></div> </div> There is a restriction on the names of the arguments: for instance, if try to call an argument <tt class="docutils literal">_call_</tt> or <tt class="docutils literal">_func_</tt> you will get a <tt class="docutils literal">NameError</tt>: <div class="codeblock python"> <div class="highlight"><pre>>>> @trace ... def f(_func_): print f ... Traceback (most recent call last): ... NameError: _func_ is overridden in def f(_func_): return _call_(_func_, _func_) </pre></div> </div> Finally, the implementation is such that the decorated function contains a copy of the original function dictionary (<tt class="docutils literal">vars(decorated_f) is not vars(f)</tt>): <div class="codeblock python"> <div class="highlight"><pre>>>> def f(): pass # the original function >>> f.attr1 = "something" # setting an attribute >>> f.attr2 = "something else" # setting another attribute >>> traced_f = trace(f) # the decorated function >>> traced_f.attr1 'something' >>> traced_f.attr2 = "something different" # setting attr >>> f.attr2 # the original attribute did not change 'something else' </pre></div> </div> </div> <div class="section" id="compatibility-notes"> <h1><a class="toc-backref" href="#id15">Compatibility notes</a></h1> Version 3.0 is a complete rewrite of the original implementation. It is mostly compatible with the past, a part for a few differences. First of all, the utilites <tt class="docutils literal">get_info</tt> and <tt class="docutils literal">new_wrapper</tt>, available in the 2.X versions, have been deprecated and they will be removed in the future. For the moment, using them raises a <tt class="docutils literal">DeprecationWarning</tt>. Incidentally, the functionality has been implemented through a decorator which makes a good example for this documentation: <div class="codeblock python"> <div class="highlight"><pre>@decorator def deprecated(func, *args, **kw): "A decorator for deprecated functions" warnings.warn( ('Calling the deprecated function %r\n' 'Downgrade to decorator 2.3 if you want to use this functionality') % func.__name__, DeprecationWarning, stacklevel=3) return func(*args, **kw) </pre></div> </div> <tt class="docutils literal">get_info</tt> has been removed since it was little used and since it had to be changed anyway to work with Python 3.0; <tt class="docutils literal">new_wrapper</tt> has been removed since it was useless: its major use case (converting signature changing decorators to signature preserving decorators) has been subsumed by <tt class="docutils literal">decorator_apply</tt> and the other use case can be managed with the <tt class="docutils literal">FunctionMaker</tt>. Finally <tt class="docutils literal">decorator</tt> cannot be used as a class decorator and the <a class="reference external" href="http://www.phyast.pitt.edu/~micheles/python/documentation.html#class-decorators-and-decorator-factories">functionality introduced in version 2.3</a> has been removed. That means that in order to define decorator factories with classes you need to define the <tt class="docutils literal">__call__</tt> method explicitly (no magic anymore). Since there is an easy way to define decorator factories by using <tt class="docutils literal">decorator_factory</tt>, there is less need to use classes to implement decorator factories. All these changes should not cause any trouble, since they were all rarely used features. Should you have any trouble, you can always downgrade to the 2.3 version. The examples shown here have been tested with Python 2.5. Python 2.4 is also supported - of course the examples requiring the <tt class="docutils literal">with</tt> statement will not work there. Python 2.6 works fine, but if you run the examples here in the interactive interpreter you will notice a few differences since <tt class="docutils literal">getargspec</tt> returns an <tt class="docutils literal">ArgSpec</tt> namedtuple instead of a regular tuple. That means that running the file <tt class="docutils literal">documentation.py</tt> under Python 2.5 will a few errors, but they are not serious. Python 3.0 is kind of supported too. Simply run the script <tt class="docutils literal">2to3</tt> on the module <tt class="docutils literal">decorator.py</tt> and you will get a version of the code running with Python 3.0 (at least, I did some simple checks and it seemed to work). However there is no support for <a class="reference external" href="http://www.python.org/dev/peps/pep-3107/">function annotations</a> yet since it seems premature at this moment (most people are still using Python 2.5). </div> <div class="section" id="licence"> <h1><a class="toc-backref" href="#id16">LICENCE</a></h1> Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: <pre class="literal-block"> Copyright (c) 2005, Michele Simionato All rights reserved. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in bytecode form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. </pre> If you use this software and you are happy with it, consider sending me a note, just to gratify my ego. On the other hand, if you use this software and you are unhappy with it, send me a patch! </div> </div> <div class="footer"> <hr class="footer" /> <a class="reference external" href="documentation.rst">View document source</a>. Generated on: 2009-08-25 12:37 UTC. Generated by <a class="reference external" href="http://docutils.sourceforge.net/">Docutils</a> from <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> source. </div> </body> </html> <pre> Michele Simionato ab893da5395b555f05aea49f8d6fd4378261e15f 3.1.2