Skip to main content

Smart random-generation state persistence for unittest

Project description

Introduction

Randomized unit tests are great. The inputs fed to your algorithm are not consistent across executions, and so each time the tests run successfully, your confidence in the algorithm grows. Unfortunately, when a test fails, this inconsistency makes it hard to duplicate the input until the problem is solved.

This library provides a simple metaclass for dropping into unittest.TestCase that addresses this. As long as a test fails, it will continue making the same random choices (via random or numpy.random) each execution. Any test that passed the previous execution, will get a fresh set of random choices.

Example

Consider the following simplified example of a test case with three functions testing some algorithms. The first and third algorithms are OK, but the second has a bug when given a number that divides by 20 (the simplified example asserts this directly).

import random


class TestMyAlgs(unittest.TestCase):
    # This "algorithm" is OK.
    def test_ok(self):
        i = random.randint(100, 200)
        self.assertLess(i, i + 1)

    # This "algorithm" fails for inputs that divide by 20.
    def test_bad(self):
        i = random.randint(100, 200)
        self.assertNotEqual(i % 20, 0, i)

    # This "algorithm" is OK too.
    def test_another_ok(self):
        i = random.randint(100, 200)
        self.assertLess(i - 1, i)

The second function will fail approximately 5% of the executions. When it does so, it might be hard (especially in a real-life case) to reproduce the problem, and eventually solve it.

Using this library, a simple change makes things easier. Simply use unittest_rand_gen_state.Saver as the metaclass:

import random

import unittest_rand_gen_state


class TestMyAlgs(unittest.TestCase, metaclass = unittest_rand_gen_state.Saver):
    ...

or, using an older Python version:

import random

import unittest_rand_gen_state


class TestMyAlgs(unittest.TestCase):
    __metaclass__ = unittest_rand_gen_state.Saver
    ...

Nothing else needs to be changed.

Once TestMyAlgs.test_bad first fails in some execution, then it will consistently fail in following executions. That is, say it fails the first time with

======================================================================
FAIL: test_bad (__main__.TestMyAlgs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/unittest_rand_gen_state/__init__.py", line 75, in wrapped
    fn(*args, **kwargs)
  File "__init__.py", line 21, in test_bad
    self.assertNotEqual(i % 20, 0, i)
AssertionError: 140

----------------------------------------------------------------------

then every following execution will fail exactly the same way, until it is fixed.

The two other tests, meanwhile, will run afresh all the while.

Download, Installation, And Bugtracking

  • The package is at PyPI.

  • The usual setup for Python libraries is used. Type:

    $ pip install UnittestRandGenState

    or

    $ sudo pip install UnittestRandGenState

  • Bugtracking is on Google Code.

Extending

The metaclass supplied by this library deals with the states of random and numpy.random, which should be sufficient for almost all tests. However, the library may be extended by dropping as a metaclass, a subclass of unittest_rand_gen_state.Saver supplying three class methods:

  • reset_state: A method specifying the meaning of resetting the random-generation state (e.g., random.seed)

  • get_state: A method specifying the query to call for getting the current random-generation state (e.g., random.getstate)

  • set_state: A method specifying the function to call to restore the random-generation state using the value returned by get_state (e.g., random.setstate)

E.g., suppose a random-generation library, groovy_rng has the functions groovy_rng.seed (setting the state to some function determined by the current time), groovy_rng.get_state, returning an int, and groovy_rng.set_state taking an int, then here is a metaclass for working with groovy_rng:

class GroovyRNGSaver(unittest_rand_gen_state.Saver):
    reset_state = groovy_rng.seed
    get_state = groovy_rng.get_state
    set_state = groovy_rng.set_state

This metaclass can be used just like in the previous example.

Changes

Changes

Version

Date

Description

0.1.4.

15/03/2013

Duplicate class-name bugfix, Python 2/3 coexistance bugfix

0.1.3

23/02/2013

Better test hashes, better support for extension

0.1.2

22/02/2013

Doc bugfix

0.1.1

20/02/2013

Less stateful design

0.1.0

18/02/2013

Initial Release

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

UnittestRandGenState-0.1.4.zip (12.7 kB view hashes)

Uploaded Source

UnittestRandGenState-0.1.4.tar.gz (6.6 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page