skip to navigation
skip to content

Not Logged In

pyutilib.autotest 2.0.1

A framework for generating test suites from external configuration files.

Overview

The pyutilib.autotest package provides a facility for automatically configuring tests that are executed with Python's unittest package. This capability is tailored for tests where one or more solvers are applied to one or more problems. This testing structure is particularly useful for evaluating the execution of external executables on datasets.

There are three main steps for configuring and applying pyutilib.autotest:

  1. Create a configuration file
  2. Create a test driver plugin
  3. Setup the Python module that will be used to apply the tests

These steps are described in the following subsections.

Creating a Configuration File

Currently, pyutilib.autotest only supports a YAML configuration file. The top-level collection in this configuration is a mapping with the following keys:

python
This section contains a list of Python expressions that are executed while setting up the test suites.
driver
This section specifies the name of the test driver that is used to execute tests.
solvers
This section contains a mapping of solver names to solver options.
problems
This section contains a mapping of problem names to problem options.
suites
This section contains a mapping of test suite names to suite configurations.

The following example illustrates the structure of a YAML configuration file. (This example is the file pyutilib.autotest/examples/autotest/example1.yml in the pyutilib.autotest distribution.)

driver: example

python:
    - import example

solvers:
    cat:
    cat2:
        name: cat
        cat_options: -n
    echo:

problems:
    p1:
        file: README.txt
    p2:
        file: example.py
    p3:
        file: LoremIpsum1.txt

suites:

    suite1:
        categories:
            - smoke
            - suite1
        solvers:
            cat:
            cat2:
            echo:
        problems:
            p1:

    suite2:
        categories:
            - nightly
            - suite2
        solvers:
            cat:
            cat2:
            echo:
        problems:
            p1:
            p2:
            p3:
        tests:
            - solver: cat
              problem: p1
            - solver: cat2
              problem: p2
            - solver: echo
              problem: p3

    suite3:
        categories:
            - suite3
        solvers:
            cat:
            catx:
                solver: cat
                cat_options: -n
        problems:
            p1:

The test driver example is defined in the example.py file, which is imported with the directives in the python section.

Within the solvers and problems sections, each solver and problem can specify additional options that are passed into the test. Note that these options are not distinguished for the test; option name conflicts will results in unpredictable behavior.

Three solvers are defined in this example, which apply the unix cat and echo commands. By default, the name of the solver is assumed to be the name of the key for the solver map. Solver cat2 illustrates how a solver name can be specified separately from the solver key.

The suites section defines one or more test suites. Each test suite consists of a mapping with the following sections:

categories
This optional section contains a list of strings that are solver categories. These categories can be used to select characteristics of the test suites that are executed.
solvers
This section contains a mapping of solvers. Note that additional options can be provided here to further customize the solver behavior. By default, the solver name is assumed to be the key value, but the solver key can be used to explicitly define the solver that is associated with a customized solver mapping.
problems
This section contains a list of problems. Note that additional options can be provided here to further customize the test behavior. By default, the problem name is assumed to be the key value, but the problem key can be used to explicitly define the problem that is associated with a customized problem mapping.
tests
This optional section contains a list of solver-problem pairs, which defines the actual tests that are defined. Note that the solver and problem values map to the keys defined in the solvers and problems sections of a test suite; thus, this section can use the names of customized solvers and problems that are not defined in the top-level solvers and problems sections. Finally, if this section is omitted, then all combinations of solvers and problems are used to create tests.

In this example, three suites are defined to illustrate different features of the test driver:

suite1
A simple test suite in which all solvers are applied to a single problem.
suite2
The tests section is specified to select a subset of all combinations of solvers and problems to test.
suite3
The catx solver is customized from cat to operate like the cat2 solver.

Creating a Test Driver

The test configuration used by pyutilib.autotest is quite generic. It specifies what combinations of solvers and problems are to be tested, along with their corresponding options. However, it does not specify how the test is performed. This is done by a test driver class.

In general, test driver classes are required to be plugins that can be dynamically created by pyutilib.autotest to execute tests. The easiest way to define a test driver plugin is to inherit from the TestDriverBase class. For example, the following test driver is used in by the earlier test configuration; this plugin is defined in pyutilib.autotest/examples/autotest/example.py.

import pyutilib.autotest
from pyutilib.component.core import *
import pyutilib.subprocess


class ExampleTestDriver(pyutilib.autotest.TestDriverBase):
    """
    This test driver executes a unix command and compares its output
    with a baseline value.
    """

    alias('example')

    def run_test(self, testcase, name, options):
        """Execute a single test in the suite"""
        name = options.suite+'_'+name
        cmd = options.solver+' '
        if not options.cat_options is None:
            cmd += options.cat_options+' '
        cmd += options.file
        print "Running test suite '%s'  test '%s'  command '%s'" % \
                (options.suite, name, cmd)
        pyutilib.subprocess.run(cmd, outfile=options.currdir+'test_'+name+".out")
        testcase.failUnlessFileEqualsBaseline(
                options.currdir+'test_'+name+".out",
                options.currdir+'test_'+name+".txt")

The alias function is used to specify the name of this plugin; this is the name of the test driver used in the test configuration file.

The run_test method executes a single test in the test suite. Note that this method is passed in testcase, which is the test suite class. Thus, this method can directly apply the unittest methods that are defined in this class (e.g. assertEquals).

In this example, a unix command-line is create from the solver name, the solver options, and the problem filename. This is executed with the pyutilib.subprocess.run function, which redirects output to a log file. This log file is then compared with a baseline file using the failUnlessFileEqualsBaseline, which is defined in the unittest extensions in pyutilib.th.

Note that a variety of other standard unit test methods can also be defined by this test driver. This driver is a ITestDriver plugin, and the API for this plugin is:

class ITestDriver(Interface):

    def setUpClass(self, cls, options):
        """Set-up the class that defines the suite of tests"""

    def tearDownClass(self, cls, options):
        """Tear-down the class that defines the suite of tests"""

    def setUp(self, testcase, options):
        """Set-up a single test in the suite"""

    def tearDown(self, testcase, options):
        """Tear-down a single test in the suite"""

    def run_test(self, testcase, name, options):
        """Execute a single test in the suite"""

Creating a Test Module

Virtually all of the work needed to create test suites is automated by pyutilib.autotest. The following test module is used in this example; (see pyutilib.autotest/examples/autotest/autotest.py):

import os
import sys
from os.path import abspath, dirname
currdir = dirname(abspath(__file__))+os.sep

import pyutilib.th as unittest
import pyutilib.autotest

if __name__ == "__main__":
    pyutilib.autotest.create_test_suites(filename=currdir+'example1.yml', _globals=globals())
    unittest.main()

The first four lines are needed to identify the current directory, where the test configuration file resides.

Note that pyutilib.th is imported as unittest, which reminds the user that this is a unittest extension package. (Specifically, this package contains hooks needed to dynamically add functions as test methods in test suites.)

The pyutilib.autotest packages is imported so the create_test_suites function can be executed. The arguments to this function are the test configuration file, and the global dictionary.

Finally, unittest.main() is executed, as in any unittest module. Tests can be executed using standard unittest command-line options. One extension to this behavior is the use of the PYUTILIB_AUTOTEST_CATEGORIES or PYUTILIB_UNITTEST_CATEGORIES environmental variables; if one of these is specified, then pyutilib.autotest assumes that this data contains a comma-separated list of categories that are used to select the test suites that are constructed. Specifically, if a test suite contains one of the specified test categories, then it will be executed.

The pyutilib_test_driver Command

The pyutilib_test_driver command can be used to execute tests defined in a configuration file without creating a test module. In practice, test modules are typically needed to support test discovery with tools like nose. However, this command provides several features that are useful when diagnosing tests.

The command-line behavior of pyutilib_test_driver extends the API of unittest.main(). The following additional options are provided to allow the user to interrogate the tests that are defined by the test configuration file:

--help-suites Print the test suites that can be executed
--help-tests=HELP_TESTS
 Print the tests in the specified test suite
--help-categories
 Print the test suite categories that can be specified

License

BSD. See the LICENSE.txt file.

Organization

Third Party Software

None.

 
File Type Py Version Uploaded on Size
pyutilib.autotest-2.0.1.tar.gz (md5) Source 2013-02-25 18KB
  • Downloads (All Versions):
  • 25 downloads in the last day
  • 156 downloads in the last week
  • 826 downloads in the last month