Skip to main content

pytest plugin to re-run tests to eliminate flaky failures

Project description

pytest-rerunfailures is a plugin for py.test that re-runs tests to eliminate intermittent failures.

License PyPI Travis

Requirements

You will need the following prerequisites in order to use pytest-rerunfailures:

  • Python 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, PyPy, or PyPy3

  • py.test 2.4.2 or newer, except Python 3.5 that needs py.test 2.7.3 or newer

Installation

To install pytest-rerunfailures:

$ pip install pytest-rerunfailures

Re-run all failures

To re-run all test failures, use the --reruns command line option with the maximum number of times you’d like the tests to run:

$ py.test --rerun 5

Re-run individual failures

To mark individual tests as flaky, and have them automatically re-run when they fail, add the flaky mark with the maximum number of times you’d like the test to run:

@pytest.mark.flaky(reruns=5)
def test_example():
    import random
    assert random.choice([True, False])

Note that when teardown fails, two reports are generated for the case, one for the test case and the other for the teardown error.

Output

Here’s an example of the output provided by the plugin when run with --reruns 2 and -r aR:

test_report.py RRF

================================== FAILURES ==================================
__________________________________ test_fail _________________________________

    def test_fail():
>       assert False
E       assert False

test_report.py:9: AssertionError
============================ rerun test summary info =========================
RERUN test_report.py::test_fail
RERUN test_report.py::test_fail
============================ short test summary info =========================
FAIL test_report.py::test_fail
======================= 1 failed, 2 rerun in 0.02 seconds ====================

Note that output will show all re-runs. Tests that fail on all the re-runs will be marked as failed. Due to a current limitation in pytest-xdist, when running tests in parallel only the final result will be included in the output.

Compatibility

  • This plugin may not be used with class, module, and package level fixtures.

  • This plugin is not compatible with pytest-xdist’s –looponfail flag.

  • This plugin is not compatible with the core –pdb flag.

  • The support for Python 3.5 is limited to pytest 2.7.3 and higher. (Will break with TypeError: Call constructor takes either 0 or 3 positional arguments for older versions, see https://github.com/pytest-dev/pytest/issues/744)

Running Tests

You will need Tox installed to run the tests against the supported Python versions.

$ pip install tox
$ tox

Tests with Python 3.2 will fail if virtualenv 14 or higher is used, since they dropped support for Python 3.2 (will break with SyntaxError: invalid syntax near empty_fill = u'∙'). Therefore:

$ pip uninstall virtualenv
$ pip install virtualenv==13.1.2

They will also fail if pip 8 or higher is used, since they, too, dropped support for Python 3.2. Make sure to use easy_install instead for Python 3.2.

Resources

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pytest-rerunfailures-1.0.2.tar.gz (6.8 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page