Skip to main content

Testimony inspects and reports on the python test cases.

Project description

Are you tired of managing your test cases in a test case management tool and your test code in a python automation framework? Testimony can help to use the python automation framework as a test case repository tool.

The project testimony is written to inspect and report on the python test cases. There are several reporting features in this program.

The parameter options are:

  1. print - List all test cases

  2. summary - Summary of number of automated cases vs. manual cases

  3. validate_docstring - Validate docstrings. This does the following:s

  • Reports test cases with invalid docstrings

  • Reports test cases with missing docstrings

  • Reports test cases that does not have minimal required docstrings (This will enforce that all test cases have a minimal set of docstrings). Currently testimony enforces Feature, Test, Assert as mandatory docstrings for each test case”

  1. bugs - Test cases affected by Bugs and the corresponding Bug list

  2. manual - List all manual test cases

  3. auto - List all auto test cases

  4. tags - Prints all test cases with the specified tags

Note: 1. testimony returns a non-zero error code when the test case docstrings does not follow the intended rules, returns zero otherwise 2. testimony also parses different folders under the given folder to verify the test files 3. testimony also displays optional colored outputs when termcolor is installed

Advantages

  1. Avoids a separate test case management tool to document test cases by using the python automation framework for the same

  2. Enforces test cases to follow a defined standard

  3. Runs in integration with tools like Travis to get a report as and when you check in code

  4. Saves a lot of time from the conventional way of writing test cases using a test management tool

Installation

You could install testimony from PyPI using pip:

pip install testimony

Requirements

1) Install testimony following the steps on the installation section

2) Pre-requisites

Expected Docstring format:

"""@Test: Log in as a valid user

@Feature: Login

@Setup: Navigate to abc.com

@Steps:

1. Launch the url
2. Log in with valid user credentials

@Assert: Log in successful

@BZ: #1234567

@Status: Manual (REMOVE this field once automated)

@Tags: T1, T2, T3

"""

3) Optional color formatting - If termcolor package is installed, output will be printed in colored text

Usage:

$ testimony -h
usage: testimony [-h] [-n] [-t TAGS] REPORT PATH [PATH ...]

Inspects and report on the Python test cases.

positional arguments:
    REPORT                report type, possible values: print, summary,
                      validate_docstring, bugs, manual, auto, tags
    PATH                  a list of paths to look for tests cases

    optional arguments:
    -h, --help            show this help message and exit
    -j, --json            JSON output
    -n, --nocolor         Do not use color option
    -t [TAGS [TAGS ...]], --tags [TAGS [TAGS ...]]
                      space separated tags to search.  Note: Always run this
                      only in the root of the project where test cases are
                      stored
$ testimony print /home/testimony/tests/

Fetching Test Path /home/testimony/tests/

Scanning test_sample.py...

TC 1
Feture: Login - Positive
Test: Login with right credentials
Steps:
    1. Login to the application with valid credentials
Assert: Login is successful
Bug: 123456
Statues: Manual

TC 2
test_positive_login_2: Docstring missing. Please update.

TC 3
Feature: Login - Positive
Test: Login with Latin credentials
Steps:
    1. Login to the application with valid Latin credentials
Assert: Login is successful

TC 4
Feature: Login - Positive
Test: Login with Credentials having special characters
Steps:
    1. Login to the application with valid credentials having
    special characters
Assert: Activation key is created
Status: Manual

TC 5
Steps:
    1. Login to the application with invalid credentials
BZ: 123456
Status: Manual

$ echo $?
255
$ testimony summary /home/testimony/tests/

Fetching Test Path /home/testimony/tests/

Total Number of test cases:      4
Total Number of automated cases: 2
Total Number of manual cases:    2
Test cases with no docstrings:   1


$ echo $?
255
$ testimony validate_docstring /home/testimony/tests/

Fetching Test Path /home/testimony/tests/

Scanning test_sample.py...
test_positive_login_1: Invalid DocString: Feture: Login - Positive
test_positive_login_1: Invalid DocString: Bug: 123456
test_positive_login_1: Invalid DocString: Statues: Manual
test_positive_login_1: Need feature, test and assert at the minimum
test_positive_login_2: Docstring missing. Please update.
test_negative_login_5: Need feature, test and assert at the minimum
Total Number of invalid docstrings:  3
Test cases with no docstrings:   1
Test cases missing minimal docstrings:  2

$ echo $?
255
$ testimony bugs /home/testimony/tests/

Fetching Test Path /home/estimony/tests/

Scanning test_sample.py...

Total Number of test cases affected by bugs: 1

Bug list:
123456

$ echo $?
255
 $ testimony manual /home/testimony/tests/

Fetching Test Path /home/estimony/tests/

Scanning test_sample.py...
Feature: Login - Positive
Test: Login with Credentials having special characters
Steps:
    1. Login to the application with valid credentials having
    special characters
Assert: Activation key is created
Status: Manual
Steps:
    1. Login to the application with invalid credentials
BZ: 123456
Status: Manual

$ echo $?
255
$ testimony auto /home/testimony/tests/

Fetching Test Path /home/estimony/tests/

Scanning test_sample.py...
Feture: Login - Positive
Test: Login with right credentials
Steps:
    1. Login to the application with valid credentials
Assert: Login is successful
Bug: 123456
Statues: Manual
Feature: Login - Positive
Test: Login with Latin credentials
Steps:
    1. Login to the application with valid Latin credentials
Assert: Login is successful

$ echo $?
255
$ testimony tags tests/ --tag t1
    ['tests.test_sample.Testsample1.test_positive_login_1',
     'tests.test_sample.Testsample1.test_positive_login_3']

$ testimony tags tests/ --tag t1 t2
    ['tests.test_sample.Testsample1.test_positive_login_1',
     'tests.test_sample.Testsample1.test_positive_login_3',
     'tests.test_sample.Testsample1.test_negative_login_5']

Success scenario in which testimony returns 0

$ testimony validate_docstring /home/tests/ui/sample/

Fetching Test Path home/tests/ui/sample/

Scanning test_activationkey.py...
Total Number of invalid docstrings:  0
Test cases with no docstrings:   0
Test cases missing minimal docstrings:  0

$ echo $?
0

Having termcolor installed, testimony produces colored output by default. It can be disabled by:

$ testimony auto /home/apple/tests/login/ --nocolor

(or)

$ testimony auto /home/apple/tests/login/ -n

Testimony supports json output format to integrate with other applications easily. This can be done by adding –json or -j to any of the testimony commands as shown below:

$ testimony summary --json tests/
    [{"auto_count": 2, "manual_count": 2, "auto_percent": 50.0, "no_docstring": 1, "path": "tests/", "tc_count": 4, "manual_percent": 50.0}]

    $ testimony summary -j tests/
    [{"auto_count": 2, "manual_count": 2, "auto_percent": 50.0, "no_docstring": 1, "path": "tests/", "tc_count": 4, "manual_percent": 50.0}]

Known Issues

None

Version History

  • Version 1.0.1

- Major code refactor for modularizing the code
- Added Tags support
- If @Test is not present, the first line of docstring will be used instead
  • Version 1.0.0

- json support now incorporated
  • Version 0.3.0

- Bug fix: Manual vs. automated test count is wrong when the test cases are written with "status" tag vs. "Status"
  • Version 0.2.0

- fix to check the tests starting with test_ rather than just test
- Testimony will return error code when docstrings are missing, incorrect docstrings found, minimal docstrings not present
- Make validate_docstring return a 0 success return code if no errors are found
- Organized Constants
- Now testimony accepts --nocolor or --n argument to avoid color output
- testimony will now not error out if termcolor is not installed.
- Make termcolor an optional dependency
- Add Travis configuration to automatically run pep8 when testimony is updated
- Get tests from subfolders of the given path
  • Version 0.1.0

- Initial Release

Author

This software is developed by Suresh Thirugn.

Contributors

Project details


Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page