goodtables 2.0.0

goodtables-py
=============

| |Travis|
| |Coveralls|
| |PyPi|
| |Gitter|

| Goodtables is a framework to validate tabular data. It can check the
structure
| of your data (e.g. all rows have the same number of columns), and its
contents
| (e.g. all dates are valid).

Features
--------

- **Structural checks**: Ensure that there are no empty rows, no blank
headers, etc.
- **Content checks**: Ensure that the values have the correct types
("string", "number", "date", etc.), that their format is valid
("string must be an e-mail"), and that they respect the constraints
("age must be a number greater than 18").
- **Support for multiple tabular formats**: CSV, Excel files,
LibreOffice, Data Package, etc.
- **Parallelized validations for multi-table datasets**
- **Command line interface**

Table of Contents
-----------------

- `Features <#features>`__
- `Getting Started <#getting-started>`__

- `Installing <#installing>`__
- `Running on CLI <#running-on-cli>`__
- `Running on Python <#running-on-python>`__

- `Validation <#validation>`__

- `Basic checks <#basic-checks>`__
- `Structural checks <#structural-checks>`__
- `Content checks <#content-checks>`__
- `Advanced checks <#advanced-checks>`__
- `Validation report format <#validation-report-format>`__

- `Developer documentation <#developer-documentation>`__

- `Semantic versioning <#semantic-versioning>`__
- `Validate <#validate>`__

- `FAQ <#faq>`__

- `How can I add a new custom
check? <#how-can-i-add-a-new-custom-check>`__
- `How can I add support for a new tabular file
type? <#how-can-i-add-support-for-a-new-tabular-file-type>`__

- `Changelog <#changelog>`__

Getting Started
---------------

Installing
~~~~~~~~~~

::

pip install goodtables
pip install goodtables[ods] # If you need LibreOffice's ODS file support

Running on CLI
~~~~~~~~~~~~~~

::

goodtables data.csv

Use ``goodtables --help`` to see the different options.

Running on Python
~~~~~~~~~~~~~~~~~

.. code:: python

from goodtables import validate

report = validate('invalid.csv')
report['valid'] # false
report['table-count'] # 1
report['error-count'] # 3
report['tables'][0]['valid'] # false
report['tables'][0]['source'] # 'invalid.csv'
report['tables'][0]['errors'][0]['code'] # 'blank-header'

| You can read a more in depth explanation on using goodtables with
Python on
| the `developer documentation <#developer-documentation>`__ section.
Check also
| the `examples <examples>`__ folder for other examples.

Validation
----------

Basic checks
~~~~~~~~~~~~

The basic checks can't be disabled, as they deal with goodtables being
able to read the files.

+------------------+-------------------------------------------------------------------------+
| check | description |
+==================+=========================================================================+
| io-error | Data reading error because of IO error. |
+------------------+-------------------------------------------------------------------------+
| http-error | Data reading error because of HTTP error. |
+------------------+-------------------------------------------------------------------------+
| source-error | Data reading error because of not supported or inconsistent contents. |
+------------------+-------------------------------------------------------------------------+
| scheme-error | Data reading error because of incorrect scheme. |
+------------------+-------------------------------------------------------------------------+
| format-error | Data reading error because of incorrect format. |
+------------------+-------------------------------------------------------------------------+
| encoding-error | Data reading error because of an encoding problem. |
+------------------+-------------------------------------------------------------------------+

Structural checks
~~~~~~~~~~~~~~~~~

These checks validate that the structure of the file are valid.

+--------------------+-----------------------------------------------------------------------------------+
| check | description |
+====================+===================================================================================+
| blank-header | There is a blank header name. All cells in the header row must have a value. |
+--------------------+-----------------------------------------------------------------------------------+
| duplicate-header | There are multiple columns with the same name. All column names must be unique. |
+--------------------+-----------------------------------------------------------------------------------+
| blank-row | Rows must have at least one non-blank cell. |
+--------------------+-----------------------------------------------------------------------------------+
| duplicate-row | Rows can't be duplicated. |
+--------------------+-----------------------------------------------------------------------------------+
| extra-value | A row has more columns than the header. |
+--------------------+-----------------------------------------------------------------------------------+
| missing-value | A row has less columns than the header. |
+--------------------+-----------------------------------------------------------------------------------+

Content checks
~~~~~~~~~~~~~~

These checks validate the contents of the file. To use them, you need to
pass a `Table
Schema <https://specs.frictionlessdata.io/table-schema/>`__. If you
don't have a schema, goodtables can infer it if you use the
``infer_schema`` option.

If your schema only covers part of the data, you can use the
``infer_fields`` to infer the remaining fields.

Lastly, if the order of the fields in the data is different than in your
schema, enable the ``order_fields`` option.

+-----------------------------+-------------------------------------------------------------------------------------------+
| check | description |
+=============================+===========================================================================================+
| schema-error | Schema is not valid. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| non-matching-header | The header's name in the schema is different from what's in the data. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| extra-header | The data contains a header not defined in the schema. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| missing-header | The data doesn't contain a header defined in the schema. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| type-or-format-error | The value can’t be cast based on the schema type and format for this field. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| required-constraint | This field is a required field, but it contains no value. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| pattern-constraint | This field value's should conform to the defined pattern. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| unique-constraint | This field is a unique field but it contains a value that has been used in another row. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| enumerable-constraint | This field value should be equal to one of the values in the enumeration constraint. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| minimum-constraint | This field value should be greater or equal than constraint value. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| maximum-constraint | This field value should be less or equal than constraint value. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| minimum-length-constraint | A length of this field value should be greater or equal than schema constraint value. |
+-----------------------------+-------------------------------------------------------------------------------------------+
| maximum-length-constraint | A length of this field value should be less or equal than schema constraint value. |
+-----------------------------+-------------------------------------------------------------------------------------------+

Advanced checks
~~~~~~~~~~~~~~~

+----------------------------------------------+---------------------------------------------------------------------------------------------------+
| check | description |
+==============================================+===================================================================================================+
| `blacklisted-value <#blacklisted-value>`__ | Ensure there are no cells with the blacklisted values. |
+----------------------------------------------+---------------------------------------------------------------------------------------------------+
| `deviated-value <#deviated-value>`__ | Ensure numbers are within a number of standard deviations from the average. |
+----------------------------------------------+---------------------------------------------------------------------------------------------------+
| `sequential-value <#sequential-value>`__ | Ensure numbers are sequential. |
+----------------------------------------------+---------------------------------------------------------------------------------------------------+
| `truncated-value <#truncated-value>`__ | Detect values that were potentially truncated. |
+----------------------------------------------+---------------------------------------------------------------------------------------------------+
| `custom-constraint <#custom-constraint>`__ | Defines a constraint based on the values of other columns (e.g. ``value * quantity == total``). |
+----------------------------------------------+---------------------------------------------------------------------------------------------------+

blacklisted-value
^^^^^^^^^^^^^^^^^

Sometimes we have to check for some values we don't want to have in out
dataset. It accepts following options:

+-------------+---------------+------------------------------+
| option | type | description |
+=============+===============+==============================+
| column | int/str | Column number or name |
+-------------+---------------+------------------------------+
| blacklist | list of str | List of blacklisted values |
+-------------+---------------+------------------------------+

Consider the following CSV file:

.. code:: csv

id,name
1,John
2,bug
3,bad
5,Alex

Let's check that the ``name`` column doesn't contain rows with ``bug``
or ``bad``:

.. code:: python

from goodtables import validate

report = validate('data.csv', checks=[
{'blacklisted-value': {'column': 'id', 'blacklist': ['bug', 'bad']}},
])
# error on row 3 with code "blacklisted-value"
# error on row 4 with code "blacklisted-value"

deviated-value
^^^^^^^^^^^^^^

This check helps to find outlines in a column containing positive
numbers. It accepts following options:

+------------+-----------+---------------------------------------------------------------------------+
| option | type | description |
+============+===========+===========================================================================+
| column | int/str | Column number or name |
+------------+-----------+---------------------------------------------------------------------------+
| average | str | Average type, either "mean", "median" or "mode" |
+------------+-----------+---------------------------------------------------------------------------+
| interval | int | Values must be inside range ``average ± standard deviation * interval`` |
+------------+-----------+---------------------------------------------------------------------------+

Consider the following CSV file:

.. code:: csv

temperature
1
-2
7
0
1
2
5
-4
100
8
3

We use ``median`` to get an average of the column values and allow
interval of 3 standard deviations. For our case median is ``2.0`` and
standard deviation is ``29.73`` so all valid values must be inside the
``[-87.19, 91.19]`` interval.

.. code:: python

report = validate('data.csv', checks=[
{'deviated-value': {'column': 'temperature', 'average': 'median', 'interval': 3}},
])
# error on row 10 with code "deviated-value"

sequential-value
^^^^^^^^^^^^^^^^

This checks is for pretty common case when a column should have integers
that sequentially increment. It accepts following options:

+----------+-----------+-------------------------+
| option | type | description |
+==========+===========+=========================+
| column | int/str | Column number or name |
+----------+-----------+-------------------------+

Consider the following CSV file:

.. code:: csv

id,name
1,one
2,two
3,three
5,five

Let's check if the ``id`` column contains sequential integers:

.. code:: python

from goodtables import validate

report = validate('data.csv', checks=[
{'sequential-value': {'column': 'id'}},
])
# error on row 5 with code "sequential-value"

truncated-value
^^^^^^^^^^^^^^^

Some database or spreadsheet software (like MySQL or Excel) could cutoff
values on saving. There are some well-known heuristics to find this bad
values. See
https://github.com/propublica/guides/blob/master/data-bulletproofing.md
for more detailed information.

Consider the following CSV file:

.. code:: csv

id,amount,comment
1,14000000,good
2,2147483647,bad
3,32767,bad
4,234234234,bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbad

To detect all probably truncated values we could use ``truncated-value``
check:

.. code:: python

report = validate('data.csv', checks=[
'truncated-value',
])
# error on row 3 with code "truncated-value"
# error on row 4 with code "truncated-value"
# error on row 5 with code "truncated-value"

custom-constraint
^^^^^^^^^^^^^^^^^

With Table Schema we could create constraints for an individual field
but sometimes it's not enough. With a custom constraint check every row
could be checked against given limited python expression in which
variable names resolve to column values. See list of `available
operators <https://github.com/danthedeckie/simpleeval#operators>`__. It
accepts following options:

.. raw:: html

<dl>
<dt>constraint (str)</dt>
<dd>Constraint definition (e.g. <code>col1 + col2 == col3</code>)</dd>
</dl>

Consider csv file like this:

.. code:: csv

id,name,salary,bonus
1,Alex,1000,200
2,Sam,2500,500
3,Ray,1350,500
4,John,5000,1000

Let's say our business rule is to be shy on bonuses:

.. code:: python

report = validate('data.csv', checks=[
{'custom-constraint': {'constraint': 'salary > bonus * 4'}},
])
# error on row 4 with code "custom-constraint"

Validation report format
~~~~~~~~~~~~~~~~~~~~~~~~

| The validation report follows the JSON Schema defined on
| `goodtables/schemas/report.json <goodtables/schemas/report.json>`__.
As an example, this is
| the report generated by running ``goodtables --row-limit 5 --json`` on
the file
| `data/datapackages/invalid/datapackage.json <data/datapackage/invalid/datapackage.json>`__:

.. code:: json

{
"time": 0.015,
"valid": false,
"error-count": 2,
"table-count": 2,
"tables": [
{
"datapackage": "data/datapackages/invalid/datapackage.json",
"time": 0.005,
"valid": false,
"error-count": 1,
"row-count": 4,
"source": "data/datapackages/invalid/data.csv",
"headers": [
"id",
"name",
"description",
"amount"
],
"format": "inline",
"schema": "table-schema",
"errors": [
{
"code": "blank-row",
"row-number": 3,
"message": "Row 3 is completely blank"
}
]
},
{
"datapackage": "data/datapackages/invalid/datapackage.json",
"time": 0.004,
"valid": false,
"error-count": 1,
"row-count": 5,
"source": "data/datapackages/invalid/data2.csv",
"headers": [
"parent",
"comment"
],
"format": "inline",
"schema": "table-schema",
"errors": [
{
"code": "blank-row",
"row-number": 4,
"message": "Row 4 is completely blank"
}
]
}
],
"warnings": [
"Table \"data/datapackages/invalid/data2.csv\" inspection has reached 5 row(s) limit"
],
"preset": "datapackage"
}

Developer documentation
-----------------------

Semantic versioning
~~~~~~~~~~~~~~~~~~~

| We follow the `Semantic Versioning <https://semver.org/>`__
specification to define our version
| numbers. This means that we'll increase the major version number when
there's a
| breaking change. Because of this, we recommend you to explicitly
specify the
| goodtables version on your dependency list (e.g. ``setup.py`` or
| ``requirements.txt``).

Validate
~~~~~~~~

| Goodtables validates your tabular dataset to find structural and
content
| errors. Consider you have a file named ``invalid.csv``. Let's validate
it:

.. code:: python

report = validate('invalid.csv')

| We could also pass a remote URI instead of a local path. It supports
CSV, XLS,
| XLSX, ODS, JSON, and all other formats supported by the
`tabulator <https://github.com/frictionlessdata/tabulator-py/>`__
| library.

Validation report
^^^^^^^^^^^^^^^^^

| The output of the ``validate()`` method is a report dictionary. It
includes
| information if the data was valid, count of errors, list of table
reports, which
| individual checks failed, etc.

Resulting report will be looking like this:

.. code:: json

{
"time": 0.009,
"error-count": 1,
"warnings": [
"Table \"data/invalid.csv\" inspection has reached 1 error(s) limit"
],
"preset": "table",
"valid": false,
"tables": [
{
"errors": [
{
"row-number": null,
"message": "Header in column 3 is blank",
"row": null,
"column-number": 3,
"code": "blank-header"
}
],
"error-count": 1,
"headers": [
"id",
"name",
"",
"name"
],
"scheme": "file",
"row-count": 2,
"valid": false,
"encoding": "utf-8",
"time": 0.007,
"schema": null,
"format": "csv",
"source": "data/invalid"
}
],
"table-count": 1
}

| Rase report errors are standardized and described
| in
| `Data Quality
Spec <https://github.com/frictionlessdata/data-quality-spec/blob/master/spec.json>`__.
| The errors are divided in one of the following categories:

- ``source`` - data can't be loaded or parsed
- ``structure`` - general tabular errors like duplicate headers
- ``schema`` - error of checks against `Table
Schema <http://specs.frictionlessdata.io/table-schema/>`__
- ``custom`` - custom checks errors

Checks
^^^^^^

| Check is a main validation actor in goodtables. The list of enabled
checks can
| be changed using ``checks`` and ``skip_checks`` arguments. Let's
explore the options
| on an example:

.. code:: python

report = validate('data.csv') # by default structure and schema (if available) checks
report = validate('data.csv', checks=['structure']) # only structure checks
report = validate('data.csv', checks=['schema']) # only schema (if available) checks
report = validate('data.csv', checks=['bad-headers']) # check only 'bad-headers'
report = validate('data.csv', skip_checks=['bad-headers']) # exclude 'bad-headers'

| By default a dataset will be validated against all available Data
Quality Spec
| errors. Some checks can be unavailable for validation. For example, if
the
| schema isn't provided, only the ``structure`` checks will be done.

Presets
^^^^^^^

| Goodtables support different formats of tabular datasets. They're
called
| presets. A tabular dataset is some data that can be split in a list of
data
| tables, as:

|Dataset|

| We can change the preset using the ``preset`` argument for
``validate()``. By
| default, it'll be inferred from the source, falling back to ``table``.
To validate
| a `data package <https://specs.frictionlessdata.io/data-package/>`__,
we can do:

.. code:: python

report = validate('datapackage.json') # implicit preset
report = validate('datapackage.json', preset='datapackage') # explicit preset

This will validate all tabular resources in the datapackage.

| It's also possible to validate a list of files using the "nested"
preset. To do
| so, the first argument to ``validate()`` should be a list of
dictionaries, where
| each key in the dictionary is named after a parameter on
``validate()``. For example:

.. code:: python

report = validate([{'source': 'data1.csv'}, {'source': 'data2.csv'}]) # implicit preset
report = validate([{'source': 'data1.csv'}, {'source': 'data2.csv'}], preset='nested') # explicit preset

Is similar to:

.. code:: python

report_data1 = validate('data1.csv')
report_data2 = validate('data2.csv')

| The difference is that goodtables validates multiple tables in
parallel, so
| calling using the "nested" preset should run faster.

Contributing
------------

This project follows the `Open Knowledge International coding
standards <https://github.com/okfn/coding-standards>`__.

| We recommend you to use ``virtualenv`` to isolate goodtables from the
rest of the
| packages in your machine.

To install goodtables and the development dependencies, run:

::

$ make install

To run the tests, use:

.. code:: bash

$ make test

FAQ
---

How can I add a new custom check?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To create a custom check user could use a ``check`` decorator. This way
the builtin check could be overridden (use the spec error code like
``duplicate-row``) or could be added a check for a custom error (use
``type``, ``context`` and ``position`` arguments):

.. code:: python

from goodtables import validate, check, Error

@check('custom-check', type='custom', context='body')
def custom_check(cells):
errors = []
for cell in cells:
message = 'Custom error on column {column_number} and row {row_number}'
error = Error(
'custom-error',
cell,
message
)
errors.append(error)
return errors

report = validate('data.csv', checks=['custom-check'])

For now this documentation section is incomplete. Please see builtin
checks to learn more about checking protocol.

How can I add support for a new tabular file type?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To create a custom preset user could use a ``preset`` decorator. This
way the builtin preset could be overridden or could be added a custom
preset.

.. code:: python

from tabulator import Stream
from tableschema import Schema
from goodtables import validate

@preset('custom-preset')
def custom_preset(source, **options):
warnings = []
tables = []
for table in source:
try:
tables.append({
'source': str(source),
'stream': Stream(...),
'schema': Schema(...),
'extra': {...},
})
except Exception:
warnings.append('Warning message')
return warnings, tables

report = validate(source, preset='custom-preset')

For now this documentation section is incomplete. Please see builtin
presets to learn more about the dataset extraction protocol.

Changelog
---------

v2.0
~~~~

Breaking changes:

- Checks method signature now only receives the current row's ``cells``
list
- Checks raise errors by returning an array of ``Error`` objects
- Cells have the row number in the ``row-number`` key
- Files with ZIP extension are presumed to be datapackages, so
``goodtables mydatapackage.zip`` works
- Improvements to goodtables CLI
(`#233 <https://github.com/frictionlessdata/goodtables-py/issues/233>`__)
- New ``goodtables init <data paths>`` command to create a new
``datapackage.json`` with the files passed as parameters and their
inferred schemas.

Bug fixes:

- Fix bug with ``truncated-values`` check on date fields
(`#250 <https://github.com/frictionlessdata/goodtables-py/issues/250>`__)

v1.5
~~~~

New API added:

- Validation ``source`` now could be a ``pathlib.Path``

v1.4
~~~~

Improved behaviour:

- rebased on Data Quality Spec v1
- rebased on Data Package Spec v1
- rebased on Table Schema Spec v1
- treat primary key as required/unique field

v1.3
~~~~

New advanced checks added:

- ``blacklisted-value``
- ``custom-constraint``
- ``deviated-value``
- ``sequential-value``
- ``truncated-value``

v1.2
~~~~

New API added:

- ``report.preset``
- ``report.tables[].schema``

v1.1
~~~~

New API added:

- ``report.tables[].scheme``
- ``report.tables[].format``
- ``report.tables[].encoding``

v1.0
~~~~

This version includes various big changes. A migration guide is under
development and will be published here.

v0.6
~~~~

First version of ``goodtables``.

.. |Travis| image:: https://img.shields.io/travis/frictionlessdata/goodtables-py/master.svg
:target: https://travis-ci.org/frictionlessdata/goodtables-py
.. |Coveralls| image:: http://img.shields.io/coveralls/frictionlessdata/goodtables-py.svg?branch=master
:target: https://coveralls.io/r/frictionlessdata/goodtables-py?branch=master
.. |PyPi| image:: https://img.shields.io/pypi/v/goodtables.svg
:target: https://pypi.python.org/pypi/goodtables
.. |Gitter| image:: https://img.shields.io/gitter/room/frictionlessdata/chat.svg
:target: https://gitter.im/frictionlessdata/chat
.. |Dataset| image:: data/dataset.png