TextFormats

TextFormats is a library for rapidly defining and using text formats for structured data, and allows for rapid prototyping of parsers for such file formats in Nim, Python and C/C++.

Given a format definition, expressed in a simple declarative language (TFSL, Text Formats Specification Language), the library provides functions for switching from the text representation of the data ("encoded string") to the actual data it represents ("decoded data") and vice-versa.

The definition of the formats in TFSL is human readable and reduces the requirement of complex regular expression. As opposed to lexers or regular expressions, it does not only validates and splits different parts of a format, but converts them to data in built-in scalar and compound datatypes, allowing for fine tuning of the conversion.

Python

The Python API of TextFormats is documented in the Python API manual and cheatsheet and can be installed using:

pip install textformats

If a binary package compatible with the system is available, it will be downloaded and installed. Nim installation is not required.

If no binary package is available, the source distribution is downloaded. In this case, the Nim compiler version >= 1.6.0 must be installed and the nim binary must be in PATH. Then, the pip install textformats command will automatically compile and install the package.

Example applications based on the Python API are available in the git repository here and here.

Nim

The Nim API of TextFormats is documented in the Nim API manual and cheatsheet. and is installed using:

nimble install textformats

Example applications based on the Nim API are available in the git repository here and here.

C/C++

The C API of TextFormats is documented in the C API manual and cheatsheet and is obtained by cloning the git repository. Furthermore Nim compiler version >= 1.6.0 must be installed

The C API are in the in the C directory of the git repository. Example applications based on the C API are available in the git repository here and here.

Command line tools

The CLI tools developed with TextFormats allows the use of the library from the command line (e.g. in Bash scripts). For using it, the Nim compiler version >= 1.6.0 must be installed and the textformats Nim package installed (nimble install textformats). The tools are thereby installed and compiled. They are tf_spec (work with TFSL specifications), tf_decode (convert a format to JSON), tf_encode (convert JSON to a format) and tf_validate (validate data or its text representations).

The CLI tools are documented in the CLI manual, cheatsheet. Man pages can be generated using nimble climan from the source code directory.

Examples of use of the CLI tools are given in the git repository here here.

Format specifications

The TFSL (TextFormats Specification Language) is usually input by the user as a YAML file. In alternative, the interactive Python script tf_genspec.py can be used, which allows the generation of a specification from scratch.

Several specifications are made available with the package and are contained in the git repository in the spec directory

The specification language is documented in a manual, and a cheatsheet, describing the syntax, a howto explaining how to define text representations for different kind of values: strings, numeric, boolean, list, dictionaries, etc. and a tests manual, describing how to implement specification tests.

Format specifications: an example

In multiple biological sequence analysis formats (e.g. SAM, GFA), a CIGAR string represents a list of multi-edit operations, each consisting of a length (positive integer value) and an operation code (one among a short list of possible codes).

A string representation of a CIGAR is for example "10M1D20M1I40M". The string compactly represents a list of mappings, each with two members "length" and "code". In JSON its representation would be: [{length: 10, code: "M"}, {length: 1, code: "D"}, {length: 20, code: "M"}, {length: 1, code: "I"}, {length: 40, code: "M"}].

The definition of a CIGAR in TextFormats would be:

cigar:
  list_of:
    composed_of:
    - length: {unsigned_integer: {min: 1}}
    - code: {values: [M, D, I, P] }

Once the definition is provided, the library provides the following functions:

# decoding: string representation => data
"10M1D".decode(cigar)
# => [{length: 10, code: "M"}, {length: 1, code: "D"}]

# encoding: data => string representation
[{length: 10, code: "M"}, {length: 1, code: "D"}].encode(cigar)
# => "10M1D"

# validation of string representation
"10M1D".is_valid(cigar)
# => true

# validation of data
[{length: 10, code: "M"}, {length: 1, code: "D"}].is_valid(cigar)
# => true

Furthermore, definitions can refer to each other, which allows splitting a complex definition into smaller parts, and reuse them in different contexts. For example, the previous definition could have been written as:

cigar_code: {values: [M, D, I, P]}
pos_integer: {unsigned_integer: {min: 1}}
cigar_op: {composed_of: [length: pos_integer, code: cigar_code]}
cigar: {list_of: cigar_op}

Since definitions can be re-used in different contexts and formats, they can be stored in modules, which can be imported from other specification files. The import mechanism is flexible, featuring namespaces, partial imports and redefinitions of parts of an imported module.

Finally, sometimes fine tuning of the conversion between encoded and decoded data is necessary. Thus, the following operations can be included in the definitions:

• providing more meaningful strings: e.g. in the example above of cigar operation code, one can decode the "M" to the string "replacement" and the "D" to "deletion"
• converting to different types: e.g. in some formats the symbols "+" and "-" represent the boolean values true and false, and should be converted accordingly
• add implicit values: e.g. in many formats, in a particular context of the file, multiple kind of information can be stored, and can be recognized from their different formatting; in this case, one can add a label to the decoded data, describing the type of information
• remove formatting symbols: often structured elements contain formatting constant strings such as separators, prefixes and suffixes, which must not be included in the resulting decoded data
• define default values: sometimes a given symbol or part of a string representation is missing when representing a default value.

Understanding specification errors

The tool tf_spec (see above Command line tools) can be used as tf_spec info -s <SPECFILE> to list of datatypes of a (valid) specification are output. An error will be output if the specification is invalid.

A further tool to validate the syntax of a YAML or JSON specification, which can be sometimes useful to better understand specification errors, is tf_cerberus.py, provided under scripts in the source code git repository. It is based on the Python library cerberus (which is required in order to use this tool). The script has some limitations: it is not always guaranteed that a validated specification is indeed valid (e.g. circular or invalid references are not found).

Interactive generation

An interactive script tf_genspec.py is provided under scripts in the source code git repository. It can be used to generate a TextFormats specification in YAML file. The script has some limitations: it is not always guaranteed that the generated specification is correct (e.g. the user can create circular or invalid references).

Thus the resulting output file should be tested, e.g. generating examples from each of the defined datatypes using cli/tf_spec generate_tests -s <OUTFILE>. This command would fail if the specification is invalid. Furthermore, the results can be inspected to check that the examples reflect the expectations.

Developer notes

The library is implemented using the programming language Nim. This language was used, since it combines some of the advantages of Python with those of compiled languages.

During compilation, C code is created and then compiled to binary. Besides in Nim code itself, the resulting library can be easily employed in C/C++ and in Python.

Code organization and conventions, used in the implementation of the TextFormats library, addressed to the library software developer, are documented in the developer manual

To run the unit test suite of the library, use the nimble test command from the main project source code directory. To run the CLI tools tests, first build it using nimble build or nimble install, then use the nimble clitest command. To run the C API tests, use nimble ctest. To run the Python API tests, first build the package using nimble pymake, then use nimble pytest.

Known limitations

• Only formats which are regular languages can be defined -- with at most some exceptions (JSON elements can be included, since they are parsed by the json library).
• Unsigned integers cannot be larger than the largest signed integer [^2]

[^2]: reason: JsonNode objects from the json nim standard library are used to represent decoded values both for Nim code and for passing the values through the API to functions written in other languages; however, JsonNode does not have a representation for unsigned integers)