spire-pipeline 0.5.2

# Welcome to Spire

Spire is a pipeline runner based on [YAML](https://en.wikipedia.org/wiki/YAML) files and [Ninja](https://ninja-build.org/). The pipeline descriptions (i.e. the list of steps to run) are stored in [Jinja](http://jinja.pocoo.org/)-templated YAML files which allow both readability and modularity. Using Ninja as a backend ensures fast and correct dependency handling.

## Quickstart

Install Ninja, following [official instructions](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages).

From the source directory, use `pip` to ensure that dependencies are installed along with Spire:
```
pip install .
```

A pipeline description is a templated YAML file containing a list of `steps`, each step having the following fields:
* `id`: unique name to reference this step from other steps
* `inputs`: files that must be present for the step to execute
* `outputs`: files that will be generated by the step
* `commands`: a list of commands (optional for `phony` rules, see below)
* `phony`: flag indicating whether the target is phony i.e. does not produce any output (optional, defaults to `false`)

The `id` and `outputs`, and `inputs` fields are mandatory for all steps. For non-`phony` rules, the `commands` is optional. The `inputs`, `outputs` and `commands` may be written either as lists or single values.

The YAML content can be templated using Jinja, with extra features:
* Filters to convert to YAML (`foo|yaml`) or to JSON (`foo|json`)
* Function to perform globbing in the directory where the build will be executed (`glob("*.c")`)
* The `basename` and `dirname` functions from the Python module `os.path`
* The current build directory (`build_directory`)
* The absolute path to the directory containing the current pipeline (`pipeline_directory`)
* References to the outputs or inputs of the current step or of another step, specified by its identifier (`{{ inputs("other_step_id") }}`, `{{ outputs("other_step_id") }}`). These references are always list.

The following example demonstrates most of those features. The first step looks for `.c` files and compile them to `.o`; the second step links them together to build an executable.

```yaml
steps:
- id: compile
inputs: {{ glob("*.c") }}
outputs: {{ inputs("compile")|replace(".c", ".o") }}
commands: gcc -c {{ inputs("compile")|join(" ") }}

- id: link
inputs: {{ outputs("compile") }}
outputs: foo
commands:
- gcc -o {{ outputs("link")[0] }} {{ inputs("link")|join(" ") }}
- strip {{ outputs("link")[0] }}
```

If you use Atom and the [atom-jinja2](https://atom.io/packages/atom-jinja2) package, save it as `.yml.j2` for better syntax highlighting.

The runner takes two sets of options: everything before a litteral `--` is passed to the parser, everything after is passed to the build engine. The parser expects the file name containing the pipeline description (mandatory), and a list of variable definitions that will be fed to Jinja, in the form `key=value`.

The following willl parse `build.yml.j2` with the variable `root` containing the current working directory, and will then build the target `foo`.

```bash
spire build.yml.j2 root=$(pwd) -- foo
```

## Design

The input data is a text file containing a YAML-templated description of the pipeline steps. Three entities will process this data in order to run the pipeline:
* Parser: run the template engine (e.g. globbing, root directory, number of slots for parallel jobs, etc.) and handle cross-step references
* Generator: convert the rendered description to build-engine specific file
* Build engine: run the build itself with user-specified options

Under this design, multiple build engines and their corresponding generators may be implemented; the build engine of choice is Ninja, due to its ability to handle multiple outputs per step and to its parallel build features.