tinuous 0.1.0

Global Options

-c FILE, --config FILE

Read configuration from the given file [default value: config.yml]

-l LEVEL, --log-level LEVEL

Set the log level to the given value. Possible values are “CRITICAL”, “ERROR”, “WARNING”, “INFO”, “DEBUG” (all case-insensitive) and their Python integer equivalents. [default value: INFO]

fetch Command

tinuous [<global options>] fetch [<options>]

tinuous fetch reads a configuration file telling it what repository to retrieve logs for, where to retrieve them from, and where to save them, and then it carries those steps out.

sanitize Command

tinuous [<global options>] sanitize <path> ...

Sanitize the given files, replacing all strings matching a secret regex with a series of asterisks.

Configuration

The configuration file is a YAML file containing a mapping with the following keys:

repo

The GitHub repository to retrieve logs for, in the form OWNER/NAME

ci

A mapping from the names of the CI systems from which to retrieve logs to sub-mappings containing CI-specific configuration. Including a given CI system is optional; logs will be fetched from a given system if & only if it is listed in this mapping.

The CI systems and their sub-mappings are as follows:

github

Configuration for retrieving logs from GitHub Actions. Subfields:

path

A template string that will be instantiated for each workflow run to produce the path for the directory (relative to the current working directory) under which the run’s build logs will be saved. See “Path Templates” for more information.

workflows

A list of the filenames for the workflows for which to retrieve logs. The filenames should only consist of the workflow basenames, including the file extension (e.g., test.yml, not .github/workflows/test.yml).

travis

Configuration for retrieving logs from Travis-CI.com. Subfield:

path

A template string that will be instantiated for each job of each build to produce the path for the file (relative to the current working directory) in which the job’s logs will be saved. See “Path Templates” for more information.

appveyor

Configuration for retrieving logs from Appveyor. Subfields:

path

A template string that will be instantiated for each job of each build to produce the path for the file (relative to the current working directory) in which the job’s logs will be saved. See “Path Templates” for more information.

accountName

The name of the Appveyor account to which the repository belongs on Appveyor

projectSlug

(optional) The project slug for the repository on Appveyor; if not specified, it is assumed that the slug is the same as the repository name

since

A timestamp (date, time, & timezone); only logs for builds started after the given point in time will be retrieved

As the script retrieves new build logs, it keeps track of their starting points. Once the logs for all builds for the given CI system & configuration have been fetched up to a certain point, the timestamp for the latest such build is stored in the state file and used as the new since value for the respective CI system on subsequent runs.

types

A list of build trigger event types; only logs for builds triggered by one of the given events will be retrieved

The recognized event types are:

cron

A build run on a schedule

pr

A build in response to activity on a pull request

push

A build in response to new commits

secrets

(optional) A mapping from names (used in log messages) to regexes matching secrets to sanitize

allow-secrets-regex

(optional) Any strings that match a secrets regex and also match this regex will not be sanitized. Note that allow-secrets-regex is tested against just the substring that matched a secrets regex without any surrounding text, and so lookahead and lookbehind will not work in this regex.

datalad

(optional) A sub-mapping describing integration of tinuous with Datalad. Subfields:

enabled

(optional) A boolean. If true (default false), the current directory will be converted into a Datalad dataset if it is not one already, the logs will optionally be divided up into subdatasets, and all new logs will be committed at the end of a run of tinuous fetch. path template strings may contain // separators indicating the boundaries of subdatasets.

cfg_proc

(optional) Procedure to run on the dataset & subdatasets when creating them

All fields are required unless stated otherwise.

A sample config file:

repo: datalad/datalad
ci:
  github:
    path: '{year}/{month}/{day}/{ci}/{type}/{type_id}/{commit}/{wf_name}/{number}/'
    workflows:
      - test_crippled.yml
      - test_extensions.yml
      - test_macos.yml
  travis:
    path: '{year}/{month}/{day}/{ci}/{type}/{type_id}/{commit}/{number}/{job}.txt'
  appveyor:
    path: '{year}/{month}/{day}/{ci}/{type}/{type_id}/{commit}/{number}/{job}.txt'
    accountName: mih
    projectSlug: datalad
since: 2021-01-20T00:00:00Z
types: [cron, pr, push]
secrets:
  github: '\b(v1\.)?[a-f0-9]{40}\b'
  docker-hub: '\b[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}\b'
  appveyor: '\b(v2\.)?[a-z0-9]{20}\b'
  travis: '\b[a-zA-Z0-9]{22}\b'
  aws: '\b[a-zA-Z0-9+/]{40}\b'
datalad:
  enabled: true
  cfg_proc: text2git

Path Templates

The path at which logs for a given workflow run or build job are saved is determined by instantiating the path template string given in the configuration file for the corresponding CI system. A template string is a filepath containing placeholders of the form {field}, where the available placeholders are:

All timestamps and timestamp components are in UTC.

Authentication