jobarchitect 0.1.0

Overview

This tool is intended to automate generation of scripts to run analysis on data sets. To use it, you will need a data set that has been created (or annotated) with dtool.

Installation

To install the jobarchitect package.

$ cd jobarchitect
$ python setup.py install

Use

To generate bash scripts for data analysis, first create a job template, e.g.:

$ echo "echo {input_file} > {output_file}" > job.tmpl

Then an example dataset:

$ datatool new dataset
project_name [project_name]:
dataset_name [dataset_name]: example_dataset
...

$ echo "My example data" > example_dataset/data/my_file.txt
$ datatool manifest update example_dataset/

Create an output directory:

$ mkdir output

Then you can generate analysis run scripts with:

$ sketchjob job.tmpl example_dataset/ output/
#!/bin/bash
_analyse_by_ids   --program_template="echo {input_file} > {output_file}"   --input_dataset_path=example_dataset/   --output_root=output/   e4c73fa7c34b76499ac13fc5c335fa007e9c3e8f

Try the script with:

$ sketchjob job.tmpl example_dataset/ output/ > run.sh
$ bash run.sh
$ cat output/my_file.txt
/Users/hartleym/scratch/example_dataset/data/my_file.txt

Working with Docker

$ bash build_docker_image.sh

$ sketchjob job.tmpl example_dataset/ output/ --backend=docker --image-name=jicscicomp/jobarchitect

#!/bin/bash
IMAGE_NAME=jicscicomp/jobarchitect
docker run    --rm    -v example_dataset/:/input_dataset:ro    -v output/:/output    $IMAGE_NAME    _analyse_by_ids      --program_template "echo {input_file} > {output_file}"      --input_dataset_path=/input_dataset      --output_root=/output      e4c73fa7c34b76499ac13fc5c335fa007e9c3e8f