hpc-beeflow 0.1.8

BEE is a workflow orchestration system designed to build containerized HPC applications and orchestrate workflows across HPC and cloud systems. BEE has adopted the Common Workflow Language (CWL) for specifying workflows. Complex scientific workflows specified by CWL are managed and visualized through a graph database, giving the user the ability to monitor the state of each task in the workflow. BEE runs jobs using the workload scheduler (i.e. Slurm or LSF) on the HPC system that tasks are specified to run on.

BEE workflows can be archived for provenance and reproducibility. BEE can orchestrate workflows with containerized applications or those built locally on a system. However, there are advantages to containerizing an application.

A container is a package of code (usually binaries) and all of that code’s dependencies (libraries, etc.). Once built, this container can be run on many different platforms.

Containers provide many benefits:

• Users can choose their own software stack (libraries, compilers, etc.) and not be bound by the currently installed environment on any one machine.

• Codes can be run portably across numerous platforms–all dependencies will be downloaded and installed at run time.

• Entire workflow environments can be built into one or more containers. A user can include visualization and analysis tools along with the application. They will all work together as the application runs.

• Provenance and history can be tracked by storing containers in a historical repository. At any time, an older container can be rerun (all of its dependencies are stored with it). Execution is repeatable and interactions between software components can be tracked.

• Functional testing can be performed on smaller, dissimilar machines–there is no real need to test on the actual HPC platform (performance testing obviously requires target hardware).