A set of NGS singularity recipes, built for you and easily downlable
Project description
Damona is a singularity environment manager.
Damona started as a small collections of singularity recipes to help installing third-party tools for Sequana pipelines and is now used to download singularity images but more importantly set different environments (e.g. one per pipeline).
In a nutshell, it puts together the logic of Conda environments with the reproducibility of singularity containers. We believe it could be useful for other projects and therefore decided to release it.
- Python version:
Python 3.6, 3.7.3, 3.7, 3.8
- Source:
- Issues:
Please fill a report on github
Motivation
As stated on their website, Conda is an open source package management system and environment management system. Conda provides pre-compiled releases of software; they can be installed in different local environment that do not interfer with your system. This has great advantages for developers. Different community have emerge using this framework. One of them is Bioconda, which is dedicated to bioinformatics. Although great, it is sometimes tricky to re-install an environment simply because NGS pipelines relies on many different software and different versions may be in conflicts. Another great tool is Singularity. Singularity containers can be used to package entire scientific workflows, software and libraries, and even data. It is a simpe file that can be shared between environments and guarantee exectution and reproducibility.
Originally, Conda provided pre-compiled version of a package. Nowadays, it also provides a docker and a singularity image of the tool. Singularity can package an entire conda environment. As you can see everything is there to build reproducible tools and environment.
Now, what about a software in development that depends on third-party packages You would create a conda environment and starts installing those packages. Quickly, you will install another package that will break your environment due to unresolved conlicts; this is not common but it happens. In the worst case scenario, the environment is broken. In facilities where users depends on you, it can be quite stresful and time-consuming to maintain several such environments. This is why we have moved little by little to a very light conda environment where known-to-cause-problem packages have been shipped into singularity containers. This means we have to create aliases to those singularities. The singularities can be simple executable containers or full environment containers with many executables inside. In both cases, on need to manager those containers for different users, pipelines, versions etc. This started to be cumbersome to have containers in different places and update script that generate the aliases to those executables.
That’s where damona started: we wanted to combine the conda-like environment framework to manage our singularitiy containers.
Our goal is not to replace existing registry of biocontainers such as biocontainers but to use existing images, download them and manage them locally. Although Damona has some recipes and images (on sylabs/cokelaer/damona dn https://biomics.pasteur.fr/drylab/damona), those containers are for testing and help managing and installing the third-party tools required by Sequana pipelines.
We will therefore maintain damona in the context of Sequana project. Yet, Damona may be useful for others developers who wish to have a quick and easy solution for their users when they need to install third-party libraries
Installation
The is the egg and chicken paradox. To get reproducible container with singularity, at some point you need to install singularity itself. That the first of the two software that you will need to install. Instructions are on singularity web site. This is not obvious to be honest. You need the GO language to be installed as well. I personally installed from source and it worked like a charm.
Second, you need Damona. This is a pure Python sotfware with only a few dependencies. Install it with the pip software provided with your Python installation (Python 3.X):
pip install damona --upgrade
You should be ready to go.
Quick Start
1. list available containers
By default, we provide some recipes (for testing mostly but also to complement existing registries when a tool is missing) and their images.
To get the list images available within Damona collection, just type:
damona list --from-url https://biomics.pasteur.fr/drylab/damona/registry.txt
or in short (just for that url):
damona list --from-url damona
You may retrieve images from a website where a registry exists (see developer guide to create a registry)
2. install an image
Download the one you want to use:
damona install fastqc:0.11.9
This will download the container in your ./config/damona directory and create an executable for you in ~/.config/damona/bin.
This is your base environment. By default there is only one and all images will be stored in this directory.
The binaries are in the ~./config/damona/bin directory and you may need to append this path to your PATH environmental variable. For instance under Linux, type:
export PATH=~/config/damona/bin:$PATH
That’s it. You have downloaded a reproducible container and you can try:
fastqc --versio
Check that this is the correct path:
which fastqc
3. combine two different environments
If you type:
damona env
it will list the environments you currently hosting. Since you are starting, most probably you have only the base environment. Let us create a new one:
damone env --create test1
and check that you now have 1 environment:
damona env
We want to create an alias to the previously downloaded image of fastqc tool but in the test1 environment. First we activate it by setting an environmental variable:
export DAMONA_ENV=~/.config/damona/envs/test1 export DAMONA_PATH=~/.config/damona/envs/test1/bin
then, we install the container:
damona install fastqc:0.11.9
This will not download the image again. Instead it will create an alias in ~/.config/damona/envs/test1/bin directory
Change your PATH accordingly using the DAMONA_PATH variable
If you are interested to know more, please see the User Guide and Developer guide here below.
Roadmap
Damona is pretty new but here is short roadmap
check the md5 of the downloaded file so as to avoid overwritten existing name
do we store all images in the damona/images or do we store them in individual environement (with possbile duplicates).
remove the build and develop command most probably. The develop that builds a registry could be reaplce by a simple python code that builds the registry on the fly. the registry.yaml may not be required after all. Could be a simple registry.txt file name and version are included in the name.
ability to download any image from internet if user provide the name and version to cope with different naming conventions;
Changelog
Version |
Description |
---|---|
0.4.0 |
|
0.3.X |
|
0.3.0 |
|
0.2.3 |
|
0.2.2 |
|
0.2.1 |
fixed manifest |
0.2.0 |
first working version of damona to pull image locally with binaries |
0.1.1 |
small update to fix RTD, travis, coveralls |
0.1 |
first release to test feasibility of the project |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.