Concurrent use of Multiple Conda Environments

· 411 words · 2 minute read
conda · bioconda · public-wrappers

Conda is a platform-neutral packaging system. The BioConda channel is of particular interest for bioinformatics users, as it provides access to an enormous library of ready-to-use bioinformatics applications, which can be otherwise notoriously difficult to install.

One of the problems solved by Conda, and BioConda, is the dependency hell which results from trying to install several applications, which have requirements for incompatible versions of their dependencies, for example different versions of samtools. By installing into different conda environments, such applications can coexist. However, there is a problem.

Only one conda environment may be active at any one time. So for some bioinformatics pipeline wanting to run programs from different environments, it would seem necessary to activate and deactivate conda environments dynamically, to get access to their programs. However, this is not the case, by virtue of exec-wrappers and public-wrappers. (In the interest of full disclosure, public-wrappers was written by this author.)

Exec-wrappers exists to create efficient wrapper scripts for programs in conda (or other) environments. What it does is equivalent to on-the-fly environment activation, albeit very much faster. Public-wrappers builds on this, to provide a means of publishing certain programs from a conda (or other) environment, into a public scripts directory.

By making accessible in the public script directory only certain programs, any incompatible dependencies are hidden from the user and each other, by virtue of not being on the path. They are only accessible by the top-level application for the duration of execution of its wrapper script.

Public-wrappers uses a configuration file to define which programs from which environments should be public. For example.

[conda.bifo-essential]
wrapperdir = "/stash/miniconda3/envs/bifo-essential/public-wrappers"
envdir = "/stash/miniconda3/envs/bifo-essential"
public = [
    "blast",
    "bwa",
    "samtools",
]

[conda.bifo-interactive]
wrapperdir = "/stash/miniconda3/envs/bifo-interactive/public-wrappers"
envdir = "/stash/miniconda3/envs/bifo-interactive"
public = [
    "igv",
]

With this configuration file in public-wrappers.toml, for example, the following command is sufficient to generate the desired public wrapper scripts.

$ configure-public-wrappers -c public-wrappers.toml

For ease of use, it is recommended to deploy a file such as the following in /etc/profile.d

# include public wrappers on path, where they exist
for env in `ls -r /stash/miniconda3/envs 2>/dev/null`; do
    test -d /stash/miniconda3/envs/$env/public-wrappers && {
        PATH=/stash/miniconda3/envs/$env/public-wrappers:$PATH
    }
done
export PATH

This simply prepends the public wrappers directories onto the path, so that users have immediate access to all programs published in the public wrappers configuration file.

Public-wrappers is on both PyPI and conda-forge, so may be installed easily by either of the following commands.

$ pip install public-wrappers
$ conda install public-wrappers
Share