Conda

It is much easier and more convenient to use virtual environments. Imagine, that you work on three projects, and each project utilizes different versions of Python and libraries. Let's talk about virtual environments and conda — package, dependency, and environment manager.

About virtual environments

Virtual environments are isolated and self-contained spaces within your computer where you can install specific versions of tools and libraries without affecting other projects or tools, installed on the computer.

They allow you to create an environment tailored to the needs of a particular project, preventing conflicts between different projects that might require different package versions. Virtual environments are a crucial tool for those who work on multiple projects simultaneously, enabling them to maintain project-specific dependencies and configurations in an organized and efficient manner.

Conda, in addition to being a package manager, also serves as an environment manager, making it easier to create, manage, and switch between virtual environments seamlessly.

What is conda?

Most likely, you are already acquainted with utilizing a package manager on your system. In the Linux environment, you may have encountered one or more of the following: apt, yum, or snap. As for macOS users, the package manager of choice is often homebrew , among other options.

Now, there exists Conda, a versatile package manager that sets itself apart by being entirely platform-independent. Unlike traditional package managers, Conda transcends the limitations of operating systems. It seamlessly operates on Linux, Mac, and Windows, offering a consistent and reliable experience across all three platforms.

Conda, functioning both as a package manager and environment manager, simplifies package discovery and installation. If you require a package that demands a distinct Python version, there's no need to switch to another environment manager or replace your local version of Python with the required one. Also, there is a paid version of conda for enterprise solutions, which may be useful for code organization in companies.

With Conda, you can effortlessly create a separate environment to run the required Python version alongside your usual environment.

About conda distributives

There are two distributives of conda: Anaconda and Miniconda.

Miniconda serves as a lightweight, bootstrap edition of Anaconda. It comprises essential components such as conda, Python, their respective dependencies, and a select few other useful packages, including pip and zlib. Unlike Anaconda, Miniconda offers a minimalistic setup, providing users with the flexibility to manually install additional packages as needed.

When to choose Anaconda, the recommendations from the original docs:

• If you are new to conda or Python.

• If you prefer the convenience of having Python and over 1,500 scientific packages automatically installed all at once.

• If you have the necessary time and disk space (a few minutes and 3 GB) for the installation.

• If you do not want to manually install each of the packages you intend to use.

• If you wish to utilize a curated and vetted set of packages known for their interoperability and usability.

When to choose Miniconda:

• If you don't mind individually installing the packages you need.

• If you lack the time or disk space to install over 1,500 packages altogether.

• If you want quick access to Python and the conda commands, and you can sort out the other programs later.

You can learn how to install both of the conda distributives using the manual on the official website.

How to use conda?

In this section, we will briefly cover the typical use case of conda.

It is common to use conda when you have multiple projects and each project utilizes its own version of libraries, for example, scikit-learn library. Let's imagine, that you want to check, whether the results of calculations, performed in each project, are affected by the change of scikit-learn library versions. To check this hypothesis you decided to create a virtual environment for each project and to run a test script.

Useful commands and notes:

1) First, to install the conda package you need to find the conda channel, which stores dependencies of your interest. Channels for basic dependencies, such as Python are already configured in conda. In this case, we are looking for scikit-learn package channel.

Option 1: conda search scikit-learn this will return versions of the package and the channel, where the package is located:

(base) username@laptop:~$ conda search scikit-learn
Loading channels: done
# Name                       Version           Build  Channel             
scikit-learn                  0.17.1 np110py27_blas_openblas_200  conda-forge         
scikit-learn                    0.18 np111py34_blas_openblas_203  conda-forge         
....

Option 2: Use the Anaconda website, type in the search panel the name of the package ('scikit-learn') and you can see the "owner" of the package (or artifact) — "conda-forge". This is the name of the channel, where this package is stored.

To add a new channel to your conda configuration use this command:

conda config --add channels conda-forge

2) Create conda environment:

conda create --name test_environment_v1 scikit-learn==1.1.0 python=3.10.12

With this command, we initiate an environment. We tell conda to use scikit-learn version 1.1.0 and Python 3.10.12 for this specific environment. In this step conda downloads packages and their dependencies into a separate environment called "test_environment_v1".

3) Activate and deactivate the environment:

To use, or "activate" the new environment, type the following:

conda activate test_environment_v1

To deactivate the environment, type the following:

conda deactivate

4) Create a conda recipe:

To share a resulting environment you need to create a yaml file, which contains all the specifications of the environment.

Export your active environment configuration:

conda env export > environment.yml

"environment.yml" file can be used by another person to reproduce the environment you have built:

conda create --name test_env -f environment.yml

However, it is good to manually curate the dependencies in case if you are going to share it between different operational systems. Some OS may not find dependencies, specific to original OS, where the yaml config was built.

5) Remove the environment:

To remove an environment you can simply type:

conda env remove --name test_environment_v1

First example

So, we need to create an environment and test our hypothesis about whether calculation results are dependent on the versions of libraries and Python. Let's say that the test code is written in test_scikit_learn.py script.

First, we will check the versions of Python and scikit-learn libraries and we determine, which channel we must use to download packages:

(base) username@laptop:~$ python --version # let's check version of python installed on the computer
Python 3.10.10

(base) username@laptop:~$ pip list | grep 'scikit' # returns nothing, because scikit-learn library is not installed

(base) username@laptop:~$ conda search scikit-learn==1.1.0
Loading channels: done
# Name                       Version           Build  Channel             
scikit-learn                   1.1.0 py310hffb9edd_0  conda-forge  scikit-learn                        
....

Second, we will create an environment:

(base) username@laptop:~$ conda config --add channels conda-forge # adding conda channel, where lies the package of out interest

(base) username@laptop:~$ conda create --name test_environment_v1 scikit-learn==1.1 python=3.10.12 # creating an environment for testing

Third, we will test the environment:

(base) username@laptop: conda activate test_environment_v1 # activation of test environment

(test_environment_v1) username@laptop:~$ python --version # note, the environment name has changed from base to test_environment_v1 and version of python is 3.10.12
Python 3.10.12

(test_environment_v1) username@laptop:~$ pip list| grep 'scikit' # we see installed scikit-learn package
scikit-learn       1.1.0

The last step – we will run our Python script and deactivate the environment:

(test_environment_v1) username@laptop:~$ python ./test_scikit_learn.py # now we can run the script
expected_results: 0.4
observed_results: 0.445

(test_environment_v1) username@laptop:~$ conda deactivate # now, we finished with this test, we need to deactivate the environment. 

Using this approach we can create an environment for each test case, safely run the code and assess the difference between test results.

Another example

Let's install the original version of the IGV browser. It is developed using Java, and not Python. We can find a conda command for IGV installation on the conda website.

(igv_environment) username@laptop:~$ conda install -c bioconda igv -n igv_environment # the command from conda website and the name of the environment, where we want to install the package
(igv_environment) username@laptop:~$ igv # launch browser

Note, that we don't have to install any Java dependencies for this application.

Conclusion

In this topic we covered conda environment essentials: we studies in which cases the usage of conda is useful and how one can use an environment. Also, we covered basic commands, which are used to manage the conda environment. We also demonstrated that conda can be used to install python-less packages.