How To Manage Python Dependencies With Virtual Environments

Python is a powerful programming language used across multiple disciplines—ranging from web developers to data scientists and beyond. 

What makes Python particularly popular is the availability of third-party libraries to fast-track the development of any complex solution by readily available source libraries written by others. 

As one moves beyond the simple scripting to the more complex solutions in the language, they delve themselves into a plethora of dependencies, which may lead to dependency hell if not efficiently managed. 

In this article, we shall focus on why dependency management is important and how to manage them using Python Virtual Environments.

Why Is Dependency Management Important?

Here are a few key reasons to ponder over:

• Stable builds independent of the environment are key features of agility. If a solution builds on your system, it should build on other systems and build a server.
• As updates roll in overtime for Python and its dependencies, a build should be stable now and in the future as well.
• Any relevant stakeholder should be able to make changes to the build.
• Foster multi-user collaborative projects
• Avoid dependencies conflicting with system-level dependencies, causing failure.
• Avoid dependency hell (trust me, this could be disastrous)

The list can be exhaustive, but the primary reasons to always consider are agility, multi-user collaboration, and avoidance of dependency hell.

Why Use Virtual Environments for Dependency Management?

A standard way of installing packages in Python is through the use of Pip. As solutions become more complex, it is inevitable to install multiple packages in a single project. This is often the starting point of dependency conflict, wherein two projects may wish to utilize different versions of the same package.

When pip install <package name> is used to install a package, it is installed in the global Python namespace by default. You may use pip3 install <package name> to specify the Python version (Python3 in this case) in a separate namespace of Python3. However, a better solution would be to specify the version of Python in the format below:

python2.7 -m pip install <package name>

Complex Cases

The trivial case might be the solution for simpler projects, but it does not cater to the following issues:

• Multiple projects using the same version of a package will conflict with each other.
• A dependency may conflict with system-level dependencies and crash the system.
• Multi-user collaborative projects will not be a possibility as the environment will not likely be the same on another machine.
• Difficulties in testing and debugging code from different Python and library versions.

It is with regards to avoiding these dependency issues that the use of Python Virtual Environments has become the de facto standard to manage Python dependencies. Virtual Environments create isolated contexts for installing and managing packages and their dependencies.

Virtual Environments also promote containerization and integration into cloud environments for advanced users—an emerging use case for those developers interested in taking advantage of the cloud.

Getting Started with Managing Dependencies using Virtual Environments:

In this article, we will be exploring three popular dependency management tools in Python, namely, pipenv, venv, and pyenv. Please note that virtualenv has been deprecated in Python 3.8.

Using Pipenv to Manage Dependencies

Pipenv allows one to create a distinct Virtual Environment for each project, thereby managing the dependencies in each.

1. First, check if Pipenv is installed and up-to-date using the following commands:

pipenv -version
pip install -upgrade pipenv

2. Change to the directory where you wish to create your project and execute the below command. This will create a virtual environment, along with a Pipfile to track dependencies and a Pipfile.lock that declares all the dependencies required.

pipenv –python 3.8

3. Now install the package of your choice using this:

pipenv install <packagename>

4. You may also alternatively provide a requirements.txt file to list down the packages needed to install.
5. Ensure all relevant packages have been installed by executing the following command:

pipenv run pip list

As a reminder, ensure all commands are executed in the directory where you wish to work on your project.

Common Issues and Resolution

Pipfile.lock is automatically updated whenever there is a change in dependencies. Sometimes, however, the file may not automatically update during a package installation.

This issue primarily arises when cloning a repo built on another Operating System because Pipfile.lock is system-specific. Regardless of the reason, pipfile lock command is useful in generating a new Pipfile.lock file.

The following commands are useful to know for Pipenv:

pipenv install <packagename> # pipfile.lock will be updated.

pipenv install –skip-lock <packagename> # pipfile.lock will not update.

pipfile lock # a pipfile.lock will be generated.

Using Venv to Manage Python Dependencies

Venv is included in Python 3.3+ and does not require to be installed.

1. Change to the directory where you wish to create your project and execute:

python -m venv <project_directoryname> #for Python 3.8

2. Alternatively, you can also execute the following to achieve the same results:

python -m virtualenv venv <project_directoryname> #for versions 3.3<Python<3.7

This is the only step for Venv. Please note that Venv is a lower-level tool as compared to Pipenv and is useful in circumstances where Pipenv may not meet your particular virtual environment needs.

Using Pyenv to Manage Python Dependencies

Pyenv is a version manager with the following key features:

• Setting up and changing the global Python version
• Installing multiple Python versions
• Set directory-specific and project-specific Python versions

and of course, create and manage virtual environments for dependency management. 

There are multiple ways to install Pyenv, and depending on your operating system you may follow the instructions from their official GitHub repo at https://github.com/pyenv/pyenv.

Alternatives to Manage Python Dependencies

The approaches shared are generalized for the majority. You may wish to explore the following tools as well to customize a set of our methods pertaining to your individual case: poetry, hatch, pip-tools, and micropipenv.

Conclusion

Our personal recommendation is to use Pipenv for the majority of dependency management use cases. Considering its approach of using the requirements.txt file, it makes it seamless to share the requirements file independent of the virtual environment to be used and configured across multiple machines for collaborative projects.

Let us know your thoughts in the comments. And as always, happy coding!