Tweet

TL;DR

Often, virtualenv is overkill for the basic task of installing project dependencies and keeping them isolated. We present a simple alternative consisting of:

1. adding ./.pip to your PYTHONPATH
2. using pip install -t .pip to install modules locally
3. executing python from your project's root directory

Which version would you like today?

Installing dependencies is a required step for almost any Python application. Each Python app depends on a different set of libraries, and to be sure that it behaves as expected, the best thing is to install exactly the right version of each library.

The standard practice is to ship Python projects with a requirements.txt file. This file lists the libraries that the project depends on, and a version number for each one. If present, installing the dependencies is as easy as:

$ pip install -r requirements.txt
    

So far so good! The problems start when using two or more projects with conflicting dependencies. Let's suppose project A only works with library X version 0.1, and project B uses the same library X, but only works with version 0.2. By default, pip installs libraries globally into the Python interpreter's library path. This means that issuing the

$ pip install X==0.2
    

command will make X version 0.2 available in every Python instance, overwriting version 0.1 if it was previously installed. Switching between project A and project B would require reinstalling the right version of X each time, which is time-consuming and inconvenient.

An island in the sun

One popular solution to this commonly encountered problem is virtual environments. The virtualenv framework allows you to create isolated Python environments. The dependencies for each project are kept separate from each other. However, some users find virtualenv complicated to use, so packages like virtualenvwrapper and autoenv extend its functionality in an attempt to make things easier. Other solutions include Anaconda environments in the Anaconda Python distribution, and pyvenv which is baked into the Python standard library starting from Python 3.3.

Though these are great tools, we have always felt that they represent a rather heavy and complicated toolset for what should essentially be a very simple task.

Looking at Javascript, tools like npm and Bower provide the easy, reliable and powerful package management capabilities which it feels like Python is missing. The key to their success? Both tools download a copy of the right versions of the right libraries, by default placing them in a special folder directly within your project's directory. The downloaded libraries remain local to only that project, meaning that you automatically avoid the issues described above.

Pure magic

As it turns out, there's a simple way of replicating the npm/Bower approach for Python packages, involving these easy steps:

• Add ./.pip to your PYTHONPATH.
• Use pip with -t .pip to install your libraries locally.

Then, simply execute your code from within your project directory and forget about source env/bin/active and deactivate!

The trick works because ./.pip is a relative path. As a result, if you run python from ~/dev/project_a then ~/dev/project_a/.pip gets included in that Python instance's library path. If you run python from ~/dev/project_b, then ~/dev/project_b/.pip gets included instead. This works on all major platforms: Linux, Mac and Windows.

The folder name .pip is arbitrary of course — for example, one could choose to name the folder pip_components or libs instead. However, .pip is quick to type, and the initial . hides the folder by default on Linux/Mac.

Step 1: Set the PYTHONPATH

• Mac/Linux

  The following command will permanently set the PYTHONPATH for standard terminal sessions:

  
  $ echo 'export PYTHONPATH="./.pip:$PYTHONPATH"' >> ~/.bash_profile
          

  After that, either restart your terminal or do $ source .bash_profile to make sure the PYTHONPATH is loaded into the current session. Depending on your platform, you might want to use ~/.bashrc instead of ~/.bash_profile.

• Windows

  Go to Control Panel > System and Security > System > Change Settings > Advanced > Environment Variables, and add/edit the PYTHONPATH variable either to your user variables or system variables, setting it to .\.pip or .\.pip;(...other paths...). Then restart your command prompt.

• Temporary PYTHONPATH

  If you prefer to change the PYTHONPATH only temporarily for the duration of your terminal session, you can also do $ export PYTHONPATH=./.pip on Mac/Linux or > set PYTHONPATH=.\.pip on Windows.

  On Mac/Linux, you can even set the PYTHONPATH just for the duration of a Python session: $ PYTHONPATH=./.pip python main.py.

Step 2: Install packages with "pip -t"

Now that we have set the PYTHONPATH, the only thing left to do is to install our packages into the right location using pip. For this, we use the -t or --target switch to indicate the directory into which pip should install the packages:

$ cd project_a
project_a$ pip install requests==2.7.0 -t .pip

project_a$ python
>>> import requests
>>> requests.__version__
'2.7.0'
  

Now let's do the same for another project with another version:

$ cd project_b
project_b$ pip install requests==2.6.0 -t .pip

project_b$ python
>>> import requests
>>> requests.__version__
'2.6.0'
  

This works equally well when using a requirements.txt file:

$ pip install -r requirements.txt -t .pip
  

Potential pitfalls

Different Python interpreters

You can easily run your program with different Python interpreters as follows:

$ /path/to/python main.py
  

However, there's an issue if you are switching between Python 2 and 3 and you are using packages that don't have a single code base, i.e. they compile their source code during installation using 2to3. In that case, you would have to introduce something like .pip3 and add this path in front of your PYTHONPATH when running things with Python 3.

easy_install

If you happen to have packages that have been installed globally using easy_install, you've got the issue that easy_install prepends the path to those libraries to your sys.path which gives them priority over whatever you have in .pip. The solution here is to get rid of global easy_install installations. You can easily check if something hijacks your .pip setup by running import sys;sys.path within a Python session. If there are paths in front of ./.pip, then you might need to clean up things first.