Virtual Environments Again (Draft)
by joe

Posted on 2019-06-09



Who - What - When - Where - Why


A classic approach to examining virtual environments - but not necessarily in the indicated order.


What are VEs for? Which is to say, WHY?

The standard answer is that Python programmers, developers and implementors use virtual environments to account for the separate run-time requirements of different projects or applications. This separation via virtual environments allows the multiple project or applications to run at the same time on the same hardware and operating system installation.

The different requirements may be because different versions of Python are required; or because different versions of the same package are required; or because of conflicting dependencies among packages used in the various projects or applications. Or simply to maintain neatly separated collections for other purposes.

We use virtual environments extensively, making new ones at the drop of a hat - testing something out, and then tossing it away. We think you should be comfortable using virtual environments - and disposing of them when they are no longer needed.

First Steps Are Quick & Easy

First, a few mechanics.

Set up a directory to hold your virtual environment.


$ cd
$ mkdir tmp
$ cd tmp

~/tmp$ dir
total 2
drwxrwxr-x  2 joe joe  2 Jun  9 20:30 .
drwxr-xr-x 14 joe joe 23 Jun  9 20:30 ..
~/tmp$

From that directory, create a virtual environment from scratch.

Show that it's there. Activate the virtual environment and upgrade PIP.


~/tmp$ python3 --version
Python 3.7.1
~/tmp$ python3 -m venv pug
~/tmp$ dir
total 4
drwxrwxr-x  3 joe joe  3 Jun  9 20:34 .
drwxr-xr-x 14 joe joe 23 Jun  9 20:30 ..
drwxrwxr-x  5 joe joe  7 Jun  9 20:34 pug

~/tmp$ source pug/bin/activate
(pug) ~/tmp$ pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 9.0MB/s
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-19.1.1
(pug) ~/tmp$

What Is In A Virtual Environment?

Show the top level of the virtual environment contents.


(pug) ~/tmp$ dir pug
total 6
drwxrwxr-x 5 joe joe  7 Jun  9 20:34 .
drwxrwxr-x 3 joe joe  3 Jun  9 20:34 ..
drwxrwxr-x 2 joe joe 12 Jun  9 20:37 bin
drwxrwxr-x 2 joe joe  2 Jun  9 20:34 include
drwxrwxr-x 3 joe joe  3 Jun  9 20:34 lib
lrwxrwxrwx 1 joe joe  3 Jun  9 20:34 lib64 -> lib
-rw-rw-r-- 1 joe joe 75 Jun  9 20:34 pyvenv.cfg
(pug) ~/tmp$

Show what's in the subdirectories. Nothing in the include directory.


(pug) ~/tmp$ dir pug/*
lrwxrwxrwx 1 joe joe  3 Jun  9 20:34 pug/lib64 -> lib
-rw-rw-r-- 1 joe joe 75 Jun  9 20:34 pug/pyvenv.cfg

pug/bin:
total 15
drwxrwxr-x 2 joe joe   12 Jun  9 20:37 .
drwxrwxr-x 5 joe joe    7 Jun  9 20:34 ..
-rw-r--r-- 1 joe joe 2191 Jun  9 20:34 activate
-rw-r--r-- 1 joe joe 1247 Jun  9 20:34 activate.csh
-rw-r--r-- 1 joe joe 2399 Jun  9 20:34 activate.fish
-rwxrwxr-x 1 joe joe  247 Jun  9 20:34 easy_install
-rwxrwxr-x 1 joe joe  247 Jun  9 20:34 easy_install-3.7
-rwxrwxr-x 1 joe joe  229 Jun  9 20:37 pip
-rwxrwxr-x 1 joe joe  229 Jun  9 20:37 pip3
-rwxrwxr-x 1 joe joe  229 Jun  9 20:37 pip3.7
lrwxrwxrwx 1 joe joe    7 Jun  9 20:34 python -> python3
lrwxrwxrwx 1 joe joe   22 Jun  9 20:34 python3 -> /usr/local/bin/python3

pug/include:
total 2
drwxrwxr-x 2 joe joe 2 Jun  9 20:34 .
drwxrwxr-x 5 joe joe 7 Jun  9 20:34 ..

pug/lib:
total 3
drwxrwxr-x 3 joe joe 3 Jun  9 20:34 .
drwxrwxr-x 5 joe joe 7 Jun  9 20:34 ..
drwxrwxr-x 3 joe joe 3 Jun  9 20:34 python3.7
(pug) ~/tmp$

The site-packages directory is where it's at.


(pug) ~/tmp$ dir pug/lib/python3.7/site-packages/
total 12
drwxrwxr-x 8 joe joe   9 Jun  9 20:37 .
drwxrwxr-x 3 joe joe   3 Jun  9 20:34 ..
-rw-rw-r-- 1 joe joe 126 Jun  9 20:34 easy_install.py
drwxrwxr-x 5 joe joe   7 Jun  9 20:37 pip
drwxrwxr-x 2 joe joe   9 Jun  9 20:37 pip-19.1.1.dist-info
drwxrwxr-x 5 joe joe   7 Jun  9 20:34 pkg_resources
drwxrwxr-x 2 joe joe   3 Jun  9 20:34 __pycache__
drwxrwxr-x 6 joe joe  43 Jun  9 20:34 setuptools
drwxrwxr-x 2 joe joe  11 Jun  9 20:34 setuptools-40.6.2.dist-info
(pug) ~/tmp$

Start Peeling Back the Layers

Start At The Top: pyenv.cfg. This file records the settings established when you created the virtual environment: where the base Python interpreter for this virtual environment is located; whether to reference the site-packages associated with the base interpreter when executing in this virtual environment; and what version of the Python interpreter served was referenced when this virtual environment was created.


home = /usr/local/bin
include-system-site-packages = false
version = 3.7.1

Check out our note on all this coordination and redundancy at the end of this post.

Plumbing in the bin directory

-rw-r--r-- 1 joe joe 2191 Jun  9 20:34 activate
-rw-r--r-- 1 joe joe 1247 Jun  9 20:34 activate.csh
-rw-r--r-- 1 joe joe 2399 Jun  9 20:34 activate.fish
-rwxrwxr-x 1 joe joe  247 Jun  9 20:34 easy_install
-rwxrwxr-x 1 joe joe  247 Jun  9 20:34 easy_install-3.7
-rwxrwxr-x 1 joe joe  229 Jun  9 20:37 pip
-rwxrwxr-x 1 joe joe  229 Jun  9 20:37 pip3
-rwxrwxr-x 1 joe joe  229 Jun  9 20:37 pip3.7
lrwxrwxrwx 1 joe joe    7 Jun  9 20:34 python -> python3
lrwxrwxrwx 1 joe joe   22 Jun  9 20:34 python3 -> /usr/local/bin/python3

The activate files turn the virtual environment on. Details a little later.

The easy_install and pip pip files are standard utilities that are specific to each Python interpreter - or to each virtual environment based on that interpreter.


#!/home/joe/tmp/pug/bin/python3

# -*- coding: utf-8 -*-
import re
import sys

from pip._internal import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Furthermore, pip and easy_install are just about the same file, anyway.


~/tmp/pug/bin$ diff easy_install easy_install-3.7
~/tmp/pug/bin$ diff pip pip3
~/tmp/pug/bin$ diff pip pip3.7
~/tmp/pug/bin$ diff pip3 pip3.7
~/tmp/pug/bin$ diff pip easy_install
7c7
< from pip._internal import main
---
> from setuptools.command.easy_install import main
~/tmp/pug/bin$

The symlinks point the python and python3 commands to the interpreter on which this virtual environment is based.

The Focus of the Action

Most of the usefulness and operational virtue associated with Python virtual environments arises from the isolation (or coordination) of the lib/PythonX.Y/site-packages directory.


(pug) ~/tmp$ dir pug/lib/python3.7/site-packages/
total 12
drwxrwxr-x 8 joe joe   9 Jun  9 20:37 .
drwxrwxr-x 3 joe joe   3 Jun  9 20:34 ..
-rw-rw-r-- 1 joe joe 126 Jun  9 20:34 easy_install.py
drwxrwxr-x 5 joe joe   7 Jun  9 20:37 pip
drwxrwxr-x 2 joe joe   9 Jun  9 20:37 pip-19.1.1.dist-info
drwxrwxr-x 5 joe joe   7 Jun  9 20:34 pkg_resources
drwxrwxr-x 2 joe joe   3 Jun  9 20:34 __pycache__
drwxrwxr-x 6 joe joe  43 Jun  9 20:34 setuptools
drwxrwxr-x 2 joe joe  11 Jun  9 20:34 setuptools-40.6.2.dist-info
(pug) ~/tmp$

The HOW Is In Activation

It turns out that a virtual environment essentially consists of defining a few environment variables to modify and reset the system PATH and the system prompt.

Here's the entire activate file.


# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
    # reset old environment variables
    if [ -n "${_OLD_VIRTUAL_PATH:-}" ] ; then
        PATH="${_OLD_VIRTUAL_PATH:-}"
        export PATH
        unset _OLD_VIRTUAL_PATH
    fi
    if [ -n "${_OLD_VIRTUAL_PYTHONHOME:-}" ] ; then
        PYTHONHOME="${_OLD_VIRTUAL_PYTHONHOME:-}"
        export PYTHONHOME
        unset _OLD_VIRTUAL_PYTHONHOME
    fi

    # This should detect bash and zsh, which have a hash command that must
    # be called to get it to forget past commands.  Without forgetting
    # past commands the $PATH changes we made may not be respected
    if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then
        hash -r
    fi

    if [ -n "${_OLD_VIRTUAL_PS1:-}" ] ; then
        PS1="${_OLD_VIRTUAL_PS1:-}"
        export PS1
        unset _OLD_VIRTUAL_PS1
    fi

    unset VIRTUAL_ENV
    if [ ! "$1" = "nondestructive" ] ; then
    # Self destruct!
        unset -f deactivate
    fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="/home/joe/tmp/pug"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
# this will fail if PYTHONHOME is set to the empty string (which is bad anyway)
# could use `if (set -u; : $PYTHONHOME) ;` in bash
if [ -n "${PYTHONHOME:-}" ] ; then
    _OLD_VIRTUAL_PYTHONHOME="${PYTHONHOME:-}"
    unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT:-}" ] ; then
    _OLD_VIRTUAL_PS1="${PS1:-}"
    if [ "x(pug) " != x ] ; then
	PS1="(pug) ${PS1:-}"
    else
    if [ "`basename \"$VIRTUAL_ENV\"`" = "__" ] ; then
        # special case for Aspen magic directories
        # see http://www.zetadev.com/software/aspen/
        PS1="[`basename \`dirname \"$VIRTUAL_ENV\"\``] $PS1"
    else
        PS1="(`basename \"$VIRTUAL_ENV\"`)$PS1"
    fi
    fi
    export PS1
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then
    hash -r
fi

Let's do the second part first. This is the activation code for the virtual environment.

The first step is to set the VIRTUAL_ENVIRONMENT system variable to the directory where this virtual environment is located.

Then we preserve the current system path and make a new path by pre-pending this virtual environment directory. Next we do something similar for the PYTHONHOME environment variable, i.e., preserve the current value, but unset it for this virtual environment.

Finally, we adjust the system prompt by pre-pending the name of this virtual environment and saving the current setting for restoration later.

With all of these settings in place, the net effect of an active virtual environment is that the Python interpreter symlinked in the bin directory runs and relies on the modified system PATH to locate any package modules that are imported as part of the application.

Just so you know, it is possible to supplement the packages local to the virtual environment with the packages installed for the base Python interpreter itself, but we'll leave that capability for others to explain. For our own part, we never install packages at the base Python interpreter level. It's just cleaner that way.

Now let's look at the deactivate() definition in the bin/activatefile.

This is a shell function that does what is says: it deactivates the virtual environment by setting the changed environment variables to their previous values.

When we reviewed the activation instructions, we skipped over the first step - which is based on the last instruction in the deactivate() function. That first activation command is:

deactivate nondestructive

In the definition of the deactivate() function, the parameter value "nondestructive" terminates a virtual environment while leaving the deactivate() function itself still active.

When it is used as the first command of the activate file, the effect is to terminate any existing virtual environment before setting up this virtual environment. See the "Olden Days" sidebar.

A Cautionary Note

Looking at the innards of a virtual environment - and its constituent elements - it becomes apparent that, however well this all works, it is vulnerable to tinkering by a naive user. That includes us, and we dare say, all but a handful of you as well.

We don't know for sure what happens when you install a different version of Python into the directory that once held the base version of the Python interpreter used for current virtual environments.

Does the interpreter check the "version" value in the pyvenv.cfg file against itself? Is the interpreter even a little suspicious if the creation dates of files in the virtual environment setup are earlier than the dates for itself or the directory in which it resides?

What are the symptoms one should look for if he suspects that the base interpreter and virtual environment have fallen out of synch?

How We Do It
We create and dispose of virtual environments regularly. So often, in fact, that we wrote a few helper commands to simplify the whole process.

Check out the full explanation here.

Feel free to copy and adapt our methods to suit your own requirements - recognizing, of course, that we make no claims that what we have done will work for anyone else. We suppose that it's entirely possible that our code could destroy all the Python accomplishments you have to your credit - and your dreams of software glory along with them.
Olden Days
It did not used to be this way. At least our recollection of the internals of virtual environments from a half-decade ago don't jibe with the current setup. As we recall, things used to be a lot more complicated.

The differences are tied up in the initial "deactivate nondestructive" command in the activate file.

One immediate effect of this command is that one cannot activate a virtual environment from a currently active virtual environment and return to the currently active virtual upon executing the deactivate command in the second virtual environment. We see to remember that in the olden days we could "stack" virtual environments. That capability vastly complicated the whole activate/deactivate process - as you can well imagine.

We weren't inclined to trust this language-level mechanism to start with. So you can also imagine our consternation when faced with a pot of spaghetti-looking code that was supposed to loosely couple a Python interpreter to the OS while enforcing what seemed to us to be somewhat indefinite boundaries to start with.

Abstracting the management of the mechanisms of virtual environments was tempting, but proved to be even more complicated than simply using the built-in Python command.

It was several years before we checked into the innards of virtual environments again. It all seems much cleaner and more straightforward now. Virtual environments have become a staple, even essential, element of our development process now.

Of course, our memory may be faulty, and none of this really matters much anyway. But we thought you'd like to know.

Comments

It will be some time yet before we get a comments section working here. In the meantime feel free to send comments via email. On this site our name is Joe Python. The email address is our first name at joepython.com.

Edited: 2019-06-10 17:29:49(utc) Generated: 2019-06-10 17:29:58(utc)