Simple Virtual Environments
by joe

Posted on 2019-03-10



Glendower:
I can call spirits from the vasty deep.

Hotspur:
Why, so can I, or so can any man;
But will they come when you do call for them?
Henry The Fourth
Part I Act 3, scene 1


Python Essentials - The Virtual Environment

What can we say that has not been said before? Python virtual environments make development, testing and deployment of Python applications simpler and safer. Learning to create, use and discard Python virtual environments will do more to preserve your sanity and boost your productivity than any other sub-skill you can pursue.

We are, perhaps, remiss in taking so long to address this topic. We have instead presumed not merely the capability but some advanced facility in the creation and employment of virtual environments in our previous posts.

Today we remedy that oversight.

Really Pretty Easy

It may be that we are too darned used to relying on Python virtual environments, but as we worked on this post it seems to us that they are really pretty easy to use.

We just checked our Mac development system, and we have 28 virtual environments stacked up in our venvs directory. Some of them are two years old. (We'll probably remove 6 - 10 of them in the wake of this realization.)

We are used to creating new virtual environments for even the most trivial exercise - and then removing them once we are done. We only hope that this post encourages you to do the same and reap the benefits of a truly wonderful Python facility.

My Way (well, Our Way)
We regularly call up virtual environments from the vasty deep. Our creation and use of virtual environments is so frequent that we decided to streamline the process.

The excerpt below is taken from the .bash_profile we most commonly use for our Linux and Mac installations. In it we define two bash functions (menv and genv) and an alias (denv) for making, getting and deactivating virtual environments, respectively.


# ----------- Virtual Environments ---------
#
function menv {
    if [ $1 = "" ]; then
        echo "Must supply directory name."
        return
    fi
    if [ -f ~/venvs/$1/bin/activate ]; then
        echo "venv $1 already exists"
    else
        python3 -m venv ~/venvs/$1
        echo "virtual environment $1 created"
    fi
}

function genv {
    source ~/venvs/$1/bin/activate
}

alias denv='deactivate'

Note, though, that these commands are only minimally robust. For example, this is all you get if you attempt to genv a virtual environment that does not actually exist:

$ genv demon
-bash: /home/joe/venvs/demon/bin/activate: No such file or directory
$


Shell Game
Check out the series of commands below. The which -a python command shows that, prior to creating and invoking the virtual environment, there is no python in the system PATH.

The which -a python3 command shows that, prior to creating and invoking the virtual environment, there are two python3's in the system PATH.

The which python3 command shows that, prior to creating and invoking the virtual environment, the active python3 in the system PATH is located in the /usr/local/bin/ directory.

The python3 command reveals that the version of Python in the /usr/local/bin/ directory is version 3.7.2. (This installation comes from the instructions presented in our earlier post here.)

The /usr/bin/python3 command shows that the Ubuntu distribution Python is version 3.5.2.

The menv devil command creates a virtual environment named devil.

The genv devil command activates the devil virtual environment - and changes the system prompt to make it clear that the virtual environment is active.

With the devil virtual environment activated, the which python command and the which python3 command both show that the virtual environment interpreter is accessible using either python or python3.

Finally, the python and python3 commands show that the virtual environment links back to the active system python3 in the /usr/local/bin/ directory to run Python 3.7.2 as expected.

See the "Take a Peek at How It Works" and the "Directory Structure Explains Environment Settings" sections below for further explanation of how this all fits together.


$ which -a python

$ which -a python3
/usr/local/bin/python3
/usr/bin/python3

$ which python3
/usr/local/bin/python3

$ python3
Python 3.7.2 (default, Jan 28 2019, 19:55:03)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>


$ /usr/bin/python3
Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

$ menv devil
virtual environment devil created

$ genv devil
(devil) $ 

(devil) $ which python
/home/joe/venvs/devil/bin/python

(devil) $ which python3
/home/joe/venvs/devil/bin/python3

(devil) $ python
Python 3.7.2 (default, Jan 28 2019, 19:55:03)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

(devil) $ python3
Python 3.7.2 (default, Jan 28 2019, 19:55:03)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
(devil) $


Assumptions
The first assumption behind these command definitions is that all virtual environments are kept in a venvs directory off of the user's home directory. (See the '~/venvs/$1/...' elements of both the menv and genv commands.)

The options for organizing virtual environments are endless. Some folks prefer keeping the virtual environment associated with a project under some common directory with the rest of the project files. That's the way that the PyCharm IDE prefers as well. And it is a perfectly sensible and reasonable alternative - just not the way we do it.

The second assumption is that all virtual environments are created based on the most recently installed version of Python - designated as python3 in the menv function definition. (See this post for instructions to install the most recent version of Python from source.)

Once again, the general purpose of and common usage for virtual environments is to create a working environment for any version of Python - not just the most recent. It is our habit, however, to work with only very recent releases of Python, so that at any one time the definition of python3 is known to us and appropriate for current work.

The third assumption is that, however virtual environments are created, each one is independent of the others. That means we are free to create, manage, invoke and deactivate virtual environments using other mechanisms such as Anaconda or PyCharm or directly with the '-m venv ...' command line arguments - and put them wherever we like. Our menv, genv, denv command options are convenience methods, not exclusive or limiting in any way.

Go Crazy
We recommend being profligate with virtual environments. Create one for every tutorial you follow or exercise you duplicate - and then dispose of it when the tutorial or exercise is completed. Get used to making new ones and discarding them once they have served their purpose. The investment in a new virtual environment is small - typically less than 20 megabytes on our Mac or Linux boxes and VMs.

Take a Peek at How It Works
The terminal listing below shows how your operating environment is adjusted when you activate a virtual environment. It is a git diff listing showing the before (in red) and after (in green) environment settings from activating a virtual environment named 'devil'.


(devil) joe@multi:~/virt$ git diff set1
diff --git a/set1 b/set1
index 7699bcb..c5d8c33 100644
--- a/set1
+++ b/set1
@@ -29,10 +29,10 @@ OLDPWD=/home/joe
 OPTERR=1
 OPTIND=1
 OSTYPE=linux-gnu
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+PATH=/home/joe/venvs/devil/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
 PIPESTATUS=([0]="0")
 PPID=18926
-PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
+PS1='(devil) ${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
 PS2='> '
 PS4='+ '
 PWD=/home/joe/virt
@@ -45,7 +45,35 @@ SSH_TTY=/dev/pts/1
 TERM=xterm-256color
 UID=1001
 USER=joe
-_=init
+VIRTUAL_ENV=/home/joe/venvs/devil
+_=devil
+_OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+_OLD_VIRTUAL_PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
+deactivate ()
+{
+    if [ -n "${_OLD_VIRTUAL_PATH:-}" ]; then
+        PATH="${_OLD_VIRTUAL_PATH:-}";
+        export PATH;
+        unset _OLD_VIRTUAL_PATH;
+    fi;
+    if [ -n "${_OLD_VIRTUAL_PYTHONHOME:-}" ]; then
+        PYTHONHOME="${_OLD_VIRTUAL_PYTHONHOME:-}";
+        export PYTHONHOME;
+        unset _OLD_VIRTUAL_PYTHONHOME;
+    fi;
+    if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ]; then
+        hash -r;
+    fi;
+    if [ -n "${_OLD_VIRTUAL_PS1:-}" ]; then
+        PS1="${_OLD_VIRTUAL_PS1:-}";
+        export PS1;
+        unset _OLD_VIRTUAL_PS1;
+    fi;
+    unset VIRTUAL_ENV;
+    if [ ! "$1" = "nondestructive" ]; then
+        unset -f deactivate;
+    fi
+}
 genv ()
 {
     source ~/venvs/$1/bin/activate
Reading through this git diff listing, we see that the mechanism behind virtual environments consists primarily of setting (and restoring) environment variables for the current operating system session.

The segment '/home/joe/venvs/devil/bin' is prepended to the system PATH. The indicator '(devil) ' is prepended to the system prompt. The environment variable VIRTUAL_ENV is set to the virtual environment directory.

The pre-existing settings for the system PATH and the system prompt are preserved in the values '_OLD_VIRTUAL_PATH' and '_OLD_VIRTUAL_PS1', and the 'deactivate()' function is defined to use those values to restore the system settings to the status quo ante when we deactivate the virtual environment.

Virtual Environment Directory Content
The image below shows the directory contents of the new ~/venvs/devil directory that functions as the Python environment for the 'devil' virtual environment.



Directory Structure Explains Environment Settings
Looking at the directory structure, you can see that the activate command for the devil virtual environment is the ~/venvs/devil/bin/activate file, matching the parameterized 'source ~/venvs/$1/bin/activate' command in the genv function definition.

You can also see that by prepending the PATH with the '/home/joe/venvs/devil/bin' directory, the virtual environment makes both the python and python3 executables work equally well to invoke the Python 3.7 interpreter. And the same is true for the pip and pip3 executables.

A Continuing Refrain
The Python community has access to quite a few more advanced mechanisms or utilities for controlling or managing virtual environments.

Recently, the utility pipenv has become available. One of its features is the management of virtual environments.

Before pipenv, we have virtual environment wrapper - a utility devoted exclusively to managing your virtual environments.

And one of the nicer side virtues of Anaconda it organizing and managing virtual environments.

We have, at one time or another, tried several of the approaches to managing virtual environments - and they just did not work for us. In all cases, the additional overhead of learning the features and syntax for the manager seemed more difficult than simply running the 'python -m venv ... command line instruction.

This phenomenon - solving a problem by adding another level of indirection - is known as the fundamental theorem of software engineering. You'll encounter our references to this phenomenon repeatedly on this site. As much as anything, D. J. Wheeler's observation cum caution is a guiding principle behind our production here:

All problems in computer science can be solved by another level of indirection.

But that usually will create another problem.


The simple mechanism that we present in this post is admittedly yet another level of indirection (or abstraction). But we think it is appropriately scaled to the specific task of managing and using Python virtual environments.
Still More Behind the Scenes
This post covers the "simplified" use of virtual environments by explaining how, and to a certain extent why, we use virtual environments the way we do.

This is not the full story on virtual environments. For example, it is possible to set up your virtual environment so that you use the site packages from the reference Python installation but override specific packages by installing different versions in the virtual environment's site packages directory. (See PEP 405.)

But that is not what we do. We keep our reference Python installation site packages pristine and make each of our virtual environments completely independent with respect to installed packages.

Our view is that introducing interdependencies, while conceivably more efficient in some ways, carries a management cost far beyond the value of such efficiencies.
Update pip
Looking over the site-packages directory in the Mac file listing reminds us that upon creating a new virtual environment, you will want to upgrade the pip package first thing. Immediately upon creation of a new virtual environment, activate the environment and run the following command:

$ pip install --upgrade pip

Comments

It will be some time yet before we get a comments section working here. In the meantime feel free to send comments via email. On this site our name is Joe Python. The email address is our first name at joepython.com.

Edited: 2019-03-15 19:28:35(utc) Generated: 2019-06-10 17:29:58(utc)