Virtualenvwrapper for your production server

Virtualenvwrapper is a popular tool for the Django developer who works on several different projects at the same time.
I have not seen much on the web about how this tool can also help simplify your production setup, in particular your Fabric and crontab scripts. So here's a quick writeup...

About virtualenvwrapper

From the author:

(Virtualenvwrapper)... includes wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

In fact, once you've set it up, switching from one Django project to another is as easy as:

workon <project_name>

I will admit that when a colleague first introduced me to this package, I couldn't really see the point; but then at the time I was working on a long contract, dealing with just a single site. In the year since finishing that job, I have worked on maybe 20 Django projects - often several in the same day - and I am now a complete convert to this wonderful tool!

I will not write anything more on setting up & using virtualenvwrapper; there are several excellent tutorials on this subject. Instead, I want to talk about...

Virtualenvwrapper for your production server

So why would you use virtualenvwrapper in production? The main justification - handling multiple Django projects in parallel - often does not apply, as you might have only one site (and hence one virtualenv) on a client's server. Well, I have 3 very good reasons:

1. Easy command-line management

You want to ssh into your production server and run a management command. Let's say that your site is hosted on a Webfaction server; your project will then usually be stored under a path like:

/home/adoro/webapps/hbcc_prod/hartsbourneCC/hartsbourneCC

A quick explanation about how our project is set up in Webfaction (and why our folder path might appear complicated):

We have declared a Django webapp called hbcc_prod.
Within it is a virtualenv called hartsbourneCC.
Within that is our project, also called hartsbourneCC.

Instead of remembering that long path, with virtualenvwrapper we can simply type:

workon hartsbourneCC

... and not have to remember paths. Ok, that's not a killer feature... let's try something else!

2. Fabric

Fabric is right now probably the most popular tool for deploying a Django site into production. I'm going to take a very simple example: using Fabric to collect static files.

If you have set up a virtualenv, you might use something like:

from fabric.context_managers import prefix, cd
from fabric.api import env, run

def collect_static_files(*args):

    with cd('/home/adoro/webapps/hbcc_prod/hartsbourneCC/hartsbourneCC'):
        with prefix('source ../bin/activate')):
            run('./manage.py collectstatic --noinput -v0')

So our function is:

Changing directories to the project directory.
Activating the virtualenv.
Running the command.

Now, if we are using virtualenvwrapper, we could write:

from fabric.context_managers import prefix
from fabric.api import env, run

def collect_static_files(*args):
    with prefix('workon hartsbourneCC'):
        run('./manage.py collectstatic --noinput -v0')

Which is shorter & easier to read. Also, we are now following good DRY (Don't Repeat Yourself) principles, by having the location of the project, and of its virtualenv, defined in only one place - in the definition of the virtualenvwrapper on the server. We no longer store (duplicate) this information in our Fabric script.

Note: the above example hardcodes the name of the virtualenv in the function; a more (re)usable piece of code would store this information in the env environment dictionary.

3. Crontab

If we have a regular job to run for our project, we might write the following crontab command:

0 6 * * * cd /home/adoro/webapps/hbcc_prod/hartsbourneCC/hartsbourneCC && source ../bin/activate && ./manage.py cleanup

If we are using virtualenvwrapper, we can instead write:

SHELL=/bin/bash

0 0 * * * source $HOME/.bashrc && workon hartsbourneCC && ./manage.py cleanup

Once again, we have something that is more DRY-ish, as well as being (IMO) easier to read. A couple of explanations are required:

SHELL is required to switch to a shell that will work with a virtualenvwrapper.
source $HOME/.bashrc runs the virtualenvwrapper initialisation script (to be totally accurate, I should put this into its own script, and call that from either .bashrc when initialising a session, or when running a crontab line. But I'm lazy...).

But it don't work...

There is one downside to the above: at first, I would get error messages like:

Traceback (most recent call last):
File "/usr/lib/python2.7/logging/handlers.py", line 78, in emit
    self.doRollover()
File "/usr/lib/python2.7/logging/handlers.py", line 140, in doRollover
    os.remove(dfn)
OSError: [Errno 2] No such file or directory: '/home/adoro/.virtualenvs/hook.log.1'
Logged from file hook_loader.py, line 134

It turns out that virtualenwrapper does not like at all having 2 environments initialised at the exact same moment, which was happening when you have 2 crontabs running at the same minute. So if this is likely to happen, add a random `sleep` in front of one of the jobs, as a quick-n-dirty way of avoiding clashes:

SHELL=/bin/bash
    
*/10 * * * * sleep 5; source $HOME/.bashrc && workon hartsbourneCC && ./manage.py run_polling    
0 0 * * * source $HOME/.bashrc && workon hartsbourneCC && ./manage.py cleanup