Managing Your Python Installation with Git

Python has a great deployment and installation system, in the form of distutils and setuptools.

Unfortunately, Python's uninstallation system is not so much poor, as much as non-existent.

This means that your default Python installation tends to get cluttered up with lots of detritus, and you will have to rely on shell tools such as find and grep to locate problems (old, conflicting, broken, missing or simply unwanted packages) and remove them.

More importantly, there is really no easy way of figuring out what packages are installed in your Python path, how they were built and how there go there.

Well, it struck me one day that I could just place my entire Python installation under Git management.

Then every change I make to it will be documented, logged and tracked.

Folks, let me tell you this: once you experience Pythoneering this way, there really is no going back!

Furthermore, by the magic of Git branching, I can easily step into an experimental branch to test the viability of a package, and then merge the changes to the master branch if I am satisfied.

Even better, I can maintain multiple branches simultaneously, each having a different suite of packages (or even different versions of the same package), and I can step between them with a simple "git checkout <BRANCH>".

And, like most developers, I always have a need for a clean Python installation to test my code, and so I a maintain a branch called vanilla that is the immediate descendent of the clean Python installation.

I am aware of virtualenv that replicates some of this behavior, specifically the ability to select among different variants of Python installations, as defined by the available packages. But the Git-based approach differs in one important way: it changes the Python environment for the whole system, not just the current shell. This is neither a plus nor minus, depending on your needs. You may prefer this if you want to install an environment for external processes (e.g., run through Condor or MPI). But you may also want to not in any way disturb or interfere with other Python clients.

Of course, there is nothing stopping you using a hybrid approach: e.g., managing the primary system Python installation with Git, and using virtualenv for development or testing. Or, even better, use virtualenv to create multiple Python environments, and then use Git to manage each them.

The Git-based or hybrid Git + virtualenv approach to managing your Python installation(s) offer a number of clear advantages over virtualenv alone:

• Versioning history -- something breaks? Revert or reset! Want to remove package entirely? Use git show or git diff to identify exactly what it added/changed so as to be able to remove it cleanly with no traces!

• Documentation -- I am finding this sooooo much more useful than I anticipated. For example, I had previously built and installed SciPy on my Leopard box. I remember having a little bit of difficulty building it, and usually I write a small text note to myself (or post on this site) about my experiences, so I do not have to hunt all over the place for solutions the next time I face the same situation. But this time I did not. But I did include a detailed Git commit message. So when I had to build SciPy again on Leopard, instead of trawling through the web for instructions, I simply:

  $ git log
  

And Git told me:

commit dc5abbab5cf147a28c73474e0908f08c085be0cc
Author: Jeet Sukumaran <jeetsukumaran@gmail.com>
Date:   Wed Sep 24 05:46:36 2008 -0500

    Added Sciy 0.6.

    Built with: "python setup.py build_src build_clib --fcompiler=gnu95 build_ext --fcompiler=gnu95
    Installed with: "python setup.py install".

commit dde31dca78f3b7576634d67c3c2c33fbd63fb55f
Author: Jeet Sukumaran <jeetsukumaran@gmail.com>
Date:   Wed Sep 24 05:36:56 2008 -0500

    Added NumPy 1.1.1.

commit d280ca0301c84f90b9932a2c8c5fbcb58dfb0720
Author: Jeet Sukumaran <jeetsukumaran@gmail.com>
Date:   Wed Sep 24 05:15:03 2008 -0500

    Added fftw 3.1.2 ("./configure=/opt/python-2.5.2; make; make install").

You cannot deny that this is pretty cool!

So, what are the downsides? Well, all the extra book-keeping, for one thing. After every change to your Python installation, you have to remember to add the files and commit them, ideally with a detailed and useful commit message. Well, you do not have to, but then your Git-managed Python installation will degenerate to a normal/basic Python installation. It would be great if distutils or setuptools could have a post-installation hook where they could automatically take care of the business of adding the files and committing, perhaps with a commit message based on information in the setup.py file combined with the configure and build settings of the package.