My Python Development Environment

Dan Poirier

This is how to set up the One True Development Environment for Python.

Haha, just kidding, there is no such thing. Here's one way to do it that works for me, and an attempt to explain the benefits of doing it this way.

Different developers working on the same project can choose different ways of doing these things, so we don't want to set up the project so it can only be worked on this way. The approach I use has worked well in a variety of settings.

This post is inspired by Jacob Kaplan-Moss's post My Python Development Environment, 2020 Edition.

Common Requirements

  • Projects use different Python versions.
  • Python 2 is dead. This assumes a recent Python 3 version. (Say, 3.6 and later?)
  • As far as possible, works the same on Mac and on different Linux distributions (though I only do Python development on Ubuntu).
  • No dependency on any particular IDE or editor.

My experience is primarily on Ubuntu. Please let me know if anything here doesn't work on other Linuxes, or on Mac.

Installing Python

Assumptions:

  • We need different versions of Python for different projects.
  • We don't want to be bothered with the changes our operating system might have made to how Python is installed, or to risk breaking our operating system by messing with the system Python.

Therefore, we will not use the operating system installed copy of Python, even if it happens to be the version we want.

Compile It Myself?

For a while, I was using an Ansible role to download source, compile, and install each version of Python that I needed (on Linux), and using stow to put them all under /usr/local. Then if I wanted Python 3.8, I could just run python3.8 and it would run the right one.

Except that if I had one project that required Python 3.8.1, and another that needed Python 3.8.2, I was out of luck. When I build and install Python vX.Y.Z, the installer creates executables pythonX and pythonX.Y, but not pythonX.Y.Z. If I had multiple Python 3.8's installed, I didn't know which 3.8.x version I'd get by running python3.8. (Actually, stow would stop me from installing multiple Python 3.8's in the first place, since there would be duplicate files.)

To create the right virtualenv, I would have to type something like:

$ /usr/local/stow/python3.8.6/bin/python -m venv ...

Pythonz?

So, I switched to using pythonz. It automates the install — and uninstall — of any version x.y.z of python I want (just run pythonz install 3.7.7).

Pythonz doesn't put all those Python versions on the path. Instead, I can find a particular executable by running:

$ pythonz locate 3.7.7
/home/dpoirier/.pythonz/pythons/CPython-3.7.7/bin/python3

So I can run a specific version using something like this:

$ $(pythonz locate 3.7.7)

where pythonz locate 3.7.7 prints the complete path to the Python 3.7.7 executable, and the $( ) captures the output and executes it.

I could use that when creating a virtualenv for a project, and not have to deal with pythonz afterward. Still, it always seemed inelegant.

Pyenv

Now I'm trying pyenv. Like pythonz, I can easily install multiple Python versions (pyenv install 3.7.7). But it takes a completely different approach to selecting which version to run. You put pyenv's shims directory first on your path, so that when you run any python command, you are running pyenv's shim executable for that command. That shim executable figures out the right actual python executable to run, and invokes it for you.

You can tell the shim which version you want at any given time in several ways.

  • You can set PYENV_VERSION in your shell.
  • You can put the version in a .python-version file in your current directory.
  • You can have a .python-version file in any parent directory and it'll use the first one it finds, working its way up.
  • Finally, you can configure a default version to use as a fallback.

This looks like it'll fit into my existing workflow pretty well. I already use direnv, so I can just set PYENV_VERSION in my .envrc file and get the version I want without changing anything in the project's files in source control. Or, I could create a .python-version file in the project, which shouldn't affect any user not using pyenv, but whose meaning should be pretty obvious.

Creating a virtualenv

Python has had built-in support for creating virtual environments since version 3.3, so we'll use that to create our virtualenv.

Where should we put our virtualenv, though? I used virtualenvwrapper for a long time, which puts all of your virtual environments in one directory ($HOME/.virtualenvs by default).

But virtual environments tended to accumulate in my virtualenvs directory from projects I hadn't touched in years, and it bugged me.

More recently, if I'm working on a project in .../projectname (my top-level directory where I cloned the project from git), then I create the virtualenv at .../projectname.venv. Anytime I'm cleaning up an old project, I'll see the virtualenv next to it and remember to clean that up too.

(I don't like to put the virtualenv actually inside my Python project, because then my IDE feels compelled to index everything in it, do searches through all of it, etc. Otherwise, I might just put my virtualenv at .../projectname/venv.)

Putting my virtualenv where virtualenvwrapper doesn't expect it does mean I can't use virtualenvwrapper's workon command to switch virtualenvs, but that's okay. I had already stopped using the rest of virtualenvwrapper when python -m venv started working.

I use direnv already, so I just have a little script that creates a virtualenv at ../projectname.venv and also adds a line like . ../project.venv/bin/activate to my .envrc file. Then anytime I change to that directory, my virtual environment is already activated.

(I'm aware of pyenv-virtualenv and pyenv-virtualenvwrapper, but these look like they also hide away the virtual environment directories somewhere I'll forget about them, so for now, I'm not using them.)

Installing Packages Into the virtualenv

I've played with pip-tools for installing Python packages into virtual environments, but somehow, most of my projects still just use [pip]{.title-ref} to install requirements:

$ pip install -r requirements.txt

What About "tox"?

tox needs to be able to find each version of Python mentioned in tox.ini, and it doesn't know to ask pyenv for them. But you can expose as many python versions as you want using pyenv local. So I can, for example, set .python-version to:

3.7.5
3.8.6

and have tox work for test environments py37 and py38.

Where Things Stand

I think this is one of the best environments I've used over the years, but as you've probably gathered, I'm always looking for ways to improve it. I continue to watch pipenv and pip-tools, I keep an eye on Python releases for improvements to what Python provides out of the box, and I look for ways to better integrate with my favorite IDE, Pycharm.