Jacob Kaplan-Moss

My Python Development Environment, 2020 Edition

For years I’ve noodled around with various setups for a Python development environment. A couple of years ago I wrote about a setup I finally liked; this is an update to that post.

Bad news: this stuff still isn’t stable, and I’ve had to make some changes. Good news: the general concepts still hold, and the new tools a generally a bit better. If you’re curious about the changes and why I made them, there’s a section at the very end about that.

My setup pieces together pyenv, poetry, and pipx. It’s probably a tad more complex than is ideal for most Python users, but for the things I need, it’s perfect.

My Requirements

I do have somewhat specific (maybe unusual?) requirements:

The Setup

1. pyenv

Why? I need to run multiple Python versions, isolated from the system Python. pyenv makes it easy to install, manage, and switch between those multiple Pythons.

On my Mac, I installed pyenv from Homebrew (brew install pyenv). On Linux, I used the Github installation technique documented in the installation instructions, which was easy and went smoothly.

Then, I installed some Python versions:

$ pyenv install 3.8.0
$ pyenv install 3.7.4
$ pyenv install 3.6.9
$ pyenv install 2.7.16
$ pyenv install pypy3.6-7.1.1

And made sure my default Python was set to the latest and greatest:

$ pyenv global 3.8.0

2. pipx

Why? pipx lets me install Python-based CLI stuff (youtube-dl, awscli, doc2dash, etc.) without those projects’ dependencies messing up my global Python.

Installing pipx is easy:

$ python -m pip install pipx

(The documentation suggests installing with --user. That’s probably a good idea for most people. I didn’t because I know I’m going to keep my global namespace clean, but YMMV.)

From there, I can easily install isolated CLI utils:

$ pipx install visidata

pipx protiop: if you have tools with optional dependencies, like visidata, you can inject them into the pipx virtualenv like so:

$ pipx inject visidata pandas

3. Poetry

Why? Poetry handles dependency- and virtual-environment-management in a way that’s very intuitive (to me), and fits perfectly with my desired workflow.

The documentation covers a few different ways to install Poetry. Because I’m using pipx, I use that:

$ pipx install poetry==1.0.0b4

Note that I’m using the 1.0 pre-release (1.0.0b4 at the time of this writing). It’s a tradeoff: the pre-releases have generally better ergonomics and fewer bugs, but are bleeding edge. I’ve already had one bug break some automated releases. So think carefully about using the stable version vs the new shiny.

To start new projects, I just make a directory and type poetry init. This guides me to create a pyproject.toml. Then I’ll run poetry install, which’ll create a managed virtualenv for me and install stuff into it.

To work on existing projects, I clone a repository and then run poetry install.

If you’re new to Poetry, and want to see an example of what a project using it looks like, my pinboard-to-sqlite repo is a good fairly simple example.

Converting to Poetry from a requirements.txt is a bit frustrating: I have to manually consult the requirements file, and run poetry add {package}, possibly with a version specifier, until I get an appropriate pyproject.toml.

If I need to switch Python versions, I run pyenv local <version> in my project directory.

Deployment

When it comes time to deploy, Heroku’s native buildpacks don’t understand pyproject.toml 😠. However, I’m increasingly moving to Docker-based deploys, even to Heroku. Getting Poetry going in a Docker image is a bit finicky, but it’s not bad. I do something like this:

FROM python:3.7

WORKDIR /code

RUN pip install -U pip && \
    pip install poetry

COPY poetry.lock pyproject.toml ./
COPY src/ ./src/

# Install poetry globally - with the current version of
# poetry, there is a known issue where poetry config will
# not create config.toml: https://github.com/sdispater/poetry/issues/1179
# As such, we create it ourselves.
RUN mkdir -p ${HOME}/.config/pypoetry/ && \
    touch ${HOME}/.config/pypoetry/config.toml && \
    poetry config settings.virtualenvs.create false && \

# Set PRODUCTION to anything to invoke installation with --no-dev
ARG PRODUCTION
RUN poetry install ${PRODUCTION:+--no-dev}

Changes since 2018

If you read my 2018 version, you’ll note that while the general architecture remains the same, two out of the three tools have changed: