Building Python Platform Wheels for Packages with Binary Extensions

Published on Last updated

Note: This guide is for Python packages with binary extensions (C/C++/Fortran). If your package doesn’t have these, this guide is not for you. You can simply use python setup.py sdist bdist_wheel and upload both the .tar.gz and .whl files to PyPI.

Update 8/20: These days Travis also supports a Windows build environment, which means it is no longer needed to do those builds separately on AppVeyor (as described below). To see an example of using Travis for all platforms, check out CleverCSV.

Intro

Python distribution packages come in two flavors:

  • Source distributions: ending in .tar.gz
  • Wheels: ending in .whl

If your package contains C code, your users will need to compile that C code locally when they install the package. This is burdensome, especially on Windows where a C compiler is not typically installed and where Visual Studio Build Tools “Requires 2.3 GB to 60 GB of available hard disk space, depending on installed features.”

This is why wheels exist, they allow you to distribute your package with precompiled C code. The problem is that it’s difficult to compile your package on all the different platforms and Python versions in existence, especially when you don’t have computers to spare for each different platform. One solution is to use Continuous Integration providers to build the wheels for you. This is a brief guide to doing just that.

Just a heads up: Python packaging is complicated, this is a setup that worked for me after a lot of trial and error and hopefully works for you too. But keep in mind that your mileage may vary.

setup.py

I’m going to assume that you have a Python package that can be installed with setup.py only, and that all requirements are defined in the setup.py file (it’s where they’re supposed to be, apparently). I’ll also assume that the extra_requires dictionary has a tests key that lists the dependencies needed for running the tests (if any). If we agree on this it’ll make it a lot easier to handle dependencies on the CI servers. I used to define my project dependencies with Poetry using a pyproject.toml file, but this would not work reliably on Travis because I use cleo in my package and there’s this bug.

I’m also going to assume you have a publically-available open-source repo. If not, some of these steps may be different (and I don’t know which). Finally, I’ll assume you have your tests in a tests directory. None of these assumptions are strictly necessary to get things to work, and you may well have a different package structure. This just means you’ll have to change some of the commands below to fit your needs. The commands for uploading a package in the setup below point to the live PyPI servers. During testing you may want to change this to the TestPyPI servers. This can be done by adding the following flag to the twine upload commands: --repository-url https://test.pypi.org/legacy/. Note that this implies that you create an account and API tokens on the TestPyPI servers.

cibuildwheel

We’ll be using an excellent package called cibuildwheel to make our lives substantially easier. Note that a lot of what I’m writing here is combining the cibuildwheel documentation and the deployment example here, but with some extras that I needed to get things working for me. It took me 99 commits to get both Travis and AppVeyor working (with some additional debugging of the package in between), so hopefully this guide will save you some time!

cibuildwheel has support for various CI/CD providers, but here we will only treat Travis and AppVeyor for respectively Linux/Mac and Windows. I briefly tried Azure Pipelines, which is supposed to be able to do all platforms, but found it hard to use.

As there’s no real point in explaining all the trial and error I went through, I’ll just jump to the end result and explain why the configurations are the way they are. If you run into any problems adapting this to your own project, please let me know.

Travis

I’ll go through the .travis.yml file block by block to explain how it works. Let’s dive in:

matrix:
  include:
    - language: python
      dist: xenial
      sudo: required
      python: "3.7"
      services:
      - docker

    - os: osx

This defines the two platforms that we want to run cibuildwheel on. For Linux, we require the Docker service to be enabled because cibuildwheel creates the Linux wheels inside a docker container (using manylinux). We choose to run cibuildwheel on Python 3.7, which requires us to specify the xenial ubuntu distribution on Travis. For OSX we need no other specifications.

env:
  global:
    - TWINE_USERNAME=__token__
    - secure: "<your encrypted api token>"
    - CIBW_SKIP="cp27-* cp33-* cp34-* cp35-*"
    - CIBW_TEST_COMMAND="python -m unittest discover -f -s {project}/tests"
    - CC="gcc"
    - PYTHON=python3
    - PIP=pip3

The twine username is set to __token__ because we will use an API token to upload to PyPI. API tokens are a beta feature of PyPI, but they’re recommended and I haven’t had any issues with them. See this PyPI page for more info. This way we can avoid having to add our (encrypted) PyPI password to Travis. This token is added using the secure keyword and is created as follows:

  1. Install the travis command line client (gem install travis)
  2. Authenticate the travis command line client: travis login --org (the --org flag is needed because we’re using the free version of Travis).
  3. Go to PyPI to create an API token (under account settings). This token will be specific to Travis, so you can give it a name like “Travis”. I also recommend using the token for only one project by setting the scope. Copy the API token.
  4. Create the encrypted token using: travis encrypt TWINE_PASSWORD="<your API token>" --org. Make sure your API token contains the pypi- prefix.
  5. Copy the generated code to the .travis.yml file, as above.

Moving on, the CIBW_SKIP environment variable tells the cibuildwheel package to skip certain Python versions. Because my package uses f-strings, it is Python 3.6+, so I’m disabling all the older Python versions. Sidenote: The cibuildwheel package also has variables to set only the versions that you want to build. See the README for more info.

The CIBW_TEST_COMMAND is added to ensure that all wheels that cibuildwheel creates are working as expected. The {project} part of the command will be replaced by the correct path by cibuildwheel. The CC="gcc" setting is necessary to avoid problems where Travis doesn’t find a C compiler. Finally, the PYTHON and PIP variables are used for safety, to ensure we really have the Python 3 version in Travis.

install:
  - if [ "${TRAVIS_OS_NAME:-}" == "osx" ]; then
      brew update;
      brew upgrade python;
    fi
  - $PIP install -e .[tests]
  - $PYTHON -m unittest discover -v -f -s ./tests
  - find . -type f -iname '*.so' -print -delete

These install commands are run before we use cibuildwheel to make sure the package actually works correctly. If any unit tests fail, we want to know that before we start building all the wheels. For MacOS we upgrade Python using HomeBrew before starting (this was copied from the websockets repository). Next, we install our package in editable mode with the tests extra requirements. As I mentioned above, I’m assuming your setup.py file has an extra_requires dictionary with the test dependencies set by the tests key. If you don’t need any additional testing dependencies, simply use extra_requires = {'tests': []} or amend the line above. The final line of the file removes the shared objects that were created when installing the package. I found that this was needed to avoid clashes when cibuildwheel creates the wheel with the same Python version as is running in Travis.

The next section of the .travis.yml file actually runs cibuildwheel:

script:
  - $PIP install cibuildwheel==0.12.0
  - cibuildwheel --output-dir wheelhouse
  - ls wheelhouse

This section is relatively straightforward: it installs cibuildwheel, runs it, and lists the output wheels for inspection.

deploy:
  skip_cleanup: true
  provider: script
  script: $PIP install twine && $PYTHON -m twine upload --verbose --skip-existing wheelhouse/*
  on:
    branch: master
    tag: true

Finally, this section uploads the wheels to PyPI. Note that this last step is only run when Travis is triggered by a git tag (tag: true) and only on the master branch. The --skip-existing flag ensures that twine doesn’t give errors if this is run without a version update (for instance when you have tags that don’t bump the version). That’s it for Travis!

AppVeyor

AppVeyor allows us to build the wheels for Windows and it’s free for open source projects as well (yay!). If you’re comfortable giving AppVeyor access to your GitHub repositories, go ahead and do so. I was a bit skeptical because I hadn’t used it before, so decided to go the slightly harder route and create an account without GitHub accesss. This means that the repository is a “generic git repository” in AppVeyor, which changes some of the settings below. The main benefit to using GitHub is that you won’t have to set up webhooks yourself or point AppVeyor to the location of the appveyor.yml configuration file.

First, create a new project on AppVeyor and create it as a “Generic Git” project. The clone URL can still be a GitHub URL even if you don’t use the official GitHub route, so that’s what I used here. In the project settings, change the following items:

  • Default branch: master (for me at least)
  • Branches to build: master (I used only the master branch, since I use Travis for the regular testing, this is only for releases).
  • Since we’re using a generic git repository, we need to set the location of the appveyor.yml config file. This is done in the “Custom configuration .yml file name” field and should be set to the URL of your appveyor.yml config. For me, this was the raw GitHub URL of the file.
  • Check skip branches without appveyor.yml
  • Check Rolling builds if you want to cancel a build when a new commit arrives. This is generally the right thing to do.
  • Don’t forget to click “Save” at the bottom.

To automate the build on AppVeyor, we’ll set up a webhook on GitHub. In the settings page on AppVeyor, copy the “Webhook URL”. Then, go to GitHub and go to the repository settings and click on Webhooks on the left. Add a webhook as follows:

  • Payload URL: the URL you just copied from AppVeyor
  • Content type: should be on application/x-www-form-urlencoded
  • Under “Which events would you like to trigger this webhook?” I selected:
    • Branch or tag creation
    • Branch or tag deletion
    • Pull requests
    • Pushes
    • Repositories

This should trigger the AppVeyor builds when you push to your GitHub repo, as with Travis.

As above with Travis, I’ll go through the appveyor.yml file block by block to explain what it’s doing and why. Again, a lot of this is based on the cibuildwheel documentation.

platform:
  - x64

Set the platform that we want to use, we’ll need it below.

environment:
  global:
    CIBW_SKIP: cp27-* cp33-* cp34-* cp35-*
    CIBW_TEST_COMMAND: "python -m unittest discover -f -s {project}/tests"
    TWINE_USERNAME: __token__
    TWINE_PASSWORD:
      secure: "<secure api token>"
  matrix:
    - PYTHON_VERSION: 3.7

This sets the environment variables for our build. You’ll notice that it’s very similar to the environment we used for Travis. The version of Python set in the matrix block is the version we want to use to run cibuildwheel. The TWINE_PASSWORD variable is set to a secure token similarly to what we did for Travis:

  1. Go to PyPI and create an API token for AppVeyor.
  2. Copy the API token (including the pypi- prefix) and go to https://ci.appveyor.com/tools/encrypt.
  3. Encrypt the API token and copy the encrypted value that you’ll get. (Be slightly surprised that AppVeyor seems to submit your API token in plaintext and wonder about the security implications of that.)

  4. Copy the encrypted token into the appveyor.yml file, as indicated above.

    init:
- set PY_VER=%PYTHON_VERSION:.=%
- set PYTHON=C:\PYTHON%PY_VER%
- if %PLATFORM%==x64 (set PYTHON=%PYTHON%-x64)
- set PATH=%PYTHON%;%PYTHON%\Scripts;%PATH%
- python --version

This initialization block sets the desired version of Python (3.7 in our case) as the first Python executable in the PATH variable, which makes it the default Python version for the python command. This initialization block is also the reason we need to specify the platform setting above.

before_build:
  - pip install -e .[tests]
  - python -m unittest discover -v -f -s ./tests

Installs the package and runs the unit tests. We want to do this to ensure we don’t go building the wheels when the unit tests fail on Windows.

build_script:
  - pip install cibuildwheel==0.12.0
  - cibuildwhweel --output-dir wheelhouse

As for Travis, this bit actually runs cibuildwheel to build the wheels for the different versions of Python and different platforms.

after_build:
  - ps: $env:GIT_TAG=$(git tag --points-at)
  - >
    IF NOT "%GIT_TAG%" == ""
    (
    pip install twine
    &&
    python -m twine upload --verbose --skip-existing wheelhouse/*.whl
    )

This final block does the automatic deployment of the wheels to PyPI. While AppVeyor has an environment variable that marks if the Git commit was tagged (APPVEYOR_REPO_TAG), this didn’t work for me during testing. It’s possible that was because I’m using a “Generic Git” project on AppVeyor. Nonetheless, this simply pulls the git tag from the repo itself and tests if it’s not empty.

Conclusion

While it took me some time to get everything up and running, all of this would be much harder without the cibuildwheel package. Hopefully this guide makes it easier for you to get started with cibuildwheel and have your Python packages compiled and distributed with ease!