Sitemap
ITNEXT

ITNEXT is a platform for IT developers & software engineers to share knowledge, connect, collaborate, learn and experience next-gen technologies.

Python Packaging with Setuptools

12 min readMar 5, 2023

--

This article will explain the best practices for creating a Python package.

It will cover:

  • Why we want to package our code
  • Evolution of Python packaging
  • PyScaffold
  • Useful configurations in setup.cfg
  • Build and publish
  • Alternative tools
Press enter or click to view image in full size
Photo by Jiawei Zhao on Unsplash

Why do we want to package our code?

One of the major strengths of Python is its vast package ecosystem. Everything is available at the fingertips of the developer with a simple pip install and import. Packages are a way to make code reusable and easy to share. Writing libraries instead of a collection scripts forces you to think about design and clear interfaces.

Too often, some code is stashed away in some repo. You need to clone it, install the requirements, and in general, it is not convenient to use. We should strive to do better.

Evolution of Python Packaging

requirements.txt

Most novices will start their Python journey with the requirements.txt and some loosely coupled scripts.

# Project structure
├── requirements.txt
├── script_1.py
└── script_2.py
# requirements.txt
numpy
pandas==1.5.3
# Installation
pip install -r requirements.txt

Its simplicity is the main appeal, and it might be easy just to share those scripts. And it’s fine as long as these scripts work on their own. But it will be inconvenient to integrate them into another project. What then usually happens is that people will copy paste your script into their project, duplicate your code and make their own modifications.

If you want to make your code more reusable, you want to write a library, package and publish it. This way, everyone can just pip install and import your library.

Python packaging is notorious for not being well documented. There is no definitive way to do things. Even the official documentation (which has no central location) might contain outdated information. Over time, many different workflows and tools have evolved. I do have to say that the documentation has improved a lot in recent years. So how would you create a package?

Distutils vs setuptools

One might encounter these two names often in conjunction and it confused me a lot at the beginning. But it’s simply a historic thing, Python started out with distutils as a build-in library, It was a collection of tools for packaging, etc.. To improve on it, setuptools was created as a pip-installable package. Today, distutils is officially deprecated starting with Python version 3.10, as decided in PEP 632.

src-layout vs flat layout

Before starting your project, you have to make a decision on your project structure. When I started with Python, figuring out the project structure was a major challenge. There are multiple ways to do it. It helped when I read this article, which identified two major matterns. There is the src-layout and the flat-layout.

src vs flat-layout

The difference between the two is the location of the library code. In the src-layout, all the library code is stored in a src folder, separate from all the other boilerplate files, like tests and documentation. This clear separation is useful for automation and package discovery. E.g. you won’t accidentally ship the boilerplate code. The downside of this is that while developing, you need to do an editable install, because of the additional misdirection of the src folder, that would end up in your import path. (E.g. only import src.mypkg would work)

In the flat structure, you are able to directly execute the code (e.g. with python -m package.script), since everything is located in the top-level folder. But you need to be more careful with package discovery.

Both solutions are valid, for our following examples, we will exclusively use the src-layout.

setup.py

With setuptools, the minimal setup you can have is a single setup.py . We can move the dependencies in requirements.txt to the install_requires argument. Moreover, we need to pass some more parameters to account for the src-style layout. With packages and packages_dir, we remove src from the import path and specify that all packages are under src. For more details, you can read my article on package discovery.

# Project structure
├── setup.py
└── src
└── tutorial
├── __init__.py
├── script_1.py
└── script_2.py
# setup.py
import setuptools
from setuptools import setup

setup(
name="python_example",
install_requires=[
"numpy",
"pandas==1.5.3",
],
package_dir={"": "src"},
packages = setuptools.find_packages(
where='src',
),
)
# Editable install
pip install -e .

This setup gets the job done. We can build and publish this package already. But … it is not the best practice. And we will find out why in the next sections.

setup.py + setup.cfg

One problem with setup.py is that it is executable. To read the metadata, setuptools need to execute this script. For automation tools it is too complicated and slow. For example, a dependency resolver would be much faster if it could read the dependencies from a machine readable configuration file than executing python code. This is why setup.cfg was introduced.

# Project structure
├── setup.cfg
├── setup.py
└── src
└── tutorial
├── __init__.py
├── script_1.py
└── script_2.py
# setup.py
from setuptools import setup

setup()

All arguments available in setup.py can be defined in setup.cfg. setup.py will automatically read the setup.cfg, when it is available.

# setup.cfg
[metadata]
name = python_example

[options]
packages = find_namespace:
include_package_data = True
package_dir =
=src

[options.packages.find]
where = src

The declarative setup.cfg is easier to parse automatically and it does not require to run Python code. It is considered to be bad practice to execute code in setup.py . My recommendation is, that you start out with setup.cfg and if you find that you reached its limitations, you can fallback to setup.py . pybind11 is an example where you have to rely on setup.py to build your C++ code.

The build dependency issue

There is one more problem, which again goes back to setup.py. It is the build dependency issue:

  • You can’t run setup.py without knowing its dependencies
  • You can’t know the dependencies without running setup.py

This is why PEP 518 was introduced, in their words:

[…] when a project chooses to use setuptools, the use of an executable file like setup.py becomes an issue. You can’t execute a setup.py file without knowing its dependencies, but currently there is no standard way to know what those dependencies are in an automated fashion without executing the setup.py file where that information is stored. It’s a catch-22 of a file not being runnable without knowing its own contents which can’t be known programmatically unless you run the file.

Source: https://peps.python.org/pep-0518/#rationale

The solution is a new file named pyproject.toml.

setup.py + setup.cfg + pyproject.toml

# Project structure
├── pyproject.toml
├── setup.cfg
├── setup.py
└── src
└── tutorial
├── __init__.py
├── script_1.py
└── script_2.py
[build-system]
requires = ["setuptools>=46.1.0", "setuptools_scm[toml]>=5"]
build-backend = "setuptools.build_meta"

In pyproject.toml we can specify the version of the build system. During the installation of the package, it will create an isolated environment and install the correct setuptools version.

pip install creates a isolated venv according to pyproject.toml

setuptools is only one out of many build backends. PEP 518 standardized the interface to the build backend, which enables the community to build alternative backends.

Press enter or click to view image in full size
frontend and backend, source: https://peps.python.org/pep-0517/

Use a Project Generator: PyScaffold

Now we know that we need to create a bunch of files when starting a project. Luckily, we do not have to remember every detail. We can use a project generator like PyScaffold. It sets up the perfect project structure for you and includes the all best practices. The maintainers will stay up to date with the newest packaging developments and update the project templates accordingly.

Get Insight in Plain Sight’s stories in your inbox

Join Medium for free to get updates from this writer.

You can simply install it with:

pip install pyscaffold
putup python_example
Press enter or click to view image in full size
pyscaffold generated files

Useful Configurations in setup.cfg

Now that we have a proper project structure, we will discuss a few configurations that are regularly used.

Package Name

In setup.cfg you can give your package a name. This will determine the name that will appear under PyPI and you would install it with pip install <name> .

setup.cfg

We have to make a distinction here between the PyPI package and the Python package as in a collection of modules (Python files). The name you use to import the code is not related to the setup.cfg name, but is determined by your project structure under the src folder.

Project structure with multiple packages
Import names

A prominent example is opencv. You would install it with pip install opencv-python but import cv2.

Version

setup.cfg

The versioning follows the Semantic Versioning. In short:

  • MAJOR: incompatible API changes
  • MINOR: add functionality, backwards compatible
  • PATCH: backwards compatible bug fixes

But most projects don’t follow these rules religiously and you should think about what makes sense from the perspective of the user.

A best practice is that you should upgrade the version on every change and should not overwrite already published packages. You might introduce breaking changes and other people’s build pipelines will break. But they will not be able to revert to a previous working version since you have overwritten it. This will block them until they adapt their codebase to your new interface. It would be easier for a 3rd party to revert to a previous version to fix their builds. Then, they can adapt their code to your changes with no pressure.

Version - PyScaffold Pitfall

When I use PyScaffold, there is one minor annoyance I always have to fix. The default project template will try to infer the version from your version control system. This assumes that you have one package per repo. If you have a monorepo with multiple packages you will encounter this error:

Press enter or click to view image in full size
Running: putup <package>
Press enter or click to view image in full size
Running: pip install -e .

To fix it, follow these steps:

putup <package_name> --force
Press enter or click to view image in full size
Remove the red parts
Press enter or click to view image in full size
Remove the red parts

install_requires

setup.cfg

You can add your dependencies under install_requires. You can use <=, ~=, >=, == to specify version ranges.

A best practice is to add all direct dependencies i.e. the packages that you import in your code. For example, you should add numpy even though it comes with opencv, because if you decide to remove opencv from your project, your code will break because there is no numpy anymore.

Another best practice is to not be too strict on the dependency versions. You might feel inclined to use the newest version of a library, but with many dependencies, constraints might be hard to fulfill by users. Given enough dependencies the dependency resolver will not be able to find a constellation where all dependencies are compatible.

extras_require

Too many dependencies and you will quickly find yourself in dependency hell. Sometimes dependencies are optional to the core functionality. Like testing libraries that will not be required at runtime. With extras_requires you can define optional dependencies under a name.

setup.cfg
Install extra dependencies
Syntax for editable installs

Horovod is a distributed learning framework that supports many deep learning frameworks. Installing all frameworks is quite impossible, so you are given the option to choose one with extras_require.

If you only use a single function from a package, you should consider implementing it yourself to reduce the number of dependencies.

Namespace Packages

Namespace packages are packages that do not contain code by themselves. They are a way to group your packages under one namespace. This is useful if you have multiple packages that you want to publish under the same name. It could be the name of your umbrella project or your organization. This way, you can also work on different packages in different repos but still publish under the same name. For example, Azure has a lot of packages that can be installed separately, but you can import them like this:

azure namespace package

PyScaffold already creates this behavior by default with the find_namespace directive. You just need to place your packages under a namespace folder. With find_namespace, __init__.py is not needed anymore, since empty folders will also be parsed as packages.

setup.cfg
Project structure
Press enter or click to view image in full size
Import path

entry_points

One more useful setting is entry_points. It will install an executable into your commandline. This is useful for creating CLIs. With PyScaffold you can simply uncomment the skeleton example to test this functionality. In combination with click, you can create your own CLIs in no time.

setup.cfg
Script available in commandline

Build and Publish

When you are finished with your package, you should consider publishing it to the PyPI, either to the public index or consider a company hosted index for your inhouse solutions.

Use the build package for building your package:

Build command

The will create two artifacts: whl and sdist:

Build artifacts

Wheel (whl) is the prefered format for packaging code. You can inspect the content with unzip -l <package>.whl. The wheel contains your files in a format that can be easily installed by unzipping the contents into the site-packages. More on wheels in PEP 427.

The source distribution (sdist) is more like a copy of your project. It can also be installed, but it serves more of an archiving purpose.

You might run into some tutorials which still run setup.py manually. But this behavior is deprecated.

Deprecated way of build and install

The last step is to use twine to publish to a PyPI:

pip install twine
twine upload dist/*

Alternative Tools

What we covered previously was the basic packaging stack:

  • frontend: pip
  • backend: setuptools
  • build: build
  • publish: twine

Due to the bumpy history of Python packaging many other tools have been created to offer a better experience. There are many, so I will only cover a few notable ones.

Cookiecutter

If you are not happy with the PyScaffold project template, you can use Cookiecutter. Cookiecutter is a project where people can define their own reusable project templates. The idea is to create a cookiecutter and once you have it, you can use it to cut out new cookies (projects). You can either use other people’s templates or as an advanced user create your personal template.

Poetry

Poetry is super popular right now and it is inspired by Rust’s Cargo package manager. It replaces the whole stack of pip, setuptools and etc. to offer a single commandline tool that can do everything.

PDM

Similar to poetry, pdm is inspired by another package manager, that is npm. One notable feature is that it supports PEP 582, which removes the need of virtualenvs. This is an exciting new direction for Python. Maybe the concept of virtualenvs is not required at all.

Update: PEP 582 was rejected. The rational being that there are already too many ways to affect the module search paths and having one more adds just more complexity compared to the small benefit for new users. Read in more detail here: https://discuss.python.org/t/pep-582-python-local-packages-directory/963/429

Final words

There are many ways to do packaging. My personal recommendation is to use PyScaffold with the regular packaging stack — pip, setuptools, build, twine. If you know what you are doing consider cookiecutter. Or if you want to have a more streamlined experience and are willing to be a little experimental, go with Poetry and PDM. I say experimental but they are well established and are received well in the community. A dream would be that one day we would have a definitive way to do things, so we do not have to carefully compare different tools anymore.

--

--

Insight in Plain Sight
Insight in Plain Sight

Written by Insight in Plain Sight

Computer Vision Engineer for Self-Driving Cars

ITNEXT
ITNEXT

Published in ITNEXT

ITNEXT is a platform for IT developers & software engineers to share knowledge, connect, collaborate, learn and experience next-gen technologies.