Python Packaging — pyproject.toml

In this article, we are going to learn how to package our Python library/program for distribution(for others to use our package, like the ones we are installing and using via pip)

Quick overview

Python modules

A Python file, provided it only relies on the standard library, can be redistributed and reused. You will also need to ensure it’s written for the right version of Python, and only relies on the standard library.

This is great for sharing simple scripts and snippets between people who both have compatible Python versions (such as via email, StackOverflow, or GitHub gists).

Python source distributions

If your code consists of multiple Python files, it’s usually organized into a directory structure. Any directory containing Python files can comprise an Import Package.

Because packages consist of multiple files, they are harder to distribute.

So long as your code contains nothing but pure Python code, and you know your deployment environment supports your version of Python, then you can use Python’s native packaging tools to create a source Distribution Package, or sdist for short.

Python’s sdists are compressed archives (.tar.gz files) containing one or more packages or modules

Python binary distributions

So much of Python’s practical power comes from its ability to integrate with the software ecosystem, in particular libraries written in C, C++, Fortran, Rust, and other languages.

Not all developers have the right tools or experiences to build these components written in these compiled languages, so Python created the Wheel, a package format designed to ship libraries with compiled artifacts. In fact, Python’s package installer, pip, always prefers wheels because installation is always faster, so even pure-Python packages work better with wheels.

src

pyproject.toml

We are going to use pyproject.toml file for building and distributing our Python packages.

Note: We can also use setup.py, setup.cfg to achieve the same.

The pyproject.toml file is a configuration file for Python projects. The file is written in the TOML (Tom’s Obvious, Minimal Language) format, which is designed to be simple and human-readable.

Key Sections in pyproject.toml

• [build-system]: Specifies the build backend and dependencies required for building the project.
• [project]: (PEP 621) Metadata about the project, such as name, version, authors, and dependencies.
• [tool.<tool_name>]: Configuration for various tools like pytest, black, etc.

Build Tools

• setuptools: Traditional Python build tool.
• poetry: Modern dependency management and packaging.
• flit: Lightweight tool for publishing Python packages.

Sample project

Lets create our Python project which we are going to distribute.

Before creating our Project we have to decide the project structure(layout).

There are two types of layouts:

1. src layout — The project should contain a src directory under the project root and all modules and packages meant for distribution are placed inside this directory

project_root_directory
├── pyproject.toml  # AND/OR setup.cfg, setup.py
├── ...
└── src/
    └── mypkg/
        ├── __init__.py
        ├── ...
        ├── module.py
        ├── subpkg1/
        │   ├── __init__.py
        │   ├── ...
        │   └── module1.py
        └── subpkg2/
            ├── __init__.py
            ├── ...
            └── module2.py

This layout is very handy when you wish to use automatic discovery, since you don’t have to worry about other Python files or folders in your project root being distributed by mistake.

2. flat layout — The package folder(s) are placed directly under the project root (also known as “adhoc”)

project_root_directory
├── pyproject.toml  # AND/OR setup.cfg, setup.py
├── ...
└── mypkg/
    ├── __init__.py
    ├── ...
    ├── module.py
    ├── subpkg1/
    │   ├── __init__.py
    │   ├── ...
    │   └── module1.py
    └── subpkg2/
        ├── __init__.py
        ├── ...
        └── module2.py

Reference: https://setuptools.pypa.io/en/stable/userguide/package_discovery.html

Get Logesh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Lets choose src layout.

SUPERPROJECT/
├── src/
│   └── super_package/
│       ├── __init__.py
│       └── super_module.py
├── pyproject.toml
└── README.md

super_module.py content

Lets write our pyproject.toml file

build-system

• requires: Specifies dependencies needed for the build process.
• build-backend: Specifies the tool/module that handles building the project.

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

project

[project]
name = "super-pkg"   # as it would appear on PyPI
authors = [
    {name = "Logesh", email = "log@gmail.com"},
]
description = "My package description"
readme = "README.md"
requires-python = ">=3.10"
license = {text = "MIT License"}
classifiers = [
    "Programming Language :: Python :: 3",
]
dependencies = [
    "requests",
]
version = "1.0.1"

For more options check here and here

tool

For the src layout we majorly don’t need the tool section. Since all our packages meant for distribution were placed unser src dir. So the automatic discovery can find the available packages from the src dir.

If we chose the flat-layout then tool section might be helpful in providing the info about the packages path(where to find), what are all the packages been included/ excluded.

[tool.setuptools.packages.find]
where = ["src"]  # list of folders that contain the packages (["."] by default)
include = ["my_package*"]  # package names should match these glob patterns (["*"] by default)
exclude = ["my_package.tests*"]  # exclude packages matching these glob patterns (empty by default)
namespaces = false  # to disable scanning PEP 420 namespaces (true by default)

our pyproject.toml file

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "super-pkg"
authors = [
    {name = "Logesh", email = "log@gmail.com"},
]
description = "My package description"
readme = "README.md"
requires-python = ">=3.10"
license = {text = "MIT License"}
classifiers = [
    "Programming Language :: Python :: 3",
]
dependencies = [
    "requests",
]
version = "1.0.1"

Build our package

To build our Python package we can use setup.py or build module.

src

To use build module, we have install that using

pip install build

Once we have our build module we can build our package by running the build command in our project root dir.

python -m build

Command execution output

PS C:\Users\L\Desktop\SuperProject> ls


    Directory: C:\Users\L\Desktop\SuperProject


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        12/27/2024   9:12 PM                src
-a----        12/27/2024   9:19 PM            436 pyproject.toml
-a----        12/27/2024   9:22 PM             31 README.md


PS C:\Users\L\Desktop\SuperProject> python -m build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools
* Getting build dependencies for sdist...
running egg_info
creating src\super_pkg.egg-info
writing src\super_pkg.egg-info\PKG-INFO
writing dependency_links to src\super_pkg.egg-info\dependency_links.txt
writing requirements to src\super_pkg.egg-info\requires.txt
writing top-level names to src\super_pkg.egg-info\top_level.txt
writing manifest file 'src\super_pkg.egg-info\SOURCES.txt'
reading manifest file 'src\super_pkg.egg-info\SOURCES.txt'
writing manifest file 'src\super_pkg.egg-info\SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing src\super_pkg.egg-info\PKG-INFO
writing dependency_links to src\super_pkg.egg-info\dependency_links.txt
writing requirements to src\super_pkg.egg-info\requires.txt
writing top-level names to src\super_pkg.egg-info\top_level.txt
reading manifest file 'src\super_pkg.egg-info\SOURCES.txt'
writing manifest file 'src\super_pkg.egg-info\SOURCES.txt'
running check
creating super_pkg-1.0.1
creating super_pkg-1.0.1\src\super_package
creating super_pkg-1.0.1\src\super_pkg.egg-info
copying files to super_pkg-1.0.1...
copying README.md -> super_pkg-1.0.1
copying pyproject.toml -> super_pkg-1.0.1
copying src\super_package\__init__.py -> super_pkg-1.0.1\src\super_package
copying src\super_package\super_module.py -> super_pkg-1.0.1\src\super_package
copying src\super_pkg.egg-info\PKG-INFO -> super_pkg-1.0.1\src\super_pkg.egg-info
copying src\super_pkg.egg-info\SOURCES.txt -> super_pkg-1.0.1\src\super_pkg.egg-info
copying src\super_pkg.egg-info\dependency_links.txt -> super_pkg-1.0.1\src\super_pkg.egg-info
copying src\super_pkg.egg-info\requires.txt -> super_pkg-1.0.1\src\super_pkg.egg-info
copying src\super_pkg.egg-info\top_level.txt -> super_pkg-1.0.1\src\super_pkg.egg-info
copying src\super_pkg.egg-info\SOURCES.txt -> super_pkg-1.0.1\src\super_pkg.egg-info
Writing super_pkg-1.0.1\setup.cfg
Creating tar archive
removing 'super_pkg-1.0.1' (and everything under it)
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools
* Getting build dependencies for wheel...
running egg_info
writing src\super_pkg.egg-info\PKG-INFO
writing dependency_links to src\super_pkg.egg-info\dependency_links.txt
writing requirements to src\super_pkg.egg-info\requires.txt
writing top-level names to src\super_pkg.egg-info\top_level.txt
reading manifest file 'src\super_pkg.egg-info\SOURCES.txt'
writing manifest file 'src\super_pkg.egg-info\SOURCES.txt'
* Building wheel...
running bdist_wheel
running build
running build_py
creating build\lib\super_package
copying src\super_package\super_module.py -> build\lib\super_package
copying src\super_package\__init__.py -> build\lib\super_package
running egg_info
writing src\super_pkg.egg-info\PKG-INFO
writing dependency_links to src\super_pkg.egg-info\dependency_links.txt
writing requirements to src\super_pkg.egg-info\requires.txt
writing top-level names to src\super_pkg.egg-info\top_level.txt
reading manifest file 'src\super_pkg.egg-info\SOURCES.txt'
writing manifest file 'src\super_pkg.egg-info\SOURCES.txt'
installing to build\bdist.win-amd64\wheel
running install
running install_lib
creating build\bdist.win-amd64\wheel
creating build\bdist.win-amd64\wheel\super_package
copying build\lib\super_package\super_module.py -> build\bdist.win-amd64\wheel\.\super_package
copying build\lib\super_package\__init__.py -> build\bdist.win-amd64\wheel\.\super_package
running install_egg_info
Copying src\super_pkg.egg-info to build\bdist.win-amd64\wheel\.\super_pkg-1.0.1-py3.11.egg-info
running install_scripts
creating build\bdist.win-amd64\wheel\super_pkg-1.0.1.dist-info\WHEEL
creating 'C:\Users\L\Desktop\SuperProject\dist\.tmp-m6_z1ckk\super_pkg-1.0.1-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
adding 'super_package/__init__.py'
adding 'super_package/super_module.py'
adding 'super_pkg-1.0.1.dist-info/METADATA'
adding 'super_pkg-1.0.1.dist-info/WHEEL'
adding 'super_pkg-1.0.1.dist-info/top_level.txt'
adding 'super_pkg-1.0.1.dist-info/RECORD'
removing build\bdist.win-amd64\wheel
Successfully built super_pkg-1.0.1.tar.gz and super_pkg-1.0.1-py3-none-any.whl
PS C:\Users\L\Desktop\SuperProject>
PS C:\Users\L\Desktop\SuperProject>
PS C:\Users\L\Desktop\SuperProject> ls


    Directory: C:\Users\L\Desktop\SuperProject


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        12/27/2024   9:36 PM                dist
d-----        12/27/2024   9:35 PM                src
-a----        12/27/2024   9:19 PM            436 pyproject.toml
-a----        12/27/2024   9:22 PM             31 README.md


PS C:\Users\L\Desktop\SuperProject>

dist dir is created.

after building the package

Now, we can locally install this package using whl file or tar.gz file

Lets create one test script in that we can install and use this package.

Creating virtual env

python -m venv venv

create and activate venv

Now we can install our package using the whl file

pip install <path-to-whl-file-or-tar.gz-file>

our super_package is installed.

Using our package.

So far we have learned how to build and install our package locally using pyproject.toml

To upload the package to PyPi please refer,

• https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives

For extensions

• https://setuptools.pypa.io/en/stable/userguide/ext_modules.html

Hope you learned something new today!

Happie learning

Cheers!!!