The Python/C API: A Brief Introduction

CPython’s implementation of Py_Initialize in pylifecycle.c

The Python/C API brings together two of my two favorite things: Python and C.

What is it?

The Python/C API allows C programmers to embed Python directly into C code by exposing aspects of CPython internals. It provides direct access to the Python interpreter from C, acting as a bridge between the two languages.

An example

This C code calls Python’s platform.system().

On my system, this code prints out:

Platform name is Darwin

Building

I built the code above on my Mac by running:

gcc -o example example.c -I<Python include directory> /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/libpython3.8.dylib

I configured, built, and installed CPython on my system, but this isn’t necessary. You can also download a Python installer, let it perform a default install, and then locate libpython and Python.h on your system.

On my Windows machine, the Python 3.8 installer placed Python.h in:

C:\Users\<user>\AppData\Local\Programs\Python\Python38\include 

And python38.lib is in:

C:\Users\<user>\AppData\Local\Programs\Python\Python38\libs

But why embed Python in C?

Why reinvent wheels that already roll well in Python? When execution at the speed of pure C isn’t necessary (and perhaps rapid development time is), embedding Python is an extremely effective tool for leveraging functionality that already exists in Python. My first foray into working with the Python/C API was driven by exactly this situation: I had a great deal of well-tested Python code that I did not want to duplicate in my C application.

Python internals are also fascinating and worth getting to know in their own right.

Performance

As an interpreted language, Python is generally slower than C. In the system.platform() example above, the calls into the Python/C API still have to be interpreted, meaning that the performance of these calls will be similar to the performance of calling system.platform() from Python. If Python isn’t a good choice for your application, it’s unlikely that the C API will change that (and in fact, the extra assignments required may make it an even worse choice).

Get Ashley Wans’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The same C API allows programmers to create Python extensions in C which do not need to be interpreted because they are compiled straight to C. This results in highly performant code, but this is different than the example of embedded Python that I share above.

Backwards compatibility

The C code above will not compile if the user’s system only has Python 2.7 installed.

The C API is necessarily updated when the rest of Python is updated. As a result, backwards compatibility isn’t always maintained between versions. This is something to consider when choosing to use existing Python code or to recreate the same functionality in C.

Multi-threading

Multi-threading using the C API is possible, but it can be complicated by Python’s Global Interpreter Lock.

The code to manage the GIL is straight-forward enough:

PyGILState_STATE gstate;
...
void example_function(void)
{
    PyGILState_Ensure();
    ...
    PyGILState_Release(gstate);
}

In practice, all of my multi-threaded C applications that call Python have required a bit of debugging before I fully understood how the GIL was functioning in context.

Reference Counting

Lines 35–37 in example.c provide an example of the reference counting API:

// Decrement reference counts for objects we are
// done using to allow garbage collection to dispose of them
Py_XDECREF(system_function);
Py_XDECREF(system_obj);
Py_XDECREF(str);

The docs explain:

Since Python makes heavy use of malloc() and free(), it needs a strategy to avoid memory leaks as well as the use of freed memory. The chosen method is called reference counting. The principle is simple: every object contains a counter, which is incremented when a reference to the object is stored somewhere, and which is decremented when a reference to it is deleted. When the counter reaches zero, the last reference to the object has been deleted and the object is freed.

The documentation on extending and embedding Python provides a thorough description of reference counting.

Ultimately

When execution speed isn’t at issue, the Python/C API is an extremely useful tool for leveraging existing functionality in Python.