Pybind11 demystified. Ch 3: Your first extension

2 min readAug 29, 2022

We’re not far away from having our first extension. Let’s make the xxmodule.c example, a real Python extension instead of a built-in module.

We’ll leave the Python interpreter binary untouched and build the extension as a shared library. Python import statement will load the library using dl_open and make it an actual Python module that you can use in the interpreter.

# Copy the xxmodule.c file to a new location# Assuming that you're in the debug folder
cd ../.. 
cp cpython/Modules/xxmodule.c .# Also create some shortcuts to the compile binary
ln -s cpython/debug/python python
ln -s cpython/debug/python-config python-config

Still remember how to build a shared library? Yes, with --fPIC --shared command. But this time, we must also tell the compiler where to find the header files using -ILIBRARY_PATH flags. So here is the complete command that you can copy.

gcc -shared -fPIC -Icpython/Include -Icpython/debug xxmodule.c -o xx$(bash python-config --extension-suffix)

In practice, you might also want -Wall to display all the warnings (sometimes they’re really helpful!) and -O3 for code optimization.

The generated shared library file is called xx.cpython-312d-x86_64-linux-gnu.so according to my settings. The name matters, as Python will look for the exact name when loading modules. The 312d word in the name means this library will not work for Python versions other than 3.12. (try import xx from a different Python version, and you’ll get aModuleNotFoundError error)

In practice, you can write a simpledistutils script that builds the module against the local installed Python version uponpip install.

Also, note that the module name is xx and it requires thePyInit_xx symbol defined in the library.

./python
>>> import xx
>>> xx.bug([1,2,3])
1

Looks great.

And you can try renaming the module to something like xx2.

cp xx.cpython-312d-x86_64-linux-gnu.so xx2.cpython-312d-x86_64-linux-gnu.so
./python
>>> import xx2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_xx2)

OK. So in the xxmodule.c file, let’s make a copy of the PyInit_xx function and name it PyInit_xx2 . Remember to also copy the line with PyMODINIT_FUNC . Now compile it, and the import should work.

gcc -shared -fPIC -Icpython/Include -Icpython/debug xxmodule.c -o xx2$(bash python-config --extension-suffix)
./python
>>> import xx2
>>> xx2.__doc__

Nice.

You can also use LD_DEBUG=files ./python to verify when and which library file has been loaded by the runtime linker.

Now you have the foundation to understand Python extensions. They’re the same thing as built-in modules but distributed and loaded differently. They’re pretty easy to write and are one of the main reasons Python is popular, despite pure Python code being slow.

In the next chapter, we’ll return to the topic of pybind11 and see how the C++ templated type system helps when writing Python extensions.

>> Chapter 4: Passing arguments

Pybind11 demystified. Ch 3: Your first extension

Written by Xianbo QIAN