PEP 803: Stable ABI for Free-Threaded Builds (GH-4556)

Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

Author: 4 people

.github/CODEOWNERS

@@ -678,6 +678,7 @@ peps/pep-0799.rst  @pablogsal
 peps/pep-0800.rst  @JelleZijlstra
 peps/pep-0801.rst  @warsaw
 peps/pep-0802.rst  @AA-Turner
+peps/pep-0803.rst  @encukou
 # ...
 peps/pep-2026.rst  @hugovk
 # ...

peps/pep-0803.rst

@@ -0,0 +1,333 @@
+PEP: 803
+Title: Stable ABI for Free-Threaded Builds
+Author: Petr Viktorin <encukou@gmail.com>
+Discussions-To: Pending
+Status: Draft
+Type: Standards Track
+Requires: 703, 793, 697
+Created: 19-Aug-2025
+Python-Version: 3.15
+
+
+Abstract
+========
+
+Version 3.15 of the Stable ABI will be compatible with both free-threaded and
+GIL-enabled builds.
+To allow this, the :c:type:`PyObject` internal structure and related APIs
+will be removed from version 3.15 of the Limited API, requiring migration to
+new API for common tasks like defining modules and most classes.
+
+
+Terminology
+===========
+
+This PEP uses “GIL-enabled build” as an antonym to “free-threaded build”,
+that is, an interpreter or extension built without ``Py_GIL_DISABLED``.
+
+
+Motivation
+==========
+
+The Stable ABI is currently not available for free-threaded builds.
+Extensions will fail to build when :c:macro:`Py_LIMITED_API` is defined,
+and extensions built for GIL-enabled builds of CPython will fail to load
+(or crash) on free-threaded builds.
+
+In its `acceptance post <https://discuss.python.org/t/84319/123>`__
+for :pep:`779`, the Steering Council stated that it “expects that Stable ABI
+for free-threading should be prepared and defined for Python 3.15”.
+
+This PEP proposes the Stable ABI for free-threading.
+
+
+Background
+----------
+
+Python's Stable ABI, as defined in :pep:`384` and :pep:`652`, provides a way to
+compile extension modules that can be loaded on multiple minor versions of the
+CPython interpreter.
+Several projects use this to limit the number of
+:ref:`wheels <packaging:binary-distribution-format>` (binary artefacts)
+that need to be built and distributed for each release, and/or to make it
+easier to test with pre-release versions of Python.
+
+With free-threading builds (:pep:`703`) being on track to eventually become
+the default (:pep:`779`), we need a way to make the Stable ABI available
+to those builds.
+
+To build against the Stable ABI, the extension must use a *Limited API*,
+that is, only a subset of the functions, structures, etc. that CPython
+exposes.
+The Limited API is versioned, and building against Limited API 3.X
+yields an extension that is ABI-compatible with CPython 3.X and *any* later
+version (though bugs in CPython sometimes cause incompatibilities in practice).
+Also, the Limited API is not “stable”: newer versions may remove API that
+were a part of older versions.
+
+This PEP proposes the most significant such removal to date.
+
+
+Rationale
+=========
+
+The design in this PEP makes several assumptions:
+
+One ABI
+   A single compiled extension module should support both
+   free-threaded and GIL-enabled builds.
+
+No backwards compatibility
+   The new limited API will not support CPython 3.14 and below.
+   Projects that need this support can build separate extensions specifically
+   for the 3.14 free-threaded interpreter, and for older stable ABI versions.
+
+API changes are OK
+   The new Limited API may require extension authors to make significant
+   changes to their code.
+   Projects that cannot do this (yet) can continue using Limited API 3.14,
+   which will yield extensions compatible with GIL-enabled builds only.
+
+No extra configuration
+   We do not introduce new “knobs” that influence what API is available
+   and what the ABI is compatible with.
+
+
+Specification
+=============
+
+
+Opaque PyObject
+---------------
+
+Version 3.15 of the Limited API will:
+
+- make the following structures *opaque* (or in C terminology, *incomplete
+  types*):
+
+  - :c:type:`PyObject`
+  - :c:type:`PyVarObject`
+  - :c:type:`!PyModuleDef_Base`
+  - :c:type:`PyModuleDef`
+
+- no longer include the following macros:
+
+  - :c:macro:`PyObject_HEAD`
+  - :c:macro:`!_PyObject_EXTRA_INIT`
+  - :c:macro:`PyObject_HEAD_INIT`
+  - :c:macro:`PyObject_VAR_HEAD`
+
+- no longer include these function-like macros:
+
+  - :c:func:`Py_SIZE`
+  - :c:func:`Py_SET_SIZE`
+  - :c:func:`Py_SET_TYPE`
+
+
+Implications
+^^^^^^^^^^^^
+
+Making the ``PyObject``, ``PyVarObject`` and ``PyModuleDef`` structures
+opaque means:
+
+- Their fields may not be directly accessed.
+
+  For example, instead of ``o->ob_type``, extensions must use
+  ``Py_TYPE(o)``.
+  This usage has been the preferred practice for some time.
+
+- Their size and alignment will not be available.
+  Expressions such as ``sizeof(PyObject)`` will no longer work.
+
+- They cannot be embedded in other structures.
+  This mainly affects instance structs of extension-defined types,
+  which will need to be defined using API added in :pep:`697` -- that is,
+  using a ``struct`` *without* ``PyObject`` (or other base class struct) at
+  the beginning, with :c:func:`PyObject_GetTypeData` calls needed to access
+  the memory.
+
+- Variables of these types cannot be created.
+  This mainly affects static ``PyModuleDef`` variables needed to define
+  extension modules.
+  Extensions will need to switch to API added in :pep:`793`.
+
+The following functions will become unusable in practice (in the new Limited
+API), since an extension cannot create valid, statically allocated, input
+for them. To ease the transition for extension developers,
+they will not yet be removed from the Limited API:
+
+- :c:func:`PyModuleDef_Init`
+- :c:func:`PyModule_Create`, :c:func:`PyModule_Create2`
+- :c:func:`PyModule_FromDefAndSpec`, :c:func:`PyModule_FromDefAndSpec2`
+
+
+New Export Hook (PEP 793)
+-------------------------
+
+Implementation of this PEP requires :pep:`793` (``PyModExport``:
+A new entry  point for C extension modules) to be
+accepted, providing a new “export hook” for defining extension modules.
+Using the new hook will become mandatory in Limited API 3.15.
+
+
+Runtime ABI checks
+------------------
+
+Users -- or rather the tools they use for building and installing extensions --
+will continue to be responsible for not putting incompatible extensions on
+Python's import paths.
+This decision makes sense since tools typically have much richer metadata than
+what CPython can check.
+
+However, CPython will add a line of defense against outdated or misconfigured
+tools, or human mistakes, in the form of a new *module slot* containing
+basic ABI information.
+This information will be checked when a module is loaded, and incompatible
+extensions will be rejected.
+The specifics are left to the C API working group
+(see `issue 72 <https://github.com/capi-workgroup/decisions/issues/72>`__).
+
+This slot will become *mandatory* with the new export hook added in
+:pep:`793`.
+(That PEP currently says “there are no required slots”; it will be updated.)
+
+
+Check for older ``abi3``
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Additionally, in free-threaded builds, :c:func:`PyModuleDef_Init` will detect
+extensions using the pre-free-threading Stable ABI, emit an informative
+message when one is loaded, *and* raise an exception.
+(Implementation note: A message will be printed before raising the exception,
+because extensions that attempt to handle an exception using incompatible ABI
+will likely crash and lose the exception's message.)
+
+This check for older ``abi3`` relies on internal bit patterns and may be
+removed in future CPython versions, if the internal object layout needs
+to change.
+
+
+Wheel tags
+----------
+
+PyPA build tools should not need changes: if they allow the user to set the
+limited API version, setting it to 3.15 should define :c:macro:`Py_LIMITED_API`
+to ``0x030f0000``.
+The resulting wheel should be tagged with the Python-ABI tag ``cp315-abi3``.
+
+Installers and other tools should assume that wheels with a *python tag*
+``cp315`` and above and ABI tag ``abi3`` are compatible with free-threaded
+builds.
+
+
+New API
+-------
+
+Implementing this PEP will make it possible to build extensions that
+can be successfully loaded on free-threaded Python, but not necessarily ones
+that are thread-safe without a GIL.
+
+Limited API to allow thread-safety without a GIL -- presumably ``PyMutex``, ``PyCriticalSection``, and
+similar -- will be added via the C API working group, or in a follow-up PEP.
+
+
+Backwards Compatibility
+=======================
+
+Limited API 3.15 will not be backwards-compatible with older CPython releases,
+due to removed structs and functions.
+
+Extension authors who cannot switch may continue to use Limited API 3.14
+and below.
+For compatibility with free-threaded builds, they can compile using
+version-specific ABI -- for example, compile on CPython 3.15 without defining
+``Py_LIMITED_API``.
+
+
+Security Implications
+=====================
+
+None known.
+
+
+How to Teach This
+=================
+
+A porting guide will need to explain how to move to APIs added in
+:pep:`697` (Limited C API for Extending Opaque Types)
+and :pep:`793` (``PyModExport``).
+
+
+Reference Implementation
+========================
+
+This PEP combines several pieces, implemented individually:
+
+- Opaque ``PyObject`` is available in CPython main branch after defining the
+  ``_Py_OPAQUE_PYOBJECT`` macro.
+  Implemented in GitHub pull request `python/cpython#136505
+  <https://github.com/python/cpython/pull/136505>`__.
+- For ``PyModExport``, see :pep:`793`.
+- For a version-checking slot, see GitHub pull request `python/cpython#137212
+  <https://github.com/python/cpython/pull/137212>`__.
+- For a check for older ``abi3``, see GitHub pull request `python/cpython#137957
+  <https://github.com/python/cpython/pull/137957>`__.
+- For wheel tags, there is no implementation yet.
+- A porting guide is not yet written.
+
+
+Rejected Ideas
+==============
+
+
+Add an alternative stable ABI for free-threading
+------------------------------------------------
+
+It would be possible to:
+
+- Keep the current stable ABI (“``abi3``”) unchanged (except additions, as done
+  in each release). Extensions would need no code changes and builds would be
+  compatible with old and new GIL-enabled CPython versions.
+- Add a new stable ABI (“``abi3t``”) specifically for free-threading.
+  Extensions would need no code changes and builds would be
+  compatible with free-threaded CPython (3.14 and above).
+- Defining an additional macro (“``Py_OPAQUE_PYOBJECT``”) would make
+  ``PyObject`` opaque as in this PEP. Extensions would need code changes as in
+  this PEP, and compiled extensions (“``abi3.abi3t``”) would be compatible with
+  all builds of CPython 3.15+.
+
+This scheme was rejected as too complex.
+It would also make the free-threading memory layout of ``PyObject`` part
+of the stable ABI, preventing future adjustments.
+
+
+Shim for compatibility with CPython 3.14
+----------------------------------------
+
+The issue that prevents compatibility with Python 3.14 is that with
+opaque ``PyObject`` and ``PyModuleDef``, it is not feasible to initialize
+an extension module.
+The solution, :pep:`793`, is only being added in Python 3.15.
+
+It is possible to work around this using the fact that the 3.14 ABIs (both
+free-threading and GIL-enabled) are “frozen”, so it is possible for an
+extension to query the running interpreter, and for 3.14, use
+a ``struct`` definition corresponding to the detected build's ``PyModuleDef``.
+
+This is too onerous to support and test in CPython's Limited API.
+It would also require adding a new wheel tag (e.g. ``abi3t``) that all install
+tools would need to recognize. (This PEP's ``cp315-abi3`` is incompatible
+with Python 3.14.)
+
+
+Open Issues
+===========
+
+[See discussion for now.]
+
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.