diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 1c58bc21620..42c10c3dcb3 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -678,6 +678,7 @@ peps/pep-0799.rst @pablogsal peps/pep-0800.rst @JelleZijlstra peps/pep-0801.rst @warsaw peps/pep-0802.rst @AA-Turner +peps/pep-0803.rst @encukou # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-0803.rst b/peps/pep-0803.rst new file mode 100644 index 00000000000..232089d7ec2 --- /dev/null +++ b/peps/pep-0803.rst @@ -0,0 +1,333 @@ +PEP: 803 +Title: Stable ABI for Free-Threaded Builds +Author: Petr Viktorin +Discussions-To: Pending +Status: Draft +Type: Standards Track +Requires: 703, 793, 697 +Created: 19-Aug-2025 +Python-Version: 3.15 + + +Abstract +======== + +Version 3.15 of the Stable ABI will be compatible with both free-threaded and +GIL-enabled builds. +To allow this, the :c:type:`PyObject` internal structure and related APIs +will be removed from version 3.15 of the Limited API, requiring migration to +new API for common tasks like defining modules and most classes. + + +Terminology +=========== + +This PEP uses “GIL-enabled build” as an antonym to “free-threaded build”, +that is, an interpreter or extension built without ``Py_GIL_DISABLED``. + + +Motivation +========== + +The Stable ABI is currently not available for free-threaded builds. +Extensions will fail to build when :c:macro:`Py_LIMITED_API` is defined, +and extensions built for GIL-enabled builds of CPython will fail to load +(or crash) on free-threaded builds. + +In its `acceptance post `__ +for :pep:`779`, the Steering Council stated that it “expects that Stable ABI +for free-threading should be prepared and defined for Python 3.15”. + +This PEP proposes the Stable ABI for free-threading. + + +Background +---------- + +Python's Stable ABI, as defined in :pep:`384` and :pep:`652`, provides a way to +compile extension modules that can be loaded on multiple minor versions of the +CPython interpreter. +Several projects use this to limit the number of +:ref:`wheels ` (binary artefacts) +that need to be built and distributed for each release, and/or to make it +easier to test with pre-release versions of Python. + +With free-threading builds (:pep:`703`) being on track to eventually become +the default (:pep:`779`), we need a way to make the Stable ABI available +to those builds. + +To build against the Stable ABI, the extension must use a *Limited API*, +that is, only a subset of the functions, structures, etc. that CPython +exposes. +The Limited API is versioned, and building against Limited API 3.X +yields an extension that is ABI-compatible with CPython 3.X and *any* later +version (though bugs in CPython sometimes cause incompatibilities in practice). +Also, the Limited API is not “stable”: newer versions may remove API that +were a part of older versions. + +This PEP proposes the most significant such removal to date. + + +Rationale +========= + +The design in this PEP makes several assumptions: + +One ABI + A single compiled extension module should support both + free-threaded and GIL-enabled builds. + +No backwards compatibility + The new limited API will not support CPython 3.14 and below. + Projects that need this support can build separate extensions specifically + for the 3.14 free-threaded interpreter, and for older stable ABI versions. + +API changes are OK + The new Limited API may require extension authors to make significant + changes to their code. + Projects that cannot do this (yet) can continue using Limited API 3.14, + which will yield extensions compatible with GIL-enabled builds only. + +No extra configuration + We do not introduce new “knobs” that influence what API is available + and what the ABI is compatible with. + + +Specification +============= + + +Opaque PyObject +--------------- + +Version 3.15 of the Limited API will: + +- make the following structures *opaque* (or in C terminology, *incomplete + types*): + + - :c:type:`PyObject` + - :c:type:`PyVarObject` + - :c:type:`!PyModuleDef_Base` + - :c:type:`PyModuleDef` + +- no longer include the following macros: + + - :c:macro:`PyObject_HEAD` + - :c:macro:`!_PyObject_EXTRA_INIT` + - :c:macro:`PyObject_HEAD_INIT` + - :c:macro:`PyObject_VAR_HEAD` + +- no longer include these function-like macros: + + - :c:func:`Py_SIZE` + - :c:func:`Py_SET_SIZE` + - :c:func:`Py_SET_TYPE` + + +Implications +^^^^^^^^^^^^ + +Making the ``PyObject``, ``PyVarObject`` and ``PyModuleDef`` structures +opaque means: + +- Their fields may not be directly accessed. + + For example, instead of ``o->ob_type``, extensions must use + ``Py_TYPE(o)``. + This usage has been the preferred practice for some time. + +- Their size and alignment will not be available. + Expressions such as ``sizeof(PyObject)`` will no longer work. + +- They cannot be embedded in other structures. + This mainly affects instance structs of extension-defined types, + which will need to be defined using API added in :pep:`697` -- that is, + using a ``struct`` *without* ``PyObject`` (or other base class struct) at + the beginning, with :c:func:`PyObject_GetTypeData` calls needed to access + the memory. + +- Variables of these types cannot be created. + This mainly affects static ``PyModuleDef`` variables needed to define + extension modules. + Extensions will need to switch to API added in :pep:`793`. + +The following functions will become unusable in practice (in the new Limited +API), since an extension cannot create valid, statically allocated, input +for them. To ease the transition for extension developers, +they will not yet be removed from the Limited API: + +- :c:func:`PyModuleDef_Init` +- :c:func:`PyModule_Create`, :c:func:`PyModule_Create2` +- :c:func:`PyModule_FromDefAndSpec`, :c:func:`PyModule_FromDefAndSpec2` + + +New Export Hook (PEP 793) +------------------------- + +Implementation of this PEP requires :pep:`793` (``PyModExport``: +A new entry point for C extension modules) to be +accepted, providing a new “export hook” for defining extension modules. +Using the new hook will become mandatory in Limited API 3.15. + + +Runtime ABI checks +------------------ + +Users -- or rather the tools they use for building and installing extensions -- +will continue to be responsible for not putting incompatible extensions on +Python's import paths. +This decision makes sense since tools typically have much richer metadata than +what CPython can check. + +However, CPython will add a line of defense against outdated or misconfigured +tools, or human mistakes, in the form of a new *module slot* containing +basic ABI information. +This information will be checked when a module is loaded, and incompatible +extensions will be rejected. +The specifics are left to the C API working group +(see `issue 72 `__). + +This slot will become *mandatory* with the new export hook added in +:pep:`793`. +(That PEP currently says “there are no required slots”; it will be updated.) + + +Check for older ``abi3`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +Additionally, in free-threaded builds, :c:func:`PyModuleDef_Init` will detect +extensions using the pre-free-threading Stable ABI, emit an informative +message when one is loaded, *and* raise an exception. +(Implementation note: A message will be printed before raising the exception, +because extensions that attempt to handle an exception using incompatible ABI +will likely crash and lose the exception's message.) + +This check for older ``abi3`` relies on internal bit patterns and may be +removed in future CPython versions, if the internal object layout needs +to change. + + +Wheel tags +---------- + +PyPA build tools should not need changes: if they allow the user to set the +limited API version, setting it to 3.15 should define :c:macro:`Py_LIMITED_API` +to ``0x030f0000``. +The resulting wheel should be tagged with the Python-ABI tag ``cp315-abi3``. + +Installers and other tools should assume that wheels with a *python tag* +``cp315`` and above and ABI tag ``abi3`` are compatible with free-threaded +builds. + + +New API +------- + +Implementing this PEP will make it possible to build extensions that +can be successfully loaded on free-threaded Python, but not necessarily ones +that are thread-safe without a GIL. + +Limited API to allow thread-safety without a GIL -- presumably ``PyMutex``, ``PyCriticalSection``, and +similar -- will be added via the C API working group, or in a follow-up PEP. + + +Backwards Compatibility +======================= + +Limited API 3.15 will not be backwards-compatible with older CPython releases, +due to removed structs and functions. + +Extension authors who cannot switch may continue to use Limited API 3.14 +and below. +For compatibility with free-threaded builds, they can compile using +version-specific ABI -- for example, compile on CPython 3.15 without defining +``Py_LIMITED_API``. + + +Security Implications +===================== + +None known. + + +How to Teach This +================= + +A porting guide will need to explain how to move to APIs added in +:pep:`697` (Limited C API for Extending Opaque Types) +and :pep:`793` (``PyModExport``). + + +Reference Implementation +======================== + +This PEP combines several pieces, implemented individually: + +- Opaque ``PyObject`` is available in CPython main branch after defining the + ``_Py_OPAQUE_PYOBJECT`` macro. + Implemented in GitHub pull request `python/cpython#136505 + `__. +- For ``PyModExport``, see :pep:`793`. +- For a version-checking slot, see GitHub pull request `python/cpython#137212 + `__. +- For a check for older ``abi3``, see GitHub pull request `python/cpython#137957 + `__. +- For wheel tags, there is no implementation yet. +- A porting guide is not yet written. + + +Rejected Ideas +============== + + +Add an alternative stable ABI for free-threading +------------------------------------------------ + +It would be possible to: + +- Keep the current stable ABI (“``abi3``”) unchanged (except additions, as done + in each release). Extensions would need no code changes and builds would be + compatible with old and new GIL-enabled CPython versions. +- Add a new stable ABI (“``abi3t``”) specifically for free-threading. + Extensions would need no code changes and builds would be + compatible with free-threaded CPython (3.14 and above). +- Defining an additional macro (“``Py_OPAQUE_PYOBJECT``”) would make + ``PyObject`` opaque as in this PEP. Extensions would need code changes as in + this PEP, and compiled extensions (“``abi3.abi3t``”) would be compatible with + all builds of CPython 3.15+. + +This scheme was rejected as too complex. +It would also make the free-threading memory layout of ``PyObject`` part +of the stable ABI, preventing future adjustments. + + +Shim for compatibility with CPython 3.14 +---------------------------------------- + +The issue that prevents compatibility with Python 3.14 is that with +opaque ``PyObject`` and ``PyModuleDef``, it is not feasible to initialize +an extension module. +The solution, :pep:`793`, is only being added in Python 3.15. + +It is possible to work around this using the fact that the 3.14 ABIs (both +free-threading and GIL-enabled) are “frozen”, so it is possible for an +extension to query the running interpreter, and for 3.14, use +a ``struct`` definition corresponding to the detected build's ``PyModuleDef``. + +This is too onerous to support and test in CPython's Limited API. +It would also require adding a new wheel tag (e.g. ``abi3t``) that all install +tools would need to recognize. (This PEP's ``cp315-abi3`` is incompatible +with Python 3.14.) + + +Open Issues +=========== + +[See discussion for now.] + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.