diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 3d085b3b8ac..aeeea8425a3 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -684,6 +684,8 @@ peps/pep-0804.rst @pradyunsg peps/pep-0806.rst @JelleZijlstra peps/pep-0807.rst @dstufft # ... +peps/pep-0809.rst @zooba +# ... peps/pep-2026.rst @hugovk # ... peps/pep-3000.rst @gvanrossum diff --git a/peps/pep-0809.rst b/peps/pep-0809.rst new file mode 100644 index 00000000000..1ae06473b9b --- /dev/null +++ b/peps/pep-0809.rst @@ -0,0 +1,433 @@ +PEP: 809 +Title: Stable ABI for the Future +Author: Steve Dower +Discussions-To: Pending +Status: Draft +Type: Standards Track +Requires: 703, 793, 697 +Created: 19-Sep-2025 +Python-Version: 3.15 + +Abstract +======== + +The Stable ABI as ``abi3`` can no longer be preserved, and requires replacement. +``abi2026`` will be the first replacement, providing resolution of current known +incompatibilities, with planned retirement after at least 10 years. The next ABI +(for example, ``abi2031``) will have at least five years of overlap with the +preceding one. + +Long-term stability will be enabled through a mechanism for runtime ABI +discovery, allowing extensions to be run with earlier releases that support the +same ABI. Changes and additions during the lifespan of an ABI can be added as +*interfaces*, allowing them to be discovered at runtime so that callers can +choose suitable fallback behaviour. Currently, such additions prevent extensions +from loading at all on earlier runtimes. + +The ``abi3`` ABI will be retained in GIL-enabled builds for at least five years, +after which time it may be retired (with only ``abi2026`` and later being +available). It's possible that GIL-enabled builds will be retired completely +before then. Free-threaded builds do not have ``abi3``, and so their first +Stable ABI would be ``abi2026``. + + +Terminology +=========== + +This PEP uses "GIL-enabled build" as an antonym to "free-threaded build", +that is, an interpreter or extension built without ``Py_GIL_DISABLED``. + + +Motivation +========== + +The Stable ABI is currently not available for free-threaded builds. +Extensions will fail to build when :c:macro:`Py_LIMITED_API` is defined. +Likewise, extensions built for GIL-enabled builds of CPython will fail to load +(or crash) on free-threaded builds. + +In its `acceptance post `__ +for :pep:`779`, the Steering Council stated that it "expects that Stable ABI +for free-threading should be prepared and defined for Python 3.15". + +This PEP proposes a Stable ABI that will be compatible with all variants of 3.15 +and later, allowing package developers to produce a single build of their +extensions. + + +Related PEPs +============ + +:pep:`803` is an alternative to this proposal, and much of the background text +used in this PEP is deliberately identical. We thank the PEP 803 author for the +use of their text. + +Background +========== + +Python's Stable ABI, as defined in :pep:`384` and :pep:`652`, provides a way to +compile extension modules that can be loaded on multiple minor versions of the +CPython interpreter. +Several projects use this to limit the number of +:ref:`wheels ` (binary artefacts) +that need to be built and distributed for each release, and/or to make it +easier to test with pre-release versions of Python. + +With free-threading builds (:pep:`703`) being on track to eventually become +the default (:pep:`779`), we need a way to make the Stable ABI available +to those builds. + +To build against the Stable ABI, the extension must use a *Limited API*, +that is, only a subset of the functions, structures, etc. that CPython +exposes. +The Limited API is versioned, and building against Limited API 3.X +yields an extension that is ABI-compatible with CPython 3.X and *any* later +version (though bugs in CPython sometimes cause incompatibilities in practice). +Also, the Limited API is not "stable": newer versions may remove API items that +were available in older versions. + +This PEP proposes a significant change to versioning of both the Limited API +and the Stable ABI. The goal is to enable long-term management of stability +and compatibility, while also allowing users of the limited subsets to have +access to innovations in later Python releases. + + +Rationale +========= + +The design in this PEP makes several assumptions: + +One ABI + A single compiled extension module should support both + free-threaded and GIL-enabled builds. + +No backwards compatibility + The new limited API will not be supported by CPython 3.14 and below. + Projects that need this support can build separate extensions specifically + for the 3.14 free-threaded interpreter, and for older stable ABI versions. + +API changes are OK + The new Limited API may require extension authors to make significant + changes to their code. + Projects that cannot do this (yet) can continue using Limited API 3.14, + which will yield extensions compatible with GIL-enabled builds only. + +No extra configuration + We do not introduce new "knobs" that influence what API is available + and what the ABI is compatible with. + + +Specification +============= + +Note that much of the specification is identical to :pep:`803`, and readers +should refer to that proposal for details. The ABI Stability, Build-time Macros +and Interfaces API sections are unique to this proposal. + +ABI Stability +------------- + +The Stable ABI will be frozen for a duration of at least 10 years. When a new +version of the Stable ABI is frozen, the existing version will continue to be +supported for at least 5 years. This allows ample migration time for package +maintainers (and other users) to migrate their entire range of supported +releases simultaneously. However, if the Python core development team sees no +reason to replace the current Stable ABI, freezing a new version may be +deferred. + +New Stable ABIs are defined using the PEP process, with their name reflecting +the release year of the first runtime that supports it. + +When a Stable ABI is frozen, the year becomes the name of the ABI. For example, +we anticipate that the first ABI under this scheme will be ``abi2026``, and will +be supported by all releases until at least 2036. If support is dropped in 2036, +then ``abi2031`` would be the migration target, which allows package developers +to support at least five years of releases after their own migration. + +While frozen, no ABI changes are permitted at all. Additions are not permitted, +nor are removals, modifications, or drastic semantic changes. Critically, an +extension module compiled against a particular ABI must load successfully +(as in, all imported symbols are satisfied on all supported platforms) against +any Python version supporting that ABI, earlier or later. + +Semantic changes that cannot be detected at runtime via existing compatible ABI +are not permitted. That is, the APIs to detect whether a particular behaviour is +expected on the current Python release must have been available on all earlier +releases that support the ABI. + + +Opaque PyObject +--------------- + +Version 3.15 of the Limited API will make a number of structures opaque, such +that users of them cannot make any assumptions about their size or layout. The +details may be found in :pep:`803`, and the proposal here is identical. + + +New Export Hook (PEP 793) +------------------------- + +Implementation of this PEP requires :pep:`793` (``PyModExport``: +A new entry point for C extension modules) to be +accepted, providing a new “export hook” for defining extension modules. +Using the new hook will become mandatory in Limited API 3.15. + +This proposal is identical to that of :pep:`803`. + + +Runtime ABI checks +------------------ + +See :pep:`803` for details. This proposal is identical. + +Build-time macros +----------------- + +We require :c:macro:`Py_LIMITED_API` to be defined to ``0x03ff_YYYY`` - that is, +the high word is a constant ``0x03ff``, while the low word is the ABI name +(year) as a hexadecimal value. While this results in a decimal value that is not +the same as the year, we consider that to be unimportant as the value is an +arbitrary label and more likely to be specified as a constant (in a ``cc`` +command line) than a calculated value. + +The use of ``0x03ff`` as the constant is intended to allow compatibility with +earlier runtimes. The same constant when used with headers only supporting +``abi3`` will select the "most complete" version of ABI3 available in that +release. For example, using ``0x03ff2026`` in 3.15+ would select ``abi2026``, +while in 3.10 will select the version of ABI3 that works for 3.10-3.14. + +Wheel tags +---------- + +Wheels should be tagged with the ABI tag ``abi2026``. No changes to Python or +platform tags are needed. It is perhaps worth noting that releases tagged for +``cp314`` or earlier will never be compatible with ``abi2026``, as it was not +present, and so a wheel tagged ``py3-abi2026-`` is not going to cause a +wheel using the new Stable ABI to be loaded by an older release. + + +New API +------- + +Implementing this PEP will make it possible to build extensions that +can be successfully loaded on free-threaded Python, but not necessarily ones +that are thread-safe without a GIL. + +Limited API to allow thread-safety without a GIL -- presumably ``PyMutex``, +``PyCriticalSection``, and similar -- will be added via the C API working group, +or in a follow-up PEP. + + +Interfaces API +-------------- + +A new *interfaces* API will be added to Python and the new Limited API. This API +is to satisfy the "semantic changes are detectable on all releases" requirement +from the ABI Stability section above. That is, consumers [#Consumers]_ will be +able to adopt a new API immediately, compile for the Limited API with the latest +release, and retain binary compatibility for all releases supporting that ABI. + +In short, the primary API is :c:func:`!PyObject_GetInterface`, which delegates +to a new native-only type slot to fill in a C struct containing either data or +function pointers. Because the C struct definition is embedded into the +extension, rather than obtained at runtime, an extension module can be aware of +later structs while running against releases of Python that do not provide it. + +If the call to ``PyObject_GetInterface`` requests a struct that is not available +on the current version, or is not available for the provided object, the call +fails safely. The caller may then use fallback logic (for example, using +abstract Python APIs) or abort, based on their preference. + +For example, if a new API were to be added during ``abi2026``'s life that allows +more efficient access to an ``int`` object's internal data, rather than adding a +new API, we would create a new interface: a struct containing a function pointer +to copy the data to a new location, and a previously unused index/name for that +interface. The caller can call ``PyObject_GetInterface(int_object, &intf_struct)`` +first; if it succeeds, call (a hypothetical) +``(*intf_struct.copy_bits)(&intf_struct, dest, sizeof(dest))``; if it fails, +they can use ``PyObject_CallMethod(int_object, "to_bytes", ...)`` to perform the +same operation, but less efficiently. The final result of this example is a +single extension module that is binary compatible with *all* releases supporting +``abi2026`` but is more efficient when running against newer releases of Python. + +Overview complete, here is the full specification of each new API: + +.. code-block:: c + + // Abstract API to request an interface for an object (or type). + PyAPI_FUNC(int) PyObject_GetInterface(PyObject *obj, void *intf); + + // API to release an interface. + PyAPI_FUNC(int) PyInterface_Release(void *intf); + + // Expected layout of the start of each interface. Actual interface structs + // will add additional function pointers or data. + typedef struct PyInterface_Base { + // sizeof(self), for additional validation that the caller is passing + // the correct structure. + Py_ssize_t size; + + // Unique identifier for the struct. Details below. + uint64_t name; + + // Function to release the struct (e.g. to decref any PyObject fields). + // Should only be invoked by PyInterface_Release(), not directly. + int (*release)(struct PyInterface_Base *intf); + } PyInterface_Base; + + // Type slot definition for PyTypeObject field. + typedef int (*Py_getinterfacefunc)(PyObject *o, PyInterface_Base *intf); + + +The unique identifier for the struct is a 64-bit integer defined as a macro (to +ensure that compiled extension modules embed the value, rather than trying to +discover it at runtime). The top 32 bits are the namespace, and implementers +defining their own structs should choose a unique value for themselves. Zero +is reserved for CPython. + +The interface name is to identify the struct layout, and so any defined object +can reuse an interface name from another namespace, provided the struct matches. +This is intentional, as it allows third-party types to implement the same +interfaces as core types without having to rely on sharing the implementation. +To be clear, an interface defined for CPython may be used by other extension +modules without changing the name or the name's namespace. + +For example, consider a hypothetical interface to implement +:c:func:`!PyDict_GetItemString`. The core ``dict`` type may do internal +optimizations to locate entries by string key, while an external type can use +the same interface to do their own optimization. To the caller, it appears to +use the same interface, and so the caller is compatible with a broader range of +types than if it were using (for example) CPython's concrete object APIs. + +Interface names cannot be removed from headers at any time, and structure +definitions can only be removed when all Stable ABI versions supporting them are +fully retired. However, objects may stop returning a particular interface if it +is no longer recommended or reliable, even if earlier releases did return them. +Runtime deprecation warnings may be used if appropriate, no particular rule is +specified. + +Interface structures are fixed and cannot be changed. When a change is required, +a new interface should be defined with a new name. The fields added to a struct +for an interface are public API and should be documented. Fields that are not +intended for direct use should begin with an underscore, but otherwise cannot be +made "private". Interfaces may provide a mix of data and function pointers, or +use strong ``PyObject *`` references to avoid race conditions. + +After retrieving an interface, the interface must remain valid until it is +released, even if the reference to the object is freed. The behaviour of the +interface may handle changes to the underlying object however appropriate, but +probably should document its choices. It would not be unreasonable to have two +similar interfaces that handle these kind of changes differently (e.g. one +interface that locks the object for the lifetime of the interface, while another +does not). + +The process of adding new Limited APIs changes somewhat: rather than having an +ABI that grows with each release, new APIs may be added as a real function for +when the Limited API is not in use, but should be added as a static inline +function for the Limited API. This static inline function should use an +interface to detect the functionality at runtime, and include an abstract +fallback or suitable exception. + +This means that consumers can adopt a new API immediately, compile for the +Limited API with the latest release, and retain binary compatibility for all +releases that support the same Stable ABI. + +At the next Stable ABI freeze, the API can either be promoted to the new Stable +ABI/Limited API as a real function, or retained as an interface. + + +Backwards Compatibility +======================= + +Limited API 3.15 will not be backwards-compatible with older CPython releases, +due to removed structs and functions. + +Extension authors who cannot switch may continue to use Limited API 3.14 +and below for use on the GIL enabled build. + +No changes to ``abi3`` will be made to the GIL enabled build, and all existing +symbols will remain available, even though these are no longer available under +new Stable ABIs. + +Making free-threaded builds the default/only release for CPython will be a +backwards-incompatible change, and extension authors will need to have migrated. + + +Security Implications +===================== + +None known. + + +How to Teach This +================= + +The native ABI of Python can be described as a periodically updated standard or +specification, identified by year, similar to other languages. Any extension +module can use this ABI, and declares which ABI they expect as part of their +distribution information. Any Python implementation may choose to support a +particular ABI version, and any extension also supporting that version should be +usable. + +Migrating from ``abi3`` to a new ABI may involve source code changes, but can +be treated as a one-time task. In many, if not most, cases, source code will be +compatible with both ``abi3`` and the new ABI, simplifying production of builds +for old releases and current releases. In general, ``abi3`` builds should be +built with the oldest supported CPython runtime, and new ABI builds should be +built with the latest CPython runtime (or another compatible runtime). + +Migrating from one ABI (e.g. ``abi2026``) to the next (e.g. ``abi2031``) should +be a manual task. There is enough overlap between ABI updates that most projects +only need to support one at a time, and can update all of their builds at once +if their own support matrix allows. There is no expectation for package +maintainers to immediately support each new ABI. + +Forward-and-backward compatibility is ensured by dynamic interface detection. +Code using recently added limited API functions will run on older releases, +though potentially at lower performance. See the documentation for new functions +to find information about any Limited API-specific nuances. + +Non-C callers should use the interfaces mechanism directly to get access to new +features without artificially limiting their compatibility to newer releases. +The names and struct layouts of interfaces are guaranteed stable for all time, +though it should not be assumed that an interface will be available for all +time, and suitable fallback code (either an alternative implementation or error +handling) should be included. + + +Reference Implementation +======================== + +See :pep:`803` for links to reference implementations for the aspects inherited +from that PEP. + +The reference implementation of interfaces is +`zooba/cpython#44 `__. + + +Rejected Ideas +============== + +[See discussion for now.] + + +Open Issues +=========== + +[See discussion for now.] + + +Footnotes +========= + +.. [#Consumers] We use the word "consumer" to include anyone who codes against + ("consumes") the C API. This is predominantly developers of native extension + modules (sometimes "package developers"), but also includes developers of + apps that host CPython and those who interact at runtime with CPython's + interfaces (such as debuggers or cross-runtime proxy tools). + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.