diff --git a/peps/pep-0780.rst b/peps/pep-0780.rst index c9af953cf68..b488e3872cc 100644 --- a/peps/pep-0780.rst +++ b/peps/pep-0780.rst @@ -23,37 +23,31 @@ Abstract ======== This PEP defines using ABI features as environment markers for project -dependencies, through a new ``sys_abi_features`` environment marker and -:data:`!sys.abi_features` attribute in the :mod:`sys` module. -:pep:`508` (later moved to :ref:`packaging:dependency-specifiers`) introduced -environment markers to specify dependencies based on rules that describe -when the dependency should be used. -This PEP extends the environment markers to allow specifying dependencies -based on specific ABI features of the Python interpreter. -For this, it defines a set of `ABI Features`_ and specifies how they are made -available via an `addition to the Python Standard Library `_ -in the form of a new attribute :data:`!sys.abi_features`, as well as for -`environment markers `_ as a new marker variable, -``sys_abi_features``. +dependencies, through a new ``sys_abi_features`` environment marker. :pep:`508` +(later moved to :ref:`packaging:dependency-specifiers`) introduced environment +markers to specify dependencies based on rules that describe when the +dependency should be used. This PEP extends the environment markers to allow +specifying dependencies based on specific ABI features of the Python +interpreter. For this, it defines a set of `ABI Features`_ and specifies how +they are made available for `environment markers `_ +as a new marker variable, ``sys_abi_features``. Motivation ========== In 2015, :pep:`508` established environment markers to specify dependencies -based on environment conditions. -The development of free-threaded CPython [#python-free-threading]_ -has underlined the need for an environment marker to discriminate between -different ABI features that the interpreter was built with. -For example, currently there is -no way to distinguish between a GIL-enabled and a free-threaded CPython -interpreter with an environment marker. This leads to real world issues for the -adoption of free-threading and its incremental rollout. When a Python package -is being made compatible with free-threaded CPython, it also needs all its -build and runtime dependencies to be compatible. Capturing the first version of -a dependency that is compatible precisely in metadata is currently not -possible, and increasing the minimum version of a dependency also for the -GIL-enabled build is usually undesirable since it unnecessarily limits -compatibility between packages. +based on environment conditions. The development of free-threaded CPython +[#python-free-threading]_ has underlined the need for an environment marker to +discriminate between different ABI features that the interpreter was built +with. For example, currently there is no way to distinguish between a +GIL-enabled and a free-threaded CPython interpreter with an environment marker. +This leads to real world issues for the adoption of free-threading and its +incremental rollout. When a Python package is being made compatible with +free-threaded CPython, it also needs all its build and runtime dependencies to +be compatible. Capturing the first version of a dependency that is compatible +precisely in metadata is currently not possible, and increasing the minimum +version of a dependency also for the GIL-enabled build is usually undesirable +since it unnecessarily limits compatibility between packages. Some concrete examples of such issues have been discussed in the `Environment marker for free-threading`_ Discourse thread: @@ -61,9 +55,8 @@ marker for free-threading`_ Discourse thread: - Cython has (experimental) support for free-threading only in its master branch, and is used by a lot of projects that already publish ``cp313t`` wheels. Picking up the wrong Cython version is causing a lot of obscure build - failures and runtime crashes. It would be beneficial if the metadata - could express that (c.f. `Require Cython Pre-release for Free-Threaded - Python`_). + failures and runtime crashes. It would be beneficial if the metadata could + express that (c.f. `Require Cython Pre-release for Free-Threaded Python`_). - CFFI has no support for free-threading yet, and `Armin Rigo, one of the maintainers, has stated `__ @@ -79,11 +72,12 @@ marker for free-threading`_ Discourse thread: While these concrete examples may be addressed later this year by Cython and CFFI making compatible releases, the same issue is going to repeat further up the stack. The free-threading rollout is expected to take several years, and an -environment marker for free-threading will make that rollout significantly easier. +environment marker for free-threading will make that rollout significantly +easier. Another important ABI feature that is not yet covered by environment markers is the bitness of the interpreter. In most cases, the ``sys_platform`` or -``platform_system`` markers are enough, because there's only a single bitness +``platform_system`` markers are enough, because there is only a single bitness in use per platform. This is not the case on Windows however: both 32-bit and 64-bit Python interpreters are widely used on x86-64 Windows. Not being able to distinguish between the two may be relevant for packages that provide compiled @@ -100,31 +94,18 @@ Rationale ========= The intention of this PEP is to introduce its core features with minimal impact -on the existing ecosystem. -The existing grammar proposed in :pep:`508` lends itself to a straightforward -extension to include the new environment marker. - -Adding ``sys.abi_features`` to the Python Standard Library ----------------------------------------------------------- - -The Python standard library already has mechanisms to provide information about -the ABI of the interpreter, like the :data:`!sys.abiflags` attribute and the -:mod:`sysconfig` module, but these do not provide all the `ABI Features`_ that -are relevant for environment markers. -The new attribute :data:`!sys.abi_features` is a natural extension to the -existing mechanisms, and it is a simple and intuitive way to provide this -information, including the information from :data:`!sys.abiflags`. +on the existing ecosystem. The existing grammar proposed in :pep:`508` lends +itself to a straightforward extension to include the new environment marker. A Forward Looking View on Free-Threaded Python ---------------------------------------------- -:pep:`703`, the accepted proposal for free threading, states that the -rollout of free-threaded Python should be gradual, which has been clarified -by the Python Steering Council in `the PEP 703 acceptance post`_ to mean a -three stage process over multiple releases. It is therefore important to make -sure that the mechanisms in this PEP are useable for Python interpreters where -either free-threading or non-free-threading could be the default or the only -option. +:pep:`703`, the accepted proposal for free threading, states that the rollout +of free-threaded Python should be gradual, which has been clarified by the +Python Steering Council in `the PEP 703 acceptance post`_ to mean a three stage +process over multiple releases. It is therefore important to make sure that the +mechanisms in this PEP are useable for Python interpreters where either +free-threading or non-free-threading could be the default or the only option. At the time of writing, free-threaded Python is in Phase I: experimental phase. In this phase, there is an acute need for the proposed environment markers to @@ -135,11 +116,11 @@ As the number of packages with support increases, and particularly during Phase II: Supported-but-not-default phase, we still anticipate a strong need for the environment markers to help with the transition. -As free-threaded Python enters into Phase III: Default phase, the need for -the environment markers will decrease, though at this point it is not clear -that the GIL-enabled Python will be completely phased out (it may remain -available as a non standard build option). If it persists, the inverse need for -the ABI feature detection may arise. +As free-threaded Python enters into Phase III: Default phase, the need for the +environment markers will decrease, though at this point it is not clear that +the GIL-enabled Python will be completely phased out (it may remain available +as a non standard build option). If it persists, the inverse need for the ABI +feature detection may arise. Indeed, in all three phases it may be necessary for package authors to choose specific versions of their dependencies based on the ABI features, with a shift @@ -169,18 +150,17 @@ ABI Features ------------ ABI features are intrinsic properties of the Python interpreter, expressed as -simple, understandable strings. -However, not all features are equally applicable to all Python interpreters or -Python versions. For example, the distinction between free-threaded and -GIL-enabled interpreters is only relevant for CPython 3.13 onwards, but the bitness -of the interpreter is relevant for all interpreters. - -All interpreters MUST handle the following ABI features as stated. -ABI features that are restricted to particular interpreters MUST NOT be -provided by other interpreters. -The features are subdivided into groups and for each group there MUST be -exactly one feature present, except when the group is marked as optional, in -which case there MUST be at most one feature present. +simple, understandable strings. However, not all features are equally +applicable to all Python interpreters or Python versions. For example, the +distinction between free-threaded and GIL-enabled interpreters is only relevant +for CPython 3.13 onwards, but the bitness of the interpreter is relevant for +all interpreters. + +All interpreters MUST handle the following ABI features as stated. ABI features +that are restricted to particular interpreters MUST NOT be provided by other +interpreters. The features are subdivided into groups and for each group there +MUST be exactly one feature present, except when the group is marked as +optional, in which case there MUST be at most one feature present. ``free-threading`` or ``gil-enabled`` (only CPython) If the Python interpreter is free-threaded, the ``free-threading`` feature @@ -191,42 +171,14 @@ which case there MUST be at most one feature present. ``debug`` (only CPython, optional) This ABI feature is reserved for the ``--with-pydebug`` build of CPython. If the interpreter is a CPython interpreter with ``Py_DEBUG`` capabilities, - the ``debug`` feature MUST be present. - On POSIX systems, this corresponds to ``"d" in sys.abiflags``. + the ``debug`` feature MUST be present. On POSIX systems, this corresponds + to the Python expression ``"d" in sys.abiflags``. ``32-bit`` or ``64-bit`` (optional) The bitness of the interpreter, that is, whether it is a 32-bit or 64-bit build [#bitness]_. If the bitness is unknown or neither 32-bit nor 64-bit, this feature MUST NOT be present. -.. _pep-780-sys.abi_features: - -The :data:`!sys.abi_features` Attribute ---------------------------------------- - -Making the ABI features available in an easily accessible, expressive, -standardized way is useful beyond the scope of environment markers. -For example, ``"32-bit" in sys.abi_features`` is much more expressive than the -current standard test of comparing ``sys.maxsize`` with ``2**32``, which can -be found more than ten thousand times on GitHub. -If one wants to determine whether the interpreter is a debug build, there is -currently no standardized, cross platform way to do so, and ``sys.abiflags`` is -not available on Windows. -Hence, the `ABI Features`_ listed above are added to the Python standard -library. - -Since they are all the result of compile time choices describing basic features -of the interpreter, the most intuitive place to put them is in ``sys``. -Since there is no intrinsic order, nor a possibility for duplication, they are -added as a ``frozenset`` of strings. - -All Python interpreters MUST provide the ``sys.abi_features`` attribute as a -``frozenset`` of strings, which MUST contain only the `ABI Features`_ that are -defined in this PEP or in a subsequent PEP. - -An example value would be ``sys.abi_features == frozenset({"free-threading", -"debug", "32-bit"})`` on a free-threaded debug build for win32. - .. _pep-780-sys_abi_features: The ``sys_abi_features`` Environment Marker @@ -234,8 +186,7 @@ The ``sys_abi_features`` Environment Marker To make ABI features available in dependency specifications, a new environment marker variable, ``sys_abi_features``, is added to the format of dependency -specifiers with the same semantics as the ``sys.abi_features`` attribute -proposed above. +specifiers. To do this, we need to extend the grammar laid out in :pep:`508` and maintained in the :ref:`packaging:dependency-specifiers` and document the possible values. @@ -262,26 +213,21 @@ Like the grammar, also the overview table of environment markers in - Python equivalent - Sample values * - ``sys_abi_features`` - - ``sys.abi_features`` [#sys-abi-features]_ - - ``frozenset()``, ``frozenset({'free-threading', '64-bit'})``, - ``frozenset({'gil-enabled', 'debug', '32-bit'})`` + - no direct equivalent available + - ``{'free-threading', '64-bit'}``, + ``{'gil-enabled', 'debug', '32-bit'}`` With these additions, ABI features can be used in dependency specifications via the ``in`` operator to test for the presence of a feature, or the ``not in`` operator to test for the absence of a feature. -Note that the presence of ``sys.abi_features`` in the Python standard library -makes implementation particularly easy for new Python versions, but its absence -in older versions does not prevent the implementation of the new environment -markers, as demonstrated in the `Reference Implementation`_. - Examples ======== Require Cython Pre-release for Free-Threaded Python ---------------------------------------------------- -To require a pre-release of Cython only for a free-threaded Python -interpreter, the following dependency specification can be used:: +To require a pre-release of Cython only for a free-threaded Python interpreter, +the following dependency specification can be used:: cython >3.1.0a1; "free-threading" in sys_abi_features cython ==3.0.*; "free-threading" not in sys_abi_features @@ -307,9 +253,9 @@ This is a pure extension to the existing environment markers and does not affect existing environment markers or dependency specifications, hence there are no direct backwards compatibility concerns. -However, the introduction of the feature has implications for a -number of ecosystem tools, especially those which attempt to support -examination of data in ``pyproject.toml`` and ``requirements.txt``. +However, the introduction of the feature has implications for a number of +ecosystem tools, especially those which attempt to support examination of data +in ``pyproject.toml`` and ``requirements.txt``. Audit and Update Tools ---------------------- @@ -318,10 +264,9 @@ A wide range of tools understand Python dependency data as expressed in ``requirements.txt`` files. (e.g., Dependabot, Tidelift, etc) Such tools inspect dependency data and, in some cases, offer tool-assisted or -fully automated updates. -It is our expectation that no such tools would support the new environment -markers at first, and broad ecosystem support could take many months or even -some number of years to arrive. +fully automated updates. It is our expectation that no such tools would support +the new environment markers at first, and broad ecosystem support could take +many months or even some number of years to arrive. As a result, users of the new environment markers would experience a degradation in their workflows and tool support at the time that they start @@ -343,22 +288,15 @@ dependencies may be specified here, just as they may be specified in How to Teach This ================= -The use of environment markers is well established and communicated chiefly -in :ref:`packaging:dependency-specifiers`. -The new environment marker can be introduced in the same document. -Additionally, both for package authors and users, free-threading specific -guidance can be provided at the `Python free-threading guide`_. -The new ``sys.abi_features`` attribute will be documented in the Python -standard library documentation. - +The use of environment markers is well established and communicated chiefly in +:ref:`packaging:dependency-specifiers`. The new environment marker can be +introduced in the same document. Additionally, both for package authors and +users, free-threading specific guidance can be provided at the +`Python free-threading guide`_. Reference Implementation ======================== -The reference implementation for the ``sys.abi_features`` attribute can be -found in `Add abi_features to sys -`__. - The reference implementation for the environment markers is available in a fork of the ``packaging`` library at `Environment markers for ABI features `__. @@ -367,8 +305,8 @@ of the ``packaging`` library at `Environment markers for ABI features also available. Since ``pip`` uses a vendored copy of ``packaging`` internally, we also provide -`a patched version of pip`__, which replaces the vendored ``packaging`` with the -reference implementation linked above. +`a patched version of pip`__, which replaces the vendored ``packaging`` with +the reference implementation linked above. __ https://github.com/zklaus/pip/tree/env-marker-free-threading @@ -382,6 +320,7 @@ In an early discussion of the topic (`Environment marker for free-threading`_), the idea of a general extension mechanism for environment markers was brought up. While it is appealing to forego a whole PEP process should the need for new environment markers arise in the future, there are two main challenges. + First, a completely dynamic mechanism would present difficulties for tools that rely on static analysis of dependency specifications. @@ -392,11 +331,6 @@ Second, the introduction of a dynamic mechanism would require a more complex implementation in the packaging library, which would be a significant departure from the current approach. -Having said that, the new ``sys.abi_features`` attribute provides a natural -extension point for any new ABI features, even if specific to a subset of -interpreters, should the need arise to add such new features with a subsequent -PEP. - Open Issues =========== @@ -409,16 +343,15 @@ Other Tooling ------------- The reference implementation is based on the ``packaging`` library and ``pip``. We have confirmed that this allows for building and installing packages with -several build backends. -It is possible that other tools should be added to the reference -implementation. +several build backends. It is possible that other tools should be added to the +reference implementation. Footnotes ========= -.. [#python-free-threading] Python experimental support for free threading - is available in Python 3.13 and later. For more information, see `Python +.. [#python-free-threading] Python experimental support for free threading is + available in Python 3.13 and later. For more information, see `Python experimental support for free threading`_. .. [#bitness] While there are some related environment markers available, such @@ -426,11 +359,6 @@ Footnotes not sufficient to reliably determine the bitness of the interpreter, particularly on platforms that allow the execution of either kind of binary. -.. [#sys-abi-features] This is contingent on the availability of the - ``sys.abi_features`` attribute in the Python standard library. In - versions that don't provide this attribute, the marker will be constructed - from other available information, such as the ``sys.abiflags`` string. - .. _Python experimental support for free threading: https://docs.python.org/3/howto/free-threading-python.html .. _Python free-threading guide: https://py-free-threading.github.io/ @@ -442,9 +370,9 @@ Footnotes Acknowledgements ================ -Thanks to Filipe Laíns for the suggestion of the ``abi_features`` attribute -and to Stephen Rosen for the Backwards Compatibility section of :pep:`735`, -which served as a template for the corresponding section in this PEP. +Thanks to Filipe Laíns for the suggestion of the ``abi_features`` attribute and +to Stephen Rosen for the Backwards Compatibility section of :pep:`735`, which +served as a template for the corresponding section in this PEP. Copyright =========