From f8e07039e904d8a591b411b07affe08f774cbe53 Mon Sep 17 00:00:00 2001 From: Klaus Zimmermann Date: Mon, 14 Apr 2025 13:28:28 -0600 Subject: [PATCH 1/2] Remove CPython addition from the PEP --- peps/pep-0780.rst | 119 +++++++++++----------------------------------- 1 file changed, 27 insertions(+), 92 deletions(-) diff --git a/peps/pep-0780.rst b/peps/pep-0780.rst index c9af953cf68..e801035bd2c 100644 --- a/peps/pep-0780.rst +++ b/peps/pep-0780.rst @@ -23,18 +23,15 @@ Abstract ======== This PEP defines using ABI features as environment markers for project -dependencies, through a new ``sys_abi_features`` environment marker and -:data:`!sys.abi_features` attribute in the :mod:`sys` module. +dependencies, through a new ``sys_abi_features`` environment marker. :pep:`508` (later moved to :ref:`packaging:dependency-specifiers`) introduced environment markers to specify dependencies based on rules that describe when the dependency should be used. This PEP extends the environment markers to allow specifying dependencies based on specific ABI features of the Python interpreter. For this, it defines a set of `ABI Features`_ and specifies how they are made -available via an `addition to the Python Standard Library `_ -in the form of a new attribute :data:`!sys.abi_features`, as well as for -`environment markers `_ as a new marker variable, -``sys_abi_features``. +available for `environment markers `_ as a new +marker variable, ``sys_abi_features``. Motivation ========== @@ -44,16 +41,15 @@ based on environment conditions. The development of free-threaded CPython [#python-free-threading]_ has underlined the need for an environment marker to discriminate between different ABI features that the interpreter was built with. -For example, currently there is -no way to distinguish between a GIL-enabled and a free-threaded CPython -interpreter with an environment marker. This leads to real world issues for the -adoption of free-threading and its incremental rollout. When a Python package -is being made compatible with free-threaded CPython, it also needs all its -build and runtime dependencies to be compatible. Capturing the first version of -a dependency that is compatible precisely in metadata is currently not -possible, and increasing the minimum version of a dependency also for the -GIL-enabled build is usually undesirable since it unnecessarily limits -compatibility between packages. +For example, currently there is no way to distinguish between a GIL-enabled and +a free-threaded CPython interpreter with an environment marker. This leads to +real world issues for the adoption of free-threading and its incremental +rollout. When a Python package is being made compatible with free-threaded +CPython, it also needs all its build and runtime dependencies to be compatible. +Capturing the first version of a dependency that is compatible precisely in +metadata is currently not possible, and increasing the minimum version of a +dependency also for the GIL-enabled build is usually undesirable since it +unnecessarily limits compatibility between packages. Some concrete examples of such issues have been discussed in the `Environment marker for free-threading`_ Discourse thread: @@ -61,9 +57,8 @@ marker for free-threading`_ Discourse thread: - Cython has (experimental) support for free-threading only in its master branch, and is used by a lot of projects that already publish ``cp313t`` wheels. Picking up the wrong Cython version is causing a lot of obscure build - failures and runtime crashes. It would be beneficial if the metadata - could express that (c.f. `Require Cython Pre-release for Free-Threaded - Python`_). + failures and runtime crashes. It would be beneficial if the metadata could + express that (c.f. `Require Cython Pre-release for Free-Threaded Python`_). - CFFI has no support for free-threading yet, and `Armin Rigo, one of the maintainers, has stated `__ @@ -79,11 +74,12 @@ marker for free-threading`_ Discourse thread: While these concrete examples may be addressed later this year by Cython and CFFI making compatible releases, the same issue is going to repeat further up the stack. The free-threading rollout is expected to take several years, and an -environment marker for free-threading will make that rollout significantly easier. +environment marker for free-threading will make that rollout significantly +easier. Another important ABI feature that is not yet covered by environment markers is the bitness of the interpreter. In most cases, the ``sys_platform`` or -``platform_system`` markers are enough, because there's only a single bitness +``platform_system`` markers are enough, because there is only a single bitness in use per platform. This is not the case on Windows however: both 32-bit and 64-bit Python interpreters are widely used on x86-64 Windows. Not being able to distinguish between the two may be relevant for packages that provide compiled @@ -104,17 +100,6 @@ on the existing ecosystem. The existing grammar proposed in :pep:`508` lends itself to a straightforward extension to include the new environment marker. -Adding ``sys.abi_features`` to the Python Standard Library ----------------------------------------------------------- - -The Python standard library already has mechanisms to provide information about -the ABI of the interpreter, like the :data:`!sys.abiflags` attribute and the -:mod:`sysconfig` module, but these do not provide all the `ABI Features`_ that -are relevant for environment markers. -The new attribute :data:`!sys.abi_features` is a natural extension to the -existing mechanisms, and it is a simple and intuitive way to provide this -information, including the information from :data:`!sys.abiflags`. - A Forward Looking View on Free-Threaded Python ---------------------------------------------- @@ -169,11 +154,11 @@ ABI Features ------------ ABI features are intrinsic properties of the Python interpreter, expressed as -simple, understandable strings. -However, not all features are equally applicable to all Python interpreters or -Python versions. For example, the distinction between free-threaded and -GIL-enabled interpreters is only relevant for CPython 3.13 onwards, but the bitness -of the interpreter is relevant for all interpreters. +simple, understandable strings. However, not all features are equally +applicable to all Python interpreters or Python versions. For example, the +distinction between free-threaded and GIL-enabled interpreters is only relevant +for CPython 3.13 onwards, but the bitness of the interpreter is relevant for +all interpreters. All interpreters MUST handle the following ABI features as stated. ABI features that are restricted to particular interpreters MUST NOT be @@ -199,34 +184,6 @@ which case there MUST be at most one feature present. build [#bitness]_. If the bitness is unknown or neither 32-bit nor 64-bit, this feature MUST NOT be present. -.. _pep-780-sys.abi_features: - -The :data:`!sys.abi_features` Attribute ---------------------------------------- - -Making the ABI features available in an easily accessible, expressive, -standardized way is useful beyond the scope of environment markers. -For example, ``"32-bit" in sys.abi_features`` is much more expressive than the -current standard test of comparing ``sys.maxsize`` with ``2**32``, which can -be found more than ten thousand times on GitHub. -If one wants to determine whether the interpreter is a debug build, there is -currently no standardized, cross platform way to do so, and ``sys.abiflags`` is -not available on Windows. -Hence, the `ABI Features`_ listed above are added to the Python standard -library. - -Since they are all the result of compile time choices describing basic features -of the interpreter, the most intuitive place to put them is in ``sys``. -Since there is no intrinsic order, nor a possibility for duplication, they are -added as a ``frozenset`` of strings. - -All Python interpreters MUST provide the ``sys.abi_features`` attribute as a -``frozenset`` of strings, which MUST contain only the `ABI Features`_ that are -defined in this PEP or in a subsequent PEP. - -An example value would be ``sys.abi_features == frozenset({"free-threading", -"debug", "32-bit"})`` on a free-threaded debug build for win32. - .. _pep-780-sys_abi_features: The ``sys_abi_features`` Environment Marker @@ -234,8 +191,7 @@ The ``sys_abi_features`` Environment Marker To make ABI features available in dependency specifications, a new environment marker variable, ``sys_abi_features``, is added to the format of dependency -specifiers with the same semantics as the ``sys.abi_features`` attribute -proposed above. +specifiers. To do this, we need to extend the grammar laid out in :pep:`508` and maintained in the :ref:`packaging:dependency-specifiers` and document the possible values. @@ -262,19 +218,14 @@ Like the grammar, also the overview table of environment markers in - Python equivalent - Sample values * - ``sys_abi_features`` - - ``sys.abi_features`` [#sys-abi-features]_ - - ``frozenset()``, ``frozenset({'free-threading', '64-bit'})``, - ``frozenset({'gil-enabled', 'debug', '32-bit'})`` + - no direct equivalent available + - ``{'free-threading', '64-bit'}``, + ``{'gil-enabled', 'debug', '32-bit'}`` With these additions, ABI features can be used in dependency specifications via the ``in`` operator to test for the presence of a feature, or the ``not in`` operator to test for the absence of a feature. -Note that the presence of ``sys.abi_features`` in the Python standard library -makes implementation particularly easy for new Python versions, but its absence -in older versions does not prevent the implementation of the new environment -markers, as demonstrated in the `Reference Implementation`_. - Examples ======== @@ -348,17 +299,10 @@ in :ref:`packaging:dependency-specifiers`. The new environment marker can be introduced in the same document. Additionally, both for package authors and users, free-threading specific guidance can be provided at the `Python free-threading guide`_. -The new ``sys.abi_features`` attribute will be documented in the Python -standard library documentation. - Reference Implementation ======================== -The reference implementation for the ``sys.abi_features`` attribute can be -found in `Add abi_features to sys -`__. - The reference implementation for the environment markers is available in a fork of the ``packaging`` library at `Environment markers for ABI features `__. @@ -382,6 +326,7 @@ In an early discussion of the topic (`Environment marker for free-threading`_), the idea of a general extension mechanism for environment markers was brought up. While it is appealing to forego a whole PEP process should the need for new environment markers arise in the future, there are two main challenges. + First, a completely dynamic mechanism would present difficulties for tools that rely on static analysis of dependency specifications. @@ -392,11 +337,6 @@ Second, the introduction of a dynamic mechanism would require a more complex implementation in the packaging library, which would be a significant departure from the current approach. -Having said that, the new ``sys.abi_features`` attribute provides a natural -extension point for any new ABI features, even if specific to a subset of -interpreters, should the need arise to add such new features with a subsequent -PEP. - Open Issues =========== @@ -426,11 +366,6 @@ Footnotes not sufficient to reliably determine the bitness of the interpreter, particularly on platforms that allow the execution of either kind of binary. -.. [#sys-abi-features] This is contingent on the availability of the - ``sys.abi_features`` attribute in the Python standard library. In - versions that don't provide this attribute, the marker will be constructed - from other available information, such as the ``sys.abiflags`` string. - .. _Python experimental support for free threading: https://docs.python.org/3/howto/free-threading-python.html .. _Python free-threading guide: https://py-free-threading.github.io/ From 46450272deb66b6234b05388b5a5da3a0acf0170 Mon Sep 17 00:00:00 2001 From: Klaus Zimmermann Date: Mon, 14 Apr 2025 14:02:14 -0600 Subject: [PATCH 2/2] Copy editing --- peps/pep-0780.rst | 131 ++++++++++++++++++++++------------------------ 1 file changed, 62 insertions(+), 69 deletions(-) diff --git a/peps/pep-0780.rst b/peps/pep-0780.rst index e801035bd2c..b488e3872cc 100644 --- a/peps/pep-0780.rst +++ b/peps/pep-0780.rst @@ -23,33 +23,31 @@ Abstract ======== This PEP defines using ABI features as environment markers for project -dependencies, through a new ``sys_abi_features`` environment marker. -:pep:`508` (later moved to :ref:`packaging:dependency-specifiers`) introduced -environment markers to specify dependencies based on rules that describe -when the dependency should be used. -This PEP extends the environment markers to allow specifying dependencies -based on specific ABI features of the Python interpreter. -For this, it defines a set of `ABI Features`_ and specifies how they are made -available for `environment markers `_ as a new -marker variable, ``sys_abi_features``. +dependencies, through a new ``sys_abi_features`` environment marker. :pep:`508` +(later moved to :ref:`packaging:dependency-specifiers`) introduced environment +markers to specify dependencies based on rules that describe when the +dependency should be used. This PEP extends the environment markers to allow +specifying dependencies based on specific ABI features of the Python +interpreter. For this, it defines a set of `ABI Features`_ and specifies how +they are made available for `environment markers `_ +as a new marker variable, ``sys_abi_features``. Motivation ========== In 2015, :pep:`508` established environment markers to specify dependencies -based on environment conditions. -The development of free-threaded CPython [#python-free-threading]_ -has underlined the need for an environment marker to discriminate between -different ABI features that the interpreter was built with. -For example, currently there is no way to distinguish between a GIL-enabled and -a free-threaded CPython interpreter with an environment marker. This leads to -real world issues for the adoption of free-threading and its incremental -rollout. When a Python package is being made compatible with free-threaded -CPython, it also needs all its build and runtime dependencies to be compatible. -Capturing the first version of a dependency that is compatible precisely in -metadata is currently not possible, and increasing the minimum version of a -dependency also for the GIL-enabled build is usually undesirable since it -unnecessarily limits compatibility between packages. +based on environment conditions. The development of free-threaded CPython +[#python-free-threading]_ has underlined the need for an environment marker to +discriminate between different ABI features that the interpreter was built +with. For example, currently there is no way to distinguish between a +GIL-enabled and a free-threaded CPython interpreter with an environment marker. +This leads to real world issues for the adoption of free-threading and its +incremental rollout. When a Python package is being made compatible with +free-threaded CPython, it also needs all its build and runtime dependencies to +be compatible. Capturing the first version of a dependency that is compatible +precisely in metadata is currently not possible, and increasing the minimum +version of a dependency also for the GIL-enabled build is usually undesirable +since it unnecessarily limits compatibility between packages. Some concrete examples of such issues have been discussed in the `Environment marker for free-threading`_ Discourse thread: @@ -96,20 +94,18 @@ Rationale ========= The intention of this PEP is to introduce its core features with minimal impact -on the existing ecosystem. -The existing grammar proposed in :pep:`508` lends itself to a straightforward -extension to include the new environment marker. +on the existing ecosystem. The existing grammar proposed in :pep:`508` lends +itself to a straightforward extension to include the new environment marker. A Forward Looking View on Free-Threaded Python ---------------------------------------------- -:pep:`703`, the accepted proposal for free threading, states that the -rollout of free-threaded Python should be gradual, which has been clarified -by the Python Steering Council in `the PEP 703 acceptance post`_ to mean a -three stage process over multiple releases. It is therefore important to make -sure that the mechanisms in this PEP are useable for Python interpreters where -either free-threading or non-free-threading could be the default or the only -option. +:pep:`703`, the accepted proposal for free threading, states that the rollout +of free-threaded Python should be gradual, which has been clarified by the +Python Steering Council in `the PEP 703 acceptance post`_ to mean a three stage +process over multiple releases. It is therefore important to make sure that the +mechanisms in this PEP are useable for Python interpreters where either +free-threading or non-free-threading could be the default or the only option. At the time of writing, free-threaded Python is in Phase I: experimental phase. In this phase, there is an acute need for the proposed environment markers to @@ -120,11 +116,11 @@ As the number of packages with support increases, and particularly during Phase II: Supported-but-not-default phase, we still anticipate a strong need for the environment markers to help with the transition. -As free-threaded Python enters into Phase III: Default phase, the need for -the environment markers will decrease, though at this point it is not clear -that the GIL-enabled Python will be completely phased out (it may remain -available as a non standard build option). If it persists, the inverse need for -the ABI feature detection may arise. +As free-threaded Python enters into Phase III: Default phase, the need for the +environment markers will decrease, though at this point it is not clear that +the GIL-enabled Python will be completely phased out (it may remain available +as a non standard build option). If it persists, the inverse need for the ABI +feature detection may arise. Indeed, in all three phases it may be necessary for package authors to choose specific versions of their dependencies based on the ABI features, with a shift @@ -160,12 +156,11 @@ distinction between free-threaded and GIL-enabled interpreters is only relevant for CPython 3.13 onwards, but the bitness of the interpreter is relevant for all interpreters. -All interpreters MUST handle the following ABI features as stated. -ABI features that are restricted to particular interpreters MUST NOT be -provided by other interpreters. -The features are subdivided into groups and for each group there MUST be -exactly one feature present, except when the group is marked as optional, in -which case there MUST be at most one feature present. +All interpreters MUST handle the following ABI features as stated. ABI features +that are restricted to particular interpreters MUST NOT be provided by other +interpreters. The features are subdivided into groups and for each group there +MUST be exactly one feature present, except when the group is marked as +optional, in which case there MUST be at most one feature present. ``free-threading`` or ``gil-enabled`` (only CPython) If the Python interpreter is free-threaded, the ``free-threading`` feature @@ -176,8 +171,8 @@ which case there MUST be at most one feature present. ``debug`` (only CPython, optional) This ABI feature is reserved for the ``--with-pydebug`` build of CPython. If the interpreter is a CPython interpreter with ``Py_DEBUG`` capabilities, - the ``debug`` feature MUST be present. - On POSIX systems, this corresponds to ``"d" in sys.abiflags``. + the ``debug`` feature MUST be present. On POSIX systems, this corresponds + to the Python expression ``"d" in sys.abiflags``. ``32-bit`` or ``64-bit`` (optional) The bitness of the interpreter, that is, whether it is a 32-bit or 64-bit @@ -231,8 +226,8 @@ Examples Require Cython Pre-release for Free-Threaded Python ---------------------------------------------------- -To require a pre-release of Cython only for a free-threaded Python -interpreter, the following dependency specification can be used:: +To require a pre-release of Cython only for a free-threaded Python interpreter, +the following dependency specification can be used:: cython >3.1.0a1; "free-threading" in sys_abi_features cython ==3.0.*; "free-threading" not in sys_abi_features @@ -258,9 +253,9 @@ This is a pure extension to the existing environment markers and does not affect existing environment markers or dependency specifications, hence there are no direct backwards compatibility concerns. -However, the introduction of the feature has implications for a -number of ecosystem tools, especially those which attempt to support -examination of data in ``pyproject.toml`` and ``requirements.txt``. +However, the introduction of the feature has implications for a number of +ecosystem tools, especially those which attempt to support examination of data +in ``pyproject.toml`` and ``requirements.txt``. Audit and Update Tools ---------------------- @@ -269,10 +264,9 @@ A wide range of tools understand Python dependency data as expressed in ``requirements.txt`` files. (e.g., Dependabot, Tidelift, etc) Such tools inspect dependency data and, in some cases, offer tool-assisted or -fully automated updates. -It is our expectation that no such tools would support the new environment -markers at first, and broad ecosystem support could take many months or even -some number of years to arrive. +fully automated updates. It is our expectation that no such tools would support +the new environment markers at first, and broad ecosystem support could take +many months or even some number of years to arrive. As a result, users of the new environment markers would experience a degradation in their workflows and tool support at the time that they start @@ -294,11 +288,11 @@ dependencies may be specified here, just as they may be specified in How to Teach This ================= -The use of environment markers is well established and communicated chiefly -in :ref:`packaging:dependency-specifiers`. -The new environment marker can be introduced in the same document. -Additionally, both for package authors and users, free-threading specific -guidance can be provided at the `Python free-threading guide`_. +The use of environment markers is well established and communicated chiefly in +:ref:`packaging:dependency-specifiers`. The new environment marker can be +introduced in the same document. Additionally, both for package authors and +users, free-threading specific guidance can be provided at the +`Python free-threading guide`_. Reference Implementation ======================== @@ -311,8 +305,8 @@ of the ``packaging`` library at `Environment markers for ABI features also available. Since ``pip`` uses a vendored copy of ``packaging`` internally, we also provide -`a patched version of pip`__, which replaces the vendored ``packaging`` with the -reference implementation linked above. +`a patched version of pip`__, which replaces the vendored ``packaging`` with +the reference implementation linked above. __ https://github.com/zklaus/pip/tree/env-marker-free-threading @@ -349,16 +343,15 @@ Other Tooling ------------- The reference implementation is based on the ``packaging`` library and ``pip``. We have confirmed that this allows for building and installing packages with -several build backends. -It is possible that other tools should be added to the reference -implementation. +several build backends. It is possible that other tools should be added to the +reference implementation. Footnotes ========= -.. [#python-free-threading] Python experimental support for free threading - is available in Python 3.13 and later. For more information, see `Python +.. [#python-free-threading] Python experimental support for free threading is + available in Python 3.13 and later. For more information, see `Python experimental support for free threading`_. .. [#bitness] While there are some related environment markers available, such @@ -377,9 +370,9 @@ Footnotes Acknowledgements ================ -Thanks to Filipe Laíns for the suggestion of the ``abi_features`` attribute -and to Stephen Rosen for the Backwards Compatibility section of :pep:`735`, -which served as a template for the corresponding section in this PEP. +Thanks to Filipe Laíns for the suggestion of the ``abi_features`` attribute and +to Stephen Rosen for the Backwards Compatibility section of :pep:`735`, which +served as a template for the corresponding section in this PEP. Copyright =========