From 4c107fe0ae2e59c87558ab5b14f85c3b5ca92e11 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 23 May 2025 14:53:58 +0200 Subject: [PATCH 01/15] Merge pull request #21 from jaimergp/pep-725-vers PEP 725: Add details about `vers` ranges and `abstract/` types --- peps/pep-0725.rst | 271 ++++++++++++++++++++++++++++++---------------- 1 file changed, 175 insertions(+), 96 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 7c9d83f949b..227d04970a6 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -1,6 +1,7 @@ PEP: 725 Title: Specifying external dependencies in pyproject.toml Author: Pradyun Gedam , + Jaime Rodríguez-Guerra , Ralf Gommers Discussions-To: https://discuss.python.org/t/31888 Status: Draft @@ -26,7 +27,7 @@ are for specifying three types of dependencies: 3. ``dependencies``, needed at runtime on the host machine but not needed at build time. Cross compilation is taken into account by distinguishing build and host dependencies. -Optional build-time and runtime dependencies are supported too, in a manner analogies +Optional build-time and runtime dependencies are supported too, in a manner analogous to how that is supported in the ``[project]`` table. @@ -41,14 +42,14 @@ this PEP are to: - Enable tools to automatically map external dependencies to packages in other packaging repositories, -- Make it possible to include needed dependencies in error messages emitting by +- Make it possible to include needed dependencies in error messages emitted by Python package installers and build frontends, - Provide a canonical place for package authors to record this dependency information. -Packaging ecosystems like Linux distros, Conda, Homebrew, Spack, and Nix need +Packaging ecosystems like Linux distros, conda, Homebrew, Spack, and Nix need full sets of dependencies for Python packages, and have tools like pyp2spec_ -(Fedora), Grayskull_ (Conda), and dh_python_ (Debian) which attempt to +(Fedora), Grayskull_ (conda), and dh_python_ (Debian) which attempt to automatically generate dependency metadata for their own package managers from the metadata in upstream Python packages. External dependencies are currently handled manually, because there is no metadata for this in ``pyproject.toml`` or any other @@ -101,14 +102,14 @@ Multiple types of external dependencies can be distinguished: concrete packages. E.g., a C++ compiler, BLAS, LAPACK, OpenMP, MPI. Concrete packages are straightforward to understand, and are a concept present -in virtually every package management system. Virtual packages are a concept +in every package management system. Virtual packages are a concept also present in a number of packaging systems -- but not always, and the -details of their implementation varies. +details of their implementation vary. Cross compilation ----------------- -Cross compilation is not yet (as of August 2023) well-supported by stdlib +Cross compilation is not yet (as of May 2025) well-supported by stdlib modules and ``pyproject.toml`` metadata. It is however important when translating external dependencies to those of other packaging systems (with tools like ``pyp2spec``). Introducing support for cross compilation immediately @@ -121,15 +122,17 @@ Terminology This PEP uses the following terminology: - *build machine*: the machine on which the package build process is being - executed + executed. - *host machine*: the machine on which the produced artifact will be installed - and run + and run. - *build dependency*: dependency for building the package that needs to be present at build time and itself was built for the build machine's OS and - architecture + architecture. - *host dependency*: dependency for building the package that needs to be present at build time and itself was built for the host machine's OS and - architecture + architecture. +- *runtime dependency*: a library or other software component needed when + the Python package is used after it's installed. Note that this terminology is not consistent across build and packaging tools, so care must be taken when comparing build/host dependencies in @@ -137,7 +140,7 @@ so care must be taken when comparing build/host dependencies in Note that "target machine" or "target dependency" is not used in this PEP. That is typically only relevant for cross-compiling compilers or other such advanced -scenarios [#gcc-cross-terminology]_, [#meson-cross]_ - this is out of scope for +scenarios [#gcc-cross-terminology]_, [#meson-cross]_ -- this is out of scope for this PEP. Finally, note that while "dependency" is the term most widely used for packages @@ -149,7 +152,7 @@ Build and host dependencies ''''''''''''''''''''''''''' Clear separation of metadata associated with the definition of build and target -platforms, rather than assuming that build and target platform will always be +platforms, rather than assuming that build and host platform will always be the same, is important [#pypackaging-native-cross]_. Build dependencies are typically run during the build process - they may be @@ -178,22 +181,29 @@ cause of a build failure when a package fails to build from an sdist. Specifying external dependencies -------------------------------- -Concrete package specification through PURL -''''''''''''''''''''''''''''''''''''''''''' +Concrete package specification +'''''''''''''''''''''''''''''' -The two types of concrete packages are supported by PURL_ (Package URL), which -implements a scheme for identifying packages that is meant to be portable +PURLs implement a scheme for identifying packages that is meant to be portable across packaging ecosystems. Its design is:: scheme:type/namespace/name@version?qualifiers#subpath The ``scheme`` component is a fixed string, ``pkg``, and of the other -components only ``type`` and ``name`` are required. As an example, a package -URL for the ``requests`` package on PyPI would be:: +components only ``type`` and ``name`` are required. - pkg:pypi/requests +Since external dependencies are likely to be typed by hand, we propose a PURL +derivative that, in the name of ergonomics and user-friendliness, introduces a +number of changes, discussed in the following sections. In this derivative, +we replace the ``pkg`` scheme with ``dep``. Hence, we will refer to them as +``dep:`` URLs. -Adopting PURL to specify external dependencies in ``pyproject.toml`` solves a +As an example, a ``dep:`` URL for the ``requests`` package on PyPI would be:: + + dep:pypi/requests + # equivalent to pkg:pypi/requests + +Adopting PURL-compatible strings to specify external dependencies in ``pyproject.toml`` solves a number of problems at once - and there are already implementations of the specification in Python and multiple languages. PURL is also already supported by dependency-related tooling like SPDX (see @@ -203,29 +213,70 @@ and the `Sonatype OSS Index `__; not having to wait years before support in such tooling arrives is valuable. For concrete packages without a canonical package manager to refer to, either -``pkg:generic/pkg-name`` can be used, or a direct reference to the VCS system +``dep:generic/dep-name`` can be used, or a direct reference to the VCS system that the package is maintained in (e.g., -``pkg:github/user-or-org-name/pkg-name``). Which of these is more appropriate -is situation-dependent. This PEP recommends using ``pkg:generic`` when the -package name is unambiguous and well-known (e.g., ``pkg:generic/git`` or -``pkg:generic/openblas``), and using the VCS as the PURL type otherwise. +``dep:github/user-or-org-name/dep-name``). Which of these is more appropriate +is situation-dependent. This PEP recommends using ``dep:generic`` when the +package name is unambiguous and well-known (e.g., ``dep:generic/git`` or +``dep:generic/openblas``), and using the VCS as the type otherwise. Virtual package specification -''''''''''''''''''''''''''''' +'''''''''''''''''''''''''''''' + +PURL does not offer support for virtual or virtual dependency specification yet. +A `proposal to add a virtual type `__ +is being discussed for revision 1.1. + +In the meantime, we propose adding a new *type* to our ``dep:`` derivative, the ``virtual`` +type, which can take two *namespaces*: -There is no ready-made support for virtual packages in PURL or another -standard. There are a relatively limited number of such dependencies though, -and adopting a scheme similar to PURL but with the ``virtual:`` rather than -``pkg:`` scheme seems like it will be understandable and map well to Linux -distros with virtual packages and to the likes of Conda and Spack. +- ``interface``: for components such as BLAS or MPI. +- ``compiler``: for compiled languages like C or Rust. -The two known virtual package types are ``compiler`` and ``interface``. +The *name* should be the most common name for the interface or language, lowercased. +Some examples include:: + + dep:virtual/compiler/c + dep:virtual/compiler/c++ + dep:virtual/compiler/rust + dep:virtual/interface/blas + dep:virtual/interface/lapack + +Since there are a relatively limited number of such dependencies, +it seems like it will be understandable and map well to Linux +distros with virtual packages and to the likes of conda and Spack. Versioning '''''''''' +PURLs support fixed versions via the ``@`` component of the URL. For example, +``numpy===2.0`` can be expressed as ``pkg:pypi/numpy@2.0``. + Support in PURL for version expressions and ranges beyond a fixed version is -still pending, see the Open Issues section. +available via the ``vers`` scheme. Users are supposed to couple a ``pkg:`` +URL with a ``vers:`` URL. For example, to express ``numpy>=2.0``, the PURL +equivalent would be ``pkg:pypi/numpy`` plus ``vers:pypi/>=2.0``. This +can be expressed as a two-item list: ``["pkg:pypi/numpy", +"vers:pypi/>=2.0"]``. Additionally, a new proposal to add the ``vers`` +URL in a PURL qualifier is being `discussed `__, +but the current direction requires percent-encoding and the repetition of some components for +disambiguation. + +Since this is not very ergonomic, the version field in ``dep:`` URLs accepts +version range specifiers too, with these rules: + +- The ``vers:`` scheme is omitted. +- The *type* is omitted and assumed to match the PURL *type*. If there's no match, + the type is assumed to be ``pypi``. +- When no operator is present, the field is understood as a version literal. +- Otherwise, it is considered a version range specifier. + +Some examples: + +- ``dep:pypi/numpy@2.0``: ``numpy`` pinned at exactly version 2.0. +- ``dep:pypi/numpy@>=2.0``: ``numpy`` with version greater or equal than 2.0. +- ``dep:virtual/interface/lapack@>=3.7.1``: any package implementing the + LAPACK interface for version greater or equal than ``3.7.1``. Dependency specifiers ''''''''''''''''''''' @@ -237,7 +288,7 @@ this is pragmatic: dependency specifiers are already used for other metadata in ``pyproject.toml``, any tooling that is used with ``pyproject.toml`` is likely to already have a robust implementation to parse it. And we do not expect to need the extra possibilities that PURL qualifiers provide (e.g. to specify a -Conan or Conda channel, or a RubyGems platform). +Conan or conda channel, or a RubyGems platform). Usage of core metadata fields ----------------------------- @@ -252,11 +303,39 @@ metadata fields. The ``optional-dependencies`` content from ``[external]`` would need to either reuse ``Provides-Extra`` or require a new ``Provides-External-Extra`` field. Neither seems desirable. +The ``dep:`` URLs MUST be processed prior to their inclusion in the core +metadata by following these rules: + +- If the *type* is not ``virtual``, the ``dep`` *scheme* MUST + be replaced with ``pkg``. +- If present, as per the + `Requires-External syntax rules `__ + the *version* field MUST be split from the URL and added next to it, + between parentheses and after a space, with the following + transformations: + + - If the value is a literal (it contains no operators), it MUST be kept as is. + + - Otherwise, the value MUST be transformed into a ``vers:`` URL. The *type* of the + PURL MUST be prepended to the value, followed by a forward slash. If the + *type* does not have a matching ``vers`` type, ``pypi`` MUST be used. + +Some examples: + +- ``dep:generic/gcc@14`` becomes ``pkg:generic/gcc (14)``. +- ``dep:generic/libarrow@>=19`` becomes ``pkg:generic/libarrow (vers:pypi/>=19)``. +- ``dep:virtual/compiler/c`` stays ``dep:virtual/compiler/c``. +- ``dep:virtual/interface/blas>=3,<4`` becomes + ``dep:virtual/interface/blas (vers:pypi/>=3,<4)``. +- For completeness, ``dep:pypi/requests`` would become ``pkg:pypi/requests`` and + ``dep:pypi/django@>=5`` would become ``pkg:pypi/django (vers:pypi/>=5)``, but + these examples are unlikely to be seen in ``Requires-External``. + Differences between sdist and wheel metadata '''''''''''''''''''''''''''''''''''''''''''' A wheel may vendor its external dependencies. This happens in particular when -distributing wheels on PyPI or other Python package indexes - and tools like +distributing wheels on PyPI or other Python package indexes -- and tools like auditwheel_, delvewheel_ and delocate_ automate this process. As a result, a ``Requires-External`` entry in an sdist may disappear from a wheel built from that sdist. It is also possible that a ``Requires-External`` entry remains in a @@ -285,7 +364,7 @@ maintain, and should not be reflected in the ``[external]`` table. It is not possible to specify this in a reasonable way that works across distros, hence only the canonical name should be used in ``[external]``. -The intended meaning of using a PURL or virtual dependency is "the full package +The intended meaning of using a ``dep:`` string is "the full package with the name specified". It will depend on the context in which the metadata is used whether the split is relevant. For example, if ``libffi`` is a host dependency and a tool wants to prepare an environment for building a wheel, @@ -330,54 +409,54 @@ to be present on the system already. ``build-requires``/``optional-build-requires`` '''''''''''''''''''''''''''''''''''''''''''''' -- Format: Array of PURL_ strings (``build-requires``) and a table - with values of arrays of PURL_ strings (``optional-build-requires``) +- Format: Array of ``dep:`` strings (``build-requires``) and a table + with values of arrays of ``dep:`` strings (``optional-build-requires``) - `Core metadata`_: N/A The (optional) external build requirements needed to build the project. For ``build-requires``, it is a key whose value is an array of strings. Each string represents a build requirement of the project and MUST be formatted as -either a valid PURL_ string or a ``virtual:`` string. +a valid ``dep:`` string. For ``optional-build-requires``, it is a table where each key specifies an extra set of build requirements and whose value is an array of strings. The -strings of the arrays MUST be valid PURL_ strings. +strings of the arrays MUST be valid ``dep:`` strings. ``host-requires``/``optional-host-requires`` '''''''''''''''''''''''''''''''''''''''''''' -- Format: Array of PURL_ strings (``host-requires``) and a table - with values of arrays of PURL_ strings (``optional-host-requires``) +- Format: Array of ``dep:`` strings (``host-requires``) and a table + with values of arrays of ``dep:`` strings (``optional-host-requires``) - `Core metadata`_: N/A The (optional) external host requirements needed to build the project. For ``host-requires``, it is a key whose value is an array of strings. Each string represents a host requirement of the project and MUST be formatted as -either a valid PURL_ string or a ``virtual:`` string. +a valid ``dep:`` string. For ``optional-host-requires``, it is a table where each key specifies an extra set of host requirements and whose value is an array of strings. The -strings of the arrays MUST be valid PURL_ strings. +strings of the arrays MUST be valid ``dep:`` strings. ``dependencies``/``optional-dependencies`` '''''''''''''''''''''''''''''''''''''''''' -- Format: Array of PURL_ strings (``dependencies``) and a table - with values of arrays of PURL_ strings (``optional-dependencies``) +- Format: Array of ``dep:`` strings (``dependencies``) and a table + with values of arrays of ``dep:`` strings (``optional-dependencies``) - `Core metadata`_: ``Requires-External``, N/A The (optional) runtime dependencies of the project. For ``dependencies``, it is a key whose value is an array of strings. Each string represents a dependency of the project and MUST be formatted as either a -valid PURL_ string or a ``virtual:`` string. Each string maps directly to a -``Requires-External`` entry in the `core metadata`_. +valid ``dep:`` string. Each string maps directly to a ``Requires-External`` +entry in the `core metadata`_. For ``optional-dependencies``, it is a table where each key specifies an extra and whose value is an array of strings. The strings of the arrays MUST be valid -PURL_ strings. Optional dependencies do not map to a core metadata field. +``dep:`` strings. Optional dependencies do not map to a core metadata field. Examples -------- @@ -391,13 +470,13 @@ cryptography 39.0: [external] build-requires = [ - "virtual:compiler/c", - "virtual:compiler/rust", - "pkg:generic/pkg-config", + "dep:virtual/compiler/c", + "dep:virtual/compiler/rust", + "dep:generic/pkg-config", ] host-requires = [ - "pkg:generic/openssl", - "pkg:generic/libffi", + "dep:generic/openssl", + "dep:generic/libffi", ] SciPy 1.10: @@ -406,15 +485,15 @@ SciPy 1.10: [external] build-requires = [ - "virtual:compiler/c", - "virtual:compiler/cpp", - "virtual:compiler/fortran", - "pkg:generic/ninja", - "pkg:generic/pkg-config", + "dep:virtual/compiler/c", + "dep:virtual/compiler/cpp", + "dep:virtual/compiler/fortran", + "dep:generic/ninja", + "dep:generic/pkg-config", ] host-requires = [ - "virtual:interface/blas", - "virtual:interface/lapack", # >=3.7.1 (can't express version ranges with PURL yet) + "dep:virtual/interface/blas", + "dep:virtual/interface/lapack@>=3.7.1", ] Pillow 10.1.0: @@ -423,24 +502,24 @@ Pillow 10.1.0: [external] build-requires = [ - "virtual:compiler/c", + "dep:virtual/compiler/c", ] host-requires = [ - "pkg:generic/libjpeg", - "pkg:generic/zlib", + "dep:generic/libjpeg", + "dep:generic/zlib", ] [external.optional-host-requires] extra = [ - "pkg:generic/lcms2", - "pkg:generic/freetype", - "pkg:generic/libimagequant", - "pkg:generic/libraqm", - "pkg:generic/libtiff", - "pkg:generic/libxcb", - "pkg:generic/libwebp", - "pkg:generic/openjpeg", # add >=2.0 once we have version specifiers - "pkg:generic/tk", + "dep:generic/lcms2", + "dep:generic/freetype", + "dep:generic/libimagequant", + "dep:generic/libraqm", + "dep:generic/libtiff", + "dep:generic/libxcb", + "dep:generic/libwebp", + "dep:generic/openjpeg@>=2.0", + "dep:generic/tk", ] @@ -453,13 +532,13 @@ NAVis 1.4.0: [external] build-requires = [ - "pkg:generic/XCB; platform_system=='Linux'", + "dep:generic/XCB; platform_system=='Linux'", ] [external.optional-dependencies] nat = [ - "pkg:cran/nat", - "pkg:cran/nat.nblast", + "dep:cran/nat", + "dep:cran/nat.nblast", ] Spyder 6.0: @@ -468,9 +547,9 @@ Spyder 6.0: [external] dependencies = [ - "pkg:cargo/ripgrep", - "pkg:cargo/tree-sitter-cli", - "pkg:golang/github.com/junegunn/fzf", + "dep:cargo/ripgrep", + "dep:cargo/tree-sitter-cli", + "dep:golang/github.com/junegunn/fzf", ] jupyterlab-git 0.41.0: @@ -479,12 +558,12 @@ jupyterlab-git 0.41.0: [external] dependencies = [ - "pkg:generic/git", + "dep:generic/git", ] [external.optional-build-requires] dev = [ - "pkg:generic/nodejs", + "dep:generic/nodejs", ] PyEnchant 3.2.2: @@ -496,7 +575,7 @@ PyEnchant 3.2.2: # libenchant is needed on all platforms but only vendored into wheels on # Windows, so on Windows the build backend should remove this external # dependency from wheel metadata. - "pkg:github/AbiWord/enchant", + "dep:github/AbiWord/enchant", ] @@ -545,7 +624,7 @@ there are parts that do have a reference implementation: with ``tomllib``. 2. The PURL specification, as a key part of this spec, has a Python package with a reference implementation for constructing and parsing PURLs: - `packageurl-python`_. + `packageurl-python`_. The ``dep:`` URLs can be handled with this package. There are multiple possible consumers and use cases of this metadata, once that metadata gets added to Python packages. Tested metadata for all of the @@ -584,18 +663,17 @@ Open Issues Version specifiers for PURLs ---------------------------- -Support in PURL for version expressions and ranges is still pending. The pull -request at `vers implementation for PURL`_ seems close to being merged, at -which point this PEP could adopt it. +Support in PURL for version expressions and ranges is now available via the +discussed ``vers:`` scheme. However, the ergonomics of having to use two URLs +is undesirable. Thus why we propose a joint ``dep:`` scheme. Versioning of virtual dependencies ----------------------------------- +----------------------------------- -Once PURL supports version expressions, virtual dependencies can be versioned -with the same syntax. It must be better specified however what the version -scheme is, because this is not as clear for virtual dependencies as it is for -PURLs (e.g., there can be multiple implementations, and abstract interfaces may -not be unambiguously versioned). E.g.: +While virtual dependencies can be versioned with the same syntax, it must be better +specified however what the version scheme is, because this is not as clear for +virtual types as it is for PURL types (e.g., there can be multiple implementations, +and virtual interfaces may not be unambiguously versioned). E.g.: - OpenMP: has regular ``MAJOR.MINOR`` versions of its standard, so would look like ``>=4.5``. @@ -619,17 +697,17 @@ Who defines canonical names and canonical package structure? Similarly to the logistics around versioning is the question about what names are allowed and where they are described. And then who is in control of that description and responsible for maintaining it. Our tentative answer is: there -should be a central list for virtual dependencies and ``pkg:generic`` PURLs, +should be a central list for ``dep:generic`` and ``dep:virtual`` URLs, maintained as a PyPA project. See https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml/31888/62. TODO: once that list/project is prototyped, include it in the PEP and close this open issue. Syntax for virtual dependencies -------------------------------- +-------------------------------- The current syntax this PEP uses for virtual dependencies is -``virtual:type/name``, which is analogous to but not part of the PURL spec. +``dep:virtual/(compiler|interface)/name``, which is analogous to but not part of the PURL spec. This open issue discusses supporting virtual dependencies within PURL: `purl-spec#222 `__. @@ -695,5 +773,6 @@ CC0-1.0-Universal license, whichever is more permissive. .. _auditwheel: https://github.com/pypa/auditwheel .. _delocate: https://github.com/matthew-brett/delocate .. _delvewheel: https://github.com/adang1345/delvewheel +.. _verspurl: https://github.com/package-url/purl-spec/issues/386 .. _rgommers/external-deps-build: https://github.com/rgommers/external-deps-build .. _Reference LAPACK: https://github.com/Reference-LAPACK/lapack From 24e7969bd01188352f7c364f8e3e38b631495059 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 26 May 2025 15:39:37 +0200 Subject: [PATCH 02/15] 725: Move dep -> purl conversion rules to specification (#22) --- peps/pep-0725.rst | 97 +++++++++++++++++++++++++++-------------------- 1 file changed, 55 insertions(+), 42 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 227d04970a6..5896b932e1c 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -23,10 +23,10 @@ three keys: "build-requires", "host-requires" and "dependencies". These are for specifying three types of dependencies: 1. ``build-requires``, build tools to run on the build machine -2. ``host-requires``, build dependencies needed for host machine but also needed at build time. +2. ``host-requires``, build dependencies needed for the host machine but also needed at build time. 3. ``dependencies``, needed at runtime on the host machine but not needed at build time. -Cross compilation is taken into account by distinguishing build and host dependencies. +Cross compilation is taken into account by distinguishing between build and host dependencies. Optional build-time and runtime dependencies are supported too, in a manner analogous to how that is supported in the ``[project]`` table. @@ -184,7 +184,7 @@ Specifying external dependencies Concrete package specification '''''''''''''''''''''''''''''' -PURLs implement a scheme for identifying packages that is meant to be portable +A `PURL`_ implements a scheme for identifying packages that is meant to be portable across packaging ecosystems. Its design is:: scheme:type/namespace/name@version?qualifiers#subpath @@ -194,9 +194,13 @@ components only ``type`` and ``name`` are required. Since external dependencies are likely to be typed by hand, we propose a PURL derivative that, in the name of ergonomics and user-friendliness, introduces a -number of changes, discussed in the following sections. In this derivative, -we replace the ``pkg`` scheme with ``dep``. Hence, we will refer to them as -``dep:`` URLs. +number of changes (further discussed below): + +- Support for virtual packages via a new ``virtual`` type. +- Allow version ranges (and not just literals) in the ``version`` field. + +In this derivative, we replace the ``pkg`` scheme with ``dep``. Hence, +we will refer to them as ``dep:`` URLs. As an example, a ``dep:`` URL for the ``requests`` package on PyPI would be:: @@ -253,13 +257,17 @@ PURLs support fixed versions via the ``@`` component of the URL. For example, ``numpy===2.0`` can be expressed as ``pkg:pypi/numpy@2.0``. Support in PURL for version expressions and ranges beyond a fixed version is -available via the ``vers`` scheme. Users are supposed to couple a ``pkg:`` -URL with a ``vers:`` URL. For example, to express ``numpy>=2.0``, the PURL -equivalent would be ``pkg:pypi/numpy`` plus ``vers:pypi/>=2.0``. This -can be expressed as a two-item list: ``["pkg:pypi/numpy", -"vers:pypi/>=2.0"]``. Additionally, a new proposal to add the ``vers`` -URL in a PURL qualifier is being `discussed `__, -but the current direction requires percent-encoding and the repetition of some components for +available via ``vers`` URIs (`see specification `__):: + + vers:type/version-constraint|version-constraint|... + +Users are supposed to couple a ``pkg:`` URL with a ``vers:`` URL. For example, +to express ``numpy>=2.0``, the PURL equivalent would be ``pkg:pypi/numpy`` plus +``vers:pypi/>=2.0``. This can be expressed as a two-item list: +``["pkg:pypi/numpy", "vers:pypi/>=2.0"]``. Additionally, a new proposal to add +the ``vers`` URL in a PURL qualifier is being `discussed +`__, but the current +direction requires percent-encoding and the repetition of some components for disambiguation. Since this is not very ergonomic, the version field in ``dep:`` URLs accepts @@ -294,7 +302,7 @@ Usage of core metadata fields ----------------------------- The `core metadata`_ specification contains one relevant field, namely -``Requires-External``. This has no well-defined semantics in core metadata 2.1; +``Requires-External``. This has no well-defined semantics as of core metadata 2.4; this PEP chooses to reuse the field for external runtime dependencies. The core metadata specification does not contain fields for any metadata in ``pyproject.toml``'s ``[build-system]`` table. Therefore the ``build-requires`` @@ -303,33 +311,9 @@ metadata fields. The ``optional-dependencies`` content from ``[external]`` would need to either reuse ``Provides-Extra`` or require a new ``Provides-External-Extra`` field. Neither seems desirable. -The ``dep:`` URLs MUST be processed prior to their inclusion in the core -metadata by following these rules: - -- If the *type* is not ``virtual``, the ``dep`` *scheme* MUST - be replaced with ``pkg``. -- If present, as per the - `Requires-External syntax rules `__ - the *version* field MUST be split from the URL and added next to it, - between parentheses and after a space, with the following - transformations: - - - If the value is a literal (it contains no operators), it MUST be kept as is. - - - Otherwise, the value MUST be transformed into a ``vers:`` URL. The *type* of the - PURL MUST be prepended to the value, followed by a forward slash. If the - *type* does not have a matching ``vers`` type, ``pypi`` MUST be used. - -Some examples: - -- ``dep:generic/gcc@14`` becomes ``pkg:generic/gcc (14)``. -- ``dep:generic/libarrow@>=19`` becomes ``pkg:generic/libarrow (vers:pypi/>=19)``. -- ``dep:virtual/compiler/c`` stays ``dep:virtual/compiler/c``. -- ``dep:virtual/interface/blas>=3,<4`` becomes - ``dep:virtual/interface/blas (vers:pypi/>=3,<4)``. -- For completeness, ``dep:pypi/requests`` would become ``pkg:pypi/requests`` and - ``dep:pypi/django@>=5`` would become ``pkg:pypi/django (vers:pypi/>=5)``, but - these examples are unlikely to be seen in ``Requires-External``. +The ``dep:`` URLs must be converted into ``pkg`` and ``vers`` URIs prior to their +inclusion in ``Requires-External``. The rules are discussed in the specification +section below. Differences between sdist and wheel metadata '''''''''''''''''''''''''''''''''''''''''''' @@ -452,7 +436,25 @@ The (optional) runtime dependencies of the project. For ``dependencies``, it is a key whose value is an array of strings. Each string represents a dependency of the project and MUST be formatted as either a valid ``dep:`` string. Each string maps directly to a ``Requires-External`` -entry in the `core metadata`_. +entry in the `core metadata`_, once converted into ``pkg`` / ``vers`` URIs +(whenever possible) using the following rules: + +- If the *type* is not ``virtual``, the ``dep`` *scheme* MUST + be replaced with ``pkg``. +- If present, as per the + `Requires-External syntax rules `__ + the *version* field MUST be processed using these rules: + + - Split the version field from the URL and remove the ``@`` symbol. + + - If the value is a literal (it contains no operators), it MUST be kept as is. + + - Otherwise, the value MUST be transformed into a ``vers:`` URL. The *type* of the + PURL MUST be prepended to the value, followed by a forward slash. If the + *type* does not have a matching ``vers`` type, ``pypi`` MUST be used. + + - Wrap the resulting value in parentheses and add it next to the URL, separated by + a space. For ``optional-dependencies``, it is a table where each key specifies an extra and whose value is an array of strings. The strings of the arrays MUST be valid @@ -578,6 +580,17 @@ PyEnchant 3.2.2: "dep:github/AbiWord/enchant", ] +Core-metadata entries of ``external.dependencies`` items: + +- ``dep:generic/gcc@14`` becomes ``pkg:generic/gcc (14)``. +- ``dep:generic/libarrow@>=19`` becomes ``pkg:generic/libarrow (vers:pypi/>=19)``. +- ``dep:virtual/compiler/c`` stays ``dep:virtual/compiler/c``. +- ``dep:virtual/interface/blas>=3,<4`` becomes + ``dep:virtual/interface/blas (vers:pypi/>=3,<4)``. +- For completeness, ``dep:pypi/requests`` would become ``pkg:pypi/requests`` and + ``dep:pypi/django@>=5`` would become ``pkg:pypi/django (vers:pypi/>=5)``, but + these examples are unlikely to be seen in ``Requires-External``. + Backwards Compatibility ======================= From 852137b55344a65b75f06d090ff02628cfe6d2a7 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 8 Jul 2025 15:39:05 +0200 Subject: [PATCH 03/15] Rephrase PURL + vers discussion (#28) --- peps/pep-0725.rst | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 5896b932e1c..de9bac72c17 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -263,14 +263,13 @@ available via ``vers`` URIs (`see specification `__):: Users are supposed to couple a ``pkg:`` URL with a ``vers:`` URL. For example, to express ``numpy>=2.0``, the PURL equivalent would be ``pkg:pypi/numpy`` plus -``vers:pypi/>=2.0``. This can be expressed as a two-item list: -``["pkg:pypi/numpy", "vers:pypi/>=2.0"]``. Additionally, a new proposal to add -the ``vers`` URL in a PURL qualifier is being `discussed -`__, but the current -direction requires percent-encoding and the repetition of some components for -disambiguation. - -Since this is not very ergonomic, the version field in ``dep:`` URLs accepts +``vers:pypi/>=2.0``. This can be done with: + +- A two-item list: ``["pkg:pypi/numpy", "vers:pypi/>=2.0"]``. +- A `percent-encoded `__ + URL qualifier: ``pkg:pypi/numpy?vers=vers:pypi%2F%3E%3D2.0``. + +Since none of these options are very ergonomic, the version field in ``dep:`` URLs accepts version range specifiers too, with these rules: - The ``vers:`` scheme is omitted. From c763ba23ba0299e8334e9d03254eb9c46ea93595 Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Tue, 8 Jul 2025 18:52:48 +0200 Subject: [PATCH 04/15] PEP 725: remove resolved open issues (#25) * PEP 725: remove resolved open issues * PEP 725: cross-link to PEP 770 (SBOMs) --- peps/pep-0725.rst | 43 +++++++++++++++++-------------------------- 1 file changed, 17 insertions(+), 26 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index de9bac72c17..6f8c32add00 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -57,10 +57,13 @@ standard location. Enabling automating this conversion is a key benefit of this PEP, making packaging Python packages for distros easier and more reliable. In addition, the authors envision other types of tools making use of this information, e.g., dependency analysis tools like Repology_, Dependabot_ and libraries.io_. + Software bill of materials (SBOM) generation tools may also be able to use this information, e.g. for flagging that external dependencies listed in ``pyproject.toml`` but not contained in wheel metadata are likely vendored -within the wheel. +within the wheel. :pep:`770` ("Improving measurability of Python packages with +Software Bill-of-Materials"), which standardizes how SBOMs are included in +wheels, contains an instructive section on how that PEP differs from this one. Packages with external dependencies are typically hard to build from source, and error messages from build failures tend to be hard to decipher for end @@ -668,16 +671,22 @@ proposed using specific library and header names as external dependencies. This is too granular; using package names is a well-established pattern across packaging ecosystems and should be preferred. +Adding a ``host-requires`` key under ``[build-system]`` +------------------------------------------------------- -Open Issues -=========== +Adding ``host-requires`` for host dependencies that are on PyPI in order to +better support name mapping to other packaging systems with support for +cross-compiling seems useful in principle, for the same reasons as this PEP +adds a ``host-requires`` under the ``[external]`` table. However, it isn't +necessary to include in this PEP, and hence the authors prefer to keep the +scope of this PEP limited - a future PEP on cross compilation may want to +tackle this. `This issue `__ +contains more arguments in favor and against adding ``host-requires`` under +``[build-system]`` as part of this PEP. -Version specifiers for PURLs ----------------------------- -Support in PURL for version expressions and ranges is now available via the -discussed ``vers:`` scheme. However, the ergonomics of having to use two URLs -is undesirable. Thus why we propose a joint ``dep:`` scheme. +Open Issues +=========== Versioning of virtual dependencies ----------------------------------- @@ -715,24 +724,6 @@ https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproje TODO: once that list/project is prototyped, include it in the PEP and close this open issue. -Syntax for virtual dependencies --------------------------------- - -The current syntax this PEP uses for virtual dependencies is -``dep:virtual/(compiler|interface)/name``, which is analogous to but not part of the PURL spec. -This open issue discusses supporting virtual dependencies within PURL: -`purl-spec#222 `__. - -Should a ``host-requires`` key be added under ``[build-system]``? ------------------------------------------------------------------ - -Adding ``host-requires`` for host dependencies that are on PyPI in order to -better support name mapping to other packaging systems with support for -cross-compiling may make sense. -`This issue `__ tracks this topic -and has arguments in favor and against adding ``host-requires`` under -``[build-system]`` as part of this PEP. - References ========== From c81b14102c7acc20d8ead941dfe34d68f323627c Mon Sep 17 00:00:00 2001 From: jaimergp Date: Thu, 10 Jul 2025 15:28:50 +0200 Subject: [PATCH 05/15] 725: Add pyproject-external examples to Reference Implementation (#29) --- peps/pep-0725.rst | 65 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 63 insertions(+), 2 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 6f8c32add00..a453c118320 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -636,10 +636,12 @@ there will not be code implementing the metadata spec as a whole. However, there are parts that do have a reference implementation: 1. The ``[external]`` table has to be valid TOML and therefore can be loaded - with ``tomllib``. + with ``tomllib``. This table can be further processed with the + `pyproject-external`_ package, demonstrated below. 2. The PURL specification, as a key part of this spec, has a Python package with a reference implementation for constructing and parsing PURLs: - `packageurl-python`_. The ``dep:`` URLs can be handled with this package. + `packageurl-python`_. This package is wrapped in `pyproject-external`_ + to provide ``dep:``-specific validation and handling. There are multiple possible consumers and use cases of this metadata, once that metadata gets added to Python packages. Tested metadata for all of the @@ -648,6 +650,64 @@ wheels can be found in `rgommers/external-deps-build`_. This metadata has been validated by using it to build wheels from sdists patched with that metadata in clean Docker containers. +Example +------- + +Given a ``pyproject.toml`` with this ``[external]`` table: + +.. code-block:: toml + + [external] + build-requires = [ + "dep:virtual/compiler/c", + "dep:virtual/compiler/rust", + "dep:generic/pkg-config", + ] + host-requires = [ + "dep:generic/openssl", + "dep:generic/libffi", + ] + +You can use ``pyproject_external.External`` to parse it and manipulate it: + +.. code-block:: python + + >>> from pyproject_external import External + >>> external = External.from_pyproject_path("./pyproject.toml") + >>> external.validate() + >>> external.to_dict() + {'external': {'build_requires': ['dep:virtual/compiler/c', 'dep:virtual/compiler/rust', 'dep:generic/pkg-config'], 'host_requires': ['dep:generic/openssl', 'dep:generic/libffi']}} + >>> external.build_requires + [DepURL(type='virtual', namespace='compiler', name='c', version=None, qualifiers={}, subpath=None), DepURL(type='virtual', namespace='compiler', name='rust', version=None, qualifiers={}, subpath=None), DepURL(type='generic', namespace=None, name='pkg-config', version=None, qualifiers={}, subpath=None)] + >>> external.build_requires[0] + DepURL(type='virtual', namespace='compiler', name='c', version=None, qualifiers={}, subpath=None) + +Note the proposed ``[external]`` table was well-formed. With invalid contents such as: + +.. code-block:: toml + + [external] + build-requires = [ + "dep:this-is-missing-the-type", + "pkg:not-a-dep-url" + ] + +You would fail the validation: + +.. code-block:: python + + >>> external = External.from_pyproject_data( + { + "external": { + "build_requires": [ + "dep:this-is-missing-the-type", + "pkg:not-a-dep-url" + ] + } + } + ) + ValueError: purl is missing the required type component: 'dep:this-is-missing-the-type'. + Rejected Ideas ============== @@ -778,4 +838,5 @@ CC0-1.0-Universal license, whichever is more permissive. .. _delvewheel: https://github.com/adang1345/delvewheel .. _verspurl: https://github.com/package-url/purl-spec/issues/386 .. _rgommers/external-deps-build: https://github.com/rgommers/external-deps-build +.. _pyproject-external: https://github.com/jaimergp/pyproject-external .. _Reference LAPACK: https://github.com/Reference-LAPACK/lapack From a1750c3962960a2b375187d013539b92892ca12a Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 11 Jul 2025 13:50:18 +0200 Subject: [PATCH 06/15] 725: Integrate with PEP 735 (#30) --- peps/pep-0725.rst | 65 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 63 insertions(+), 2 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index a453c118320..87c405296c1 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -19,13 +19,19 @@ runtime dependencies in a ``pyproject.toml`` file for packaging-related tools to consume. This PEP proposes to add an ``[external]`` table to ``pyproject.toml`` with -three keys: "build-requires", "host-requires" and "dependencies". These -are for specifying three types of dependencies: +seven keys. "build-requires", "host-requires" and "dependencies" are +are for specifying three types of required dependencies: 1. ``build-requires``, build tools to run on the build machine 2. ``host-requires``, build dependencies needed for the host machine but also needed at build time. 3. ``dependencies``, needed at runtime on the host machine but not needed at build time. +These three keys also have their optional counterparts (``optional-build-requires``, +``optional-host-requires``, ``optional-dependencies``), which play the same role and +have the same format as ``optional-dependencies`` key in the ``[project]`` table. +Finally ``dependency-groups`` offers the same functionality as :pep:`735` but for external +dependencies. + Cross compilation is taken into account by distinguishing between build and host dependencies. Optional build-time and runtime dependencies are supported too, in a manner analogous to how that is supported in the ``[project]`` table. @@ -371,6 +377,39 @@ consistency between Python dependencies and external dependencies, we choose to add it implicitly. Python development headers must be assumed to be necessary when an ``[external]`` table contains one or more compiler packages. +Dependency groups +''''''''''''''''' + +This PEP has chosen to include the :pep:`735` key ``dependency-groups`` under +the ``[external]`` table too. This decision is motivated by the need of having +similar functionality for external metadata. The top-level table cannot be used +for external dependencies because it's expected to have PEP 508 strings, while +we have chosen to rely on ``dep:`` URLs for the external dependencies. Conflating +both would raise significant backwards compatibility issues with existing usage. + +Optional dependencies versus dependency groups +'''''''''''''''''''''''''''''''''''''''''''''' + +Three ``optional-*`` keys are available in the ``[external]`` table, for build, host +and runtime dependencies. These are not required, and they are grouped in categories. +In the case of ``optional-build-requires`` and ``optional-host-requires``, the metadata +will not make it to the distributed metadata (e.g. contents of ``*.dist-info``). + +The dependency groups introduced in :pep:`735` share some of these features, so one +could argue why they are needed as an additional key under ``[external]``, or why can't +the ``optional-*`` keys purpose be fulfilled by ``dependency-groups``. + +The ``optional-build-requires`` and ``optional-host-requires`` keys have a very specific +meaning: they are optional dependencies to be used as build tools or as linked libraries, +respectively. If present, they may be used by the build machinery, but if absent no error +will be raised. ``[external]``-aware tools will inspect these fields to provision or analyze +the dependency tree as needed, taking this meaning into account (which is especially important +in cross-compiling scenarios). The ``dependency-groups`` field doesn't provide this nuance +and choosing a naming convention (or reserving a few keywords) doesn't seem practical or robust. + +About the overlaps between ``dependency-groups`` and ``optional-dependencies``, there is one +key distinction: ``dependency-groups`` will not make it to distributed metadata. It is hence +useful to provide external dependency metadata for e.g. development or testing. Specification ============= @@ -462,6 +501,18 @@ For ``optional-dependencies``, it is a table where each key specifies an extra and whose value is an array of strings. The strings of the arrays MUST be valid ``dep:`` strings. Optional dependencies do not map to a core metadata field. +``dependency-groups`` +''''''''''''''''''''' + +- Format: A table where each key is the name of the group, and the values are + arrays of ``dep:`` strings, tables, or a mix of both. +- `Core metadata`_: N/A + +PEP 735 -style dependency groups, but using ``dep:`` URLs instead of PEP 508 strings as +dependency specifiers. Every other detail (e.g. group inclusion, name normalization) +follows the official `Dependency Groups specification +`__. + Examples -------- @@ -582,6 +633,16 @@ PyEnchant 3.2.2: "dep:github/AbiWord/enchant", ] +Example with dependency groups: + +.. code:: toml + + [external.dependency-groups] + dev = [ + "dep:generic/catch2", + "dep:generic/valgrind", + ] + Core-metadata entries of ``external.dependencies`` items: - ``dep:generic/gcc@14`` becomes ``pkg:generic/gcc (14)``. From 44b2131e44ae1c4ecfff70b3f3b6e04bfca86216 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 18 Jul 2025 10:03:17 +0200 Subject: [PATCH 07/15] 725: PEP 735 integration, part two (#31) * Clarify optional-dependencies vs (project|external) * Precise dependency-groups contents * Simplify optional dependencies vs dependency groups section --- peps/pep-0725.rst | 60 ++++++++++++++++++++++++----------------------- 1 file changed, 31 insertions(+), 29 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 87c405296c1..87e3f0438f2 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -19,17 +19,17 @@ runtime dependencies in a ``pyproject.toml`` file for packaging-related tools to consume. This PEP proposes to add an ``[external]`` table to ``pyproject.toml`` with -seven keys. "build-requires", "host-requires" and "dependencies" are -are for specifying three types of required dependencies: +seven keys. "build-requires", "host-requires" and "dependencies" +are for specifying three types of *required* dependencies: 1. ``build-requires``, build tools to run on the build machine 2. ``host-requires``, build dependencies needed for the host machine but also needed at build time. 3. ``dependencies``, needed at runtime on the host machine but not needed at build time. -These three keys also have their optional counterparts (``optional-build-requires``, -``optional-host-requires``, ``optional-dependencies``), which play the same role and -have the same format as ``optional-dependencies`` key in the ``[project]`` table. -Finally ``dependency-groups`` offers the same functionality as :pep:`735` but for external +These three keys also have their *optional* ``external`` counterparts (``optional-build-requires``, +``optional-host-requires``, ``optional-dependencies``), which have the same role that +``project.optional-dependencies`` plays for ``project.dependencies``. Finally, +``dependency-groups`` offers the same functionality as :pep:`735` but for external dependencies. Cross compilation is taken into account by distinguishing between build and host dependencies. @@ -383,33 +383,35 @@ Dependency groups This PEP has chosen to include the :pep:`735` key ``dependency-groups`` under the ``[external]`` table too. This decision is motivated by the need of having similar functionality for external metadata. The top-level table cannot be used -for external dependencies because it's expected to have PEP 508 strings, while -we have chosen to rely on ``dep:`` URLs for the external dependencies. Conflating -both would raise significant backwards compatibility issues with existing usage. +for external dependencies because it's expected to have PEP 508 strings (and tables +for group includes), while we have chosen to rely on ``dep:`` URLs for the external +dependencies. Conflating both would raise significant backwards compatibility +issues with existing usage. + +Strictly speaking, the ``dependency-groups`` schema allows us to define external +dependencies in per-group sub-tables:: + + [dependency-groups] + dev = [ + "pytest", + { external = ["dep:cargo/ripgrep"] }, + ] + +However, this has the same problem: we are mixing different types of dependency +specifiers in the same data structure. We believe it's cleaner to separate concerns +in different top-level tables, hence why we still prefer to have +``external.dependency-groups``. Optional dependencies versus dependency groups '''''''''''''''''''''''''''''''''''''''''''''' -Three ``optional-*`` keys are available in the ``[external]`` table, for build, host -and runtime dependencies. These are not required, and they are grouped in categories. -In the case of ``optional-build-requires`` and ``optional-host-requires``, the metadata -will not make it to the distributed metadata (e.g. contents of ``*.dist-info``). - -The dependency groups introduced in :pep:`735` share some of these features, so one -could argue why they are needed as an additional key under ``[external]``, or why can't -the ``optional-*`` keys purpose be fulfilled by ``dependency-groups``. - -The ``optional-build-requires`` and ``optional-host-requires`` keys have a very specific -meaning: they are optional dependencies to be used as build tools or as linked libraries, -respectively. If present, they may be used by the build machinery, but if absent no error -will be raised. ``[external]``-aware tools will inspect these fields to provision or analyze -the dependency tree as needed, taking this meaning into account (which is especially important -in cross-compiling scenarios). The ``dependency-groups`` field doesn't provide this nuance -and choosing a naming convention (or reserving a few keywords) doesn't seem practical or robust. - -About the overlaps between ``dependency-groups`` and ``optional-dependencies``, there is one -key distinction: ``dependency-groups`` will not make it to distributed metadata. It is hence -useful to provide external dependency metadata for e.g. development or testing. +The rationale for having ``external.dependency-groups`` is identical for the +rationale given in :pep:`735` for introducing ``[dependency-groups]``. The +intended usage and semantics of inclusion/exclusion into Core Metadata +is thus identical to ``[dependency-groups]``. + +``external.optional-dependencies`` will show up in Core Metadata. +``external.dependency-groups`` will not. Specification ============= From 67cb38c2ed2d0566e6f76a3382b384a65a8835a4 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Thu, 24 Jul 2025 18:36:31 +0200 Subject: [PATCH 08/15] 725: Deprecate `Requires-External` and propose replacement fields (#27) --- peps/pep-0725.rst | 274 ++++++++++++++++++++++++++++++++-------------- 1 file changed, 189 insertions(+), 85 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 87e3f0438f2..f5e031740ea 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -306,44 +306,6 @@ to already have a robust implementation to parse it. And we do not expect to need the extra possibilities that PURL qualifiers provide (e.g. to specify a Conan or conda channel, or a RubyGems platform). -Usage of core metadata fields ------------------------------ - -The `core metadata`_ specification contains one relevant field, namely -``Requires-External``. This has no well-defined semantics as of core metadata 2.4; -this PEP chooses to reuse the field for external runtime dependencies. The core -metadata specification does not contain fields for any metadata in -``pyproject.toml``'s ``[build-system]`` table. Therefore the ``build-requires`` -and ``host-requires`` content also does not need to be reflected in core -metadata fields. The ``optional-dependencies`` content from ``[external]`` -would need to either reuse ``Provides-Extra`` or require a new -``Provides-External-Extra`` field. Neither seems desirable. - -The ``dep:`` URLs must be converted into ``pkg`` and ``vers`` URIs prior to their -inclusion in ``Requires-External``. The rules are discussed in the specification -section below. - -Differences between sdist and wheel metadata -'''''''''''''''''''''''''''''''''''''''''''' - -A wheel may vendor its external dependencies. This happens in particular when -distributing wheels on PyPI or other Python package indexes -- and tools like -auditwheel_, delvewheel_ and delocate_ automate this process. As a result, a -``Requires-External`` entry in an sdist may disappear from a wheel built from -that sdist. It is also possible that a ``Requires-External`` entry remains in a -wheel, either unchanged or with narrower constraints. ``auditwheel`` does not -vendor certain allow-listed dependencies, such as OpenGL, by default. In -addition, ``auditwheel`` and ``delvewheel`` allow a user to manually exclude -dependencies via a ``--exclude`` or ``--no-dll`` command-line flag. This is -used to avoid vendoring large shared libraries, for example those from CUDA. - -``Requires-External`` entries generated from external dependencies in -``pyproject.toml`` in a wheel are therefore allowed to be narrower than those -for the corresponding sdist. They must not be wider, i.e. constraints must not -allow a version of a dependency for a wheel that isn't allowed for an sdist, -nor contain new dependencies that are not listed in the sdist's metadata at -all. - Canonical names of dependencies and ``-dev(el)`` split packages ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' @@ -377,8 +339,67 @@ consistency between Python dependencies and external dependencies, we choose to add it implicitly. Python development headers must be assumed to be necessary when an ``[external]`` table contains one or more compiler packages. +New Core Metadata fields +------------------------ + +Two new Core Metadata fields are proposed: + +- ``Requires-External-Dep``. An external requirement expressed as a ``dep:`` + URL. Mimics the transition from ``Requires`` to ``Requires-Dist``. We + chose the ``-Dep`` suffix to emphasize that the value is not a regular + Python specifier (distribution), but a ``dep:`` URL. +- ``Provides-External-Extra``. An *extra* group that carries external dependencies + (as found in ``Requires-External-Dep``) only. + +Since the Core Metadata specification does not contain fields for any metadata in +``pyproject.toml``'s ``[build-system]`` table, the ``build-requires`` +and ``host-requires`` content do not need to be reflected in existing core +metadata fields. + +Additionally, this PEP also proposes to deprecate the ``Requires-External`` field. +The reasons being: + +- Avoiding confusion with the newly proposed fields. +- Avoiding potential incompatibilities with existing usage (even if limited). +- Low penetration in the ecosystem: + + - There is no direct correspondence to a field in the ``pyproject.toml`` metadata. + - Mainstream build backends like ``setuptools`` (see `pypa/setuptools#4220`_), + ``hatch`` (see `pypa/hatch#1712`_), ``flit`` (see `pypa/flit#353`_), or ``poetry`` + do not offer ways to specify it or require a plugin (e.g. `poetry-external-dependencies`_). + ``maturin`` does seem to support it since 0.7.0 (see `PyO3/maturin@5b0e4808`_), + but it's not directly documented. Other backends like ``scikit-build-core`` or + ``meson-python`` returned no results for ``External-Requires``. + - The field is not included in the `PyPI JSON API responses`_. + +Note about the differences between sdist and wheel metadata +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +A wheel may vendor its external dependencies. This happens in particular when +distributing wheels on PyPI or other Python package indexes -- and tools like +auditwheel_, delvewheel_ and delocate_ automate this process. As a result, a +``Requires-External`` entry in an sdist may disappear from a wheel built from +that sdist. It is also possible that a ``Requires-External`` entry remains in a +wheel, either unchanged or with narrower constraints. ``auditwheel`` does not +vendor certain allow-listed dependencies, such as OpenGL, by default. In +addition, ``auditwheel`` and ``delvewheel`` allow a user to manually exclude +dependencies via a ``--exclude`` or ``--no-dll`` command-line flag. This is +used to avoid vendoring large shared libraries, for example those from CUDA. + +``Requires-External`` entries generated from external dependencies in +``pyproject.toml`` MAY differ between an sdist and its corresponding wheel(s). +However, the ``Requires-External`` entries in the wheels MUST always specify +a narrower set of dependencies. + +Note that this does not imply that the field must be marked as Dynamic, since this +distinction only applies to wheels built from an sdist by a build backend. In +particular, wheels built from other wheels do not need to satisfy this constraint (see +`message 179 in this DPO thread +`__ +.) + Dependency groups -''''''''''''''''' +----------------- This PEP has chosen to include the :pep:`735` key ``dependency-groups`` under the ``[external]`` table too. This decision is motivated by the need of having @@ -419,8 +440,36 @@ Specification If metadata is improperly specified then tools MUST raise an error to notify the user about their mistake. -Details -------- +Changes in Core Metadata +------------------------ + +Deprecations +'''''''''''' + +The ``External-Requires`` Core Metadata field will be marked as *obsolete* and its +usage will be discouraged. + +Additions +''''''''' + +Two new fields are added to Core Metadata: + +- ``Requires-External-Dep``. An external requirement expressed as a ``dep:`` URL. +- ``Provides-External-Extra``. An *extra* group that carries external dependencies + (as found in ``Requires-External-Dep``) only. + +Version bump +'''''''''''' + +Given that the proposed changes are purely additive, the Core Metadata +version will be bumped to 2.5. + +This will only impact PyPI and tools that want to support external runtime dependencies, +and require no changes otherwise. + + +Changes in ``pyproject.toml`` +----------------------------- Note that ``pyproject.toml`` content is in the same format as in :pep:`621`. @@ -472,36 +521,24 @@ strings of the arrays MUST be valid ``dep:`` strings. - Format: Array of ``dep:`` strings (``dependencies``) and a table with values of arrays of ``dep:`` strings (``optional-dependencies``) -- `Core metadata`_: ``Requires-External``, N/A +- `Core metadata`_: ``Requires-External-Dep``, ``Provides-External-Extra`` The (optional) runtime dependencies of the project. For ``dependencies``, it is a key whose value is an array of strings. Each string represents a dependency of the project and MUST be formatted as either a -valid ``dep:`` string. Each string maps directly to a ``Requires-External`` -entry in the `core metadata`_, once converted into ``pkg`` / ``vers`` URIs -(whenever possible) using the following rules: - -- If the *type* is not ``virtual``, the ``dep`` *scheme* MUST - be replaced with ``pkg``. -- If present, as per the - `Requires-External syntax rules `__ - the *version* field MUST be processed using these rules: - - - Split the version field from the URL and remove the ``@`` symbol. +valid ``dep:`` string. Each string must be added to `Core Metadata`_ as a +``Requires-External-Dep`` field. - - If the value is a literal (it contains no operators), it MUST be kept as is. - - - Otherwise, the value MUST be transformed into a ``vers:`` URL. The *type* of the - PURL MUST be prepended to the value, followed by a forward slash. If the - *type* does not have a matching ``vers`` type, ``pypi`` MUST be used. - - - Wrap the resulting value in parentheses and add it next to the URL, separated by - a space. - -For ``optional-dependencies``, it is a table where each key specifies an extra +For ``optional-dependencies``, it is a table where each key specifies an *extra* and whose value is an array of strings. The strings of the arrays MUST be valid -``dep:`` strings. Optional dependencies do not map to a core metadata field. +``dep:`` strings. For each ``optional-dependencies`` group: + +- The name of the group MUST be added to `Core Metadata`_ as a + ``Provides-External-Extra`` field. +- The ``dep:`` URLs in that group MUST be added to `Core Metadata`_ as a + ``Requires-External-Dep`` field, with the corresponding ``; extra == 'name'`` + environment marker. ``dependency-groups`` ''''''''''''''''''''' @@ -521,7 +558,10 @@ Examples These examples show what the ``[external]`` content for a number of packages is expected to be. -cryptography 39.0: +cryptography 39.0 +''''''''''''''''' + +``pyproject.toml`` content: .. code:: toml @@ -536,7 +576,10 @@ cryptography 39.0: "dep:generic/libffi", ] -SciPy 1.10: +SciPy 1.10 +'''''''''' + +``pyproject.toml`` content: .. code:: toml @@ -553,7 +596,10 @@ SciPy 1.10: "dep:virtual/interface/lapack@>=3.7.1", ] -Pillow 10.1.0: +Pillow 10.1.0 +''''''''''''' + +``pyproject.toml`` content: .. code:: toml @@ -580,7 +626,10 @@ Pillow 10.1.0: ] -NAVis 1.4.0: +NAVis 1.4.0 +''''''''''' + +``pyproject.toml`` content: .. code:: toml @@ -598,7 +647,18 @@ NAVis 1.4.0: "dep:cran/nat.nblast", ] -Spyder 6.0: +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Provides-External-Extra: nat + Requires-External-Dep: dep:cran/nat; extra == 'nat' + Requires-External-Dep: dep:cran/nat.nblast; extra == 'nat' + +Spyder 6.0 +'''''''''' + +``pyproject.toml`` content: .. code:: toml @@ -609,7 +669,18 @@ Spyder 6.0: "dep:golang/github.com/junegunn/fzf", ] -jupyterlab-git 0.41.0: +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Requires-External-Dep: dep:cargo/ripgrep + Requires-External-Dep: dep:cargo/tree-sitter-cli + Requires-External-Dep: dep:golang/github.com/junegunn/fzf + +jupyterlab-git 0.41.0 +''''''''''''''''''''' + +``pyproject.toml`` content: .. code:: toml @@ -623,7 +694,16 @@ jupyterlab-git 0.41.0: "dep:generic/nodejs", ] -PyEnchant 3.2.2: +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Requires-External-Dep: dep:generic/git + +PyEnchant 3.2.2 +''''''''''''''' + +``pyproject.toml`` content: .. code:: toml @@ -635,7 +715,10 @@ PyEnchant 3.2.2: "dep:github/AbiWord/enchant", ] -Example with dependency groups: +With dependency groups +'''''''''''''''''''''' + +``pyproject.toml`` content: .. code:: toml @@ -645,25 +728,23 @@ Example with dependency groups: "dep:generic/valgrind", ] -Core-metadata entries of ``external.dependencies`` items: - -- ``dep:generic/gcc@14`` becomes ``pkg:generic/gcc (14)``. -- ``dep:generic/libarrow@>=19`` becomes ``pkg:generic/libarrow (vers:pypi/>=19)``. -- ``dep:virtual/compiler/c`` stays ``dep:virtual/compiler/c``. -- ``dep:virtual/interface/blas>=3,<4`` becomes - ``dep:virtual/interface/blas (vers:pypi/>=3,<4)``. -- For completeness, ``dep:pypi/requests`` would become ``pkg:pypi/requests`` and - ``dep:pypi/django@>=5`` would become ``pkg:pypi/django (vers:pypi/>=5)``, but - these examples are unlikely to be seen in ``Requires-External``. - Backwards Compatibility ======================= -There is no impact on backwards compatibility, as this PEP only adds new, +There is no major impact on backwards compatibility, as this PEP primarily adds new, optional metadata. In the absence of such metadata, nothing changes for package authors or packaging tooling. +The only change introduced in this PEP that has impact on existing projects is the +deprecation of the ``External-Requires`` Core Metadata field. We estimate the impact +of this deprecation to be negligible, given the its low penetration in the ecosystem +(see Rationale). + +The field will still be recognized by existing tools such as `setuptools-ext`_ +but its usage will be discouraged in the `Python Packaging User Guide`_, similar to +what it's done for obsolete fields like ``Requires`` (deprecated in favor of +``Requires-Dist``). Security Implications ===================== @@ -807,6 +888,22 @@ tackle this. `This issue `__ contains more arguments in favor and against adding ``host-requires`` under ``[build-system]`` as part of this PEP. +Reusing the ``Requires-External`` field in Core Metadata +-------------------------------------------------------- + +The `Core Metadata`_ specification contains one relevant field, namely +``Requires-External``. While at first sight it would be a good candidate +to record the ``external.dependencies`` table, the authors have decided +to not re-use this field to propagate the external runtime dependencies metadata. + +The ``Requires-External`` field has very loosely defined semantics as of +version 2.4. Essentially: ``name [(version)][; environment marker]`` +(with square brackets denoting optional fields). It is not defined what valid strings for ``name`` are; +the example in the specification uses both "C" as a language name, and "libpng" as a package name. +Tightening up the semantics would be backwards incompatible, and leaving it as is seems +unsatisfactory. The ``dep:`` URLs would need +to be decomposed to fit in this syntax. + Open Issues =========== @@ -880,7 +977,7 @@ CC0-1.0-Universal license, whichever is more permissive. .. _PyPI: https://pypi.org -.. _core metadata: https://packaging.python.org/specifications/core-metadata/ +.. _Core Metadata: https://packaging.python.org/specifications/core-metadata/ .. _setuptools: https://setuptools.readthedocs.io/ .. _setuptools metadata: https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata .. _SPDX: https://spdx.dev/ @@ -903,3 +1000,10 @@ CC0-1.0-Universal license, whichever is more permissive. .. _rgommers/external-deps-build: https://github.com/rgommers/external-deps-build .. _pyproject-external: https://github.com/jaimergp/pyproject-external .. _Reference LAPACK: https://github.com/Reference-LAPACK/lapack +.. _setuptools-ext: https://pypi.org/project/setuptools-ext/ +.. _PyPI JSON API responses: https://docs.pypi.org/api/json/ +.. _pypa/hatch#1712: https://github.com/pypa/hatch/issues/1712 +.. _pypa/flit#353: https://github.com/pypa/flit/issues/353 +.. _pypa/setuptools#4220: https://github.com/pypa/setuptools/discussions/4220#discussioncomment-8930671 +.. _poetry-external-dependencies: https://pypi.org/project/poetry-external-dependencies/ +.. _PyO3/maturin@5b0e4808: https://github.com/PyO3/maturin/commit/5b0e4808bb8852fe796cd2848932a35fbb14de8b From df3b561614670bc5b82feba1ebe15f45529335d1 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 25 Jul 2025 11:01:41 +0200 Subject: [PATCH 09/15] clarify host vs target (#32) --- peps/pep-0725.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index f5e031740ea..df082b08b73 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -135,11 +135,11 @@ This PEP uses the following terminology: - *host machine*: the machine on which the produced artifact will be installed and run. - *build dependency*: dependency for building the package that needs to be - present at build time and itself was built for the build machine's OS and + present at build time and itself was built for the *build* machine's OS and architecture. - *host dependency*: dependency for building the package that needs to be - present at build time and itself was built for the host machine's OS and - architecture. + present at build time and itself was built for the *host* machine's OS and + architecture. These are usually libraries the project needs to link to. - *runtime dependency*: a library or other software component needed when the Python package is used after it's installed. @@ -147,7 +147,7 @@ Note that this terminology is not consistent across build and packaging tools, so care must be taken when comparing build/host dependencies in ``pyproject.toml`` to dependencies from other package managers. -Note that "target machine" or "target dependency" is not used in this PEP. That +Note that "target machine" or "target dependency" are not used in this PEP. That is typically only relevant for cross-compiling compilers or other such advanced scenarios [#gcc-cross-terminology]_, [#meson-cross]_ -- this is out of scope for this PEP. @@ -160,7 +160,7 @@ build-time dependencies is ``build-requires``. Hence this PEP uses the keys Build and host dependencies ''''''''''''''''''''''''''' -Clear separation of metadata associated with the definition of build and target +Clear separation of metadata associated with the definition of build and host platforms, rather than assuming that build and host platform will always be the same, is important [#pypackaging-native-cross]_. From 64a1b70ecb4c117b258ccfecfcb7df5e6c7dbfba Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 25 Jul 2025 12:39:11 +0200 Subject: [PATCH 10/15] Reword 'cross-compiling compilers' (#33) --- peps/pep-0725.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index df082b08b73..91b37b65b45 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -148,7 +148,7 @@ so care must be taken when comparing build/host dependencies in ``pyproject.toml`` to dependencies from other package managers. Note that "target machine" or "target dependency" are not used in this PEP. That -is typically only relevant for cross-compiling compilers or other such advanced +is typically only relevant for cross-compiling a compiler or other such advanced scenarios [#gcc-cross-terminology]_, [#meson-cross]_ -- this is out of scope for this PEP. From a51a28e5177acc5db5e78d1eb2c8b84f6c89ddd4 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 25 Jul 2025 13:44:33 +0200 Subject: [PATCH 11/15] Mention elfdeps (#35) * Mention elfdeps * pre-commit * Apply suggestions from code review Co-authored-by: Ralf Gommers * Move elfdeps mention a bit later * Update peps/pep-0725.rst Co-authored-by: jaimergp --------- Co-authored-by: Ralf Gommers --- peps/pep-0725.rst | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 91b37b65b45..a46faceb18d 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -59,10 +59,13 @@ full sets of dependencies for Python packages, and have tools like pyp2spec_ automatically generate dependency metadata for their own package managers from the metadata in upstream Python packages. External dependencies are currently handled manually, because there is no metadata for this in ``pyproject.toml`` or any other -standard location. Enabling automating this conversion is a key benefit of -this PEP, making packaging Python packages for distros easier and more reliable. In addition, the -authors envision other types of tools making use of this information, e.g., -dependency analysis tools like Repology_, Dependabot_ and libraries.io_. +standard location. Other tools resort to extracting dependencies from extension +modules and shared libraries inside Python packages, like elfdeps_ (Fedora). +Enabling automating this type of conversion by only using explicitly annotated metadata +is a key benefit of this PEP, making packaging Python packages for distros easier +and more reliable. In addition, the authors envision other types of tools +making use of this information, e.g., dependency analysis tools like Repology_, +Dependabot_ and libraries.io_. Software bill of materials (SBOM) generation tools may also be able to use this information, e.g. for flagging that external dependencies listed in @@ -250,7 +253,7 @@ The *name* should be the most common name for the interface or language, lowerca Some examples include:: dep:virtual/compiler/c - dep:virtual/compiler/c++ + dep:virtual/compiler/cxx dep:virtual/compiler/rust dep:virtual/interface/blas dep:virtual/interface/lapack @@ -900,7 +903,7 @@ The ``Requires-External`` field has very loosely defined semantics as of version 2.4. Essentially: ``name [(version)][; environment marker]`` (with square brackets denoting optional fields). It is not defined what valid strings for ``name`` are; the example in the specification uses both "C" as a language name, and "libpng" as a package name. -Tightening up the semantics would be backwards incompatible, and leaving it as is seems +Tightening up the semantics would be backwards incompatible, and leaving it as is seems unsatisfactory. The ``dep:`` URLs would need to be decomposed to fit in this syntax. @@ -1007,3 +1010,4 @@ CC0-1.0-Universal license, whichever is more permissive. .. _pypa/setuptools#4220: https://github.com/pypa/setuptools/discussions/4220#discussioncomment-8930671 .. _poetry-external-dependencies: https://pypi.org/project/poetry-external-dependencies/ .. _PyO3/maturin@5b0e4808: https://github.com/PyO3/maturin/commit/5b0e4808bb8852fe796cd2848932a35fbb14de8b +.. _elfdeps: https://github.com/python-wheel-build/elfdeps/ From c2e6bf64722bf546f3a1a1b3ef158e70149cda85 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Thu, 31 Jul 2025 13:29:50 +0200 Subject: [PATCH 12/15] Address @tobiasdiez's feedback in #15 (#38) --- peps/pep-0725.rst | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index a46faceb18d..0571058b21b 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -137,14 +137,16 @@ This PEP uses the following terminology: executed. - *host machine*: the machine on which the produced artifact will be installed and run. -- *build dependency*: dependency for building the package that needs to be - present at build time and itself was built for the *build* machine's OS and - architecture. -- *host dependency*: dependency for building the package that needs to be - present at build time and itself was built for the *host* machine's OS and - architecture. These are usually libraries the project needs to link to. -- *runtime dependency*: a library or other software component needed when - the Python package is used after it's installed. +- *build dependency*: package required only during the build process. It must + be available at build time and is built for the *build* machine's OS and + architecture. Typical examples include compilers, code generators, and + build tools. +- *host dependency*: package needed during the build and often also at runtime. + It must be available during the build and is built for the *host* machine's OS + and architecture. These are usually libraries the project links against. +- *runtime dependency*: package required only when the package is used after + installation. It is not required at build time but must be available on + the *host* machine at runtime. Note that this terminology is not consistent across build and packaging tools, so care must be taken when comparing build/host dependencies in From 6e99291e28dda040ed9926950caa74e0aa736059 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 12 Aug 2025 15:10:41 +0200 Subject: [PATCH 13/15] 725: Extend Rejected ideas (#42) * Extend Rejected ideas * Apply suggestions from code review Co-authored-by: Ralf Gommers * Address feedback in -dev/devel split * Address feedback in Identifier indirections Co-authored-by: rgommers --------- Co-authored-by: Ralf Gommers --- peps/pep-0725.rst | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 0571058b21b..dee376a9df4 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -870,15 +870,45 @@ those, and if it's not present on the system then install the PyPI package for it. The authors believe that specific support for this scenario is not necessary (or too complex to justify such support); a dependency provider for external dependencies can treat PyPI as one possible source for obtaining the -package. +package. An example mapping for this use case is proposed in *name mapping PEP*. Using library and header names as external dependencies ------------------------------------------------------- A previous draft PEP (`"External dependencies" (2015) `__) proposed using specific library and header names as external dependencies. This -is too granular; using package names is a well-established pattern across -packaging ecosystems and should be preferred. +is both too granular, and insufficient (e.g., headers are often unversioned; multiple +packages may provide the same header or library). Using +package names is a well-established pattern across packaging ecosystems +and should be preferred. + +Splitting host dependencies with explicit ``-dev`` or ``-devel`` suffixes +------------------------------------------------------------------------- + +This convention is not consistent across all ecosystems. Since the need +for explicit control is quite niche and we don't want to add design +complexity without enough clear use cases, we have chosen to rely solely +on the the ``build``, ``host`` and ``run`` category split, with tools +being in charge of which category applies to each case in a context-dependent way. + +If this proves to be insufficient, a future PEP could use the URL qualifier features +present in the PURL schema (``?key=value``) to implement the necessary adjustments. + +Identifier indirections +----------------------- + +Some ecosystems exhibit methods to select packages based on parametrized functions +like ``cmake("dependency")`` or ``compiler("language")``, which return package +names based on some additional context or configuration. This feature is arguably +not very common and, even when present, rarely used. Additionally, its dynamic +nature makes it prone to changing meaning over time, and relying on specific build +systems for the name resolution is in general not a good idea. + +The authors prefer static identifiers that can be mapped explicitly via well known metadata +(e.g. as proposed in *name mapping PEP*). + +Ecosystems that do implement these indirections can use them to support the infrastructure +designed to generate the mappings proposed in *name mapping PEP*. Adding a ``host-requires`` key under ``[build-system]`` ------------------------------------------------------- From dcf9bb711c9418dceab63dc537783e4da642fdc2 Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Thu, 28 Aug 2025 14:02:12 +0200 Subject: [PATCH 14/15] PEP 725: expand Specification section --- peps/pep-0725.rst | 87 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 64 insertions(+), 23 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index dee376a9df4..c40d89111d7 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -445,6 +445,42 @@ Specification If metadata is improperly specified then tools MUST raise an error to notify the user about their mistake. +DepURL +------ + +A DepURL implements a scheme for identifying packages that is meant to be +portable across packaging ecosystems. Its design is:: + + dep:type/namespace/name@version?qualifiers#subpath + +``dep:`` is a fixed string, and always present. ``type`` and ``name`` are +required, other components are optional. Component definitions: + +- ``type`` (required): MUST be either a `PURL`_ ``type``, or ``virtual``. +- ``namespace`` (optional): MUST be a `PURL`_ ``namespace``, or a namespace in + the DepURL central registry (see :pep:`700`). FIXME: PEP number +- ``name`` (required): MUST be a name that parses as a valid `PURL`_ ``name``. + Tools MAY warn or error if a name is not present in the DepURL central + registry (:pep:`700`). +- ``version`` (optional): MUST be one of: + + - A regular `version specifier`_ (PEP 440 semantics) as a single version or + version range, with the restriction that only the following operators may + be used: ``>=``, ``>``, ``<``, ``<=``, ``==``, ``,``. + - A `PURL`_ percent-encoded version string. + +- ``qualifiers`` (optional): MUST parse as a valid `PURL`_ ``qualifier``. +- ``subpath`` (optional): MUST parse as a valid `PURL`_ ``subpath``. + + +External dependency specifiers +------------------------------ + +External dependency specifiers MUST contain a DepURL, and MAY contain +environment markers with the same syntax as used in regular `dependency +specifiers`_ (as originally specified in :pep:`508`). + + Changes in Core Metadata ------------------------ @@ -459,7 +495,8 @@ Additions Two new fields are added to Core Metadata: -- ``Requires-External-Dep``. An external requirement expressed as a ``dep:`` URL. +- ``Requires-External-Dep``. An external requirement expressed as an external + dependency specifier string. - ``Provides-External-Extra``. An *extra* group that carries external dependencies (as found in ``Requires-External-Dep``) only. @@ -490,72 +527,73 @@ to be present on the system already. ``build-requires``/``optional-build-requires`` '''''''''''''''''''''''''''''''''''''''''''''' -- Format: Array of ``dep:`` strings (``build-requires``) and a table - with values of arrays of ``dep:`` strings (``optional-build-requires``) +- Format: Array of external dependency specifiers (``build-requires``) and a + table with values of arrays of external dependency strings + (``optional-build-requires``) - `Core metadata`_: N/A The (optional) external build requirements needed to build the project. For ``build-requires``, it is a key whose value is an array of strings. Each string represents a build requirement of the project and MUST be formatted as -a valid ``dep:`` string. +a valid external dependency string. For ``optional-build-requires``, it is a table where each key specifies an extra set of build requirements and whose value is an array of strings. The -strings of the arrays MUST be valid ``dep:`` strings. +strings of the arrays MUST be valid external dependency strings. ``host-requires``/``optional-host-requires`` '''''''''''''''''''''''''''''''''''''''''''' -- Format: Array of ``dep:`` strings (``host-requires``) and a table - with values of arrays of ``dep:`` strings (``optional-host-requires``) -- `Core metadata`_: N/A +- Format: Array of external dependency strings (``host-requires``) and a table + with values of arrays of external dependency strings (``optional-host-requires``) - + `Core metadata`_: N/A The (optional) external host requirements needed to build the project. For ``host-requires``, it is a key whose value is an array of strings. Each string represents a host requirement of the project and MUST be formatted as -a valid ``dep:`` string. +a valid external dependency string. For ``optional-host-requires``, it is a table where each key specifies an extra set of host requirements and whose value is an array of strings. The -strings of the arrays MUST be valid ``dep:`` strings. +strings of the arrays MUST be valid external dependency strings. ``dependencies``/``optional-dependencies`` '''''''''''''''''''''''''''''''''''''''''' -- Format: Array of ``dep:`` strings (``dependencies``) and a table - with values of arrays of ``dep:`` strings (``optional-dependencies``) +- Format: Array of external dependency strings (``dependencies``) and a table + with values of arrays of external dependency strings + (``optional-dependencies``) - `Core metadata`_: ``Requires-External-Dep``, ``Provides-External-Extra`` The (optional) runtime dependencies of the project. For ``dependencies``, it is a key whose value is an array of strings. Each -string represents a dependency of the project and MUST be formatted as either a -valid ``dep:`` string. Each string must be added to `Core Metadata`_ as a +string represents a dependency of the project and MUST be formatted as a valid +external dependency string. Each string must be added to `Core Metadata`_ as a ``Requires-External-Dep`` field. For ``optional-dependencies``, it is a table where each key specifies an *extra* and whose value is an array of strings. The strings of the arrays MUST be valid -``dep:`` strings. For each ``optional-dependencies`` group: +external dependency strings. For each ``optional-dependencies`` group: - The name of the group MUST be added to `Core Metadata`_ as a ``Provides-External-Extra`` field. -- The ``dep:`` URLs in that group MUST be added to `Core Metadata`_ as a - ``Requires-External-Dep`` field, with the corresponding ``; extra == 'name'`` - environment marker. +- The external dependency specifiers in that group MUST be added to `Core + Metadata`_ as a ``Requires-External-Dep`` field, with the corresponding ``; + extra == 'name'`` environment marker. ``dependency-groups`` ''''''''''''''''''''' - Format: A table where each key is the name of the group, and the values are - arrays of ``dep:`` strings, tables, or a mix of both. + arrays of external dependency strings, tables, or a mix of both. - `Core metadata`_: N/A -PEP 735 -style dependency groups, but using ``dep:`` URLs instead of PEP 508 strings as -dependency specifiers. Every other detail (e.g. group inclusion, name normalization) -follows the official `Dependency Groups specification -`__. +PEP 735 -style dependency groups, but using external dependency specifiers +instead of PEP 508 strings. Every other detail (e.g. group inclusion, name +normalization) follows the official `dependency groups specification`_. Examples -------- @@ -1017,6 +1055,9 @@ CC0-1.0-Universal license, whichever is more permissive. .. _setuptools metadata: https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata .. _SPDX: https://spdx.dev/ .. _PURL: https://github.com/package-url/purl-spec/ +.. _version specifier: https://packaging.python.org/en/latest/specifications/version-specifiers/ +.. _dependencies specifier: https://packaging.python.org/en/latest/specifications/dependency-specifiers/ +.. _dependency groups specification: https://packaging.python.org/en/latest/specifications/dependency-groups/ .. _packageurl-python: https://pypi.org/project/packageurl-python/ .. _vers: https://github.com/package-url/purl-spec/blob/version-range-spec/VERSION-RANGE-SPEC.rst .. _vers implementation for PURL: https://github.com/package-url/purl-spec/pull/139 From 01380f06e7a856066788733cf1e576a45df94b76 Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Thu, 28 Aug 2025 14:21:56 +0200 Subject: [PATCH 15/15] PEP 725: improve content of Examples section --- peps/pep-0725.rst | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index c40d89111d7..4765bd4560a 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -598,7 +598,8 @@ normalization) follows the official `dependency groups specification`_. Examples -------- -These examples show what the ``[external]`` content for a number of packages is +These examples show what the ``[external]`` table content for a number of +packages, and the corresponding ``PKG-INFO``/``METADATA`` content (if any) is expected to be. cryptography 39.0 @@ -619,6 +620,8 @@ cryptography 39.0 "dep:generic/libffi", ] +``PKG-INFO`` / ``METADATA`` content: N/A. + SciPy 1.10 '''''''''' @@ -639,6 +642,8 @@ SciPy 1.10 "dep:virtual/interface/lapack@>=3.7.1", ] +``PKG-INFO`` / ``METADATA`` content: N/A. + Pillow 10.1.0 ''''''''''''' @@ -668,6 +673,7 @@ Pillow 10.1.0 "dep:generic/tk", ] +``PKG-INFO`` / ``METADATA`` content: N/A. NAVis 1.4.0 ''''''''''' @@ -752,12 +758,22 @@ PyEnchant 3.2.2 [external] dependencies = [ - # libenchant is needed on all platforms but only vendored into wheels on - # Windows, so on Windows the build backend should remove this external - # dependency from wheel metadata. - "dep:github/AbiWord/enchant", + # libenchant is needed on all platforms but vendored into wheels + # distributed on PyPI for Windows. Hence choose to encode that in + # the metadata. Note: there is no completely unambiguous way to do + # this; another choice is to leave out the environment marker in the + # source distribution and either live with the unnecessary ``METADATA`` + # entry in the distributed Windows wheels, or to apply a patch to this + # metadata when building those wheels. + "dep:github/AbiWord/enchant; platform_system!='Windows'", ] +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Requires-External-Dep: dep:github/AbiWord/enchant; platform_system!="Windows" + With dependency groups '''''''''''''''''''''' @@ -771,6 +787,7 @@ With dependency groups "dep:generic/valgrind", ] +``PKG-INFO`` / ``METADATA`` content: N/A. Backwards Compatibility =======================