diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 7c9d83f949b..4765bd4560a 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -1,6 +1,7 @@ PEP: 725 Title: Specifying external dependencies in pyproject.toml Author: Pradyun Gedam , + Jaime Rodríguez-Guerra , Ralf Gommers Discussions-To: https://discuss.python.org/t/31888 Status: Draft @@ -18,15 +19,21 @@ runtime dependencies in a ``pyproject.toml`` file for packaging-related tools to consume. This PEP proposes to add an ``[external]`` table to ``pyproject.toml`` with -three keys: "build-requires", "host-requires" and "dependencies". These -are for specifying three types of dependencies: +seven keys. "build-requires", "host-requires" and "dependencies" +are for specifying three types of *required* dependencies: 1. ``build-requires``, build tools to run on the build machine -2. ``host-requires``, build dependencies needed for host machine but also needed at build time. +2. ``host-requires``, build dependencies needed for the host machine but also needed at build time. 3. ``dependencies``, needed at runtime on the host machine but not needed at build time. -Cross compilation is taken into account by distinguishing build and host dependencies. -Optional build-time and runtime dependencies are supported too, in a manner analogies +These three keys also have their *optional* ``external`` counterparts (``optional-build-requires``, +``optional-host-requires``, ``optional-dependencies``), which have the same role that +``project.optional-dependencies`` plays for ``project.dependencies``. Finally, +``dependency-groups`` offers the same functionality as :pep:`735` but for external +dependencies. + +Cross compilation is taken into account by distinguishing between build and host dependencies. +Optional build-time and runtime dependencies are supported too, in a manner analogous to how that is supported in the ``[project]`` table. @@ -41,25 +48,31 @@ this PEP are to: - Enable tools to automatically map external dependencies to packages in other packaging repositories, -- Make it possible to include needed dependencies in error messages emitting by +- Make it possible to include needed dependencies in error messages emitted by Python package installers and build frontends, - Provide a canonical place for package authors to record this dependency information. -Packaging ecosystems like Linux distros, Conda, Homebrew, Spack, and Nix need +Packaging ecosystems like Linux distros, conda, Homebrew, Spack, and Nix need full sets of dependencies for Python packages, and have tools like pyp2spec_ -(Fedora), Grayskull_ (Conda), and dh_python_ (Debian) which attempt to +(Fedora), Grayskull_ (conda), and dh_python_ (Debian) which attempt to automatically generate dependency metadata for their own package managers from the metadata in upstream Python packages. External dependencies are currently handled manually, because there is no metadata for this in ``pyproject.toml`` or any other -standard location. Enabling automating this conversion is a key benefit of -this PEP, making packaging Python packages for distros easier and more reliable. In addition, the -authors envision other types of tools making use of this information, e.g., -dependency analysis tools like Repology_, Dependabot_ and libraries.io_. +standard location. Other tools resort to extracting dependencies from extension +modules and shared libraries inside Python packages, like elfdeps_ (Fedora). +Enabling automating this type of conversion by only using explicitly annotated metadata +is a key benefit of this PEP, making packaging Python packages for distros easier +and more reliable. In addition, the authors envision other types of tools +making use of this information, e.g., dependency analysis tools like Repology_, +Dependabot_ and libraries.io_. + Software bill of materials (SBOM) generation tools may also be able to use this information, e.g. for flagging that external dependencies listed in ``pyproject.toml`` but not contained in wheel metadata are likely vendored -within the wheel. +within the wheel. :pep:`770` ("Improving measurability of Python packages with +Software Bill-of-Materials"), which standardizes how SBOMs are included in +wheels, contains an instructive section on how that PEP differs from this one. Packages with external dependencies are typically hard to build from source, and error messages from build failures tend to be hard to decipher for end @@ -101,14 +114,14 @@ Multiple types of external dependencies can be distinguished: concrete packages. E.g., a C++ compiler, BLAS, LAPACK, OpenMP, MPI. Concrete packages are straightforward to understand, and are a concept present -in virtually every package management system. Virtual packages are a concept +in every package management system. Virtual packages are a concept also present in a number of packaging systems -- but not always, and the -details of their implementation varies. +details of their implementation vary. Cross compilation ----------------- -Cross compilation is not yet (as of August 2023) well-supported by stdlib +Cross compilation is not yet (as of May 2025) well-supported by stdlib modules and ``pyproject.toml`` metadata. It is however important when translating external dependencies to those of other packaging systems (with tools like ``pyp2spec``). Introducing support for cross compilation immediately @@ -121,23 +134,27 @@ Terminology This PEP uses the following terminology: - *build machine*: the machine on which the package build process is being - executed + executed. - *host machine*: the machine on which the produced artifact will be installed - and run -- *build dependency*: dependency for building the package that needs to be - present at build time and itself was built for the build machine's OS and - architecture -- *host dependency*: dependency for building the package that needs to be - present at build time and itself was built for the host machine's OS and - architecture + and run. +- *build dependency*: package required only during the build process. It must + be available at build time and is built for the *build* machine's OS and + architecture. Typical examples include compilers, code generators, and + build tools. +- *host dependency*: package needed during the build and often also at runtime. + It must be available during the build and is built for the *host* machine's OS + and architecture. These are usually libraries the project links against. +- *runtime dependency*: package required only when the package is used after + installation. It is not required at build time but must be available on + the *host* machine at runtime. Note that this terminology is not consistent across build and packaging tools, so care must be taken when comparing build/host dependencies in ``pyproject.toml`` to dependencies from other package managers. -Note that "target machine" or "target dependency" is not used in this PEP. That -is typically only relevant for cross-compiling compilers or other such advanced -scenarios [#gcc-cross-terminology]_, [#meson-cross]_ - this is out of scope for +Note that "target machine" or "target dependency" are not used in this PEP. That +is typically only relevant for cross-compiling a compiler or other such advanced +scenarios [#gcc-cross-terminology]_, [#meson-cross]_ -- this is out of scope for this PEP. Finally, note that while "dependency" is the term most widely used for packages @@ -148,8 +165,8 @@ build-time dependencies is ``build-requires``. Hence this PEP uses the keys Build and host dependencies ''''''''''''''''''''''''''' -Clear separation of metadata associated with the definition of build and target -platforms, rather than assuming that build and target platform will always be +Clear separation of metadata associated with the definition of build and host +platforms, rather than assuming that build and host platform will always be the same, is important [#pypackaging-native-cross]_. Build dependencies are typically run during the build process - they may be @@ -178,22 +195,33 @@ cause of a build failure when a package fails to build from an sdist. Specifying external dependencies -------------------------------- -Concrete package specification through PURL -''''''''''''''''''''''''''''''''''''''''''' +Concrete package specification +'''''''''''''''''''''''''''''' -The two types of concrete packages are supported by PURL_ (Package URL), which -implements a scheme for identifying packages that is meant to be portable +A `PURL`_ implements a scheme for identifying packages that is meant to be portable across packaging ecosystems. Its design is:: scheme:type/namespace/name@version?qualifiers#subpath The ``scheme`` component is a fixed string, ``pkg``, and of the other -components only ``type`` and ``name`` are required. As an example, a package -URL for the ``requests`` package on PyPI would be:: +components only ``type`` and ``name`` are required. + +Since external dependencies are likely to be typed by hand, we propose a PURL +derivative that, in the name of ergonomics and user-friendliness, introduces a +number of changes (further discussed below): + +- Support for virtual packages via a new ``virtual`` type. +- Allow version ranges (and not just literals) in the ``version`` field. + +In this derivative, we replace the ``pkg`` scheme with ``dep``. Hence, +we will refer to them as ``dep:`` URLs. + +As an example, a ``dep:`` URL for the ``requests`` package on PyPI would be:: - pkg:pypi/requests + dep:pypi/requests + # equivalent to pkg:pypi/requests -Adopting PURL to specify external dependencies in ``pyproject.toml`` solves a +Adopting PURL-compatible strings to specify external dependencies in ``pyproject.toml`` solves a number of problems at once - and there are already implementations of the specification in Python and multiple languages. PURL is also already supported by dependency-related tooling like SPDX (see @@ -203,29 +231,73 @@ and the `Sonatype OSS Index `__; not having to wait years before support in such tooling arrives is valuable. For concrete packages without a canonical package manager to refer to, either -``pkg:generic/pkg-name`` can be used, or a direct reference to the VCS system +``dep:generic/dep-name`` can be used, or a direct reference to the VCS system that the package is maintained in (e.g., -``pkg:github/user-or-org-name/pkg-name``). Which of these is more appropriate -is situation-dependent. This PEP recommends using ``pkg:generic`` when the -package name is unambiguous and well-known (e.g., ``pkg:generic/git`` or -``pkg:generic/openblas``), and using the VCS as the PURL type otherwise. +``dep:github/user-or-org-name/dep-name``). Which of these is more appropriate +is situation-dependent. This PEP recommends using ``dep:generic`` when the +package name is unambiguous and well-known (e.g., ``dep:generic/git`` or +``dep:generic/openblas``), and using the VCS as the type otherwise. Virtual package specification -''''''''''''''''''''''''''''' +'''''''''''''''''''''''''''''' -There is no ready-made support for virtual packages in PURL or another -standard. There are a relatively limited number of such dependencies though, -and adopting a scheme similar to PURL but with the ``virtual:`` rather than -``pkg:`` scheme seems like it will be understandable and map well to Linux -distros with virtual packages and to the likes of Conda and Spack. +PURL does not offer support for virtual or virtual dependency specification yet. +A `proposal to add a virtual type `__ +is being discussed for revision 1.1. -The two known virtual package types are ``compiler`` and ``interface``. +In the meantime, we propose adding a new *type* to our ``dep:`` derivative, the ``virtual`` +type, which can take two *namespaces*: + +- ``interface``: for components such as BLAS or MPI. +- ``compiler``: for compiled languages like C or Rust. + +The *name* should be the most common name for the interface or language, lowercased. +Some examples include:: + + dep:virtual/compiler/c + dep:virtual/compiler/cxx + dep:virtual/compiler/rust + dep:virtual/interface/blas + dep:virtual/interface/lapack + +Since there are a relatively limited number of such dependencies, +it seems like it will be understandable and map well to Linux +distros with virtual packages and to the likes of conda and Spack. Versioning '''''''''' +PURLs support fixed versions via the ``@`` component of the URL. For example, +``numpy===2.0`` can be expressed as ``pkg:pypi/numpy@2.0``. + Support in PURL for version expressions and ranges beyond a fixed version is -still pending, see the Open Issues section. +available via ``vers`` URIs (`see specification `__):: + + vers:type/version-constraint|version-constraint|... + +Users are supposed to couple a ``pkg:`` URL with a ``vers:`` URL. For example, +to express ``numpy>=2.0``, the PURL equivalent would be ``pkg:pypi/numpy`` plus +``vers:pypi/>=2.0``. This can be done with: + +- A two-item list: ``["pkg:pypi/numpy", "vers:pypi/>=2.0"]``. +- A `percent-encoded `__ + URL qualifier: ``pkg:pypi/numpy?vers=vers:pypi%2F%3E%3D2.0``. + +Since none of these options are very ergonomic, the version field in ``dep:`` URLs accepts +version range specifiers too, with these rules: + +- The ``vers:`` scheme is omitted. +- The *type* is omitted and assumed to match the PURL *type*. If there's no match, + the type is assumed to be ``pypi``. +- When no operator is present, the field is understood as a version literal. +- Otherwise, it is considered a version range specifier. + +Some examples: + +- ``dep:pypi/numpy@2.0``: ``numpy`` pinned at exactly version 2.0. +- ``dep:pypi/numpy@>=2.0``: ``numpy`` with version greater or equal than 2.0. +- ``dep:virtual/interface/lapack@>=3.7.1``: any package implementing the + LAPACK interface for version greater or equal than ``3.7.1``. Dependency specifiers ''''''''''''''''''''' @@ -237,41 +309,7 @@ this is pragmatic: dependency specifiers are already used for other metadata in ``pyproject.toml``, any tooling that is used with ``pyproject.toml`` is likely to already have a robust implementation to parse it. And we do not expect to need the extra possibilities that PURL qualifiers provide (e.g. to specify a -Conan or Conda channel, or a RubyGems platform). - -Usage of core metadata fields ------------------------------ - -The `core metadata`_ specification contains one relevant field, namely -``Requires-External``. This has no well-defined semantics in core metadata 2.1; -this PEP chooses to reuse the field for external runtime dependencies. The core -metadata specification does not contain fields for any metadata in -``pyproject.toml``'s ``[build-system]`` table. Therefore the ``build-requires`` -and ``host-requires`` content also does not need to be reflected in core -metadata fields. The ``optional-dependencies`` content from ``[external]`` -would need to either reuse ``Provides-Extra`` or require a new -``Provides-External-Extra`` field. Neither seems desirable. - -Differences between sdist and wheel metadata -'''''''''''''''''''''''''''''''''''''''''''' - -A wheel may vendor its external dependencies. This happens in particular when -distributing wheels on PyPI or other Python package indexes - and tools like -auditwheel_, delvewheel_ and delocate_ automate this process. As a result, a -``Requires-External`` entry in an sdist may disappear from a wheel built from -that sdist. It is also possible that a ``Requires-External`` entry remains in a -wheel, either unchanged or with narrower constraints. ``auditwheel`` does not -vendor certain allow-listed dependencies, such as OpenGL, by default. In -addition, ``auditwheel`` and ``delvewheel`` allow a user to manually exclude -dependencies via a ``--exclude`` or ``--no-dll`` command-line flag. This is -used to avoid vendoring large shared libraries, for example those from CUDA. - -``Requires-External`` entries generated from external dependencies in -``pyproject.toml`` in a wheel are therefore allowed to be narrower than those -for the corresponding sdist. They must not be wider, i.e. constraints must not -allow a version of a dependency for a wheel that isn't allowed for an sdist, -nor contain new dependencies that are not listed in the sdist's metadata at -all. +Conan or conda channel, or a RubyGems platform). Canonical names of dependencies and ``-dev(el)`` split packages ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' @@ -285,7 +323,7 @@ maintain, and should not be reflected in the ``[external]`` table. It is not possible to specify this in a reasonable way that works across distros, hence only the canonical name should be used in ``[external]``. -The intended meaning of using a PURL or virtual dependency is "the full package +The intended meaning of using a ``dep:`` string is "the full package with the name specified". It will depend on the context in which the metadata is used whether the split is relevant. For example, if ``libffi`` is a host dependency and a tool wants to prepare an environment for building a wheel, @@ -306,6 +344,100 @@ consistency between Python dependencies and external dependencies, we choose to add it implicitly. Python development headers must be assumed to be necessary when an ``[external]`` table contains one or more compiler packages. +New Core Metadata fields +------------------------ + +Two new Core Metadata fields are proposed: + +- ``Requires-External-Dep``. An external requirement expressed as a ``dep:`` + URL. Mimics the transition from ``Requires`` to ``Requires-Dist``. We + chose the ``-Dep`` suffix to emphasize that the value is not a regular + Python specifier (distribution), but a ``dep:`` URL. +- ``Provides-External-Extra``. An *extra* group that carries external dependencies + (as found in ``Requires-External-Dep``) only. + +Since the Core Metadata specification does not contain fields for any metadata in +``pyproject.toml``'s ``[build-system]`` table, the ``build-requires`` +and ``host-requires`` content do not need to be reflected in existing core +metadata fields. + +Additionally, this PEP also proposes to deprecate the ``Requires-External`` field. +The reasons being: + +- Avoiding confusion with the newly proposed fields. +- Avoiding potential incompatibilities with existing usage (even if limited). +- Low penetration in the ecosystem: + + - There is no direct correspondence to a field in the ``pyproject.toml`` metadata. + - Mainstream build backends like ``setuptools`` (see `pypa/setuptools#4220`_), + ``hatch`` (see `pypa/hatch#1712`_), ``flit`` (see `pypa/flit#353`_), or ``poetry`` + do not offer ways to specify it or require a plugin (e.g. `poetry-external-dependencies`_). + ``maturin`` does seem to support it since 0.7.0 (see `PyO3/maturin@5b0e4808`_), + but it's not directly documented. Other backends like ``scikit-build-core`` or + ``meson-python`` returned no results for ``External-Requires``. + - The field is not included in the `PyPI JSON API responses`_. + +Note about the differences between sdist and wheel metadata +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +A wheel may vendor its external dependencies. This happens in particular when +distributing wheels on PyPI or other Python package indexes -- and tools like +auditwheel_, delvewheel_ and delocate_ automate this process. As a result, a +``Requires-External`` entry in an sdist may disappear from a wheel built from +that sdist. It is also possible that a ``Requires-External`` entry remains in a +wheel, either unchanged or with narrower constraints. ``auditwheel`` does not +vendor certain allow-listed dependencies, such as OpenGL, by default. In +addition, ``auditwheel`` and ``delvewheel`` allow a user to manually exclude +dependencies via a ``--exclude`` or ``--no-dll`` command-line flag. This is +used to avoid vendoring large shared libraries, for example those from CUDA. + +``Requires-External`` entries generated from external dependencies in +``pyproject.toml`` MAY differ between an sdist and its corresponding wheel(s). +However, the ``Requires-External`` entries in the wheels MUST always specify +a narrower set of dependencies. + +Note that this does not imply that the field must be marked as Dynamic, since this +distinction only applies to wheels built from an sdist by a build backend. In +particular, wheels built from other wheels do not need to satisfy this constraint (see +`message 179 in this DPO thread +`__ +.) + +Dependency groups +----------------- + +This PEP has chosen to include the :pep:`735` key ``dependency-groups`` under +the ``[external]`` table too. This decision is motivated by the need of having +similar functionality for external metadata. The top-level table cannot be used +for external dependencies because it's expected to have PEP 508 strings (and tables +for group includes), while we have chosen to rely on ``dep:`` URLs for the external +dependencies. Conflating both would raise significant backwards compatibility +issues with existing usage. + +Strictly speaking, the ``dependency-groups`` schema allows us to define external +dependencies in per-group sub-tables:: + + [dependency-groups] + dev = [ + "pytest", + { external = ["dep:cargo/ripgrep"] }, + ] + +However, this has the same problem: we are mixing different types of dependency +specifiers in the same data structure. We believe it's cleaner to separate concerns +in different top-level tables, hence why we still prefer to have +``external.dependency-groups``. + +Optional dependencies versus dependency groups +'''''''''''''''''''''''''''''''''''''''''''''' + +The rationale for having ``external.dependency-groups`` is identical for the +rationale given in :pep:`735` for introducing ``[dependency-groups]``. The +intended usage and semantics of inclusion/exclusion into Core Metadata +is thus identical to ``[dependency-groups]``. + +``external.optional-dependencies`` will show up in Core Metadata. +``external.dependency-groups`` will not. Specification ============= @@ -313,8 +445,73 @@ Specification If metadata is improperly specified then tools MUST raise an error to notify the user about their mistake. -Details -------- +DepURL +------ + +A DepURL implements a scheme for identifying packages that is meant to be +portable across packaging ecosystems. Its design is:: + + dep:type/namespace/name@version?qualifiers#subpath + +``dep:`` is a fixed string, and always present. ``type`` and ``name`` are +required, other components are optional. Component definitions: + +- ``type`` (required): MUST be either a `PURL`_ ``type``, or ``virtual``. +- ``namespace`` (optional): MUST be a `PURL`_ ``namespace``, or a namespace in + the DepURL central registry (see :pep:`700`). FIXME: PEP number +- ``name`` (required): MUST be a name that parses as a valid `PURL`_ ``name``. + Tools MAY warn or error if a name is not present in the DepURL central + registry (:pep:`700`). +- ``version`` (optional): MUST be one of: + + - A regular `version specifier`_ (PEP 440 semantics) as a single version or + version range, with the restriction that only the following operators may + be used: ``>=``, ``>``, ``<``, ``<=``, ``==``, ``,``. + - A `PURL`_ percent-encoded version string. + +- ``qualifiers`` (optional): MUST parse as a valid `PURL`_ ``qualifier``. +- ``subpath`` (optional): MUST parse as a valid `PURL`_ ``subpath``. + + +External dependency specifiers +------------------------------ + +External dependency specifiers MUST contain a DepURL, and MAY contain +environment markers with the same syntax as used in regular `dependency +specifiers`_ (as originally specified in :pep:`508`). + + +Changes in Core Metadata +------------------------ + +Deprecations +'''''''''''' + +The ``External-Requires`` Core Metadata field will be marked as *obsolete* and its +usage will be discouraged. + +Additions +''''''''' + +Two new fields are added to Core Metadata: + +- ``Requires-External-Dep``. An external requirement expressed as an external + dependency specifier string. +- ``Provides-External-Extra``. An *extra* group that carries external dependencies + (as found in ``Requires-External-Dep``) only. + +Version bump +'''''''''''' + +Given that the proposed changes are purely additive, the Core Metadata +version will be bumped to 2.5. + +This will only impact PyPI and tools that want to support external runtime dependencies, +and require no changes otherwise. + + +Changes in ``pyproject.toml`` +----------------------------- Note that ``pyproject.toml`` content is in the same format as in :pep:`621`. @@ -330,121 +527,158 @@ to be present on the system already. ``build-requires``/``optional-build-requires`` '''''''''''''''''''''''''''''''''''''''''''''' -- Format: Array of PURL_ strings (``build-requires``) and a table - with values of arrays of PURL_ strings (``optional-build-requires``) +- Format: Array of external dependency specifiers (``build-requires``) and a + table with values of arrays of external dependency strings + (``optional-build-requires``) - `Core metadata`_: N/A The (optional) external build requirements needed to build the project. For ``build-requires``, it is a key whose value is an array of strings. Each string represents a build requirement of the project and MUST be formatted as -either a valid PURL_ string or a ``virtual:`` string. +a valid external dependency string. For ``optional-build-requires``, it is a table where each key specifies an extra set of build requirements and whose value is an array of strings. The -strings of the arrays MUST be valid PURL_ strings. +strings of the arrays MUST be valid external dependency strings. ``host-requires``/``optional-host-requires`` '''''''''''''''''''''''''''''''''''''''''''' -- Format: Array of PURL_ strings (``host-requires``) and a table - with values of arrays of PURL_ strings (``optional-host-requires``) -- `Core metadata`_: N/A +- Format: Array of external dependency strings (``host-requires``) and a table + with values of arrays of external dependency strings (``optional-host-requires``) - + `Core metadata`_: N/A The (optional) external host requirements needed to build the project. For ``host-requires``, it is a key whose value is an array of strings. Each string represents a host requirement of the project and MUST be formatted as -either a valid PURL_ string or a ``virtual:`` string. +a valid external dependency string. For ``optional-host-requires``, it is a table where each key specifies an extra set of host requirements and whose value is an array of strings. The -strings of the arrays MUST be valid PURL_ strings. +strings of the arrays MUST be valid external dependency strings. ``dependencies``/``optional-dependencies`` '''''''''''''''''''''''''''''''''''''''''' -- Format: Array of PURL_ strings (``dependencies``) and a table - with values of arrays of PURL_ strings (``optional-dependencies``) -- `Core metadata`_: ``Requires-External``, N/A +- Format: Array of external dependency strings (``dependencies``) and a table + with values of arrays of external dependency strings + (``optional-dependencies``) +- `Core metadata`_: ``Requires-External-Dep``, ``Provides-External-Extra`` The (optional) runtime dependencies of the project. For ``dependencies``, it is a key whose value is an array of strings. Each -string represents a dependency of the project and MUST be formatted as either a -valid PURL_ string or a ``virtual:`` string. Each string maps directly to a -``Requires-External`` entry in the `core metadata`_. +string represents a dependency of the project and MUST be formatted as a valid +external dependency string. Each string must be added to `Core Metadata`_ as a +``Requires-External-Dep`` field. -For ``optional-dependencies``, it is a table where each key specifies an extra +For ``optional-dependencies``, it is a table where each key specifies an *extra* and whose value is an array of strings. The strings of the arrays MUST be valid -PURL_ strings. Optional dependencies do not map to a core metadata field. +external dependency strings. For each ``optional-dependencies`` group: + +- The name of the group MUST be added to `Core Metadata`_ as a + ``Provides-External-Extra`` field. +- The external dependency specifiers in that group MUST be added to `Core + Metadata`_ as a ``Requires-External-Dep`` field, with the corresponding ``; + extra == 'name'`` environment marker. + +``dependency-groups`` +''''''''''''''''''''' + +- Format: A table where each key is the name of the group, and the values are + arrays of external dependency strings, tables, or a mix of both. +- `Core metadata`_: N/A + +PEP 735 -style dependency groups, but using external dependency specifiers +instead of PEP 508 strings. Every other detail (e.g. group inclusion, name +normalization) follows the official `dependency groups specification`_. Examples -------- -These examples show what the ``[external]`` content for a number of packages is +These examples show what the ``[external]`` table content for a number of +packages, and the corresponding ``PKG-INFO``/``METADATA`` content (if any) is expected to be. -cryptography 39.0: +cryptography 39.0 +''''''''''''''''' + +``pyproject.toml`` content: .. code:: toml [external] build-requires = [ - "virtual:compiler/c", - "virtual:compiler/rust", - "pkg:generic/pkg-config", + "dep:virtual/compiler/c", + "dep:virtual/compiler/rust", + "dep:generic/pkg-config", ] host-requires = [ - "pkg:generic/openssl", - "pkg:generic/libffi", + "dep:generic/openssl", + "dep:generic/libffi", ] -SciPy 1.10: +``PKG-INFO`` / ``METADATA`` content: N/A. + +SciPy 1.10 +'''''''''' + +``pyproject.toml`` content: .. code:: toml [external] build-requires = [ - "virtual:compiler/c", - "virtual:compiler/cpp", - "virtual:compiler/fortran", - "pkg:generic/ninja", - "pkg:generic/pkg-config", + "dep:virtual/compiler/c", + "dep:virtual/compiler/cpp", + "dep:virtual/compiler/fortran", + "dep:generic/ninja", + "dep:generic/pkg-config", ] host-requires = [ - "virtual:interface/blas", - "virtual:interface/lapack", # >=3.7.1 (can't express version ranges with PURL yet) + "dep:virtual/interface/blas", + "dep:virtual/interface/lapack@>=3.7.1", ] -Pillow 10.1.0: +``PKG-INFO`` / ``METADATA`` content: N/A. + +Pillow 10.1.0 +''''''''''''' + +``pyproject.toml`` content: .. code:: toml [external] build-requires = [ - "virtual:compiler/c", + "dep:virtual/compiler/c", ] host-requires = [ - "pkg:generic/libjpeg", - "pkg:generic/zlib", + "dep:generic/libjpeg", + "dep:generic/zlib", ] [external.optional-host-requires] extra = [ - "pkg:generic/lcms2", - "pkg:generic/freetype", - "pkg:generic/libimagequant", - "pkg:generic/libraqm", - "pkg:generic/libtiff", - "pkg:generic/libxcb", - "pkg:generic/libwebp", - "pkg:generic/openjpeg", # add >=2.0 once we have version specifiers - "pkg:generic/tk", + "dep:generic/lcms2", + "dep:generic/freetype", + "dep:generic/libimagequant", + "dep:generic/libraqm", + "dep:generic/libtiff", + "dep:generic/libxcb", + "dep:generic/libwebp", + "dep:generic/openjpeg@>=2.0", + "dep:generic/tk", ] +``PKG-INFO`` / ``METADATA`` content: N/A. + +NAVis 1.4.0 +''''''''''' -NAVis 1.4.0: +``pyproject.toml`` content: .. code:: toml @@ -453,60 +687,124 @@ NAVis 1.4.0: [external] build-requires = [ - "pkg:generic/XCB; platform_system=='Linux'", + "dep:generic/XCB; platform_system=='Linux'", ] [external.optional-dependencies] nat = [ - "pkg:cran/nat", - "pkg:cran/nat.nblast", + "dep:cran/nat", + "dep:cran/nat.nblast", ] -Spyder 6.0: +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Provides-External-Extra: nat + Requires-External-Dep: dep:cran/nat; extra == 'nat' + Requires-External-Dep: dep:cran/nat.nblast; extra == 'nat' + +Spyder 6.0 +'''''''''' + +``pyproject.toml`` content: .. code:: toml [external] dependencies = [ - "pkg:cargo/ripgrep", - "pkg:cargo/tree-sitter-cli", - "pkg:golang/github.com/junegunn/fzf", + "dep:cargo/ripgrep", + "dep:cargo/tree-sitter-cli", + "dep:golang/github.com/junegunn/fzf", ] -jupyterlab-git 0.41.0: +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Requires-External-Dep: dep:cargo/ripgrep + Requires-External-Dep: dep:cargo/tree-sitter-cli + Requires-External-Dep: dep:golang/github.com/junegunn/fzf + +jupyterlab-git 0.41.0 +''''''''''''''''''''' + +``pyproject.toml`` content: .. code:: toml [external] dependencies = [ - "pkg:generic/git", + "dep:generic/git", ] [external.optional-build-requires] dev = [ - "pkg:generic/nodejs", + "dep:generic/nodejs", ] -PyEnchant 3.2.2: +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Requires-External-Dep: dep:generic/git + +PyEnchant 3.2.2 +''''''''''''''' + +``pyproject.toml`` content: .. code:: toml [external] dependencies = [ - # libenchant is needed on all platforms but only vendored into wheels on - # Windows, so on Windows the build backend should remove this external - # dependency from wheel metadata. - "pkg:github/AbiWord/enchant", + # libenchant is needed on all platforms but vendored into wheels + # distributed on PyPI for Windows. Hence choose to encode that in + # the metadata. Note: there is no completely unambiguous way to do + # this; another choice is to leave out the environment marker in the + # source distribution and either live with the unnecessary ``METADATA`` + # entry in the distributed Windows wheels, or to apply a patch to this + # metadata when building those wheels. + "dep:github/AbiWord/enchant; platform_system!='Windows'", + ] + +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Requires-External-Dep: dep:github/AbiWord/enchant; platform_system!="Windows" + +With dependency groups +'''''''''''''''''''''' + +``pyproject.toml`` content: + +.. code:: toml + + [external.dependency-groups] + dev = [ + "dep:generic/catch2", + "dep:generic/valgrind", ] +``PKG-INFO`` / ``METADATA`` content: N/A. Backwards Compatibility ======================= -There is no impact on backwards compatibility, as this PEP only adds new, +There is no major impact on backwards compatibility, as this PEP primarily adds new, optional metadata. In the absence of such metadata, nothing changes for package authors or packaging tooling. +The only change introduced in this PEP that has impact on existing projects is the +deprecation of the ``External-Requires`` Core Metadata field. We estimate the impact +of this deprecation to be negligible, given the its low penetration in the ecosystem +(see Rationale). + +The field will still be recognized by existing tools such as `setuptools-ext`_ +but its usage will be discouraged in the `Python Packaging User Guide`_, similar to +what it's done for obsolete fields like ``Requires`` (deprecated in favor of +``Requires-Dist``). Security Implications ===================== @@ -542,10 +840,12 @@ there will not be code implementing the metadata spec as a whole. However, there are parts that do have a reference implementation: 1. The ``[external]`` table has to be valid TOML and therefore can be loaded - with ``tomllib``. + with ``tomllib``. This table can be further processed with the + `pyproject-external`_ package, demonstrated below. 2. The PURL specification, as a key part of this spec, has a Python package with a reference implementation for constructing and parsing PURLs: - `packageurl-python`_. + `packageurl-python`_. This package is wrapped in `pyproject-external`_ + to provide ``dep:``-specific validation and handling. There are multiple possible consumers and use cases of this metadata, once that metadata gets added to Python packages. Tested metadata for all of the @@ -554,6 +854,64 @@ wheels can be found in `rgommers/external-deps-build`_. This metadata has been validated by using it to build wheels from sdists patched with that metadata in clean Docker containers. +Example +------- + +Given a ``pyproject.toml`` with this ``[external]`` table: + +.. code-block:: toml + + [external] + build-requires = [ + "dep:virtual/compiler/c", + "dep:virtual/compiler/rust", + "dep:generic/pkg-config", + ] + host-requires = [ + "dep:generic/openssl", + "dep:generic/libffi", + ] + +You can use ``pyproject_external.External`` to parse it and manipulate it: + +.. code-block:: python + + >>> from pyproject_external import External + >>> external = External.from_pyproject_path("./pyproject.toml") + >>> external.validate() + >>> external.to_dict() + {'external': {'build_requires': ['dep:virtual/compiler/c', 'dep:virtual/compiler/rust', 'dep:generic/pkg-config'], 'host_requires': ['dep:generic/openssl', 'dep:generic/libffi']}} + >>> external.build_requires + [DepURL(type='virtual', namespace='compiler', name='c', version=None, qualifiers={}, subpath=None), DepURL(type='virtual', namespace='compiler', name='rust', version=None, qualifiers={}, subpath=None), DepURL(type='generic', namespace=None, name='pkg-config', version=None, qualifiers={}, subpath=None)] + >>> external.build_requires[0] + DepURL(type='virtual', namespace='compiler', name='c', version=None, qualifiers={}, subpath=None) + +Note the proposed ``[external]`` table was well-formed. With invalid contents such as: + +.. code-block:: toml + + [external] + build-requires = [ + "dep:this-is-missing-the-type", + "pkg:not-a-dep-url" + ] + +You would fail the validation: + +.. code-block:: python + + >>> external = External.from_pyproject_data( + { + "external": { + "build_requires": [ + "dep:this-is-missing-the-type", + "pkg:not-a-dep-url" + ] + } + } + ) + ValueError: purl is missing the required type component: 'dep:this-is-missing-the-type'. + Rejected Ideas ============== @@ -567,35 +925,86 @@ those, and if it's not present on the system then install the PyPI package for it. The authors believe that specific support for this scenario is not necessary (or too complex to justify such support); a dependency provider for external dependencies can treat PyPI as one possible source for obtaining the -package. +package. An example mapping for this use case is proposed in *name mapping PEP*. Using library and header names as external dependencies ------------------------------------------------------- A previous draft PEP (`"External dependencies" (2015) `__) proposed using specific library and header names as external dependencies. This -is too granular; using package names is a well-established pattern across -packaging ecosystems and should be preferred. +is both too granular, and insufficient (e.g., headers are often unversioned; multiple +packages may provide the same header or library). Using +package names is a well-established pattern across packaging ecosystems +and should be preferred. + +Splitting host dependencies with explicit ``-dev`` or ``-devel`` suffixes +------------------------------------------------------------------------- +This convention is not consistent across all ecosystems. Since the need +for explicit control is quite niche and we don't want to add design +complexity without enough clear use cases, we have chosen to rely solely +on the the ``build``, ``host`` and ``run`` category split, with tools +being in charge of which category applies to each case in a context-dependent way. -Open Issues -=========== +If this proves to be insufficient, a future PEP could use the URL qualifier features +present in the PURL schema (``?key=value``) to implement the necessary adjustments. + +Identifier indirections +----------------------- + +Some ecosystems exhibit methods to select packages based on parametrized functions +like ``cmake("dependency")`` or ``compiler("language")``, which return package +names based on some additional context or configuration. This feature is arguably +not very common and, even when present, rarely used. Additionally, its dynamic +nature makes it prone to changing meaning over time, and relying on specific build +systems for the name resolution is in general not a good idea. -Version specifiers for PURLs ----------------------------- +The authors prefer static identifiers that can be mapped explicitly via well known metadata +(e.g. as proposed in *name mapping PEP*). -Support in PURL for version expressions and ranges is still pending. The pull -request at `vers implementation for PURL`_ seems close to being merged, at -which point this PEP could adopt it. +Ecosystems that do implement these indirections can use them to support the infrastructure +designed to generate the mappings proposed in *name mapping PEP*. + +Adding a ``host-requires`` key under ``[build-system]`` +------------------------------------------------------- + +Adding ``host-requires`` for host dependencies that are on PyPI in order to +better support name mapping to other packaging systems with support for +cross-compiling seems useful in principle, for the same reasons as this PEP +adds a ``host-requires`` under the ``[external]`` table. However, it isn't +necessary to include in this PEP, and hence the authors prefer to keep the +scope of this PEP limited - a future PEP on cross compilation may want to +tackle this. `This issue `__ +contains more arguments in favor and against adding ``host-requires`` under +``[build-system]`` as part of this PEP. + +Reusing the ``Requires-External`` field in Core Metadata +-------------------------------------------------------- + +The `Core Metadata`_ specification contains one relevant field, namely +``Requires-External``. While at first sight it would be a good candidate +to record the ``external.dependencies`` table, the authors have decided +to not re-use this field to propagate the external runtime dependencies metadata. + +The ``Requires-External`` field has very loosely defined semantics as of +version 2.4. Essentially: ``name [(version)][; environment marker]`` +(with square brackets denoting optional fields). It is not defined what valid strings for ``name`` are; +the example in the specification uses both "C" as a language name, and "libpng" as a package name. +Tightening up the semantics would be backwards incompatible, and leaving it as is seems +unsatisfactory. The ``dep:`` URLs would need +to be decomposed to fit in this syntax. + + +Open Issues +=========== Versioning of virtual dependencies ----------------------------------- +----------------------------------- -Once PURL supports version expressions, virtual dependencies can be versioned -with the same syntax. It must be better specified however what the version -scheme is, because this is not as clear for virtual dependencies as it is for -PURLs (e.g., there can be multiple implementations, and abstract interfaces may -not be unambiguously versioned). E.g.: +While virtual dependencies can be versioned with the same syntax, it must be better +specified however what the version scheme is, because this is not as clear for +virtual types as it is for PURL types (e.g., there can be multiple implementations, +and virtual interfaces may not be unambiguously versioned). E.g.: - OpenMP: has regular ``MAJOR.MINOR`` versions of its standard, so would look like ``>=4.5``. @@ -619,30 +1028,12 @@ Who defines canonical names and canonical package structure? Similarly to the logistics around versioning is the question about what names are allowed and where they are described. And then who is in control of that description and responsible for maintaining it. Our tentative answer is: there -should be a central list for virtual dependencies and ``pkg:generic`` PURLs, +should be a central list for ``dep:generic`` and ``dep:virtual`` URLs, maintained as a PyPA project. See https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml/31888/62. TODO: once that list/project is prototyped, include it in the PEP and close this open issue. -Syntax for virtual dependencies -------------------------------- - -The current syntax this PEP uses for virtual dependencies is -``virtual:type/name``, which is analogous to but not part of the PURL spec. -This open issue discusses supporting virtual dependencies within PURL: -`purl-spec#222 `__. - -Should a ``host-requires`` key be added under ``[build-system]``? ------------------------------------------------------------------ - -Adding ``host-requires`` for host dependencies that are on PyPI in order to -better support name mapping to other packaging systems with support for -cross-compiling may make sense. -`This issue `__ tracks this topic -and has arguments in favor and against adding ``host-requires`` under -``[build-system]`` as part of this PEP. - References ========== @@ -676,11 +1067,14 @@ CC0-1.0-Universal license, whichever is more permissive. .. _PyPI: https://pypi.org -.. _core metadata: https://packaging.python.org/specifications/core-metadata/ +.. _Core Metadata: https://packaging.python.org/specifications/core-metadata/ .. _setuptools: https://setuptools.readthedocs.io/ .. _setuptools metadata: https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata .. _SPDX: https://spdx.dev/ .. _PURL: https://github.com/package-url/purl-spec/ +.. _version specifier: https://packaging.python.org/en/latest/specifications/version-specifiers/ +.. _dependencies specifier: https://packaging.python.org/en/latest/specifications/dependency-specifiers/ +.. _dependency groups specification: https://packaging.python.org/en/latest/specifications/dependency-groups/ .. _packageurl-python: https://pypi.org/project/packageurl-python/ .. _vers: https://github.com/package-url/purl-spec/blob/version-range-spec/VERSION-RANGE-SPEC.rst .. _vers implementation for PURL: https://github.com/package-url/purl-spec/pull/139 @@ -695,5 +1089,15 @@ CC0-1.0-Universal license, whichever is more permissive. .. _auditwheel: https://github.com/pypa/auditwheel .. _delocate: https://github.com/matthew-brett/delocate .. _delvewheel: https://github.com/adang1345/delvewheel +.. _verspurl: https://github.com/package-url/purl-spec/issues/386 .. _rgommers/external-deps-build: https://github.com/rgommers/external-deps-build +.. _pyproject-external: https://github.com/jaimergp/pyproject-external .. _Reference LAPACK: https://github.com/Reference-LAPACK/lapack +.. _setuptools-ext: https://pypi.org/project/setuptools-ext/ +.. _PyPI JSON API responses: https://docs.pypi.org/api/json/ +.. _pypa/hatch#1712: https://github.com/pypa/hatch/issues/1712 +.. _pypa/flit#353: https://github.com/pypa/flit/issues/353 +.. _pypa/setuptools#4220: https://github.com/pypa/setuptools/discussions/4220#discussioncomment-8930671 +.. _poetry-external-dependencies: https://pypi.org/project/poetry-external-dependencies/ +.. _PyO3/maturin@5b0e4808: https://github.com/PyO3/maturin/commit/5b0e4808bb8852fe796cd2848932a35fbb14de8b +.. _elfdeps: https://github.com/python-wheel-build/elfdeps/