From 7a9b241b4f3644656fb4357edc03be7bbd04064e Mon Sep 17 00:00:00 2001 From: Seth Michael Larson Date: Tue, 25 Mar 2025 14:46:27 -0500 Subject: [PATCH 1/2] PEP 770: Remove mechanism for statically defining SBOM files --- peps/pep-0770.rst | 183 ++++++++-------------------------------------- 1 file changed, 32 insertions(+), 151 deletions(-) diff --git a/peps/pep-0770.rst b/peps/pep-0770.rst index 55f4429aafa..10db2152879 100644 --- a/peps/pep-0770.rst +++ b/peps/pep-0770.rst @@ -153,17 +153,11 @@ can projects begin using the new features. Even with this careful approach it's not guaranteed that tools won't break on new metadata versions and fields. To avoid this delay, simplify overall how to include SBOMs, and to give -flexibility to build backends and tools, this PEP proposes a new top-level table -in ``pyproject.toml``, ``[dist-info.files]``, to safely add data to a Python -package through a registry of reserved names that -avoids the need for new metadata fields and versions. This mechanism allows -build backends and tools to begin using the features described in this PEP -immediately after acceptance without the head-of-line blocking on other -projects adopting the PEP. - -A new top-level table was chosen over using the ``[project]`` table because, -as described in :pep:`621`, the ``[project]`` table is used for storing core -metadata and this mechanism doesn't use core metadata. +flexibility to build backends and tools, this PEP proposes using a subdirectory +under ``.dist-info`` to safely add data to a Python package while avoiding the +need for new metadata fields and versions. This mechanism allows build backends +and tools to begin using the feature described in this PEP immediate after +acceptance without the head-of-line blocking on other projects adopting the PEP. Storing files in the ``.dist-info`` or ``.data`` directory ---------------------------------------------------------- @@ -171,8 +165,7 @@ Storing files in the ``.dist-info`` or ``.data`` directory There are two top-level directories in binary distributions where files beyond the software itself can be stored: ``.dist-info`` and ``.data``. This specification chose to use the ``.dist-info`` directory for storing -subdirectories and files from the new ``[dist-info.files]`` top-level table -for two reasons: +subdirectories and files. Firstly, the ``.data`` directory has no corresponding location in the installed package, compared to ``.dist-info`` which does preserve the link between the @@ -231,11 +224,7 @@ Specification The changes necessary to implement this PEP include: -* Explicitly reserving all subdirectory names in the ``.dist-info`` directory. -* A new registry of reserved subdirectory names in the ``.dist-info`` directory. -* An optional top-level table, ``[dist-info.files]``, added to - `project source metadata <770-spec-project-source-metadata_>`_, -* An optional ``sboms`` key in the new ``[dist-info.files]`` table, +* Explicitly reserving the subdirectory ``.dist-info/sboms``. * `Additions <770-spec-project-formats_>`_ to the built distribution (wheel), and installed project specifications @@ -243,25 +232,10 @@ In addition to the above, an informational PEP will be created for tools consuming included SBOM documents and other Python package metadata to generate complete SBOM documents for Python packages. -.. _770-spec-dist-info-subdirs: - -Reserving all subdirectory names in ``.dist-info`` --------------------------------------------------- - -This PEP explicitly reserves all subdirectory names in the ``.dist-info`` -directory for future usage. - -Build backends MUST NOT create subdirectories in the ``.dist-info`` directory -beyond the names in the `registry <770-spec-dist-info-subdirs-registry>`_ -to avoid collisions with future reserved names. - -Build frontends and publishing tools MAY warn users if any ``.dist-info`` -subdirectories aren't in the registry. - .. _770-spec-dist-info-subdirs-registry: -Registry of reserved ``.dist-info`` subdirectory names ------------------------------------------------------- +Reserving the ``.dist-info/sboms`` directory +-------------------------------------------- This PEP introduces a new registry of reserved subdirectory names allowed in the ``.dist-info`` directory for the :term:`distribution archive` @@ -278,88 +252,7 @@ Subdirectory name PEP / Standard ================= ============== See :ref:`770-backwards-compat` for a complete methodology for -creating this initial set of values to avoid backwards incompatibility issues. - -.. _770-spec-project-source-metadata: - -Project source metadata ------------------------ - -This PEP specifies changes to the project's source metadata -in the ``pyproject.toml`` file: - -Add new ``[dist-info.files]`` table -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A new optional ``[dist-info.files]`` table is added for specifying paths -in the project source tree relative to ``pyproject.toml`` to file(s) which -should be included in the built project to a subdirectory of ``.dist-info``. - -This new table has only one defined optional key: ``sboms``. The value of the -``sboms`` key MUST be an array of valid glob patterns, as specified below: - -* Alphanumeric characters, underscores (``_``), hyphens (``-``) and dots (``.``) - MUST be matched verbatim. -* Special glob characters: ``*``, ``?``, ``**`` and character ranges: ``[]`` - containing only the verbatim matched characters MUST be supported. Within - ``[...]``, the hyphen indicates a locale-agnostic range (e.g. a-z, order based - on Unicode code points). Hyphens at the start or end are matched literally. -* Path delimiters MUST be the forward slash character (``/``). Patterns are - relative to the directory containing ``pyproject.toml``, therefore the leading - slash character MUST NOT be used. -* Parent directory indicators (``..``) MUST NOT be used. - -Any characters or character sequences not covered by this specification are -invalid. Projects MUST NOT use such values. Tools consuming this field SHOULD -reject invalid values with an error. - -Literal paths (e.g. ``bom.cdx.json``) are treated as valid globs which means -they can also be defined. - -Build tools: - -* MUST treat each value in the array as a glob pattern, and MUST raise an error - if the pattern contains invalid glob syntax. -* MUST include all files matched by a listed pattern in all distribution - archives under the ``.dist-info/sboms`` directory. -* MUST raise an error if any individual user-specified pattern does not match - at least one file. - -If the ``sboms`` key is present and is set to a value of an empty array, -then tools MUST NOT include any SBOM files and MUST NOT raise an error. - -Examples of valid SBOM files declarations: - -.. code-block:: toml - - [dist-info.files] - sboms = ["bom.json"] - - [dist-info.files] - sboms = ["sboms/openssl.cdx.json", "sboms/openssl.spdx.json"] - - [dist-info.files] - sboms = ["sboms/*"] - - [dist-info.files] - sboms = [] - -Examples of invalid SBOM files declarations: - -.. code-block:: toml - - [dist-info.files] - sboms = ["..\bom.json"] - -Reason: ``..`` must not be used. ``\\`` is an invalid path delimiter, ``/`` -must be used. - -.. code-block:: toml - - [dist-info.files] - sboms = ["bom{.json*"] - -Reason: ``bom{.json*`` is not a valid glob. +avoiding backwards incompatibilities with selecting this directory name. .. _770-spec-project-formats: @@ -368,12 +261,6 @@ SBOM files in project formats A few additions will be made to the existing specifications. -:term:`Project source trees ` - Per :ref:`770-spec-project-source-metadata` section, the - `Declaring Project Metadata specification `__ - will be updated to add the ``[dist-info.files]`` table - and optional ``sboms`` key. - :term:`Built distributions ` (:term:`wheels `) The wheel specification will be updated to add the new registry of reserved directory names and to reflect that if the ``.dist-info/sboms`` subdirectory @@ -420,10 +307,10 @@ SBOM standards, versions, or fields. Backwards Compatibility ======================= -Reserved ``.dist-info`` subdirectories registry ------------------------------------------------ +Reserved ``.dist-info/sboms`` subdirectory +------------------------------------------ -The new registry of reserved ``.dist-info`` subdirectories represents +The new reserved ``.dist-info/sboms`` subdirectory represents a new reservation that wasn't previously documented, thus has the potential to break assumptions being made by already existing tools. @@ -521,23 +408,6 @@ Typically tools can automatically annotate bundled dependencies when those dependencies come from a "packaging ecosystem" (such as PyPI, Linux distros, Crates.io, NPM, etc). -For packages which cannot be automatically annotated and if the package author -wishes to provide an SBOM the approach will be to generate or author SBOM files -and then include those files using ``pyproject.toml``: - -.. code-block:: toml - - [dist-info.files] - sboms = [ - "sboms/bom.cdx.json" - ] - -For projects manually specifying an SBOM document the challenge will be -keeping the document up-to-date. The CPython project has some -`customized tooling `__ -for this task, but it can likely be generalized into a tool reusable by other -projects. - What do SBOM tool authors need to know? --------------------------------------- @@ -644,11 +514,12 @@ user-involvement or knowledge. Compare this situation to license files which are almost always static. The 639-style approach was ultimately dropped in favor of defining SBOMs simply -by their presence in the ``.dist-info/sboms`` directory and using a new table in -``pyproject.toml`` called ``[dist-info.files]`` to define SBOMs in source -distributions. This approach allows users to specify static SBOM files while -still empowering build backends and tools to add their own SBOM data without the -static/dynamic conflict. +by their presence in the ``.dist-info/sboms`` directory. This approach allows +build backends and tools to add their own SBOM data without the static/dynamic +conflict. + +A future PEP will define the process for statically defining SBOM files to be +added to the ``.dist-info/sboms`` directory. Open Issues =========== @@ -659,6 +530,17 @@ Conditional project source SBOM files How can a project specify an SBOM file that is conditional? Under what circumstances would an SBOM document be conditional? +Statically defining source SBOM files +------------------------------------- + +How can SBOM files within a project source (and outside a project source tree) +be specified statically such that files are added to ``.dist-info/sboms``? + +A previous version of this PEP suggested a mechanic for +adding files into subdirectories under ``.dist-info/`` using a top-level +``pyproject.toml`` table ``[dist-info.files]``. This addition has been +refactored out of this PEP and will be proposed in a separate PEP. + References ========== @@ -686,9 +568,8 @@ Acknowledgements ================ Thanks to Karolina Surma for authoring and leading :pep:`639` to acceptance. -This PEP's initial design was heavily inspired by :pep:`639` and the new -"dist-info.files" mechanism generalizes 639's approach of using a subdirectory -under ``.dist-info``. +This PEP's initial design was heavily inspired by :pep:`639` and adopts a +similar approach of using a subdirectory under ``.dist-info`` to store files. Copyright ========= From cc3373d23e3b727547b277a646910fd8c988a745 Mon Sep 17 00:00:00 2001 From: Seth Michael Larson Date: Tue, 25 Mar 2025 15:23:53 -0500 Subject: [PATCH 2/2] dist-info.files table isn't an open issue --- peps/pep-0770.rst | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/peps/pep-0770.rst b/peps/pep-0770.rst index 10db2152879..12bac2905c2 100644 --- a/peps/pep-0770.rst +++ b/peps/pep-0770.rst @@ -156,7 +156,7 @@ To avoid this delay, simplify overall how to include SBOMs, and to give flexibility to build backends and tools, this PEP proposes using a subdirectory under ``.dist-info`` to safely add data to a Python package while avoiding the need for new metadata fields and versions. This mechanism allows build backends -and tools to begin using the feature described in this PEP immediate after +and tools to begin using the feature described in this PEP immediately after acceptance without the head-of-line blocking on other projects adopting the PEP. Storing files in the ``.dist-info`` or ``.data`` directory @@ -530,17 +530,6 @@ Conditional project source SBOM files How can a project specify an SBOM file that is conditional? Under what circumstances would an SBOM document be conditional? -Statically defining source SBOM files -------------------------------------- - -How can SBOM files within a project source (and outside a project source tree) -be specified statically such that files are added to ``.dist-info/sboms``? - -A previous version of this PEP suggested a mechanic for -adding files into subdirectories under ``.dist-info/`` using a top-level -``pyproject.toml`` table ``[dist-info.files]``. This addition has been -refactored out of this PEP and will be proposed in a separate PEP. - References ==========