diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 3ac3df987d1..4b19f2fd0b1 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -30,14 +30,14 @@ file, which specifies what direct and indirect dependencies should be installed into a virtual environment. Considering there are at least five well-known solutions to this problem in the -community (``pip freeze``, pip-tools_, uv_, Poetry_, and PDM_), there seems to +community (PDM_, ``pip freeze``, pip-tools_, Poetry_, and uv_), there seems to be an appetite for lock files in general. Those tools also vary in what locking scenarios they support. For instance, ``pip freeze`` and pip-tools only generate lock files for the current -environment while PDM and Poetry try to lock for *any* environment to some -degree. There's also concerns around the lack of secure defaults in the face of -supply chain attacks (e.g., always including hashes for files). +environment while PDM, Poetry, and uv can/try to lock for multiple environments +at once. There's also concerns around the lack of secure defaults in the face of +supply chain attacks (e.g., including hashes for files). The lack of a standard also has some drawbacks. For instance, any tooling that wants to work with lock files must choose which format to support, potentially @@ -46,8 +46,18 @@ same for cloud providers who can do dependency installations on your behalf, etc.). It also impacts portability between tools, which causes vendor lock-in. By not having compatibility and interoperability it fractures tooling around lock files where both users and tools have to choose what lock file format to -use upfront and making it costly to use/switch to other formats. Rallying -around a single format removes that cost/barrier. +use upfront and making it costly to use/switch to other formats (e.g., tooling +around auditing a lock file). Rallying around a single format removes this +cost/barrier. + +The closest the community has to a standard are pip's `requirements files`_ +which all the aforementioned tools either use directly as their file format or +export to (i.e. ``requirements.txt``). Unfortunately the format is not a +standard but is supported by convention. It's also designed very much for pip's +needs, limiting its flexibility and ease of use (e.g., it's a bespoke file +format). Lastly, it is not secure by default (e.g., file hash support is +entirely an opt-in feature, you have to tell pip to not look for other +dependencies outside of what's in the requirements file, etc.). .. note:: @@ -58,27 +68,28 @@ around a single format removes that cost/barrier. Rationale ========= -The format is designed so that a *locker* which produces the lock file -and an *installer* which consumes the lock file can be separate tools. This -allows for situations such as cloud hosting providers to use their own installer -that's optimized for their system which is independent of what locker the user -used to create their lock file. - -The file format is designed to be human-readable. This is so that the contents -of the file can be audited by a human to make sure no undesired dependencies end -up being included in the lock file. +The file format proposed by this PEP is designed to be human-readable. This is +so that the contents of the file can be audited by a human to make sure no +undesired dependencies end up being included in the lock file. The file format is also designed to not require a resolver at install time. This -greatly simplifies installers and thus reasoning about what would be installed -when consuming a lock file. It should also lead to faster installs which are -much more frequent than creating a lock file. +greatly simplifies reasoning about what would be installed when consuming a lock +file. It should also lead to faster installs which are much more frequent than +creating a lock file. + +The data in the file should be consumable by tools not written in Python. This +allows for e.g., clould hosting providers to write their own tool to perform +installations in their preferred programming language. -Finally, the lock file is meant to be flexible enough to meets the various needs -tools have for choosing what to install. That means the lock file records the -dependency graph of what _may_ be installed. This allows tools to enter the -graph at any point and still have reproducible results from that root of the -graph. Flexibility also means supporting different installation scenarios within -the same lock file (e.g., with or without test dependencies). +The file format should promote good security defaults. As the format is not +meant to be human-writable, this means having tools provide security-related +details is reasonable and not a costly burden. + +The contents of a lock file should be able to replace the vast majority of uses +of `requirements files`_ that are used as a lock file (e.g., what +pip-tools_ and ``pip freeze`` emit). This means the file format specified by +this PEP can, at minimum, act as an export target for tools which have their own +internal lock file format. ============= @@ -89,13 +100,22 @@ Specification File Name --------- -A lock file MUST be named :file:`pylock.toml`. The use of the ``.toml`` file -extension is to make syntax highlighting in editors easier and to reinforce the -fact that the file format is meant to be human-readable. +A lock file MUST be named :file:`pylock.toml` or match the regular expression +``r"pylock\.(.+)\.toml"`` if a name for the lock file is desired or if multiple +lock files exist. The use of the ``.toml`` file extension is to make syntax +highlighting in editors easier and to reinforce the fact that the file format is +meant to be human-readable. The prefix and suffix of a named file MUST be +lowercase when possible for easy detection and removal, +e.g.: + +.. code-block:: Python + + if filename.startswith("pylock.") and filename.endswith(".toml"): + name = filename.removeprefix("pylock.").removesuffix(".toml") -The lock file SHOULD be located in the directory as appropriate for the scope of -the lock file. Locking against a single ``pyproject.toml``, for instance, would -place the ``pylock.toml`` in the same directory. If the lock file covered +The lock file(s) SHOULD be located in the directory as appropriate for the scope +of the lock file. Locking against a single ``pyproject.toml``, for instance, +would place the ``pylock.toml`` in the same directory. If the lock file covered multiple projects in a monorepo, then the expectation is the ``pylock.toml`` file would be in the directory that held all the projects being locked. @@ -106,506 +126,439 @@ File Format The format of the file is TOML_. -All keys listed below are required unless otherwise noted. If two keys are -mutually exclusive to one another, then one of the keys is required while the -other is disallowed. - -Keys in tables -- including the top-level table -- SHOULD be emitted by lockers -in the order they are listed in this PEP when applicable unless another sort -order is specified to minimize noise in diffs. If the keys are not explicitly -specified in this PEP, then the keys SHOULD be sorted by lexicographic order. +Tools SHOULD write their lock files in a consistent way to minmize noise in diff +output. Keys in tables -- including the top-level table -- SHOULD be recorded in +a consistent order. As well, tools SHOULD sort arrays in consistent order. Usage +of inline tables SHOULD also be kept consistent. -As well, lockers SHOULD sort arrays in lexicographic order unless otherwise -specified for the same reason. +``metadata-version`` +==================== -``version`` -=========== - -- String -- The version of the lock file format. +- **Type**: string; value of ``"1.0"`` +- **Required?**: yes +- **Inspiration**: :ref:`packaging:core-metadata-metadata-version` +- Record the file format version that the file adheres to. - This PEP specifies the initial version -- and only valid value until future updates to the standard change it -- as ``"1.0"``. -- If an installer supports the major version but not the minor version, a tool +- If a tool supports the major version but not the minor version, a tool SHOULD warn when an unknown key is seen. -- If an installer doesn't support a major version, it MUST raise an error. - - -``hash-algorithm`` -================== - -- String -- The name of the hash algorithm used for calculating all hash values. -- Only a single hash algorithm is used for the entire file to allow hash values - to be written in inline tables for readability and compactness purposes by - only listing a single hash value instead of multiple values based on multiple - hash algorithms. -- Specifying a single hash algorithm guarantees that an algorithm that the user - prefers is used consistently throughout the file without having to audit - each file hash value separately. -- Allows for updating the entire file to a new hash algorithm without running - the risk of accidentally leaving an old hash value in the file. -- :ref:`packaging:simple-repository-api-json` and the ``hashes`` dictionary of - of the ``files`` dictionary of the Project Details dictionary specifies what - values are valid and guidelines on what hash algorithms to use. -- Failure to validate any hash values for any file that is to be installed MUST - raise an error. - +- If an tool doesn't support a major version, it MUST raise an error. -``[locker]`` -============ - -- Table -- Record of the tool that generated the lock file. -- Enough details SHOULD be provided such that the lock - file from the details in this table can be reproduced (provided the same I/O - data is available, e.g., Dependabot if only files from a repository is - necessary to run the command). - - -``locker.name`` ---------------- - -- String -- The name of the tool used to create the lock file. -- If the locker is a Python project, its normalized name SHOULD be used. - - -``locker.version`` ------------------- - -- String -- The version of the tool used. - - -``locker.run`` --------------- - -- Optional -- Inline table -- Records the command used to create the lock file. - - -``locker.run.module`` -''''''''''''''''''''' -- Optional -- String -- The module name used for running the locker (i.e. what would be passed to - ``python -m``). -- Lockers MUST specify this key if the locker can be executed via ``python -m``. - - -``locker.run.args`` -''''''''''''''''''' +``environments`` +================ -- Optional -- Array of strings -- If the locker has a CLI, the arguments to pass to the locker. -- All paths MUST be relative to the lock file so that another tool could use - the lock file's location as the current working directory. +- **Type**: Array of strings +- **Required?**: no +- **Inspiration**: uv_ +- A list of :ref:`packaging:dependency-specifiers-environment-markers` for + which the lock file is considered compatible with. +- Tools SHOULD write exclusive/non-overlapping environment markers to ease in + understanding. -``[[groups]]`` -============== +``requires-python`` +=================== -- Array of tables -- A named subset of packages as found in ``[[packages]]``. -- Act as roots into the dependency graph. -- Installers MUST allow the user to select one or more groups by name to - install all relevant packages together. -- Installers SHOULD let the user skip specifying a name if there is only one - entry in the array. +- **Type**: string +- **Required?**: no +- **Inspiration**: PDM_, Poetry_, uv_ +- Specifies the :ref:`packaging:core-metadata-requires-python` for the minimum + Python version compatibile for any environment supported by the lock file + (i.e. the minimum viable Python version for the lock file). -``groups.name`` ---------------- +``[[packages]]`` +================ -- String -- The name of the group. +- **Type**: array of tables +- **Required?**: yes +- **Inspiration**: PDM_, Poetry_, uv_ +- An array containing all packages that *may* be installed. +- Packages MAY be listed multiple times with varying data, but all packages to + be installed MUST narrow down to a single entry at install time. -``groups.project`` ------------------- +.. Identification -- Mutually-exclusive with ``requirements`` -- String -- The normalized name of a package to act as the starting point into the - dependency graph. -- Analogous to locking to the ``[project]`` table in ``pyproject.toml``. -- Installers MUST let a user specify any optional features/extras that the - package provides. -- Lockers MUST NOT allow for ambiguity by specifying multiple package versions - of the same package under the same group name when a package is listed in any - ``project`` key. +``packages.name`` +----------------- +- **Type**: string +- **Required?**: yes +- **Inspiration**: :ref:`packaging:core-metadata-name` +- The name of the package :ref:`normalized `. -``groups.requirements`` ------------------------ -- Mutually-exclusive with ``project`` -- Array of tables -- Represents the installation requirements for this group. -- Analogous to a key in ``[dependency-groups]`` in ``pyproject.toml``. -- Lockers MUST make sure that resolving any requirement for any environment does - not lead to ambiguity by having multiple values in ``[[packages]]`` match the - same requirement. -- Values in the array SHOULD be written as inline tables, sorted - lexicographically by ``name``, then by ``feature`` with the lack of that key - sorting first. +``packages.version`` +-------------------- +- **Type**: string +- **Required?**: no +- **Inspiration**: :ref:`packaging:core-metadata-version` +- The version of the package. +- The version SHOULD be specified when the version is known to be stable + (i.e. when an :ref:`sdist ` or + :ref:`wheels ` are specified). +- The version MUST NOT be included when it cannot be guaranteed to be consistent + with the code used (i.e. when a + :ref:`source tree ` is + used). -``groups.requirements.name`` -'''''''''''''''''''''''''''''' -- String -- Normalized name of the package. +.. Requirements +``packages.marker`` +------------------- -``groups.requirements.extras`` -''''''''''''''''''''''''''''''' +- **Type**: string +- **Required?**: no +- **Inspiration**: PDM_ +- The + :ref:`environment marker ` + which specify when the package should be installed. -- Optional -- Array of strings -- The names of the extras specified for the requirement - (i.e. what comes between ``[...]``). +``packages.requires-python`` +---------------------------- -``groups.requirements.version`` -''''''''''''''''''''''''''''''''' +- **Type**: string +- **Required?**: no +- **Inspiration**: :ref:`packaging:core-metadata-requires-python` +- Holds the :ref:`packaging:version-specifiers` for Python version compatibility + for the package. -- Optional -- String -- The `version specifiers`_ for the requirement. +.. Installation -``groups.requirements.marker`` -'''''''''''''''''''''''''''''''' +``packages.direct`` +------------------- -- Optional -- String -- The `environment markers`_ for the requirement. +- **Type**: boolean +- **Required?**: no; defaults to ``false`` +- **Inspiration**: :ref:`packaging:direct-url` +- Represents whether the installation is via a + :ref:`direct URL reference `. -``[[packages]]`` -================ +.. Source -- Array of tables -- The array contains all data on the nodes of the dependency graph. -- Lockers SHOULD record packages in order by ``name`` - lexicographically, ``version`` by its Python `version specifiers`_ - ordering, and then by ``groups`` following Python's sort order for lists of - strings (i.e. item by item, then by length as a tiebreaker). +``[packaging.vcs]`` +------------------- +- **Type**: table +- **Required?**: no; mutually-exclusive with ``packaging.directory``, + ``packaging.archive``, ``packaging.sdist``, and ``packaging.wheels`` +- **Inspiration**: :ref:`packaging:direct-url-data-structure` +- Record the version control system details for the + :ref:`source tree ` it + contains. +- Tools MAY choose to not support version control systems, both from a locking + and/or installation perspective. +- Tools SHOULD provide a way for users to opt out of using version control + systems. -.. Identification -``packages.name`` ------------------ +``packaging.vcs.type`` +'''''''''''''''''''''' -- String -- The `normalized name`_ of the package. +- **Type**: string; supported values specified in + :ref:`packaging:direct-url-data-structure-registered-vcs` +- **Required?**: yes +- **Inspiration**: :ref:`packaging:direct-url-data-structure-vcs` +- The type of version control system used. -``packages.version`` --------------------- +``packaging.vcs.url`` +''''''''''''''''''''' -- String -- The version of the package. +- **Type**: string +- **Required?**: if ``path`` is not specified +- **Inspiration**: :ref:`packaging:direct-url-data-structure-vcs` +- The URL to the source tree. -``packages.groups`` -------------------- +``packaging.vcs.path`` +'''''''''''''''''''''' -- Array of strings -- Associates this table with the ``group.name`` entries of the same names. +- **Type**: string +- **Required?**: if ``url`` is not specified +- **Inspiration**: :ref:`packaging:direct-url-data-structure-vcs` +- The path to the local directory of the source tree. +- If a relative path is used it MUST be relative to the location of this file. +- If the path is relative it MAY use POSIX-style path separators explicitly + for portability. -``packages.index-url`` ----------------------- +``packaging.vcs.requested-revision`` +'''''''''''''''''''''''''''''''''''' -- Optional -- String -- Stores the `project index`_ URL from the `Simple Repository API`_. -- Useful for generating Packaging URLs (aka PURLs). -- When possible, lockers SHOULD include this to assist with generating - `software bill of materials`_ (aka SBOMs). +- **Type**: string +- **Required?**: no +- **Inspiration**: :ref:`packaging:direct-url-data-structure-vcs` +- The branch/tag/ref/commit/revision/etc. that the user requested. +- This is purely informational and to facilitate writing the + :ref:`packaging:direct-url-data-structure`; it MUST NOT be used to checkout + the repository. -``packages.direct`` -------------------- +``packaging.vcs.commit-id`` +''''''''''''''''''''''''''' -- Optional (defaults to ``false``) -- Boolean -- Represents whether the installation is via a `direct URL reference`_. +- **Type**: string +- **Required?**: yes +- **Inspiration**: :ref:`packaging:direct-url-data-structure-vcs` +- The exact commit/revision number that is to be installed. +- If the VCS supports commit-hash based revision identifiers, such commit-hash + MUST be used as the commit ID in order to reference an immutable version of + the source code. -.. Requirements - -``packages.requires-python`` ----------------------------- +``packaging.vcs.subdirectory`` +'''''''''''''''''''''''''''''' -- String -- Holds the `version specifiers`_ for Python version compatibility for the - package and version. -- The value MUST match what's provided by the package version, if available, via - :ref:`packaging:core-metadata-requires-python`. +- **Type**: string +- **Required?**: no +- **Inspiration**: :ref:`packaging:direct-url-data-structure-subdirectories` +- The subdirectory within the + :ref:`source tree ` where + the project root of the project is (e.g., the location of the + ``pyproject.toml`` file). +- The path MUST be relative to the root of the source tree structure. -``[[packages.dependencies]]`` ------------------------------ +``[packaging.directory]`` +------------------------- -- Array of tables -- A record of the dependency requirements of the package and version. -- The values MUST semantically match what's provided by the package version via - :ref:`packaging:core-metadata-requires-dist` for all dependencies referenced - in the lock file (i.e all base dependencies plus all dependencies for extras - referenced in the lock file); lock files MAY list all dependencies for unused - extras if desired. -- Values in the array SHOULD be written as inline tables, sorted - lexicographically by ``name``, then by ``feature`` with the lack of that key - sorting first. +- **Type**: table +- **Required?**: no; mutually-exclusive with ``packaging.vcs``, + ``packaging.archive``, ``packaging.sdist``, and ``packaging.wheels`` +- **Inspiration**: :ref:`packaging:direct-url-data-structure-local-directory` +- Record the local directory details for the + :ref:`source tree ` it + contains. +- Tools MAY choose to not support local directories, both from a locking + and/or installation perspective. +- Tools SHOULD provide a way for users to opt out of using local directories. -``packages.dependencies.name`` -'''''''''''''''''''''''''''''' +``packaging.directory.path`` +'''''''''''''''''''''''''''' -See ``groups.requirements.name``. +- **Type**: string +- **Required?**: yes +- **Inspiration**: :ref:`packaging:direct-url-data-structure-local-directory` +- The local directory where the source tree is. +- If the path is relative it MUST be relative to the location of the lock file. +- If the path is relative it MAY use POSIX-style path separators for + portability. -``packages.dependencies.extras`` +``packaging.directory.editable`` '''''''''''''''''''''''''''''''' -See ``groups.requirements.extras``. - - -``packages.dependencies.version`` -''''''''''''''''''''''''''''''''' +- **Type**: boolean +- **Required?**: no; defaults to ``false`` +- **Inspiration**: :ref:`packaging:direct-url-data-structure-local-directory` +- A flag representing whether the source tree should be installed as editable. -See ``groups.requirements.version``. - - -``packages.dependencies.marker`` -'''''''''''''''''''''''''''''''' -See ``groups.requirements.marker``. +``packaging.directory.subdirectory`` +'''''''''''''''''''''''''''''''''''' +See ``packaging.vcs.subdirectory``. -``packages.dependencies.feature`` -''''''''''''''''''''''''''''''''' -- Optional -- String -- The optional feature/:ref:`packaging:core-metadata-provides-extra` that this - requirement is conditional on. +``[packaging.archive]`` +----------------------- +- **Type**: table +- **Required?**: no +- **Inspiration**: :ref:`packaging:direct-url-data-structure-archive` +- An archive file containing a + :ref:`packaging:source-distribution-format-source-tree`. +- Tools MAY choose to not support archive files, both from a locking + and/or installation perspective. +- Tools SHOULD provide a way for users to opt out of using archive files. -.. Installing -``packages.editable`` ---------------------- +``packaging.archive.url`` +''''''''''''''''''''''''' -- Optional (defaults to ``false``) -- Boolean -- Specifies whether the package should be installed in editable mode. +See ``packaging.vcs.url``. -``[packages.source-tree]`` --------------------------- +``packaging.archive.path`` +'''''''''''''''''''''''''' -- Optional -- Table -- For recording where to find the `source tree`_ for the package version. -- Lockers SHOULD write this table inline. -- Support for source trees by installers is optional. -- If support is provided by an installer it SHOULD be opt-in. -- If multiple source trees are provided, installers MUST prefer either the - ``vcs`` option or a file for security/reproducibility due to their commit or - hash, respectively. +See ``packaging.vcs.path``. -``packages.source-tree.vcs`` -'''''''''''''''''''''''''''' +``packaging.archive.size`` +'''''''''''''''''''''''''' -- Optional -- String -- If specifying a VCS, the type of version control system used. -- The valid values are specified by the - `registered VCSs `__ - of the direct URL data structure. +- **Type**: integer +- **Required?**: yes +- **Inspiration**: uv_, :ref:`packaging:simple-repository-api` +- The size of the archive file. -``packages.source-tree.path`` -''''''''''''''''''''''''''''' +``[packaging.archive.hashes]`` +'''''''''''''''''''''''''''''' -- Required if ``url`` is not set -- String -- A path to the source tree, which may be absolute or relative. -- If the path is relative it MUST be relative to the lock file. -- The path may either be to a directory, file archive, or VCS checkout if - ``vcs`` if is specified. +- **Type**: Table of strings +- **Required?**: yes +- **Inspiration**: PDM_, Poetry_, uv_, :ref:`packaging:simple-repository-api` +- A table listing known hash values of the file where the key is the hash + algorithm and the value is the hash value. +- The table MUST contain at least one entry. +- Hash algorithm keys SHOULD be lowercase. +- At least one secure algorithm from :py:data:`hashlib.algorithms_guaranteed` + SHOULD always be included (at time of writing, sha256 specifically is + recommended. -``packages.source-tree.url`` -'''''''''''''''''''''''''''' +``packaging.archive.subdirectory`` +'''''''''''''''''''''''''''''''''' -- Required if ``path`` is not set -- String -- A URL to a file archive containing the source tree, or a VCS checkout if - ``vcs`` is specified. +See ``packaging.vcs.subdirectory``. -``packages.source-tree.commit`` -''''''''''''''''''''''''''''''' +``packages.index`` +------------------ -- Required if ``vcs`` is set -- String -- The commit ID for the repository which represents the package and version. -- The value MUST be immutable for the VCS for security purposes - (e.g. no Git tags). +- **Type**: string +- **Required?**: no +- **Inspiration**: uv_ +- The base URL for the package index from :ref:`packaging:simple-repository-api` + where the sdist and/or wheels were found (e.g., ``https://pypi.org/simple/``). +- When possible, this SHOULD be specified to assist with generating + `software bill of materials`_ (aka SBOMs). -``packages.source-tree.size`` -''''''''''''''''''''''''''''' +``[packages.sdist]`` +-------------------- -- Optional -- Integer -- The size in bytes for the source tree if it is a file. -- Installers MUST verify the file size matches this value. +- **Type**: table +- **Required?**: no; mutually-exclusive with ``packaging.vcs``, + ``packaging.directory``, and ``packaging.archive`` +- **Inspiration**: uv_ +- Details of a :ref:`packaging:source-distribution-format-sdist` for the + package. +- Tools MAY choose to not support sdist files, both from a locking + and/or installation perspective. +- Tools SHOULD provide a way for users to opt out of using sdist files. -``packages.source-tree.hash`` -''''''''''''''''''''''''''''' +``packages.sdist.name`` +''''''''''''''''''''''' -- Required if ``url`` or ``path`` points to a file -- String -- The hash value of the file contents using the hash algorithm specified by - ``hash-algorithm``. -- Installers MUST verify the hash matches the file. +- **Type**: string +- **Required?**: no +- **Inspiration**: PDM_, Poetry_, uv_ +- The file name of the :ref:`packaging:source-distribution-format-sdist` file. +- The name SHOULD be recorded when it does not follow the standard outlined in + :ref:`packaging:source-distribution-format-sdist`. -``[packages.sdist]`` --------------------- +``packages.sdist.upload-time`` +'''''''''''''''''''''''''''''' -- Optional -- Table -- The location of a source distribution as specified by - :ref:`packaging:source-distribution-format`. -- Lockers SHOULD write the table inline. -- Support for source distributions by installers is optional. -- If support is provided by an installer it SHOULD be opt-in. +- **Type**: datetime +- **Required?**: no +- **Inspiration**: :ref:`packaging:simple-repository-api` +- The time the file was uploaded. +- The date and time MUST be recorded in UTC. ``packages.sdist.url`` '''''''''''''''''''''' -- Optional; mutually-exclusive with ``path`` -- String -- The URL to the file. +See ``packaging.archive.url``. ``packages.sdist.path`` ''''''''''''''''''''''' -- Optional; mutually-exclusive with ``url`` -- String -- A path to the file, which may be absolute or relative. -- If the path is relative it MUST be relative to the lock file. +See ``packaging.archive.path``. -``packages.sdist.upload-time`` -'''''''''''''''''''''''''''''' - -- Optional and only applicable when ``url`` is specified -- Offset date time -- The upload date and time of the file as specified by a valid ISO 8601 - date/time string for the ``.files[]."upload-time"`` field in the JSON - version of :ref:`packaging:simple-repository-api`. - ``packages.sdist.size`` ''''''''''''''''''''''' -- Optional -- Integer -- The size of the file in bytes. -- Installers MUST verify the file size matches this value. +See ``packaging.archive.size``. -``packages.sdist.hash`` -''''''''''''''''''''''' +``packages.sdist.hashes`` +''''''''''''''''''''''''' + +See ``packaging.archive.hashes``. -- String -- The hash value of the file contents using the hash algorithm specified by - ``hash-algorithm``. -- Installers MUST verify the hash matches the file. ``[[packages.wheels]]`` ----------------------- -- Optional -- Array of tables +- **Type**: array of tables +- **Required?**: no; mutually-exclusive with ``packaging.vcs``, + ``packaging.directory``, and ``packaging.archive`` +- **Inspiration**: PDM_, Poetry_, uv_ - For recording the wheel files as specified by - :ref:`packaging:binary-distribution-format` for the package version. -- Lockers SHOULD write the table inline. -- Lockers SHOULD sort the array values lexicographically by ``tag``. + :ref:`packaging:binary-distribution-format` for the package. +- Tools MUST support wheel files, both from a locking and installation + perspective. -``packages.wheels.tags`` +``packages.wheels.name`` '''''''''''''''''''''''' -- Array of string -- The uncompressed tag portion of the wheel file: Python, ABI, and platform. -- Lockers MUST make sure the tag values are unique within the - ``packages.wheels`` array. +- **Type**: string +- **Required?**: yes +- **Inspiration**: PDM_, Poetry_, uv_ +- The file name of the :ref:`packaging:binary-distribution-format` file. -``packages.wheels.build`` -''''''''''''''''''''''''' +``packages.wheels.upload-time`` +''''''''''''''''''''''''''''''' -- Optional -- String -- The build tag for the wheel file (if appropriate). +See ``packages.sdist.upload-time``. ``packages.wheels.url`` ''''''''''''''''''''''' -See ``packages.sdist.url``. +See ``packaging.archive.url``. ``packages.wheels.path`` '''''''''''''''''''''''' -See ``packages.sdist.path``. - - -``packages.wheels.upload-time`` -''''''''''''''''''''''''''''''' - -See ``packages.sdist.upload-time``. +See ``packaging.archive.path``. ``packages.wheels.size`` '''''''''''''''''''''''' -See ``packages.sdist.size``. +See ``packaging.archive.size``. -``packages.wheels.hash`` -'''''''''''''''''''''''' +``packages.wheels.hashes`` +'''''''''''''''''''''''''' + +See ``packaging.archive.hashes``. -See ``packages.sdist.hash``. ``[packages.tool]`` ------------------- -- Optional -- Table +- **Type**: table +- **Required?**: no +- **Inspiration**: :ref:`packaging:pyproject-tool-table` - Similar usage as that of the ``[tool]`` table from the - `pyproject.toml specification`_ , but at the package version level instead of - at the lock file level (which is also available via ``[tool]``). + :ref:`packaging:pyproject-toml-spec`, but at the package version level instead + of at the lock file level (which is also available via ``[tool]``). - Useful for scoping package version/release details (e.g., recording signing identities to then use to verify package integrity separately from where the package is hosted, prototyping future extensions to this file format, etc.). @@ -614,330 +567,152 @@ See ``packages.sdist.hash``. ``[tool]`` ========== -- Optional -- Table +- **Type**: table +- **Required?**: no +- **Inspiration**: :ref:`packaging:pyproject-tool-table` - Same usage as that of the equivalent ``[tool]`` table from the - `pyproject.toml specification`_. + :ref:`packaging:pyproject-toml-spec`. --------- -Examples --------- +------- +Example +------- .. code-block:: TOML - version = '1.0' - hash-algorithm = 'sha256' - - [locker] - name = 'mousebender' - version = 'pep' - run = { module = 'mousebender', args = ['lock', '--platform', 'cpython3.12-manylinux2014-x64', '--platform', 'cpython3.12-windows-x64', 'cattrs', 'numpy'] } - - [[groups]] - name = 'Default' - requirements = [ - { name = 'cattrs' }, - { name = 'numpy' }, - ] + metadata-version = "1.0" + requires-python = ">=3.9" [[packages]] - name = 'attrs' - version = '24.2.0' - groups = ['Default'] - index_url = 'https://pypi.org/simple/attrs' - direct = false - requires_python = '>=3.7' - dependencies = [ - { name = 'importlib-metadata', marker = 'python_version < "3.8"' }, - { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'benchmark' }, - { name = 'hypothesis', feature = 'benchmark' }, - { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'benchmark' }, - { name = 'pympler', feature = 'benchmark' }, - { name = 'pytest-codspeed', feature = 'benchmark' }, - { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'benchmark' }, - { name = 'pytest-xdist', extras = ['psutil'], feature = 'benchmark' }, - { name = 'pytest', version = '>=4.3.0', feature = 'benchmark' }, - { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'cov' }, - { name = 'coverage', extras = ['toml'], version = '>=5.3', feature = 'cov' }, - { name = 'hypothesis', feature = 'cov' }, - { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'cov' }, - { name = 'pympler', feature = 'cov' }, - { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'cov' }, - { name = 'pytest-xdist', extras = ['psutil'], feature = 'cov' }, - { name = 'pytest', version = '>=4.3.0', feature = 'cov' }, - { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'dev' }, - { name = 'hypothesis', feature = 'dev' }, - { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'dev' }, - { name = 'pre-commit', feature = 'dev' }, - { name = 'pympler', feature = 'dev' }, - { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'dev' }, - { name = 'pytest-xdist', extras = ['psutil'], feature = 'dev' }, - { name = 'pytest', version = '>=4.3.0', feature = 'dev' }, - { name = 'cogapp', feature = 'docs' }, - { name = 'furo', feature = 'docs' }, - { name = 'myst-parser', feature = 'docs' }, - { name = 'sphinx', feature = 'docs' }, - { name = 'sphinx-notfound-page', feature = 'docs' }, - { name = 'sphinxcontrib-towncrier', feature = 'docs' }, - { name = 'towncrier', version = '<24.7', feature = 'docs' }, - { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'tests' }, - { name = 'hypothesis', feature = 'tests' }, - { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'tests' }, - { name = 'pympler', feature = 'tests' }, - { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'tests' }, - { name = 'pytest-xdist', extras = ['psutil'], feature = 'tests' }, - { name = 'pytest', version = '>=4.3.0', feature = 'tests' }, - { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'tests-mypy' }, - { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'tests-mypy' } - ] - editable = false + name = "attrs" + version = "23.2.0" + requires-python = ">=3.7" + index = "https://pypi.org/simple/" wheels = [ - { tags = ['py3-none-any'], url = 'https://files.pythonhosted.org/packages/6a/21/5b6702a7f963e95456c0de2d495f67bf5fd62840ac655dc451586d23d39a/attrs-24.2.0-py3-none-any.whl', hash = '81921eb96de3191c8258c199618104dd27ac608d9366f5e35d011eae1867ede2', upload_time = 2024-08-06T14:37:36.958006+00:00, size = 63001 } + {name = "attrs-23.2.0-py3-none-any.whl", upload-time = 2023-12-31T06:30:30.772444Z, url = "https://files.pythonhosted.org/packages/e0/44/827b2a91a5816512fcaf3cc4ebc465ccd5d598c45cefa6703fcf4a79018f/attrs-23.2.0-py3-none-any.whl", size = 60752, hashes = {sha256 = "99b87a485a5820b23b879f04c2305b44b951b502fd64be915879d77a7e8fc6f1"} } ] [[packages]] - name = 'cattrs' - version = '24.1.2' - groups = ['Default'] - index_url = 'https://pypi.org/simple/cattrs' - direct = false - requires_python = '>=3.8' - dependencies = [ - { name = 'attrs', version = '>=23.1.0' }, - { name = 'exceptiongroup', version = '>=1.1.1', marker = 'python_version < "3.11"' }, - { name = 'typing-extensions', version = '!=4.6.3,>=4.1.0', marker = 'python_version < "3.11"' }, - { name = 'pymongo', version = '>=4.4.0', feature = 'bson' }, - { name = 'cbor2', version = '>=5.4.6', feature = 'cbor2' }, - { name = 'msgpack', version = '>=1.0.5', feature = 'msgpack' }, - { name = 'msgspec', version = '>=0.18.5', marker = 'implementation_name == "cpython"', feature = 'msgspec' }, - { name = 'orjson', version = '>=3.9.2', marker = 'implementation_name == "cpython"', feature = 'orjson' }, - { name = 'pyyaml', version = '>=6.0', feature = 'pyyaml' }, - { name = 'tomlkit', version = '>=0.11.8', feature = 'tomlkit' }, - { name = 'ujson', version = '>=5.7.0', feature = 'ujson' } - ] - editable = false + name = "cattrs" + version = "23.2.3" + requires-python = ">=3.8" + index = "https://pypi.org/simple/" wheels = [ - { tags = ['py3-none-any'], url = 'https://files.pythonhosted.org/packages/c8/d5/867e75361fc45f6de75fe277dd085627a9db5ebb511a87f27dc1396b5351/cattrs-24.1.2-py3-none-any.whl', hash = '67c7495b760168d931a10233f979b28dc04daf853b30752246f4f8471c6d68d0', upload_time = 2024-09-22T14:58:34.812643+00:00, size = 66446 } + {name = "cattrs-23.2.3-py3-none-any.whl", upload-time = 2023-11-30T22:19:19.163763Z, url = "https://files.pythonhosted.org/packages/b3/0d/cd4a4071c7f38385dc5ba91286723b4d1090b87815db48216212c6c6c30e/cattrs-23.2.3-py3-none-any.whl", size = 57474, hashes = {sha256 = "0341994d94971052e9ee70662542699a3162ea1e0c62f7ce1b4a57f563685108"} } ] [[packages]] - name = 'numpy' - version = '2.1.2' - groups = ['Default'] - index_url = 'https://pypi.org/simple/numpy' - direct = false - requires_python = '>=3.10' - dependencies = [ - - ] - editable = false - wheels = [ - { tags = ['cp312-cp312-manylinux2014_x86_64', 'cp312-cp312-manylinux_2_17_x86_64'], url = 'https://files.pythonhosted.org/packages/9b/b4/e3c7e6fab0f77fff6194afa173d1f2342073d91b1d3b4b30b17c3fb4407a/numpy-2.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', hash = '6d95f286b8244b3649b477ac066c6906fbb2905f8ac19b170e2175d3d799f4df', upload_time = 2024-10-05T18:36:20.729642+00:00, size = 16041825 }, - { tags = ['cp312-cp312-win_amd64'], url = 'https://files.pythonhosted.org/packages/4c/79/73735a6a5dad6059c085f240a4e74c9270feccd2bc66e4d31b5ca01d329c/numpy-2.1.2-cp312-cp312-win_amd64.whl', hash = '456e3b11cb79ac9946c822a56346ec80275eaf2950314b249b512896c0d2505e', upload_time = 2024-10-05T18:37:38.159022+00:00, size = 12568254 } + name = "numpy" + version = "2.0.1" + requires-python = ">=3.9" + index = "https://pypi.org/simple/" + files = [ + {name = "numpy-2.0.1-cp312-cp312-macosx_10_9_x86_64.whl", upload-time = 2024-07-21T13:37:15.810939Z, url = "https://files.pythonhosted.org/packages/64/1c/401489a7e92c30db413362756c313b9353fb47565015986c55582593e2ae/numpy-2.0.1-cp312-cp312-macosx_10_9_x86_64.whl", size = 20965374, hashes = {sha256 = "6bf4e6f4a2a2e26655717a1983ef6324f2664d7011f6ef7482e8c0b3d51e82ac"} }, + {name = "numpy-2.0.1-cp312-cp312-macosx_11_0_arm64.whl", upload-time = 2024-07-21T13:37:36.460324Z, url = "https://files.pythonhosted.org/packages/08/61/460fb524bb2d1a8bd4bbcb33d9b0971f9837fdedcfda8478d4c8f5cfd7ee/numpy-2.0.1-cp312-cp312-macosx_11_0_arm64.whl", size = 13102536, hashes = {sha256 = "7d6fddc5fe258d3328cd8e3d7d3e02234c5d70e01ebe377a6ab92adb14039cb4"} }, + {name = "numpy-2.0.1-cp312-cp312-macosx_14_0_arm64.whl", upload-time = 2024-07-21T13:37:46.601144Z, url = "https://files.pythonhosted.org/packages/c2/da/3d8debb409bc97045b559f408d2b8cefa6a077a73df14dbf4d8780d976b1/numpy-2.0.1-cp312-cp312-macosx_14_0_arm64.whl", size = 5037809, hashes = {sha256 = "5daab361be6ddeb299a918a7c0864fa8618af66019138263247af405018b04e1"} }, + {name = "numpy-2.0.1-cp312-cp312-macosx_14_0_x86_64.whl", upload-time = 2024-07-21T13:37:58.784393Z, url = "https://files.pythonhosted.org/packages/6d/59/85160bf5f4af6264a7c5149ab07be9c8db2b0eb064794f8a7bf6d/numpy-2.0.1-cp312-cp312-macosx_14_0_x86_64.whl", size = 6631813, hashes = {sha256 = "ea2326a4dca88e4a274ba3a4405eb6c6467d3ffbd8c7d38632502eaae3820587"} }, + {name = "numpy-2.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", upload-time = 2024-07-21T13:38:19.714559Z, url = "https://files.pythonhosted.org/packages/5e/e3/944b77e2742fece7da8dfba6f7ef7dccdd163d1a613f7027f4d5b/numpy-2.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", size = 13623742, hashes = {sha256 = "529af13c5f4b7a932fb0e1911d3a75da204eff023ee5e0e79c1751564221a5c8"} }, + {name = "numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", upload-time = 2024-07-21T13:38:48.972569Z, url = "https://files.pythonhosted.org/packages/2c/f3/61eee37decb58e7cb29940f19a1464b8608f2cab8a8616aba75fd/numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", size = 19242336, hashes = {sha256 = "6790654cb13eab303d8402354fabd47472b24635700f631f041bd0b65e37298a"} }, + {name = "numpy-2.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", upload-time = 2024-07-21T13:39:19.213811Z, url = "https://files.pythonhosted.org/packages/77/b5/c74cc436114c1de5912cdb475145245f6e645a6a1a29b5d08c774/numpy-2.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", size = 19637264, hashes = {sha256 = "cbab9fc9c391700e3e1287666dfd82d8666d10e69a6c4a09ab97574c0b7ee0a7"} }, + {name = "numpy-2.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", upload-time = 2024-07-21T13:39:41.812321Z, url = "https://files.pythonhosted.org/packages/da/89/c8856e12e0b3f6af371ccb90d604600923b08050c58f0cd26eac9/numpy-2.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", size = 14108911, hashes = {sha256 = "99d0d92a5e3613c33a5f01db206a33f8fdf3d71f2912b0de1739894668b7a93b"} }, + {name = "numpy-2.0.1-cp312-cp312-win32.whl", upload-time = 2024-07-21T13:39:52.932102Z, url = "https://files.pythonhosted.org/packages/15/96/310c6f6d146518479b0a6ee6eb92a537954ec3b1acfa2894d1347/numpy-2.0.1-cp312-cp312-win32.whl", size = 6171379, hashes = {sha256 = "173a00b9995f73b79eb0191129f2455f1e34c203f559dd118636858cc452a1bf"} }, + {name = "numpy-2.0.1-cp312-cp312-win_amd64.whl", upload-time = 2024-07-21T13:40:17.532627Z, url = "https://files.pythonhosted.org/packages/b5/59/f6ad378ad85ed9c2785f271b39c3e5b6412c66e810d2c60934c9f/numpy-2.0.1-cp312-cp312-win_amd64.whl", size = 16255757, hashes = {sha256 = "bb2124fdc6e62baae159ebcfa368708867eb56806804d005860b6007388df171"} }, ] ------------------------- -Expectations for Lockers ------------------------- - -- Lockers MUST make sure that entering the dependency graph via a specific group - will not lead to ambiguity for installers as to which value in - ``[[packages]]`` to install for any environment (this can be controlled for - via ``packages.version`` and ``packages.groups``). -- Lockers SHOULD try to make all logically related groups resolve together - (i.e. no ambiguity if grouped together). -- If a ``groups.project`` would have extras that cause ambiguity or installation - failure due to conflicts between the extras, the locker MAY create - separate ``groups.requirements`` entries instead, otherwise the locker MUST - raise an error. -- Lockers MAY try to lock for multiple environments in a single lock file. -- Lockers MAY try to update a lock file containing ``[tool]`` and - ``[packages.tool]`` for other tools than themselves. -- Lockers MAY want to provide a way to let users provide the information - necessary to lock for other environments, e.g., supporting a JSON - file format which specifies wheel tags and marker values. - -.. code-block:: JSON - - { - "marker-values": {"": ""}, - "wheel-tags": [""] - } - - ---------------------------- -Expectations for Installers ---------------------------- - -- Installers MAY support installation of non-binary files - (i.e. source trees and source distributions), but are not required to. -- Installers MUST provide a way to avoid non-binary file installation for - reproducibility and security purposes. -- Installers SHOULD make it opt-in to use non-binary file installation to - facilitate a secure-by-default approach. -- If a traversal of the graph leads to any ambiguity as to what package version - to install (i.e. more than one package version qualifies), an error MUST be - raised. -- Installers MUST only consider package versions included in any selected - groups (i.e. installers cannot consider packages outside of the groups - selected to install from). -- Installers MUST error out if a package version lacks a way to install into the - chosen environment. -- Installers MUST support installing into an empty environment. - - -Pseudo-Code -=========== +------------ +Installation +------------ -.. code-block:: Python +The following outlines the steps to be taken to install from a lock file +(while the requirements are prescriptive, the general steps and order are +a suggestion): + +#. Check if the metadata version specified by ``metadata-version`` is supported; + an error or warning MUST be raised as appropriate. +#. If ``requires-python`` is specified, check that the environment being + installed for meets the requirement; an error MUST be raised if it is not + met. +#. If ``environments`` is specified, check that at least one of the environment + marker expressions is satisfied; error MUST be raised if no expression is + satisfied. +#. For each package listed in ``[[packages]]``: + + #. If ``marker`` is specified, check if it is satisfied; it it isn't, + skip to the next package. + #. If ``requires-python`` is specified, check if it is satisfied; an error + MUST be raised if it isn't. + #. Check that no other instance of the package has been slated to be + installed; an error about the ambiguity MUST be raied otherwise. + #. Check that the source of the package is specified appropriately (i.e. + there are not conflicting sources in the package entry); + an error MUST be raised if any issues are found. + #. Add the package to the set of packages to install. + +#. For each package to be installed: + + - If ``vcs`` is set: + + #. Clone the repository to the commit ID specified in ``commit-id``. + #. Build the package, respecting ``subdirectory``. + #. Install. - class UnsatisfiableError(Exception): - """Raised when a requirement cannot be satisfied.""" - - - class AmbiguityError(Exception): - """Raised when a requirement has multiple solutions.""" - - - def install_packages(lock_file_contents): - # Hard-coded out of laziness. - packages = choose_packages(lock_file_contents, (GROUP_NAME, frozenset())) - - for package in packages: - tags = list(packaging.tags.sys_tags()) - for tag in tags: # Prioritize by tag order. - tag_str = str(tag) - for wheel in package["wheels"]: - if tag_str in wheel["tags"]: - break - else: - continue - break - else: - raise UnsatisfiableError( - f"No wheel for {package['name']} {package['version']}" - ) - print(f"Installing {package['name']} {package['version']} ({tag_str})") - - - def choose_packages(lock_file_data, *selected_groups): - """Select the package versions that should be installed based on the requested groups. - - 'selected_groups' is a sequence of two-item tuples, representing a group name and - optionally any requested extras if the group is a project. - """ - group_names = frozenset(operator.itemgetter(0)(group) for group in selected_groups) - available_packages = {} # The packages in the selected groups. - for pkg in lock_file_data["packages"]: - if frozenset(pkg["groups"]) & group_names: - available_packages.setdefault(pkg["name"], []).append(pkg) - selected_packages = {} # The package versions that have been selected. - handled_extras = {} # The extras that have been handled. - requirements = [] # A stack of requirements to satisfy. - - # First, get our starting list of requirements. - for group in selected_groups: - requirements.extend(gather_requirements(lock_file_data, group)) - - # Next, go through the requirements and try to find a **single** package version - # that satisfies each requirement. - while requirements: - req = requirements.pop() - # Ignore requirements whose markers disqualify it. - if not applies_to_env(req): - continue - name = req["name"] - if pkg := selected_packages.get(name): - # Safety check that the cross-section of groups doesn't cause issues. - # It somewhat assumes the locker didn't mess up such that there would be - # ambiguity by what package version was initially selected. - if not version_satisfies(req, pkg): - raise UnsatisfiableError( - f"requirement {req!r} not satisfied by " - f"{selected_packages[req['name']]!r}" - ) - if "extras" not in req: - continue - needed_extras = req["extras"] - if not (extras := handled_extras.set_default(name, set())).difference( - needed_extras - ): - continue - # This isn't optimal as we may tread over the same extras multiple times, - # but eventually the maximum set of extras for the package will be handled - # and thus the above guard will short-circuit adding any more requirements. - extras.update(needed_extras) - else: - # Raises UnsatisfiableError or AmbiguityError if no suitable, single package - # version is found. - pkg = compatible_package_version(req, available_packages[req["name"]]) - selected_packages[name] = pkg - requirements.extend(dependencies(pkg, req)) - - return selected_packages.values() - - - def gather_requirements(locked_file_data, group): - """Return a collection of all requirements for a group.""" - # Hard-coded to support `groups.requirements` out of laziness. - group_name, _extras = group - for group in locked_file_data["groups"]: - if group["name"] == group_name: - return group["requirements"] - else: - raise ValueError(f"Group {group_name!r} not found in lock file") - - - def applies_to_env(requirement): - """Check if the requirement applies to the current environment.""" - try: - markers = requirement["marker"] - except KeyError: - return True - else: - return packaging.markers.Marker(markers).evaluate() - - - def version_satisfies(requirement, package): - """Check if the package version satisfies the requirement.""" - try: - raw_specifier = requirement["version"] - except KeyError: - return True - else: - specifier = packaging.specifiers.SpecifierSet(raw_specifier) - return specifier.contains(package["version"], prereleases=True) - - - def compatible_package_version(requirement, available_packages): - """Return the package version that satisfies the requirement. - - If no package version can satisfy the requirement, raise UnsatisfiableError. If - multiple package versions can satisfy the requirement, raise AmbiguityError. - """ - possible_packages = [ - pkg for pkg in available_packages if version_satisfies(requirement, pkg) - ] - if not possible_packages: - raise UnsatisfiableError(f"No package version satisfies {requirement!r}") - elif len(possible_packages) > 1: - raise AmbiguityError(f"Multiple package versions satisfy {requirement!r}") - return possible_packages[0] - - - def dependencies(package, requirement): - """Return the dependencies of the package. - - The extras from the requirement will extend the base requirements as needed. - """ - applicable_deps = [] - extras = frozenset(requirement.get("extras", [])) - for dep in package["dependencies"]: - if "feature" not in dep or dep["feature"] in extras: - applicable_deps.append(dep) - return applicable_deps + - Else if ``directory`` is set: + + #. Build the package, respecting ``subdirectory``. + #. Install. + + - Else if ``archive`` is set: + + #. Get the file. + #. Build the package, respecting ``subdirectory``. + #. Install. + + - Else if there are entries for ``wheels``: + + #. Look for the appropriate wheel file based on ``name``; if one is not + found then move on to ``sdist`` or an error MUST be raised about a + lack of source for the project. + #. Get the file. + #. Install. + + - Else if no ``wheel`` file is found or ``sdist`` is solely set: + + #. Get the file. + #. Build the package. + #. Install. + + +---------------------------------------------------- +Semantic differences with ``requirements.txt`` files +---------------------------------------------------- + +Ignoring formatting, there are a few differences between lock files as proposed +by this PEP and those that are possible via a `requirements file`_. + +Some of the differences are in regards to security. Requiring hashes, recording +file sizes, and where a file was found -- both the index and the location of the +file itself -- help with auditing and validating the files that were locked +against. Compare that with requirements files which can optionally include +hashes, but it is an opt-in feature and can be bypassed. The optional inclusion +of a file's upload time and where the files can be found is also different. + +Being explicit about the supported Python versions and environments for the file +overall is also unique to this PEP. This is to alleviate the issue of not +knowing when a requirements file targets a specific platform. + +The ``[tool]`` tables don't have a direct correlation in requirements files. +They do support comments, but they are not inherently structured like the +``[tool]`` table is thanks to being in TOML. + +While comments in a requirements file could record details that are helpful for +auditing and understanding what the lock file contains, providing the structured +support to record such things makes auditing easier. Recording the required +Python version for a package upfront helps with this as well as erroring out +sooner if an install is going to fail. Recording the wheel file name separate +from the URL or path is also to help make reading the list of wheel files easier +as it encodes information that can be useful when understanding and auditing a +file. Recording the sdist file name is for the same reason. ======================= @@ -947,51 +722,50 @@ Backwards Compatibility Because there is no preexisting lock file format, there are no explicit backwards-compatibility concerns in terms of Python packaging standards. -As for packaging tools themselves, that will be a per-tool decision. For tools -that don't document their lock file format, they could choose to simply start -using the format internally and then transition to saving their lock files with -a name supported by this PEP. For tools with a preexisting, documented format, -they could provide an option to choose which format to emit. +As for packaging tools themselves, that will be a per-tool decision as to +whether they choose to support this PEP and in what way (i.e. as an export +target or as the primary way they record their lock file). ===================== Security Implications ===================== -The hope is that by standardizing on a lock file format that starts from a +The hope is that by standardizing on a lock file format which starts from a security-first posture it will help make overall packaging installation safer. However, this PEP does not solve all potential security concerns. One potential concern is tampering with a lock file. If a lock file is not kept in source control and properly audited, a bad actor could change the file in -nefarious ways (e.g. point to a malware version of a package). Tampering could -also occur in transit to e.g. a cloud provider who will perform an installation +nefarious ways (e.g., point to a malware version of a package). Tampering could +also occur in transit to e.g., a cloud provider who will perform an installation on the user's behalf. Both could be mitigated by signing the lock file either within the file in a ``[tool]`` entry or via a side channel external to the lock file itself. -This PEP does not do anything to prevent a user from installing an incorrect +This PEP does not do anything to prevent a user from installing incorrect packages. While including many details to help in auditing a package's inclusion, -there isn't any mechanism to stop e.g. name confusion attacks via typosquatting. -Lockers may be able to provide some UX to help with this (e.g. by providing -download counts for a package). +there isn't any mechanism to stop e.g., name confusion attacks via +typosquatting. Tools may be able to provide some UX to help with this (e.g., by +providing download counts for a package). ================= How to Teach This ================= -Users should be informed that when they ask to install some package, that +Users should be informed that when they ask to install some package, the package may have its own dependencies, those dependencies may have dependencies, and so on. Without writing down what gets installed as part of installing the package they requested, things could change from underneath them (e.g., package versions). Changes to the underlying dependencies can lead to accidental breakage of their code. Lock files help deal with that by providing a way to -write down what was (and should be) installed. +write down what was installed so you can install the exact same thing in the +future. Having what to install written down also helps in collaborating with others. By agreeing to a lock file's contents, everyone ends up with the same packages -installed. This helps make sure no one relies on e.g. an API that's only +installed. This helps make sure no one relies on e.g., an API that's only available in a certain version that not everyone working on the project has installed. @@ -1005,37 +779,32 @@ the change is on purpose and not one slipped in by a bad actor. Reference Implementation ======================== -A proof-of-concept implementing most of this PEP for wheels can be found at -https://github.com/brettcannon/mousebender/tree/pep . +A proof-of-concept implementing most of this PEP for various versions +of this PEP can be found at +https://github.com/brettcannon/mousebender/tree/pep . While the various +implementations have not matched the exact format of this PEP, the general +semantic requirements have been implemented before. + +Prior to acceptance of this PEP, the PoC will be updated. ============== Rejected Ideas ============== ---------------------------------- -A flat set of packages to install ---------------------------------- +-------------------------------------------------------- +Recording the dependency graph for installation purposes +-------------------------------------------------------- -An earlier version of this PEP proposed to use a flat set of package versions -instead of a graph. The idea was that each package version could be evaluated in -isolation as to whether it applied to an environment for installation. The hope -was that would lend itself to easier auditing as one wouldn't have to worry -about how a package version fit into the graph when looking at e.g., a diff for -a lock file. +A previous version of this PEP recorded the dependency graph of packages instead +of a set of packages to install. The idea was that by recording the dependency +graph you not only got more information, but it provided more flexibility by +supporting more features innately (e.g., platform-specific dependencies without +explicitly propagating markers). -Unfortunately this was deemed not as flexible as using a graph. For instance, -recording the graph -`assists in dependency analysis for tools like GitHub `__. -A graph also makes following how you ended up with dependencies within your lock -file from any point in the graph. It also balances out the implementation costs -a bit more between lockers and installers by alleviating the complexity off of -lockers a bit for only a minor increase in complexity for installers by -involving standard graph-traversing algorithms instead of a linear walk. - -And if the dependency graph is already being recorded for the above benefits, -then recording that same data in a flattened manner is redundant that makes -lock files larger and potentially more unruly. +In the end, though, it was deemed to add complexity that wasn't worth the cost +(e.g., it impacted the ease of auditing for details which were not necessary +for this PEP to reach its goals). ------------------------------------------------------------------------------------- @@ -1071,7 +840,7 @@ lockers. ----------------------------------------- -Requiring specific hash algorithm support +Requiring minimum hash algorithm support ----------------------------------------- It was proposed to require a baseline hash algorithm for the files. This was @@ -1083,19 +852,6 @@ simply defaulting to the baseline in tools without considering the security ramifications of that hash algorithm. ------------------------------------- -Require a URL or file path for files ------------------------------------- - -Originally references to files were required, e.g., ``packages.sdist.url`` or -``packages.sdist.path``. But at least -`one use-case `__ -surfaced during discussions about this PEP where statically specifying the -location of files would be problematic. And in earlier discussions the idea of -the location being a hint wasn't preferred. Hence the PEP now makes the data -optional, but considers the locations accurate if specified. - - ----------- File naming ----------- @@ -1153,16 +909,19 @@ deemed more important to keep using TOML. Other keys ---------- -Multiple hashes per file -======================== +A single hash algorithm for the whole file +========================================== + +Earlier versions of this PEP proposed having a single hash algrorithm be +specified per file instead of any number of algorithms per file. The thinking +was that by specifying a single algorithm it would help with auditing the file +when a specific hash algorithm was mandated for use. -An initial version of this PEP proposed supporting multiple hashes per file. The -idea was to allow one to choose which hashing algorithm they wanted to go with -when installing. But upon reflection it seemed like an unnecessary complication -as there was no guarantee the hashes provided would satisfy the user's needs. -As well, if the single hash algorithm used in the lock file wasn't sufficient, -rehashing the files involved as a way to migrate to a different algorithm didn't -seem insurmountable. +In the end there was some objection to this idea. Typically it centered around +the cost of rehashing large wheel files (e.g., PyTorch). There was also concern +about making hashing decisions upfront on the installer's behalf which they may +disagree with. In the end it was deemed better to have flexibility and let +people audit the lock file as they see fit. Hashing the contents of the lock file itself @@ -1188,12 +947,11 @@ recording the creation date of the lock file. But for some same merge conflict reasons as storing the hash of the file contents, this idea was dropped. -Recording the package indexes used -================================== +Recording the package indexes used in searching +=============================================== -Recording what package indexes were used by the locker to decide what to lock -for was considered. In the end, though, it was rejected as it was deemed -unnecessary bookkeeping. +Recording what package indexes were used to create the lock file was considered. +In the end, though, it was rejected as it was deemed unnecessary bookkeeping. Locking build requirements for sdists @@ -1210,69 +968,318 @@ could propose a solution. Open Issues =========== ----------------------------------------------- -Specify ``requires-python`` at the file level? ----------------------------------------------- +-------------- +Simplification +-------------- -The lock file formats from PDM_, Poetry_, and uv_ all specify -``requires-python`` at the top level for the absolute minimum Python version -needed for the lock file. This can be inferred, though, by examining all -``packages.requires-python`` values. The global value might also not be -accurate for all platforms depending on how environment markers influence what -package versions are installed and what their Python version requirements are. +Drop recording the package version +================================== +As this is written, the package version is optional since it can only be +reliably recorded when an sdist of wheel file is used. And since both sources +record the version in file names it is technically redundant. ---------------------- -Don't pre-parse data? ---------------------- +But having the version explicitly called out could be viewed as helping with +auditing by not having to find and parse file names (especially if an sdist +file name doesn't conform to :ref:`packaging:source-distribution-format-sdist`). -This PEP currently takes the viewpoint that if a piece of data is going to be -parsed by installers everytime they run, then trying to pre-parse as much as -possible so the TOML parser can help is a good thing. The thinking is TOML -parsers have a higher chance of being optimized, and so letting them do more -parsing leads to a faster outcome. It should also increase readability by -breaking apart data upfront more. -But in the case of doing this to wheel file names, some might consider it too -much. The question becomes whether separating out all the parts of a wheel -file name hinders readability because people are used to reading the file names -already, or by clearly separating its parts it actually helps make installers -faster, easier to write, and doesn't hinder readability. +Drop the requirement to specify the location of an sdist and/or wheels +====================================================================== -This all equally applies to requirement specifiers. +At least one person has commented how their work has unstable URLs for all +sdists and wheels. As such, they have to search for all files at install +regardless of where the file was found previously. Dropping the requirement to +provide the URL or path to a file would help solve the issue of recording +known-bad information. +To support this, though, would require installation to support finding files via +a package index or some other mechanism specified outside of this PEP. The +former adds complexity (discussed as another open issue), while the latter means +this PEP cannot fully explain the installation process. -============== -Deferred Ideas -============== + +Drop requiring file size and hashes +=================================== + +At least one person has said that their work modifies all wheels and sdists with +internal files. That means any recorded hashes and file sizes will be wrong. By +making the file size and hashes optional -- very likely through some opt-out +mechanism -- then they could continue to produce lock files that meet this PEP's +requirements. + +As it weakens security by not making hashes and file sizes mandatory, it +somewhat dilutes the purpose of this PEP. It also only works with external +projects if the creator of the lock file is external to the company modifying +the files **and** chose to leave out hashes. It also is only beneficial if +the file modifications are not idempotent, thus causing random changes in +hashes and file size. + + +Drop recording the sdist file name +================================== + +While incompatible with dropping the URL/path requirement, the package +version, and hashes, recording the sdist file name is technically not necessary +at all (right now recording the file name is optional). The file name only +encodes the project name and version, so no new info is conveyed about the file +(when the package version is provided). And if the location is recorded then +getting the file is handled regardless of the file name. + +But recording the file name can helpful when looking for an appropriate file +when the recorded file location is no longer available (while sdist file names +are now standardized thanks to :pep:`625`, that has only been true since 2020 +and thus there are many older sdists with names that may not be guessable). + + +Support installing files via a package index +============================================ + +With a package index URL and a file name, one can find the location of a file +at install-time. This not only allows recording the URL or path optional, it +could also act as a fallback if the original location is no longer valid. + +This does increase the burden on tools performing installation as they would +now have to support this fallback. It could be made as an optional feature, +although the chances are people will expect it to be implemented as it shouldn't +increase the complexity of an installer drastically. + + +Make ``packaging.wheels`` a table +================================= + +One could see writing out wheel file details as a table keyed on the file name. +For example: + +.. code-block:: TOML + + [[packages]] + name = "attrs" + version = "23.2.0" + requires-python = ">=3.7" + index = "https://pypi.org/simple/" + + [packages.wheels] + "attrs-23.2.0-py3-none-any.whl" = {upload-time = 2023-12-31T06:30:30.772444Z, url = "https://files.pythonhosted.org/packages/e0/44/827b2a91a5816512fcaf3cc4ebc465ccd5d598c45cefa6703fcf4a79018f/attrs-23.2.0-py3-none-any.whl", size = 60752, hashes = {sha256 = "99b87a485a5820b23b879f04c2305b44b951b502fd64be915879d77a7e8fc6f1"} + + [[packages]] + name = "numpy" + version = "2.0.1" + requires-python = ">=3.9" + index = "https://pypi.org/simple/" + + [packages.wheels] + "numpy-2.0.1-cp312-cp312-macosx_10_9_x86_64.whl" = {upload-time = 2024-07-21T13:37:15.810939Z, url = "https://files.pythonhosted.org/packages/64/1c/401489a7e92c30db413362756c313b9353fb47565015986c55582593e2ae/numpy-2.0.1-cp312-cp312-macosx_10_9_x86_64.whl", size = 20965374, hashes = {sha256 = "6bf4e6f4a2a2e26655717a1983ef6324f2664d7011f6ef7482e8c0b3d51e82ac"} + "numpy-2.0.1-cp312-cp312-macosx_11_0_arm64.whl" = {upload-time = 2024-07-21T13:37:36.460324Z, url = "https://files.pythonhosted.org/packages/08/61/460fb524bb2d1a8bd4bbcb33d9b0971f9837fdedcfda8478d4c8f5cfd7ee/numpy-2.0.1-cp312-cp312-macosx_11_0_arm64.whl", size = 13102536, hashes = {sha256 = "7d6fddc5fe258d3328cd8e3d7d3e02234c5d70e01ebe377a6ab92adb14039cb4"} + "numpy-2.0.1-cp312-cp312-macosx_14_0_arm64.whl" = {upload-time = 2024-07-21T13:37:46.601144Z, url = "https://files.pythonhosted.org/packages/c2/da/3d8debb409bc97045b559f408d2b8cefa6a077a73df14dbf4d8780d976b1/numpy-2.0.1-cp312-cp312-macosx_14_0_arm64.whl", size = 5037809, hashes = {sha256 = "5daab361be6ddeb299a918a7c0864fa8618af66019138263247af405018b04e1"} + "numpy-2.0.1-cp312-cp312-macosx_14_0_x86_64.whl" = {upload-time = 2024-07-21T13:37:58.784393Z, url = "https://files.pythonhosted.org/packages/6d/59/85160bf5f4af6264a7c5149ab07be9c8db2b0eb064794f8a7bf6d/numpy-2.0.1-cp312-cp312-macosx_14_0_x86_64.whl", size = 6631813, hashes = {sha256 = "ea2326a4dca88e4a274ba3a4405eb6c6467d3ffbd8c7d38632502eaae3820587"} + "numpy-2.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl" = {upload-time = 2024-07-21T13:38:19.714559Z, url = "https://files.pythonhosted.org/packages/5e/e3/944b77e2742fece7da8dfba6f7ef7dccdd163d1a613f7027f4d5b/numpy-2.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", size = 13623742, hashes = {sha256 = "529af13c5f4b7a932fb0e1911d3a75da204eff023ee5e0e79c1751564221a5c8"} + "numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl" = {upload-time = 2024-07-21T13:38:48.972569Z, url = "https://files.pythonhosted.org/packages/2c/f3/61eee37decb58e7cb29940f19a1464b8608f2cab8a8616aba75fd/numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", size = 19242336, hashes = {sha256 = "6790654cb13eab303d8402354fabd47472b24635700f631f041bd0b65e37298a"} + "numpy-2.0.1-cp312-cp312-musllinux_1_1_x86_64.whl" = {upload-time = 2024-07-21T13:39:19.213811Z, url = "https://files.pythonhosted.org/packages/77/b5/c74cc436114c1de5912cdb475145245f6e645a6a1a29b5d08c774/numpy-2.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", size = 19637264, hashes = {sha256 = "cbab9fc9c391700e3e1287666dfd82d8666d10e69a6c4a09ab97574c0b7ee0a7"} + "numpy-2.0.1-cp312-cp312-musllinux_1_2_aarch64.whl" = {upload-time = 2024-07-21T13:39:41.812321Z, url = "https://files.pythonhosted.org/packages/da/89/c8856e12e0b3f6af371ccb90d604600923b08050c58f0cd26eac9/numpy-2.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", size = 14108911, hashes = {sha256 = "99d0d92a5e3613c33a5f01db206a33f8fdf3d71f2912b0de1739894668b7a93b"} + "numpy-2.0.1-cp312-cp312-win32.whl" = {upload-time = 2024-07-21T13:39:52.932102Z, url = "https://files.pythonhosted.org/packages/15/96/310c6f6d146518479b0a6ee6eb92a537954ec3b1acfa2894d1347/numpy-2.0.1-cp312-cp312-win32.whl", size = 6171379, hashes = {sha256 = "173a00b9995f73b79eb0191129f2455f1e34c203f559dd118636858cc452a1bf"} + "numpy-2.0.1-cp312-cp312-win_amd64.whl" = {upload-time = 2024-07-21T13:40:17.532627Z, url = "https://files.pythonhosted.org/packages/b5/59/f6ad378ad85ed9c2785f271b39c3e5b6412c66e810d2c60934c9f/numpy-2.0.1-cp312-cp312-win_amd64.whl", size = 16255757, hashes = {sha256 = "bb2124fdc6e62baae159ebcfa368708867eb56806804d005860b6007388df171"} + + +It's entirely a structural change which some may (not) prefer. ---------------- -Per-file locking +Self-Referential ---------------- -An earlier version of this PEP supported two approaches to locking: *per-file* -and **per-package**. The idea for the former approach to locking was that if you -were locking for an a-priori set of environments you could lock to just the -files necessary to install into those environments. The thinking was that by -only listing a subset of files that auditing would be easier. +Record what tool created the lock file +====================================== -Unfortunately there was disagreement on how best to express upfront what the -supported environment requirements would be. Since what this PEP currently -proposes still prevents accidental success of installation into unsupported -environments, this idea has been deferred until such time someone can come up -with a representation that makes sense. +Right now the PEP does not record any details about the tool that created a +file. That's out of simplicity reasons only. Which tool is used may be +implicitly recorded by a ``[tool]`` table. +But one could record various amounts of details about the tool to help recreate +the file. Key details like tool name, the installation requirements when the +tool is hosted on PyPI (encoded as :ref:`packaging:dependency-specifiers`), and +the command used to create the file would allow another tool to re-run the tool. +It would also help discover what tool was used. --------------------------------- -Allowing for multiple lock files --------------------------------- -Before the introduction of ``[[groups]]``, this PEP proposed supporting multiple -lock files that would match the regular expression -``r"pylock\.(.+)\.toml"`` if a name for the lock file is desired or if multiple -lock files exist. But since ``[[groups]]`` subsumes a lot of the need to support -multiple lock files, this specific feature can be postponed until such time that -a need is shown to support multiple lock files. +Drop the ``[tool]`` table +========================= + +The ``[tool]`` table is included as it has been found to be very useful for +``pyproject.toml`` files. Providing similar flexibility to this PEP is done in +hopes that similar benefits will materialize. + +But some people are concerned that such a table will be too enticing to tools +and will lead to files that are tool-specific and unusable by other tools. This +could cause issues for tools trying to do installation, auditing, etc. as they +would not know what details in the ``[tool]`` table are somehow critical. + + +Restrict the ``[tool]`` table to data that is disposable +======================================================== + +The ``[tool]`` table is included as it has been found to be very useful for +``pyproject.toml`` files. Providing similar flexibility to this PEP is done in +hopes that similar benefits will materialize. + +But some people are concerned that such a table will be too enticing to tools +and will lead to files that are tool-specific and unusable by other tools. As +such, some have suggested only recording data that could be tossed at any time +and have no negative effect (e.g., caching info). That would allow another tool +to update a file and delete the ``[tool]`` tables without fear of impacting the +file adversely. + + +List the requirement inputs for the file +======================================== + +Right now the file does not record the requirements that acted as inputs to the +file. This is for simplicity reasons and to not explicitly constrain the file +in some unforeseen way (e.g., updating the file after initial creation for a +new platform that has different requirements, all without having to resolve +how to write a comprehensive set of requirements). + +But it may help in auditing and any recreation of the file if the original +requirements were somehow recorded. This could be a single string or an array +of strings if multiple requirements were used with the file. + + +-------- +Auditing +-------- + +Recording dependencies +====================== + +Recording the dependencies of a package is not necessary to install it. As such, +it has been left out of the PEP as it can be included via ``[tool]``. + +But knowing how costly a package is to include may be beneficial to users when +determining why a certain package was included in the lock file. A flexible +approach could be used to record the dependencies, e.g., as much detail as to +differentiate from any other entry for the same package in the file (inspired +by uv_). + + +Recording dependents +==================== + +Recording the dependencies of a package is not necessary to install it. As such, +it has been left out of the PEP as it can be included via ``[tool]``. + +But knowing how critical a package is to other packages may be beneficial. This +information is included by `pip-tools`_ , so there's prior art in including it. +A flexible approach could be used to record the dependencies, e.g., as much +detail as to differentiate from any other entry for the same package in the file +(inspired by uv_). + + +Including index-hosted attestatons +================================== + +:ref:`packaging:index-hosted-attestations` specifies attestation details for +files uploaded to a package index like PyPI. Including some of those details may +help detect issues with packaging when auditing the file (e.g., the publisher +suddenly changing).The key reason this isn't included in the PEP is because the +specification is entirely focused on JSON. In order to bring it to this PEP +either how to translate JSON to TOML would need to be specified, embed the +JSON payload as a string, or re-specify some or all of the attestation spec. + + +------------------------- +Expanding the feature set +------------------------- + +This PEP is currenty oriented towards standardizing on something that can +replace a ``requirements.txt`` file that acts as a lock file (e.g., what +`pip-tools`_ produces). But with an expansion of features, the file format may be +able to replace the internal lock file format used by tools like PDM_ and +Poetry_, especially when a ``pyproject.toml`` file is viewed as the ideal input +for creating a lock file. + + +Record the requirements for extras of a package +=============================================== + +A project with a ``pyprojec.toml`` file may define some extras which add +dependencies to install. In the simple case this would just be a matter of +marking an antry in ``[[packages]]`` as only applying when a specific extra is +requested. Unfortunately the simple case doesn't cover all cases. + +Consider the following example where the latest release of NumPy is 2.2.1 and +the last NumPy 1 release was 1.26.4: + +.. code-block:: TOML + + [project.optional-dependencies] + extra-1 = ["numpy"] + extra-2 = ["numpy~=1.0"] + +Individually those extras cause no issue. But extra-2 does "overpower" +extra-1 when it comes to what version of NumPy to install. That leads to the +issue of needing a way to record the fact that if extra-1 is requested on its +own then NumPy 2.2.1 should be recorded in the lock file, but if extra-2 is +specified (either on its own or in conjunction with extra-1), then NumPy 1.26.4 +should be recorded. + +There are two possible soluutions to this. + + +A single version across all extras in a single lock file +-------------------------------------------------------- + +One solution to the problem is to do what uv_ does and lock to a single version +for a package no matter what. That would mean any use of NumPy which could occur +in any scenario would use NumPy 1.26.4. Some argue that leads to consistency as +you won't be wondering what version of NumPy you will end up with based on what +extras you select. + +But this does mean that if you want the version of NumPy to vary across extras +you will need to create separate lock files for the varyious NumPy versions you +want. While not technically an issue, it is ergonomically a bit annoying when +this is necessary. But it's not known how frequently varying package versions +which depend on which extra(s) are chosen occur, and when they do occur do people +still want the variance or prefer the approach uv_ has taken. + +If this solution were to be taken, then very likely an ``extras`` key would be +added which would list the extras that the entry in ``[[package]]`` should be +used for. This works thanks to extras being additive, and thus only contributing +more packages. + + +Support boolean logic for extra selection +----------------------------------------- + +Another solution to this problem is specifying the conditions under +which a package version applies. This would mean supporting boolean logic +to fully express the conditions under which a package applies. + +But historically extras have not been expressed this way. The use of the +``extra`` clause in :ref:`packaging:core-metadata-requires-dist` is always +singular and with a ``==`` operator. This also means the operators on ``extra`` +have not been designed to treat the extras specified as a set, and so an +expression simultaneously using ``==`` and ``!=`` are not well-defined when it +comes to ``extra``. This all means that using +``extra == 'extra-1' and extra != 'extra-2'`` to appropriately express what is +needed for extra-1 to work has not been done before. It would also mean +potentially more use of an ``extra`` key as the default package version may need +to explicitly exclude all extra groups when other groups restrict what package +versions apply. + +For this to work we would either need to expand the use of the ``extra`` clause +so it can be used in ``packages.marker`` or have an ``extras`` key which +expresses a boolean expression for under which the package should be used. In +both situations the spec around ``extra`` would need to be expanded by this PEP +-- or another PEP before this one is accepted -- to lay out how boolean +expressions would work in this case. + + +Record dependency groups +======================== + +Dependency groups have the same concerns as extras mentioned above along with +lacking any preexisting clause for use in dependency specifiers. And so +dependency groups have the added issue that to use boolean expressions would +require defining a new clause type. ================ @@ -1281,7 +1288,7 @@ Acknowledgements Thanks to everyone who participated in the discussions on discuss.python.org. Also thanks to Randy Döring, Seth Michael Larson, Paul Moore, and Ofek Lev for -providing feedback on a draft version of this PEP. +providing feedback on a draft version of this PEP before going public. ========= @@ -1291,22 +1298,12 @@ Copyright This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. - -.. _core metadata: https://packaging.python.org/en/latest/specifications/core-metadata/ .. _Dependabot: https://docs.github.com/en/code-security/dependabot -.. _dependency specifiers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/ -.. _direct URL reference: https://packaging.python.org/en/latest/specifications/direct-url/ -.. _environment markers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers -.. _normalized name: https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization .. _PDM: https://pypi.org/project/pdm/ .. _pip-tools: https://pypi.org/project/pip-tools/ .. _Poetry: https://python-poetry.org/ -.. _project index: https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-list -.. _pyproject.toml specification: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-specification -.. _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ +.. _requirements file: +.. _requirements files: https://pip.pypa.io/en/stable/reference/requirements-file-format/ .. _software bill of materials: https://www.cisa.gov/sbom -.. _source tree: https://packaging.python.org/en/latest/specifications/source-distribution-format/#source-trees .. _TOML: https://toml.io/ .. _uv: https://github.com/astral-sh/uv -.. _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ -.. _wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/