From 79036e6a0b59698c89ecae0b28278b411a2fde1f Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 25 Sep 2024 11:12:53 +0100 Subject: [PATCH 01/15] PEP: Default Extras for Python Software Packages --- peps/pep-9999.rst | 502 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 502 insertions(+) create mode 100644 peps/pep-9999.rst diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst new file mode 100644 index 00000000000..32e2ec1b82f --- /dev/null +++ b/peps/pep-9999.rst @@ -0,0 +1,502 @@ +PEP: 9999 +Title: Default Extras for Python Software Packages +Author: Thomas Robitaille , Jonathan Dekhtiar +Sponsor: Pradyun Gedam +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 24-Sep-2024 + +Abstract +======== + +:pep:`508` specifies a mini-language for +declaring package dependencies. One feature of this language is the ability to +specify *extras*, which are optional components of a distribution that, when +used, install additional dependencies. This PEP proposes a mechanism to allow +one or more extras to be installed by default if none are provided explicitly. + +Motivation +========== + +Various use cases for default extras and possible solutions in this PEP were discussed +extensively on `this DPO thread `__. +These fall into two broad cases that that provide the +motivation for the present PEP. + +Recommended but not required dependencies +----------------------------------------- + +Package maintainers often use extras to declare optional dependencies that +extend the functionality or performance of a package. In some cases, it can be +difficult to determine which dependencies should be required and which should be +categorized as extras. A balance must be struck between the needs of typical +users (who may prefer most features to be available 'by default') and users who +want minimal installations without large, optional dependencies. One solution +with existing Python packaging infrastructure is for package maintainers to +define an extra called, for example, ``recommended``, which +includes all non-essential but suggested dependencies. Users are then instructed to +install the package using ``package[recommended]``, while those who prefer more +control can use ``package``. However, in practice, many users are unaware +of the ``[recommended]`` syntax, placing the burden on them to know this for a +typical installation. Having a way to have recommended dependencies be installed +by default while providing a way for users to request a more minimal installation +would satisfy this use case. The present PEP will describe a solution for this use case. + +Packages supporting multiple backends or frontends +-------------------------------------------------- + +Another common use case for using extras is to define different backends or +frontends and dependencies that need to be installed for each backend or +frontend. A package might need at least one backend or frontend to be installed +in order to be functional, but may be flexible on which backend or frontend this +is. Concrete examples of such frontends or backends include: + +* The Qt frontend library, which requires either PySide or PyQt to be installed +* BLAS/LAPACK, which have different possible implementations (e.g. OpenBLAS, and MKL) +* FFT libraries, which also have different implementations (e.g. ``scipy.fft`` and pyFFTW) + +With current packaging standards, maintainers have to either +require one of the backends or frontends, or require users +to always specify extras, e.g. ``package[backend]`` and therefore risk users +having an unusable installation if they only install ``package``. Having a +way to specify one or more default backend or frontend and providing a way to +override these defaults would provide a much better experience for users, and +the approach described in this PEP will allow this. + +Note that this PEP does not aim to address the issue of disallowing conflicting +or incompatible extras - for example if a package requires exactly one frontend +or backend package. There is currently no mechanism in Python packaging +infrastructure to disallow conflicting or incompatible extras to be installed, +and this PEP does not change that. + +Rationale +========= + +A number of possible solutions have been extensively and vigorously discussed by +the community for several years, including in `this DPO thread +`__ +as well as in numerous issues and pull requests. The solution that will be +presented below may not be everyone's preferred approach from a philosophical +point of view, however it appears to be the best compromise as it is the only +solution to a very real problem that would not break backward-compatibility of +packaging infrastructure in any significant way. It is an opt-in solution which +means that package maintainers can choose whether or not to use it, and it is +flexible enough to accommodate both of the major use cases described in +`Motivation`_. + +Specification +============= + +``Default-Extra`` Metadata Field +--------------------------------- + +A new metadata field, ``Default-Extra``, will be added to the `core package +metadata `_. +This field allows package maintainers to define an extra that is +automatically included when a user installs the package without specifying any +extras explicitly:: + + Default-Extra: recommended + +If multiple default extras are needed, one ``Default-Extra:`` entry +should be provided for each one:: + + Default-Extra: backend1 + Default-Extra: backend2 + Default-Extra: backend3 + +New key in ``[project]`` metadata table +--------------------------------------- + +A new key will be added to the ``[project]`` table in project metadata as +originally defined in :pep:`621` and now defined in the `PyPA specifications +`_. This key will be named +``default-optional-dependencies`` with the following description: + +* `TOML `_ type: Array of strings +* Corresponding core metadata field: ``Default-Extra`` + +Each string in ``default-optional-dependencies`` should be the name of an extra +defined in `optional-dependencies +`_, +and each extra in this array will be converted to a matching ``Default-Extra`` +entry in the core package metadata. Examples of valid usage which would +produce the example ``Default-Extra`` entries presented in the previous section are: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "recommended" + ] + +and: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "backend1", + "backend2", + "backend3" + ] + + +Overriding default extras +------------------------- + +If extras are explicitly given in a dependency specification, the default +extras are not installed. Otherwise, the default extras are used. + +For example, if a package +defines a ``extra1`` default extra as well as a non-default ``extra2`` +extra, then if a user was to install the package with:: + + pip install package + +the ``extra1`` dependency would be included. If the user instead uses:: + + pip install package[extra2] + +then the ``extra1`` extras would not be installed. + +If the same package is specified multiple times in a dependency tree, then if +any of these were specified without any explicit extras, the default extras +should be installed even if other occurrences of the package in the dependency +tree have explicitly specified extras. + +Note that ``package[]`` would continue to be equivalent to ``package`` and would +not be provided as a way to install without default extras (see the `Rejected +Ideas`_ section for the rationale). + +Installing without default extras +--------------------------------- + +In some cases, package maintainers may want to facilitate installing packages +without any default extras. In this case, as will be shown in more detail in +`How to teach this`_, the best approach is to define an extras which could be +called e.g. ``minimal`` or ``nodefault`` (the naming would be up to the package +maintainer) which would be an empty set of dependencies. If this extras is +specified, no default extras will be included, so that e.g. ``package[minimal]`` +would include only required dependencies and no extras. Note that this requires +no additional specification and is a natural consequence of the rule described +in `Overriding default extras`_. + +There are however valid use cases where package maintainers may not want to +provide this. For example, in the case of the multiple possible frontends or +backends, it may be that the package would not be functional without any of the +options. To take a specific example, a package may need either PyQt or PySide to +be installed but will not work if none are provided, so a package maintainer may +therefore not want to provide an option to install the package without any +extras. + +Backward Compatibility +====================== + +All package specification cases valid under :pep:`508` will remain valid. +Therefore, this proposal is fully backward-compatible with existing :pep:`508` +usage. + +Once packages start defining default extras, those defaults will only be honored +with recent versions of packaging tools which implement this PEP, but those +packages will remain fully backward-compatible with older packaging tools - with +the only difference that the default extras will not be installed automatically +for older packaging tools. + +The only conceptual backward-compatibility issue to consider is the fact that +this PEP changes extras to no longer be strictly additive, in that specifying +an extra such as ``minimal`` could result in fewer packages being installed. + +Security Implications +===================== + +There are no known security implications for this PEP. + +How to teach this +================= + +The rule above regarding only installing default extras when no extras +are explicitly specified, combined with the introduction of the +``Default-Extra:`` keyword and ``default-optional-dependencies`` metadata key +allows us to address several different use cases. Below we take a look at the +two specific use cases raised in the `Motivation`_ section and how package +maintainers should be taught to address these. + +Recommended dependencies and minimal installations +-------------------------------------------------- + +First, we consider the case of packages that want recommended +but not strictly required dependencies installed by default, while also +providing a way to only install the required dependencies. + +In order to do this, a package maintainer would define an extras called +``recommended`` containing the recommended but not required dependencies, and +would choose to have this be included as a default extras: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "recommended" + ] + + [project.optional-dependencies] + recommended = [ + "package1", + "package2" + ] + +In this specific case, a package maintainer may want to allow users to also +install the package without the recommended dependencies, in which case they +could define an empty extras: + +.. code-block:: toml + + [project.optional-dependencies] + minimal = [] + recommended = [ + "package1", + "package2" + ] + +This would then allow users to install ``package[minimal]`` which, since +there would be an extra explicitly specified, would mean the default extras +does not get installed, and since the ``minimal`` extras is empty, no +additional dependencies would be installed. + +Maintainers would have the choice as to whether to offer the capability to do a +minimal installation or not - in some cases, such as highlighted in the next +section, this might not be desirable. + +Packages requiring at least one backend or frontend +--------------------------------------------------- + +As described in `Motivation`_, some packages may support multiple backends +and/or frontends, and in some cases it may be desirable to ensure that there +is always at least one backend or frontend package installed, as the package +would be unusable otherwise. Concrete examples of this might include a GUI +application that needs a GUI library to be present to be usable but is able +to support different ones, or a package that can rely on different computational +backends but needs at least one to be installed. + +In this case, package maintainers could make the choice to define an extra +for each backend or frontend, and provide a default, e.g.: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "backend1" + ] + + [project.optional-dependencies] + backend1 = [ + "package1", + "package2" + ] + backend2 = [ + "package3" + ] + +Unlike the previous example however, maintainers would not necessarily provide a +way to do an installation without any extras since it might leave the package in +an unusable state. + +If packages can support e.g. multiple backends at the same time, and some of +the backends should always be installed, then the dependencies for these should be given +as required dependencies rather than using the default extras mechanism. + +Supporting minimal installations while not always removing default extras +------------------------------------------------------------------------- + +An additional case we consider here is where a package maintainer wants to support +minimal installations without any extras, but also wants to support having users +specify additional extras without removing the default one. Essentially, they +would want: + +* ``package[minimal]`` to give an installation without any extras +* ``package`` to install recommended dependencies (in a ``recommended`` extras) +* ``package[additional]`` to install both recommended and additional dependencies (in an ``additional`` extras) + +This could be achieved with e.g: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "recommended" + ] + + [project.optional-dependencies] + minimal = [] + recommended = [ + "package1", + "package2" + ] + additional = [ + "package[recommended]", + "package3" + ] + +The ability for a package to reference itself in the extras is supported by +existing Python packaging tools. + +Teaching package authors +------------------------ + +Packages making use of any of the approaches above should ensure that they +properly document the options available to users in terms of installation. + +Reference Implementation +======================== + +The following branch contains a modified version of the `flit +`_ package which recognizes the +``default-optional-dependencies`` key in ``pyproject.toml`` and adds the +relevant ``Default-Extra:`` metadata to the build package: + +https://github.com/astrofrog/flit/tree/default-extras-pep + +In addition, the following branch contains a modified version of `pip +`_ which implements the correct logic for +installing extras when default extras are present: + +https://github.com/astrofrog/pip/tree/default-extras-pep + +Finally, the following repository makes use of the above two branches to +test out different scenarios and use cases described in this PEP: + +https://github.com/astrofrog/default-extras-pep-examples + +The implementations above are proof-of-concepts at this time, do not include any +tests or documentation in the respective packages, and the existing changes have +not yet been reviewed by the relevant maintainers. Nevertheless, they are +functional enough to allow for interested maintainers to try these out. + +Rejected Ideas +============== + +Syntax for unselecting extras +----------------------------- + +One of the main competing approaches was as follows: instead of having defaults +be unselected if any extras were explicitly provided, default extras would need +to be explicitly unselected. + +In this picture, a new syntax for unselecting extras would be introduced as an +extension of the mini-language defined in :pep:`508`. If a package defined +default extras, users could opt out of these defaults by using a minus sign +(``-``) before the extra name. The proposed syntax update would have been as follows:: + + extras_list = (-)?identifier (wsp* ',' wsp* (-)?identifier)* + +Valid examples of this new syntax would have included, e.g.: + +* ``package[-recommended]`` +* ``package[-backend1, backend2]`` +* ``package[pdf, -svg]`` + +However, there are two main issues with this approach: + +* One would need to define a number of rules for how to interpret corner cases + such as if an extras and its negated version were both present in the same + dependency specification (e.g. ``package[pdf, -pdf]``) or if a dependency + tree included both ``package[pdf]`` and ``package[-pdf]``, and the rules would + not be intuitive to users. + +* More critically, this would introduce new syntax into dependency specification, + which means that if any package defined a dependency using the new syntax, it + and any other package depending on it would no longer be installable by existing + packaging tools, so this would be a major backward compatibility break. + +For these reasons, this alternative was not included in the final proposal. + +Adding a special entry in ``extras_require`` +-------------------------------------------- + +A potential solution that has been explored as an alternative to introducing the +new ``Default-Extra`` metadata field would be to make use of an extra with a +'special' name. + +One example would be to use an empty string:: + + Provides-Extra: + Requires-Dist: numpy ; extra == '' + +The idea would be that dependencies installed as part of the 'empty' extras +would only get installed if another extra was not specified. An implementation +of this was proposed in https://github.com/pypa/setuptools/pull/1503, but it +was found that there would be no way to make this work without breaking +compatibility with existing usage. For example, packages using setuptools via +a ``setup.py`` file can do:: + + setup( + ... + extras_require={'': ['package_a']}, + ) + + +which is valid and equivalent to having ``package_a`` being defined in +``install_requires``, so changing the meaning of the empty string requires would +break compatibility. + +In addition, no other string (such as ``'default'``) can be used as a special +string since all strings that would be a backward-compatible valid extras name +may already be used in existing packages. + +There have been suggestions of using the special ``None`` Python variable, but +again this is not possible, because even though one can use ``None`` in a ``setup.py`` file, +this is not possible in declarative files such as ``setup.cfg`` or +``pyproject.toml``, and furthermore ultimately extras names have to be converted +to strings in the package metadata. Having:: + + Provides-Extra: None + +would be indistinguishable from the string 'None' which may already be used as +an extras name in a Python package. If we were to modify the core metadata +syntax to allow non-string 'special' extras names, then we would be back to +modifying the core metadata specification, which is no better than +introducing ``Default-Extra``. + +Relying on tooling to deselect any default extras +------------------------------------------------- + +Another option to unselect extras would be to implement this at the +level of packaging tools. For instance, pip could include an option such as:: + + pip install package --no-default-extras + +This option could apply to all or specific packages, similar to +the ``--no-binary`` option, e.g.,:: + + pip install package --no-default-extras :all: + +The advantage of this approach is that tools supporting default extras could +also support unselecting them. This approach would be similar to the ``--no-install-recommends`` +option for the ``apt`` tool. + +However, this solution is not ideal because it would not allow packages to +specify themselves that they do not need some of the default extras of a +dependency. It would also carry risks for users who might disable all default +extras in a big dependency tree, potentially breaking packages in the tree that +rely on default extras at any point. Nevertheless, this PEP does not disallow +this approach and it is up to the maintainers of different packaging tools to +decide if they want to support this kind of option. + +``package[]`` disables default extras +------------------------------------- + +Another way to specify not to install any extras, including default extras, would +be to use ``package[]``. However, this would break the current assumption in packaging tools that +``package[]`` is equivalent to ``package``, and may also result +in developers overusing ``[]`` by default even when it is not needed. As +highlighted in `How to teach this`_, there may also be cases where package +maintainers do not actually want to support an installation without any extras, +for example in cases where at least one backend or frontend should be installed. + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 94551b07f0c9c69b75c111211dfd689931d99474 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Mon, 13 Jan 2025 14:55:39 +0000 Subject: [PATCH 02/15] Added entry to CODEOWNERS --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index fbacbd2388a..b8999733bac 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -653,6 +653,7 @@ peps/pep-0773.rst @zooba peps/pep-0777.rst @warsaw # ... peps/pep-0789.rst @njsmith +peps/pep-0790.rst @astrofrog @DEKHTIARJonathan # ... peps/pep-0801.rst @warsaw # ... From f9a4b44b44503e82bfaff68e31212294c8e6a50e Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Mon, 13 Jan 2025 14:58:12 +0000 Subject: [PATCH 03/15] Assign preliminary PEP number --- peps/{pep-9999.rst => pep-0790.rst} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename peps/{pep-9999.rst => pep-0790.rst} (99%) diff --git a/peps/pep-9999.rst b/peps/pep-0790.rst similarity index 99% rename from peps/pep-9999.rst rename to peps/pep-0790.rst index 32e2ec1b82f..714e1e788ee 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-0790.rst @@ -1,4 +1,4 @@ -PEP: 9999 +PEP: 790 Title: Default Extras for Python Software Packages Author: Thomas Robitaille , Jonathan Dekhtiar Sponsor: Pradyun Gedam From 7a2a97fd76467f744835ffa8af0b26515e37bad7 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Mon, 13 Jan 2025 15:06:30 +0000 Subject: [PATCH 04/15] Added sponsor to CODEOWNERS --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index b8999733bac..97d35aeb9b0 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -653,7 +653,7 @@ peps/pep-0773.rst @zooba peps/pep-0777.rst @warsaw # ... peps/pep-0789.rst @njsmith -peps/pep-0790.rst @astrofrog @DEKHTIARJonathan +peps/pep-0790.rst @astrofrog @DEKHTIARJonathan @pradyunsg # ... peps/pep-0801.rst @warsaw # ... From f4c14699089a28af4d05138933450f70560111d3 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Mon, 13 Jan 2025 15:09:26 +0000 Subject: [PATCH 05/15] Update PEP numbering --- peps/{pep-0790.rst => pep-0771.rst} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename peps/{pep-0790.rst => pep-0771.rst} (99%) diff --git a/peps/pep-0790.rst b/peps/pep-0771.rst similarity index 99% rename from peps/pep-0790.rst rename to peps/pep-0771.rst index 714e1e788ee..34678350b79 100644 --- a/peps/pep-0790.rst +++ b/peps/pep-0771.rst @@ -1,4 +1,4 @@ -PEP: 790 +PEP: 771 Title: Default Extras for Python Software Packages Author: Thomas Robitaille , Jonathan Dekhtiar Sponsor: Pradyun Gedam From bba7c553dee80a60fc2583eeef48e92c40351a24 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Mon, 13 Jan 2025 15:10:16 +0000 Subject: [PATCH 06/15] Update .github/CODEOWNERS Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 97d35aeb9b0..95fcb482b9a 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -653,7 +653,7 @@ peps/pep-0773.rst @zooba peps/pep-0777.rst @warsaw # ... peps/pep-0789.rst @njsmith -peps/pep-0790.rst @astrofrog @DEKHTIARJonathan @pradyunsg +peps/pep-0771.rst @pradyunsg # ... peps/pep-0801.rst @warsaw # ... From 2e259a30073a59cb073fa4187f8e1d109a52bb89 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Tue, 14 Jan 2025 15:12:49 +0000 Subject: [PATCH 07/15] Adressed comments by @warsaw --- peps/pep-0771.rst | 44 ++++++++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 34678350b79..89696ded847 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -2,10 +2,11 @@ PEP: 771 Title: Default Extras for Python Software Packages Author: Thomas Robitaille , Jonathan Dekhtiar Sponsor: Pradyun Gedam +Discussions-To: https://discuss.python.org (TBD) Status: Draft Type: Standards Track Topic: Packaging -Created: 24-Sep-2024 +Created: 13-Jan-2025 Abstract ======== @@ -74,16 +75,16 @@ Rationale ========= A number of possible solutions have been extensively and vigorously discussed by -the community for several years, including in `this DPO thread +the community for several years, including in `this DPO threadq `__ as well as in numerous issues and pull requests. The solution that will be -presented below may not be everyone's preferred approach from a philosophical -point of view, however it appears to be the best compromise as it is the only -solution to a very real problem that would not break backward-compatibility of -packaging infrastructure in any significant way. It is an opt-in solution which -means that package maintainers can choose whether or not to use it, and it is -flexible enough to accommodate both of the major use cases described in -`Motivation`_. +presented below: + +* does not break backward-compatibility of existing packaging infrastructure +* is an opt-in solution which means that package maintainers can choose whether or not to use it +* is flexible enough to accommodate both of the major use cases described in `Motivation`_. + +It is the only solution out of all those discussed that meets all three criteria. Specification ============= @@ -100,7 +101,7 @@ extras explicitly:: Default-Extra: recommended If multiple default extras are needed, one ``Default-Extra:`` entry -should be provided for each one:: +must be provided for each one:: Default-Extra: backend1 Default-Extra: backend2 @@ -117,7 +118,7 @@ originally defined in :pep:`621` and now defined in the `PyPA specifications * `TOML `_ type: Array of strings * Corresponding core metadata field: ``Default-Extra`` -Each string in ``default-optional-dependencies`` should be the name of an extra +Each string in ``default-optional-dependencies`` must be the name of an extra defined in `optional-dependencies `_, and each extra in this array will be converted to a matching ``Default-Extra`` @@ -161,10 +162,13 @@ the ``extra1`` dependency would be included. If the user instead uses:: then the ``extra1`` extras would not be installed. -If the same package is specified multiple times in a dependency tree, then if -any of these were specified without any explicit extras, the default extras -should be installed even if other occurrences of the package in the dependency -tree have explicitly specified extras. +If the same package is specified multiple times in an installation command or +dependency tree, the default extras must be installed if any of the instances of +the package are specified without extras. For instance:: + + pip install package package[extra2] + +should install the default extras. Note that ``package[]`` would continue to be equivalent to ``package`` and would not be provided as a way to install without default extras (see the `Rejected @@ -202,7 +206,7 @@ Once packages start defining default extras, those defaults will only be honored with recent versions of packaging tools which implement this PEP, but those packages will remain fully backward-compatible with older packaging tools - with the only difference that the default extras will not be installed automatically -for older packaging tools. +when older packaging tools are used. The only conceptual backward-compatibility issue to consider is the fact that this PEP changes extras to no longer be strictly additive, in that specifying @@ -304,7 +308,7 @@ way to do an installation without any extras since it might leave the package in an unusable state. If packages can support e.g. multiple backends at the same time, and some of -the backends should always be installed, then the dependencies for these should be given +the backends should always be installed, then the dependencies for these must be given as required dependencies rather than using the default extras mechanism. Supporting minimal installations while not always removing default extras @@ -345,7 +349,7 @@ existing Python packaging tools. Teaching package authors ------------------------ -Packages making use of any of the approaches above should ensure that they +Packages making use of any of the approaches above must ensure that they properly document the options available to users in terms of installation. Reference Implementation @@ -367,7 +371,7 @@ https://github.com/astrofrog/pip/tree/default-extras-pep Finally, the following repository makes use of the above two branches to test out different scenarios and use cases described in this PEP: -https://github.com/astrofrog/default-extras-pep-examples +https://github.com/astrofrog/pep-771-demo The implementations above are proof-of-concepts at this time, do not include any tests or documentation in the respective packages, and the existing changes have @@ -493,7 +497,7 @@ be to use ``package[]``. However, this would break the current assumption in pac in developers overusing ``[]`` by default even when it is not needed. As highlighted in `How to teach this`_, there may also be cases where package maintainers do not actually want to support an installation without any extras, -for example in cases where at least one backend or frontend should be installed. +for example in cases where at least one backend or frontend must be installed. Copyright ========= From b00018c55a8400c87761348b029b176c976018bb Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Tue, 14 Jan 2025 15:26:45 +0000 Subject: [PATCH 08/15] Fix discussions-to entry --- peps/pep-0771.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 89696ded847..d627be12e35 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -2,7 +2,7 @@ PEP: 771 Title: Default Extras for Python Software Packages Author: Thomas Robitaille , Jonathan Dekhtiar Sponsor: Pradyun Gedam -Discussions-To: https://discuss.python.org (TBD) +Discussions-To: DPO Status: Draft Type: Standards Track Topic: Packaging From 84ad8cf3e5560f58cef8c5707b52d8c3035a79a1 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Tue, 14 Jan 2025 15:34:05 +0000 Subject: [PATCH 09/15] Remove Discussions-To --- peps/pep-0771.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index d627be12e35..5b1efc24aae 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -2,7 +2,6 @@ PEP: 771 Title: Default Extras for Python Software Packages Author: Thomas Robitaille , Jonathan Dekhtiar Sponsor: Pradyun Gedam -Discussions-To: DPO Status: Draft Type: Standards Track Topic: Packaging From fd8fae857b92f0c40e8819350fb0a22b03bc5414 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 15 Jan 2025 09:31:06 +0000 Subject: [PATCH 10/15] Apply suggestions from code review by @hugovk Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0771.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 5b1efc24aae..3c3223fdd53 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -74,7 +74,7 @@ Rationale ========= A number of possible solutions have been extensively and vigorously discussed by -the community for several years, including in `this DPO threadq +the community for several years, including in `this DPO thread `__ as well as in numerous issues and pull requests. The solution that will be presented below: @@ -128,7 +128,7 @@ produce the example ``Default-Extra`` entries presented in the previous section [project] default-optional-dependencies = [ - "recommended" + "recommended", ] and: @@ -150,7 +150,7 @@ If extras are explicitly given in a dependency specification, the default extras are not installed. Otherwise, the default extras are used. For example, if a package -defines a ``extra1`` default extra as well as a non-default ``extra2`` +defines an ``extra1`` default extra as well as a non-default ``extra2`` extra, then if a user was to install the package with:: pip install package @@ -354,7 +354,7 @@ properly document the options available to users in terms of installation. Reference Implementation ======================== -The following branch contains a modified version of the `flit +The following branch contains a modified version of the `Flit `_ package which recognizes the ``default-optional-dependencies`` key in ``pyproject.toml`` and adds the relevant ``Default-Extra:`` metadata to the build package: @@ -372,7 +372,7 @@ test out different scenarios and use cases described in this PEP: https://github.com/astrofrog/pep-771-demo -The implementations above are proof-of-concepts at this time, do not include any +The implementations above are proofs-of-concept at this time, do not include any tests or documentation in the respective packages, and the existing changes have not yet been reviewed by the relevant maintainers. Nevertheless, they are functional enough to allow for interested maintainers to try these out. @@ -380,7 +380,7 @@ functional enough to allow for interested maintainers to try these out. Rejected Ideas ============== -Syntax for unselecting extras +Syntax for deselecting extras ----------------------------- One of the main competing approaches was as follows: instead of having defaults @@ -431,7 +431,7 @@ The idea would be that dependencies installed as part of the 'empty' extras would only get installed if another extra was not specified. An implementation of this was proposed in https://github.com/pypa/setuptools/pull/1503, but it was found that there would be no way to make this work without breaking -compatibility with existing usage. For example, packages using setuptools via +compatibility with existing usage. For example, packages using Setuptools via a ``setup.py`` file can do:: setup( @@ -441,7 +441,7 @@ a ``setup.py`` file can do:: which is valid and equivalent to having ``package_a`` being defined in -``install_requires``, so changing the meaning of the empty string requires would +``install_requires``, so changing the meaning of the empty string would break compatibility. In addition, no other string (such as ``'default'``) can be used as a special From d9e9e704c175344260c21af016fd409e8fdd3e16 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 15 Jan 2025 09:38:04 +0000 Subject: [PATCH 11/15] Implemented more comments by @hugovk --- .github/CODEOWNERS | 2 +- peps/pep-0771.rst | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 95fcb482b9a..6fa815b1920 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -648,12 +648,12 @@ peps/pep-0767.rst @carljm peps/pep-0768.rst @pablogsal peps/pep-0769.rst @facundobatista peps/pep-0770.rst @sethmlarson @brettcannon +peps/pep-0771.rst @pradyunsg peps/pep-0773.rst @zooba # ... peps/pep-0777.rst @warsaw # ... peps/pep-0789.rst @njsmith -peps/pep-0771.rst @pradyunsg # ... peps/pep-0801.rst @warsaw # ... diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 3c3223fdd53..43fe09ca6d7 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -178,7 +178,7 @@ Installing without default extras In some cases, package maintainers may want to facilitate installing packages without any default extras. In this case, as will be shown in more detail in -`How to teach this`_, the best approach is to define an extras which could be +`How to teach this`_, the best approach is to define an extra which could be called e.g. ``minimal`` or ``nodefault`` (the naming would be up to the package maintainer) which would be an empty set of dependencies. If this extras is specified, no default extras will be included, so that e.g. ``package[minimal]`` @@ -233,9 +233,9 @@ First, we consider the case of packages that want recommended but not strictly required dependencies installed by default, while also providing a way to only install the required dependencies. -In order to do this, a package maintainer would define an extras called +In order to do this, a package maintainer would define an extra called ``recommended`` containing the recommended but not required dependencies, and -would choose to have this be included as a default extras: +would choose to have this be included as a default extra: .. code-block:: toml @@ -252,7 +252,7 @@ would choose to have this be included as a default extras: In this specific case, a package maintainer may want to allow users to also install the package without the recommended dependencies, in which case they -could define an empty extras: +could define an empty extra: .. code-block:: toml @@ -264,8 +264,8 @@ could define an empty extras: ] This would then allow users to install ``package[minimal]`` which, since -there would be an extra explicitly specified, would mean the default extras -does not get installed, and since the ``minimal`` extras is empty, no +there would be an extra explicitly specified, would mean the default extra +does not get installed, and since the ``minimal`` extra is empty, no additional dependencies would be installed. Maintainers would have the choice as to whether to offer the capability to do a @@ -403,7 +403,7 @@ Valid examples of this new syntax would have included, e.g.: However, there are two main issues with this approach: * One would need to define a number of rules for how to interpret corner cases - such as if an extras and its negated version were both present in the same + such as if an extra and its negated version were both present in the same dependency specification (e.g. ``package[pdf, -pdf]``) or if a dependency tree included both ``package[pdf]`` and ``package[-pdf]``, and the rules would not be intuitive to users. @@ -457,7 +457,7 @@ to strings in the package metadata. Having:: Provides-Extra: None would be indistinguishable from the string 'None' which may already be used as -an extras name in a Python package. If we were to modify the core metadata +an extra name in a Python package. If we were to modify the core metadata syntax to allow non-string 'special' extras names, then we would be back to modifying the core metadata specification, which is no better than introducing ``Default-Extra``. From 4e0c2a65ba0788ed5a7364405ccd43eb64c0889d Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 15 Jan 2025 09:54:15 +0000 Subject: [PATCH 12/15] Update links in reference implementation section --- peps/pep-0771.rst | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 43fe09ca6d7..4e203d52241 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -354,26 +354,25 @@ properly document the options available to users in terms of installation. Reference Implementation ======================== -The following branch contains a modified version of the `Flit -`_ package which recognizes the -``default-optional-dependencies`` key in ``pyproject.toml`` and adds the -relevant ``Default-Extra:`` metadata to the build package: +The following repository contains a fully functional demo package +that makes use of default extras: -https://github.com/astrofrog/flit/tree/default-extras-pep +https://github.com/wheel-next/pep_771 -In addition, the following branch contains a modified version of `pip -`_ which implements the correct logic for -installing extras when default extras are present: +This makes use of modified branches of several packages, and the following +links are to these branches: -https://github.com/astrofrog/pip/tree/default-extras-pep +* `Setuptools `_ +* `pip `_ +* `importlib_metadata `_ -Finally, the following repository makes use of the above two branches to -test out different scenarios and use cases described in this PEP: +In addition, `this branch `_ +contains a modified version of the `Flit +`_ package. -https://github.com/astrofrog/pep-771-demo -The implementations above are proofs-of-concept at this time, do not include any -tests or documentation in the respective packages, and the existing changes have + +The implementations above are proofs-of-concept at this time and the existing changes have not yet been reviewed by the relevant maintainers. Nevertheless, they are functional enough to allow for interested maintainers to try these out. From f33b436f490f291f4719fe37ad3d716565b16496 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 15 Jan 2025 16:53:34 +0000 Subject: [PATCH 13/15] Fix DPO link Co-authored-by: Dustin Ingram --- peps/pep-0771.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 4e203d52241..cacc56d590a 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -20,7 +20,7 @@ Motivation ========== Various use cases for default extras and possible solutions in this PEP were discussed -extensively on `this DPO thread `__. +extensively on `this DPO thread `__. These fall into two broad cases that that provide the motivation for the present PEP. From 077ec36e96c0bf7b6e4989761260574b5dba1979 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 15 Jan 2025 16:55:47 +0000 Subject: [PATCH 14/15] Fix typos Co-authored-by: Paul Moore --- peps/pep-0771.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index cacc56d590a..27e14fddda7 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -151,7 +151,7 @@ extras are not installed. Otherwise, the default extras are used. For example, if a package defines an ``extra1`` default extra as well as a non-default ``extra2`` -extra, then if a user was to install the package with:: +extra, then if a user were to install the package with:: pip install package @@ -159,7 +159,7 @@ the ``extra1`` dependency would be included. If the user instead uses:: pip install package[extra2] -then the ``extra1`` extras would not be installed. +then the ``extra1`` extra would not be installed. If the same package is specified multiple times in an installation command or dependency tree, the default extras must be installed if any of the instances of @@ -180,7 +180,7 @@ In some cases, package maintainers may want to facilitate installing packages without any default extras. In this case, as will be shown in more detail in `How to teach this`_, the best approach is to define an extra which could be called e.g. ``minimal`` or ``nodefault`` (the naming would be up to the package -maintainer) which would be an empty set of dependencies. If this extras is +maintainer) which would be an empty set of dependencies. If this extra is specified, no default extras will be included, so that e.g. ``package[minimal]`` would include only required dependencies and no extras. Note that this requires no additional specification and is a natural consequence of the rule described From 9927fc76810e0c5178669017168144063b3053c7 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Thu, 16 Jan 2025 11:22:44 +0000 Subject: [PATCH 15/15] Use 'multiple-use' terminology for Default-Extra metadata field, and mention that Metadata-Version will need to be bumped --- peps/pep-0771.rst | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 27e14fddda7..ee5e42413b9 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -91,27 +91,32 @@ Specification ``Default-Extra`` Metadata Field --------------------------------- -A new metadata field, ``Default-Extra``, will be added to the `core package +A new multiple-use metadata field, ``Default-Extra``, will be added to the `core package metadata `_. -This field allows package maintainers to define an extra that is -automatically included when a user installs the package without specifying any -extras explicitly:: +For this field, each entry must be a string specifying an extra that will be +automatically included when the package is installed without any extras specified explicitly. - Default-Extra: recommended +Only entries already specified in a `Provides-Extra +`_ +entry can be used in a ``Default-Extra`` entry. -If multiple default extras are needed, one ``Default-Extra:`` entry -must be provided for each one:: +Examples:: + Default-Extra: recommended Default-Extra: backend1 Default-Extra: backend2 Default-Extra: backend3 +Since this introduces a new field in the core package metadata, this will require +`Metadata-Version `_ +to be bumped to the next minor version (2.5 at the time of writing). + New key in ``[project]`` metadata table --------------------------------------- A new key will be added to the ``[project]`` table in project metadata as originally defined in :pep:`621` and now defined in the `PyPA specifications -`_. This key will be named +`_. This key will be named ``default-optional-dependencies`` with the following description: * `TOML `_ type: Array of strings