diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index fbacbd2388a..6fa815b1920 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -648,6 +648,7 @@ peps/pep-0767.rst @carljm peps/pep-0768.rst @pablogsal peps/pep-0769.rst @facundobatista peps/pep-0770.rst @sethmlarson @brettcannon +peps/pep-0771.rst @pradyunsg peps/pep-0773.rst @zooba # ... peps/pep-0777.rst @warsaw diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst new file mode 100644 index 00000000000..ee5e42413b9 --- /dev/null +++ b/peps/pep-0771.rst @@ -0,0 +1,509 @@ +PEP: 771 +Title: Default Extras for Python Software Packages +Author: Thomas Robitaille , Jonathan Dekhtiar +Sponsor: Pradyun Gedam +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 13-Jan-2025 + +Abstract +======== + +:pep:`508` specifies a mini-language for +declaring package dependencies. One feature of this language is the ability to +specify *extras*, which are optional components of a distribution that, when +used, install additional dependencies. This PEP proposes a mechanism to allow +one or more extras to be installed by default if none are provided explicitly. + +Motivation +========== + +Various use cases for default extras and possible solutions in this PEP were discussed +extensively on `this DPO thread `__. +These fall into two broad cases that that provide the +motivation for the present PEP. + +Recommended but not required dependencies +----------------------------------------- + +Package maintainers often use extras to declare optional dependencies that +extend the functionality or performance of a package. In some cases, it can be +difficult to determine which dependencies should be required and which should be +categorized as extras. A balance must be struck between the needs of typical +users (who may prefer most features to be available 'by default') and users who +want minimal installations without large, optional dependencies. One solution +with existing Python packaging infrastructure is for package maintainers to +define an extra called, for example, ``recommended``, which +includes all non-essential but suggested dependencies. Users are then instructed to +install the package using ``package[recommended]``, while those who prefer more +control can use ``package``. However, in practice, many users are unaware +of the ``[recommended]`` syntax, placing the burden on them to know this for a +typical installation. Having a way to have recommended dependencies be installed +by default while providing a way for users to request a more minimal installation +would satisfy this use case. The present PEP will describe a solution for this use case. + +Packages supporting multiple backends or frontends +-------------------------------------------------- + +Another common use case for using extras is to define different backends or +frontends and dependencies that need to be installed for each backend or +frontend. A package might need at least one backend or frontend to be installed +in order to be functional, but may be flexible on which backend or frontend this +is. Concrete examples of such frontends or backends include: + +* The Qt frontend library, which requires either PySide or PyQt to be installed +* BLAS/LAPACK, which have different possible implementations (e.g. OpenBLAS, and MKL) +* FFT libraries, which also have different implementations (e.g. ``scipy.fft`` and pyFFTW) + +With current packaging standards, maintainers have to either +require one of the backends or frontends, or require users +to always specify extras, e.g. ``package[backend]`` and therefore risk users +having an unusable installation if they only install ``package``. Having a +way to specify one or more default backend or frontend and providing a way to +override these defaults would provide a much better experience for users, and +the approach described in this PEP will allow this. + +Note that this PEP does not aim to address the issue of disallowing conflicting +or incompatible extras - for example if a package requires exactly one frontend +or backend package. There is currently no mechanism in Python packaging +infrastructure to disallow conflicting or incompatible extras to be installed, +and this PEP does not change that. + +Rationale +========= + +A number of possible solutions have been extensively and vigorously discussed by +the community for several years, including in `this DPO thread +`__ +as well as in numerous issues and pull requests. The solution that will be +presented below: + +* does not break backward-compatibility of existing packaging infrastructure +* is an opt-in solution which means that package maintainers can choose whether or not to use it +* is flexible enough to accommodate both of the major use cases described in `Motivation`_. + +It is the only solution out of all those discussed that meets all three criteria. + +Specification +============= + +``Default-Extra`` Metadata Field +--------------------------------- + +A new multiple-use metadata field, ``Default-Extra``, will be added to the `core package +metadata `_. +For this field, each entry must be a string specifying an extra that will be +automatically included when the package is installed without any extras specified explicitly. + +Only entries already specified in a `Provides-Extra +`_ +entry can be used in a ``Default-Extra`` entry. + +Examples:: + + Default-Extra: recommended + Default-Extra: backend1 + Default-Extra: backend2 + Default-Extra: backend3 + +Since this introduces a new field in the core package metadata, this will require +`Metadata-Version `_ +to be bumped to the next minor version (2.5 at the time of writing). + +New key in ``[project]`` metadata table +--------------------------------------- + +A new key will be added to the ``[project]`` table in project metadata as +originally defined in :pep:`621` and now defined in the `PyPA specifications +`_. This key will be named +``default-optional-dependencies`` with the following description: + +* `TOML `_ type: Array of strings +* Corresponding core metadata field: ``Default-Extra`` + +Each string in ``default-optional-dependencies`` must be the name of an extra +defined in `optional-dependencies +`_, +and each extra in this array will be converted to a matching ``Default-Extra`` +entry in the core package metadata. Examples of valid usage which would +produce the example ``Default-Extra`` entries presented in the previous section are: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "recommended", + ] + +and: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "backend1", + "backend2", + "backend3" + ] + + +Overriding default extras +------------------------- + +If extras are explicitly given in a dependency specification, the default +extras are not installed. Otherwise, the default extras are used. + +For example, if a package +defines an ``extra1`` default extra as well as a non-default ``extra2`` +extra, then if a user were to install the package with:: + + pip install package + +the ``extra1`` dependency would be included. If the user instead uses:: + + pip install package[extra2] + +then the ``extra1`` extra would not be installed. + +If the same package is specified multiple times in an installation command or +dependency tree, the default extras must be installed if any of the instances of +the package are specified without extras. For instance:: + + pip install package package[extra2] + +should install the default extras. + +Note that ``package[]`` would continue to be equivalent to ``package`` and would +not be provided as a way to install without default extras (see the `Rejected +Ideas`_ section for the rationale). + +Installing without default extras +--------------------------------- + +In some cases, package maintainers may want to facilitate installing packages +without any default extras. In this case, as will be shown in more detail in +`How to teach this`_, the best approach is to define an extra which could be +called e.g. ``minimal`` or ``nodefault`` (the naming would be up to the package +maintainer) which would be an empty set of dependencies. If this extra is +specified, no default extras will be included, so that e.g. ``package[minimal]`` +would include only required dependencies and no extras. Note that this requires +no additional specification and is a natural consequence of the rule described +in `Overriding default extras`_. + +There are however valid use cases where package maintainers may not want to +provide this. For example, in the case of the multiple possible frontends or +backends, it may be that the package would not be functional without any of the +options. To take a specific example, a package may need either PyQt or PySide to +be installed but will not work if none are provided, so a package maintainer may +therefore not want to provide an option to install the package without any +extras. + +Backward Compatibility +====================== + +All package specification cases valid under :pep:`508` will remain valid. +Therefore, this proposal is fully backward-compatible with existing :pep:`508` +usage. + +Once packages start defining default extras, those defaults will only be honored +with recent versions of packaging tools which implement this PEP, but those +packages will remain fully backward-compatible with older packaging tools - with +the only difference that the default extras will not be installed automatically +when older packaging tools are used. + +The only conceptual backward-compatibility issue to consider is the fact that +this PEP changes extras to no longer be strictly additive, in that specifying +an extra such as ``minimal`` could result in fewer packages being installed. + +Security Implications +===================== + +There are no known security implications for this PEP. + +How to teach this +================= + +The rule above regarding only installing default extras when no extras +are explicitly specified, combined with the introduction of the +``Default-Extra:`` keyword and ``default-optional-dependencies`` metadata key +allows us to address several different use cases. Below we take a look at the +two specific use cases raised in the `Motivation`_ section and how package +maintainers should be taught to address these. + +Recommended dependencies and minimal installations +-------------------------------------------------- + +First, we consider the case of packages that want recommended +but not strictly required dependencies installed by default, while also +providing a way to only install the required dependencies. + +In order to do this, a package maintainer would define an extra called +``recommended`` containing the recommended but not required dependencies, and +would choose to have this be included as a default extra: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "recommended" + ] + + [project.optional-dependencies] + recommended = [ + "package1", + "package2" + ] + +In this specific case, a package maintainer may want to allow users to also +install the package without the recommended dependencies, in which case they +could define an empty extra: + +.. code-block:: toml + + [project.optional-dependencies] + minimal = [] + recommended = [ + "package1", + "package2" + ] + +This would then allow users to install ``package[minimal]`` which, since +there would be an extra explicitly specified, would mean the default extra +does not get installed, and since the ``minimal`` extra is empty, no +additional dependencies would be installed. + +Maintainers would have the choice as to whether to offer the capability to do a +minimal installation or not - in some cases, such as highlighted in the next +section, this might not be desirable. + +Packages requiring at least one backend or frontend +--------------------------------------------------- + +As described in `Motivation`_, some packages may support multiple backends +and/or frontends, and in some cases it may be desirable to ensure that there +is always at least one backend or frontend package installed, as the package +would be unusable otherwise. Concrete examples of this might include a GUI +application that needs a GUI library to be present to be usable but is able +to support different ones, or a package that can rely on different computational +backends but needs at least one to be installed. + +In this case, package maintainers could make the choice to define an extra +for each backend or frontend, and provide a default, e.g.: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "backend1" + ] + + [project.optional-dependencies] + backend1 = [ + "package1", + "package2" + ] + backend2 = [ + "package3" + ] + +Unlike the previous example however, maintainers would not necessarily provide a +way to do an installation without any extras since it might leave the package in +an unusable state. + +If packages can support e.g. multiple backends at the same time, and some of +the backends should always be installed, then the dependencies for these must be given +as required dependencies rather than using the default extras mechanism. + +Supporting minimal installations while not always removing default extras +------------------------------------------------------------------------- + +An additional case we consider here is where a package maintainer wants to support +minimal installations without any extras, but also wants to support having users +specify additional extras without removing the default one. Essentially, they +would want: + +* ``package[minimal]`` to give an installation without any extras +* ``package`` to install recommended dependencies (in a ``recommended`` extras) +* ``package[additional]`` to install both recommended and additional dependencies (in an ``additional`` extras) + +This could be achieved with e.g: + +.. code-block:: toml + + [project] + default-optional-dependencies = [ + "recommended" + ] + + [project.optional-dependencies] + minimal = [] + recommended = [ + "package1", + "package2" + ] + additional = [ + "package[recommended]", + "package3" + ] + +The ability for a package to reference itself in the extras is supported by +existing Python packaging tools. + +Teaching package authors +------------------------ + +Packages making use of any of the approaches above must ensure that they +properly document the options available to users in terms of installation. + +Reference Implementation +======================== + +The following repository contains a fully functional demo package +that makes use of default extras: + +https://github.com/wheel-next/pep_771 + +This makes use of modified branches of several packages, and the following +links are to these branches: + +* `Setuptools `_ +* `pip `_ +* `importlib_metadata `_ + +In addition, `this branch `_ +contains a modified version of the `Flit +`_ package. + + + +The implementations above are proofs-of-concept at this time and the existing changes have +not yet been reviewed by the relevant maintainers. Nevertheless, they are +functional enough to allow for interested maintainers to try these out. + +Rejected Ideas +============== + +Syntax for deselecting extras +----------------------------- + +One of the main competing approaches was as follows: instead of having defaults +be unselected if any extras were explicitly provided, default extras would need +to be explicitly unselected. + +In this picture, a new syntax for unselecting extras would be introduced as an +extension of the mini-language defined in :pep:`508`. If a package defined +default extras, users could opt out of these defaults by using a minus sign +(``-``) before the extra name. The proposed syntax update would have been as follows:: + + extras_list = (-)?identifier (wsp* ',' wsp* (-)?identifier)* + +Valid examples of this new syntax would have included, e.g.: + +* ``package[-recommended]`` +* ``package[-backend1, backend2]`` +* ``package[pdf, -svg]`` + +However, there are two main issues with this approach: + +* One would need to define a number of rules for how to interpret corner cases + such as if an extra and its negated version were both present in the same + dependency specification (e.g. ``package[pdf, -pdf]``) or if a dependency + tree included both ``package[pdf]`` and ``package[-pdf]``, and the rules would + not be intuitive to users. + +* More critically, this would introduce new syntax into dependency specification, + which means that if any package defined a dependency using the new syntax, it + and any other package depending on it would no longer be installable by existing + packaging tools, so this would be a major backward compatibility break. + +For these reasons, this alternative was not included in the final proposal. + +Adding a special entry in ``extras_require`` +-------------------------------------------- + +A potential solution that has been explored as an alternative to introducing the +new ``Default-Extra`` metadata field would be to make use of an extra with a +'special' name. + +One example would be to use an empty string:: + + Provides-Extra: + Requires-Dist: numpy ; extra == '' + +The idea would be that dependencies installed as part of the 'empty' extras +would only get installed if another extra was not specified. An implementation +of this was proposed in https://github.com/pypa/setuptools/pull/1503, but it +was found that there would be no way to make this work without breaking +compatibility with existing usage. For example, packages using Setuptools via +a ``setup.py`` file can do:: + + setup( + ... + extras_require={'': ['package_a']}, + ) + + +which is valid and equivalent to having ``package_a`` being defined in +``install_requires``, so changing the meaning of the empty string would +break compatibility. + +In addition, no other string (such as ``'default'``) can be used as a special +string since all strings that would be a backward-compatible valid extras name +may already be used in existing packages. + +There have been suggestions of using the special ``None`` Python variable, but +again this is not possible, because even though one can use ``None`` in a ``setup.py`` file, +this is not possible in declarative files such as ``setup.cfg`` or +``pyproject.toml``, and furthermore ultimately extras names have to be converted +to strings in the package metadata. Having:: + + Provides-Extra: None + +would be indistinguishable from the string 'None' which may already be used as +an extra name in a Python package. If we were to modify the core metadata +syntax to allow non-string 'special' extras names, then we would be back to +modifying the core metadata specification, which is no better than +introducing ``Default-Extra``. + +Relying on tooling to deselect any default extras +------------------------------------------------- + +Another option to unselect extras would be to implement this at the +level of packaging tools. For instance, pip could include an option such as:: + + pip install package --no-default-extras + +This option could apply to all or specific packages, similar to +the ``--no-binary`` option, e.g.,:: + + pip install package --no-default-extras :all: + +The advantage of this approach is that tools supporting default extras could +also support unselecting them. This approach would be similar to the ``--no-install-recommends`` +option for the ``apt`` tool. + +However, this solution is not ideal because it would not allow packages to +specify themselves that they do not need some of the default extras of a +dependency. It would also carry risks for users who might disable all default +extras in a big dependency tree, potentially breaking packages in the tree that +rely on default extras at any point. Nevertheless, this PEP does not disallow +this approach and it is up to the maintainers of different packaging tools to +decide if they want to support this kind of option. + +``package[]`` disables default extras +------------------------------------- + +Another way to specify not to install any extras, including default extras, would +be to use ``package[]``. However, this would break the current assumption in packaging tools that +``package[]`` is equivalent to ``package``, and may also result +in developers overusing ``[]`` by default even when it is not needed. As +highlighted in `How to teach this`_, there may also be cases where package +maintainers do not actually want to support an installation without any extras, +for example in cases where at least one backend or frontend must be installed. + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.