diff --git a/peps/pep-0810.rst b/peps/pep-0810.rst index 120511ba209..74082a734c5 100644 --- a/peps/pep-0810.rst +++ b/peps/pep-0810.rst @@ -335,6 +335,15 @@ An import remains lazy only if the filter function returns ``True``. If no lazy import filter is set, all *potentially lazy* imports are lazy. +Lazy objects +------------ + +Lazy modules, as well as names lazy imported from modules, are represented +by :class:`!types.LazyImportType` instances, which are resolved to the real +object (reified) before they can be used. This reification is usually done +automatically (see below), but can also be done by calling the lazy object's +``get`` method. + Lazy import mechanism --------------------- @@ -342,12 +351,13 @@ When an import is lazy, ``__lazy_import__`` is called instead of ``__import__``. ``__lazy_import__`` has the same function signature as ``__import__``. It adds the module name to ``sys.lazy_modules``, a set of fully-qualified module names which have been lazily imported at some point -(primarily for diagnostics and introspection), and returns a "lazy module -object." +(primarily for diagnostics and introspection), and returns a +:class:`!types.LazyImportType`` object for the module. The implementation of ``from ... import`` (the ``IMPORT_FROM`` bytecode implementation) checks if the module it's fetching from is a lazy module -object, and if so, returns a lazy object for each name instead. +object, and if so, returns a :class:`!types.LazyImportType` for each name +instead. The end result of this process is that lazy imports (regardless of how they are enabled) result in lazy objects being assigned to global variables. @@ -356,7 +366,7 @@ Lazy module objects do not appear in ``sys.modules``, they're just listed in the ``sys.lazy_modules`` set. Under normal operation lazy objects should only end up stored in global variables, and the common ways to access those variables (regular variable access, module attributes) will resolve lazy -imports ("reify") and replace them when they're accessed. +imports (reify) and replace them when they're accessed. It is still possible to expose lazy objects through other means, like debuggers. This is not considered a problem. @@ -364,26 +374,28 @@ debuggers. This is not considered a problem. Reification ----------- -When a lazy object is first used, it needs to be reified. This means resolving -the import at that point in the program and replacing the lazy object with the -concrete one. Reification imports the module in the same way as it would have -been if it had been imported eagerly. Notably, reification still calls -``__import__`` to resolve the import, which uses the state of the import system -(e.g. ``sys.path``, ``sys.meta_path``, ``sys.path_hooks`` and ``__import__``) -at **reification** time, **not** the state when the ``lazy import`` statement -was evaluated. - -When the module is first reified, it's removed from ``sys.lazy_modules`` (even -if there are still other unreified lazy references to it). When a package is +When a lazy object is used, it needs to be reified. This means resolving the +import at that point in the program and replacing the lazy object with the +concrete one. Reification imports the module at that point in the program. +Notably, reification still calls ``__import__`` to resolve the import, which +uses the state of the import system (e.g. ``sys.path``, ``sys.meta_path``, +``sys.path_hooks`` and ``__import__``) at **reification** time, **not** the +state when the ``lazy import`` statement was evaluated. + +When the module is reified, it's removed from ``sys.lazy_modules`` (even if +there are still other unreified lazy references to it). When a package is reified and submodules in the package were also previously lazily imported, those submodules are *not* automatically reified but they *are* added to the -reified package's globals (unless the package already assigned something else -to the name of the submodule). +reified package's globals (unless the package already assigned something +else to the name of the submodule). -If reification fails (e.g., due to an ``ImportError``), the exception is -enhanced with chaining to show both where the lazy import was defined and -where it was first accessed (even though it propagates from the code that -triggered reification). This provides clear debugging information: +If reification fails (e.g., due to an ``ImportError``), the lazy object is +*not* reified or replaced. Subsequent uses of the lazy object will re-try +the reification. Exceptions that happen during reification are raised as +normal, but the exception is enhanced with chaining to show both where the +lazy import was defined and where it was accessed (even though it propagates +from the code that triggered reification). This provides clear debugging +information: .. code-block:: python @@ -469,7 +481,7 @@ Example using ``globals()``: g = globals() print('json' in sys.modules) # False - still lazy - print(type(g['json'])) # + print(type(g['json'])) # # Explicitly reify using the get() method resolved = g['json'].get() @@ -704,7 +716,7 @@ These changes are limited to bindings explicitly made lazy: * **Error timing.** Exceptions that would have occurred during an eager import (for example ``ImportError`` or ``AttributeError`` for a missing member) now - occur at the first *use* of the lazy name. + occur at the *use* of the lazy name. .. code-block:: python @@ -714,7 +726,7 @@ These changes are limited to bindings explicitly made lazy: # With lazy import - error deferred lazy import broken_module print("Import succeeded") - broken_module.foo() # ImportError raised here on first use + broken_module.foo() # ImportError raised here on use * **Side-effect timing.** Import-time side effects in lazily imported modules occur at first use of the binding, not at module import time. @@ -727,16 +739,17 @@ These changes are limited to bindings explicitly made lazy: when it is first used. * **Proxy visibility.** Before first use, the bound name refers to a lazy proxy. Indirect introspection that touches the value may observe a proxy - lazy object representation. After first use, the name is rebound to the real - object and becomes indistinguishable from an eager import. + lazy object representation. After first use (provied the module was + imported succesfully), the name is rebound to the real object and becomes + indistinguishable from an eager import. Thread-safety and reification ----------------------------- -First use of a lazy binding follows the existing import-lock discipline. -Exactly one thread performs the import and **atomically rebinds** the -importing module's global to the resolved object. Concurrent readers -thereafter observe the real object. +Reification follows the existing import-lock discipline. Exactly one thread +performs the import and **atomically rebinds** the importing module's global +to the resolved object. Concurrent readers thereafter observe the real +object. Lazy imports are thread-safe and have no special considerations for free-threading. A module that would normally be imported in the main thread @@ -758,11 +771,11 @@ code that doesn't. Runtime performance ~~~~~~~~~~~~~~~~~~~ -After reification (first use), lazy imports have **zero overhead**. The -adaptive interpreter specializes the bytecode (typically after 2-3 accesses), -eliminating any checks. For example, ``LOAD_GLOBAL`` becomes -``LOAD_GLOBAL_MODULE``, which directly accesses the module identically to -normal imports. +After reification (provided the import was succesful), lazy imports have +**zero overhead**. The adaptive interpreter specializes the bytecode +(typically after 2-3 accesses), eliminating any checks. For example, +``LOAD_GLOBAL`` becomes ``LOAD_GLOBAL_MODULE``, which directly accesses the +module identically to normal imports. The `pyperformance suite`_ confirms the implementation is performance-neutral. @@ -1033,6 +1046,9 @@ it was first used: 1/0 ZeroDivisionError: division by zero +Exceptions during reification prevent the replacement of the lazy object, +and subsequent uses of the lazy object will retry the whole reification. + How do lazy imports affect modules with import-time side effects? ----------------------------------------------------------------- @@ -1118,8 +1134,8 @@ What's the performance overhead of lazy imports? The overhead is minimal: -- Zero overhead after first use thanks to the adaptive interpreter optimizing - the slow path away. +- Zero overhead after first use (provided the import doesn't fail) thanks to + the adaptive interpreter optimizing the slow path away. - Small one-time cost to create the proxy object. - Reification (first use) has the same cost as a regular import. - No ongoing performance penalty. @@ -1159,7 +1175,7 @@ error. If lazy imports are globally enabled, star imports will still be eager. How do lazy imports interact with import hooks and custom loaders? ------------------------------------------------------------------ -Import hooks and loaders work normally. When a lazy object is first used, +Import hooks and loaders work normally. When a lazy object is used, the standard import protocol runs, including any custom hooks or loaders that were in place at reification time. @@ -1191,7 +1207,7 @@ reifies all lazy imports in that module first. This design ensures: lazy import json g = globals() - print(type(g['json'])) # - your problem + print(type(g['json'])) # - your problem # From external code: import sys @@ -1390,11 +1406,11 @@ The best practice is still to avoid circular imports in your code design. Will lazy imports affect the performance of my hot paths? --------------------------------------------------------- -After first use, lazy imports have **zero overhead** thanks to the adaptive -interpreter. The interpreter specializes the bytecode (e.g., ``LOAD_GLOBAL`` -becomes ``LOAD_GLOBAL_MODULE``) which eliminates the lazy check on subsequent -accesses. This means once a lazy import is reified, accessing it is just as -fast as a normal import. +After first use (provided the import succeed), lazy imports have **zero +overhead** thanks to the adaptive interpreter. The interpreter specializes +the bytecode (e.g., ``LOAD_GLOBAL`` becomes ``LOAD_GLOBAL_MODULE``) which +eliminates the lazy check on subsequent accesses. This means once a lazy +import is reified, accessing it is just as fast as a normal import. .. code-block:: python