PEP 728: Improve specification (#4166)

JelleZijlstra · web-flow · commit 484c43d0a78b · 2024-12-20T00:13:09.000Z
- Specify that extra_items and closed are also supported with the
  functional syntax.
- Rewrite the rules for `closed=True` and inheritance. I attempted
  to make `closed=True` exactly equivalent to `extra_items=Never` in
  terms of inheritance. The semantics as specified in the previous
  version of the PEP felt harder to understand and less consistent.
- Fix some incorrect comments regarding expected type checker errors.
- Clarify section on assignability with Mapping
- Add section on runtime behavior. I tried to make the intended runtime
  behavior simple to implement and understand. This makes the runtime
  simpler but may make life more complicated for tools consuming the
  metadata.
- Fix typos and trailing whitespace
diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst
@@ -48,7 +48,7 @@ Another possible use case for this is a sound way to
     class Movie(TypedDict):
         name: str
         director: str
-    
+
     class Book(TypedDict):
         name: str
         author: str
@@ -194,12 +194,12 @@ to the ``extra_items`` argument. For example::
 
     class Movie(TypedDict, extra_items=bool):
         name: str
-    
+
     a: Movie = {"name": "Blade Runner", "novel_adaptation": True}  # OK
     b: Movie = {
         "name": "Blade Runner",
         "year": 1982,  # Not OK. 'int' is not assignable to 'bool'
-    }  
+    }
 
 Here, ``extra_items=bool`` specifies that items other than ``'name'``
 have a value type of ``bool`` and are non-required.
@@ -214,12 +214,12 @@ the ``extra_items`` argument::
     def f(movie: Movie) -> None:
         reveal_type(movie["name"])              # Revealed type is 'str'
         reveal_type(movie["novel_adaptation"])  # Revealed type is 'bool'
-    
+
 ``extra_items`` is inherited through subclassing::
 
     class MovieBase(TypedDict, extra_items=int | None):
         name: str
-    
+
     class Movie(MovieBase):
         year: int
 
@@ -234,38 +234,46 @@ Here, ``'year'`` in ``a`` is an extra key defined on ``Movie`` whose value type
 is ``int``. ``'other_extra_key'`` in ``b`` is another extra key whose value type
 must be assignable to the value of ``extra_items`` defined on ``MovieBase``.
 
+``extra_items`` is also supported with the functional syntax::
+
+    Movie = TypedDict("Movie", {"name": str}, extra_items=int | None)
+
 The ``closed`` Class Parameter
 ------------------------------
 
-When ``closed=True`` is set, no extra items are allowed. This is a shorthand for
+When ``closed=True`` is set, no extra items are allowed. This is equivalent to
 ``extra_items=Never``, because there can't be a value type that is assignable to
-:class:`~typing.Never`.
+:class:`~typing.Never`. It is a runtime error to use the ``closed`` and
+``extra_items`` parameters in the same TypedDict definition.
 
 Similar to ``total``, only a literal ``True`` or ``False`` is supported as the
-value of the ``closed`` argument; ``closed`` is ``False`` by default, which
-preserves the previous TypedDict behavior.
+value of the ``closed`` argument. Type checkers should reject any non-literal value.
+
+Passing ``closed=False`` explicitly requests the default TypedDict behavior,
+where arbitrary other keys may be present and subclasses may add arbitrary items.
+It is a type checker error to pass ``closed=False`` if a superclass has
+``closed=True`` or sets ``extra_items``.
 
-The value of ``closed`` is not inherited through subclassing, but the
-implicitly set ``extra_items=Never`` is. It should be an error to use the
-default ``closed=False`` when subclassing a closed TypedDict type::
+If ``closed`` is not provided, the behavior is inherited from the superclass.
+If the superclass is TypedDict itself or the superclass does not have ``closed=True``
+or the ``extra_items`` parameter, the previous TypedDict behavior is preserved:
+arbitrary extra items are allowed. If the superclass has ``closed=True``, the
+child class is also closed.
 
     class BaseMovie(TypedDict, closed=True):
         name: str
 
-    class MovieA(BaseMovie):  # Not OK. An explicit 'closed=True' is required
+    class MovieA(BaseMovie):  # OK, still closed
         pass
 
-    class MovieB(BaseMovie, closed=True):  # OK
+    class MovieB(BaseMovie, closed=True):  # OK, but redundant
         pass
 
-Setting both ``closed`` and ``extra_items`` when defining a TypedDict type
-should always be a runtime error::
-
-    class Person(TypedDict, closed=False, extra_items=bool):  # Not OK. 'closed' and 'extra_items' are incompatible
-        name: str
+    class MovieC(BaseMovie, closed=False):  # Type checker error
+        pass
 
-As a consequence of ``closed=True`` being equivalent to ``extra_items=Never``.
-The same rules that apply to ``extra_items=Never`` should also apply to
+As a consequence of ``closed=True`` being equivalent to ``extra_items=Never``,
+the same rules that apply to ``extra_items=Never`` also apply to
 ``closed=True``. It is possible to use ``closed=True`` when subclassing if the
 ``extra_items`` argument is a read-only type::
 
@@ -275,7 +283,7 @@ The same rules that apply to ``extra_items=Never`` should also apply to
     class MovieClosed(Movie, closed=True):  # OK
         pass
 
-    class MovieNever(Movie, extra_items=Never):  # Not OK. 'closed=True' is preferred
+    class MovieNever(Movie, extra_items=Never):  # OK, but 'closed=True' is preferred
         pass
 
 This will be further discussed in
@@ -286,6 +294,10 @@ is assumed to allow non-required extra items of value type ``ReadOnly[object]``
 during inheritance or assignability checks. This preserves the existing behavior
 of TypedDict.
 
+``closed`` is also supported with the functional syntax::
+
+    Movie = TypedDict("Movie", {"name": str}, closed=True)
+
 Interaction with Totality
 -------------------------
 
@@ -315,7 +327,7 @@ function parameters still apply::
 
     class Movie(TypedDict, extra_items=int):
         name: str
-    
+
     def f(**kwargs: Unpack[Movie]) -> None: ...
 
     # Should be equivalent to:
@@ -356,7 +368,7 @@ unless it is declared to be ``ReadOnly`` in the superclass::
 
     class Parent(TypedDict, extra_items=int | None):
         pass
-    
+
     class Child(Parent, extra_items=int): # Not OK. Like any other TypedDict item, extra_items's type cannot be changed
 
 Second, ``extra_items=T`` effectively defines the value type of any unnamed
@@ -378,20 +390,17 @@ added in a subclass, all of the following conditions should apply:
 
   - The item's value type is :term:`typing:consistent` with ``T``
 
-- If ``extra_items`` is not overriden, the subclass inherits it as-is.
+- If ``extra_items`` is not overridden, the subclass inherits it as-is.
 
 For example::
 
     class MovieBase(TypedDict, extra_items=int | None):
         name: str
-    
-    class AdaptedMovie(MovieBase):  # Not OK. 'bool' is not assignable to 'int | None'
-        adapted_from_novel: bool
- 
-    class MovieRequiredYear(MovieBase):  # Not OK. Required key 'year' is not known to 'Parent'
+
+    class MovieRequiredYear(MovieBase):  # Not OK. Required key 'year' is not known to 'MovieBase'
         year: int | None
 
-    class MovieNotRequiredYear(MovieBase):  # Not OK. 'int | None' is not assignable to 'int'
+    class MovieNotRequiredYear(MovieBase):  # Not OK. 'int | None' is not consistent with 'int'
         year: NotRequired[int]
 
     class MovieWithYear(MovieBase):  # OK
@@ -478,7 +487,7 @@ checks::
     class MovieDetails(TypedDict, extra_items=int | None):
         name: str
         year: NotRequired[int]
-    
+
     details: MovieDetails = {"name": "Kill Bill Vol. 1", "year": 2003}
     movie: Movie = details  # Not OK. While 'int' is assignable to 'int | None',
                             # 'int | None' is not assignable to 'int'
@@ -502,7 +511,7 @@ possible for an item to have a :term:`narrower <typing:narrow>` type than the
 
     class Movie(TypedDict, extra_items=ReadOnly[str | int]):
         name: str
-    
+
     class MovieDetails(TypedDict, extra_items=int):
         name: str
         year: NotRequired[int]
@@ -522,19 +531,19 @@ enforced::
 
     class MovieExtraStr(TypedDict, extra_items=str):
         name: str
-    
+
     extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
     extra_str: MovieExtraStr = {"name": "No Country for Old Men", "description": ""}
     extra_int = extra_str  # Not OK. 'str' is not assignable to extra items type 'int'
     extra_str = extra_int  # Not OK. 'int' is not assignable to extra items type 'str'
-    
+
 A non-closed TypedDict type implicitly allows non-required extra keys of value
 type ``ReadOnly[object]``. Applying the assignability rules between this type
 and a closed TypedDict type is allowed::
 
     class MovieNotClosed(TypedDict):
         name: str
-    
+
     extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
     not_closed: MovieNotClosed = {"name": "No Country for Old Men"}
     extra_int = not_closed  # Not OK.
@@ -578,17 +587,13 @@ arguments of this type when constructed by calling the class object::
 Interaction with Mapping[KT, VT]
 --------------------------------
 
-A TypedDict type can be assignable to ``Mapping[KT, VT]`` types other than
-``Mapping[str, object]`` as long as all value types of the items on the
-TypedDict type is :term:`typing:assignable` to ``VT``. This is an extension of this
+A TypedDict type is :term:`typing:assignable` to a type of the form ``Mapping[str, VT]``
+when all value types of the items in the TypedDict
+are assignable to ``VT``. For the purpose of this rule, a
+TypedDict that does not have ``extra_items=`` or ``closed=`` set is considered
+to have an item with a value of type ``object``. This extends the current
 assignability rule from the `typing spec
-<https://typing.readthedocs.io/en/latest/spec/typeddict.html#assignability>`__:
-
-    * A TypedDict with all ``int`` values is not :term:`typing:assignable` to
-      ``Mapping[str, int]``, since there may be additional non-``int`` values
-      not visible through the type, due to :term:`typing:structural`
-      assignability. These can be accessed using the ``values()`` and
-      ``items()`` methods in ``Mapping``,
+<https://typing.readthedocs.io/en/latest/spec/typeddict.html#assignability>`__.
 
 For example::
 
@@ -598,6 +603,10 @@ For example::
     extra_str: MovieExtraStr = {"name": "Blade Runner", "summary": ""}
     str_mapping: Mapping[str, str] = extra_str  # OK
 
+    class MovieExtraInt(TypedDict, extra_items=int):
+        name: str
+
+    extra_int: MovieExtraInt = {"name": "Blade Runner", "year": 1982}
     int_mapping: Mapping[str, int] = extra_int  # Not OK. 'int | str' is not assignable with 'int'
     int_str_mapping: Mapping[str, int | str] = extra_int  # OK
 
@@ -611,7 +620,7 @@ and ``items()`` on such TypedDict types::
 Interaction with dict[KT, VT]
 -----------------------------
 
-Note that because the presence of ``extra_items`` on a closed TypedDict type
+Because the presence of ``extra_items`` on a closed TypedDict type
 prohibits additional required keys in its :term:`typing:structural`
 :term:`typing:subtypes <subtype>`, we can determine if the TypedDict type and
 its structural subtypes will ever have any required key during static analysis.
@@ -636,8 +645,8 @@ For example::
     def f(x: IntDict) -> None:
         v: dict[str, int] = x  # OK
         v.clear()  # OK
-    
-    not_required_num_dict: IntDictWithNum = {"num": 1, "bar": 2} 
+
+    not_required_num_dict: IntDictWithNum = {"num": 1, "bar": 2}
     regular_dict: dict[str, int] = not_required_num_dict  # OK
     f(not_required_num_dict)  # OK
 
@@ -652,10 +661,28 @@ because such dict can be a subtype of dict::
 
     class CustomDict(dict[str, int]):
         pass
-    
+
     not_a_regular_dict: CustomDict = {"num": 1}
     int_dict: IntDict = not_a_regular_dict  # Not OK
 
+Runtime behavior
+----------------
+
+At runtime, it is an error to pass both the ``closed`` and ``extra_items``
+arguments in the same TypedDict definition, whether using the class syntax or
+the functional syntax. For simplicity, the runtime does not check other invalid
+combinations involving inheritance.
+
+For introspection, the ``closed`` and ``extra_items`` arguments are mapped to
+two new attributes on the resulting TypedDict object: ``__closed__`` and
+``__extra_items__``. These attributes reflect exactly what was passed to the
+TypedDict constructor, without considering superclasses.
+
+If ``closed`` is not passed, the value of ``__closed__`` is None. If ``extra_items``
+is not passed, the value of ``__extra_items__`` is the new sentinel object
+``typing.NoExtraItems``. (It cannot be ``None``, because ``extra_items=None`` is a
+valid definition that indicates all extra items must be ``None``.)
+
 How to Teach This
 =================
 
@@ -673,7 +700,7 @@ Because ``extra_items`` is an opt-in feature, no existing codebase will break
 due to this change.
 
 Note that ``closed`` and ``extra_items`` as keyword arguments do not collide
-with othere keys when using something like
+with other keys when using something like
 ``TD = TypedDict("TD", foo=str, bar=int)``, because this syntax has already
 been removed in Python 3.13.
 
