From 5fe22230c83e365bcc98da3ae29f958260e89e5f Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Fri, 25 Apr 2025 20:14:02 -0400 Subject: [PATCH 1/3] Address feedback from Carl Discussion: https://discuss.python.org/t/pep-728-typeddict-with-typed-extra-items/45443/153 --- peps/pep-0728.rst | 83 ++++++++++++++++++++++++++++++----------------- 1 file changed, 53 insertions(+), 30 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 4880e0bd360..4d93419d7f1 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -36,8 +36,8 @@ The current behavior of TypedDict prevents users from defining a TypedDict type when it is expected that the type contains no extra items. Due to the possible presence of extra items, type checkers cannot infer more -precise return types for ``.items()`` and ``.values()`` on a TypedDict. This can -also be resolved by +precise return types for ``.items()`` and ``.values()`` on a TypedDict. +This can be resolved by `defining a closed TypedDict type `__. Another possible use case for this is a sound way to @@ -126,12 +126,11 @@ that the old typing behavior can be supported in combination with ``Unpack``. Rationale ========= -A type that allows extra items of type ``str`` on a TypedDict can be loosely -described as the intersection between the TypedDict and ``Mapping[str, str]``. +Suppose we want a type that allows extra items of type ``str`` on a TypedDict. `Index Signatures `__ -in TypeScript achieve this: +in TypeScript allow this: .. code-block:: typescript @@ -140,9 +139,8 @@ in TypeScript achieve this: [key: string]: string } -This proposal aims to support a similar feature without introducing general -intersection of types or syntax changes, offering a natural extension to the -existing assignability rules. +This proposal aims to support a similar feature without syntax changes, +offering a natural extension to the existing assignability rules. We propose to add a class parameter ``extra_items`` to TypedDict. It accepts a :term:`typing:type expression` as the argument; when it is present, @@ -512,10 +510,11 @@ checks:: movie: Movie = details # Not OK. 'year' is not required in 'Movie', # so it shouldn't be required in 'MovieWithYear' either -Because ``'year'`` is absent in ``Movie``, ``extra_items`` is considered the -corresponding key. ``'year'`` being required violates this rule: +Given that ``'year'`` is not-required and non-read-only in A (``Movie``), +it shouldn’t be required in B (``MovieWithYear``) either, according to this rule: - * For each required key in ``A``, the corresponding key is required in ``B``. + * For each non-required key in ``A``, if the item is not read-only in ``A``, + the corresponding key is not required in ``B``. When ``extra_items`` is specified to be read-only on a TypedDict type, it is possible for an item to have a :term:`narrower ` type than the @@ -606,9 +605,6 @@ still holds true. Operations with arbitrary str keys (instead of string literals or other expressions with known string values) should generally be rejected. -This means that indexed accesses and assignments with arbitrary keys can still -be rejected even when ``extra_items`` is specified. - Operations that already apply to ``NotRequired`` items should generally also apply to extra items, following the same rationale from the `typing spec `__: @@ -617,9 +613,10 @@ apply to extra items, following the same rationale from the `typing spec cases potentially unsafe operations may be accepted if the alternative is to generate false positive errors for idiomatic code. -Some operations are allowed due to the TypedDict being -:term:`typing:assignable` to ``Mapping[str, VT]`` or ``dict[str, VT]``. -The two following sections will expand on that. +Some operations, including indexed accesses and assignments with arbitrary str keys, +can be allowed due to the TypedDict being :term:`typing:assignable` to +``Mapping[str, VT]`` or ``dict[str, VT]``. The two following sections will expand +on that. Interaction with Mapping[str, VT] --------------------------------- @@ -628,8 +625,8 @@ A TypedDict type is :term:`typing:assignable` to a type of the form ``Mapping[st when all value types of the items in the TypedDict are assignable to ``VT``. For the purpose of this rule, a TypedDict that does not have ``extra_items=`` or ``closed=`` set is considered -to have an item with a value of type ``object``. This extends the current -assignability rule from the `typing spec +to have an item with a value of type ``ReadOnly[object]``. This extends the +current assignability rule from the `typing spec `__. For example:: @@ -647,12 +644,26 @@ For example:: int_mapping: Mapping[str, int] = extra_int # Not OK. 'int | str' is not assignable with 'int' int_str_mapping: Mapping[str, int | str] = extra_int # OK -Type checkers should be able to infer the precise return types of ``values()`` -and ``items()`` on such TypedDict types:: +Type checkers should be able to infer the precise signatures of ``values()``, +``items()``, etc. on such TypedDict types:: + + def foo(movie: MovieExtraInt) -> None: + reveal_type(movie.items()) # Revealed type is 'dict_items[str, str | int]' + reveal_type(movie.values()) # Revealed type is 'dict_values[str, str | int]' + +By extension of this assignability rule, indexed accesses with arbitrary str keys +can be allowed as long as ``extra_items`` or ``closed=True`` is specified. +For example:: + + def bar(movie: MovieExtraInt, key: str) -> None: + reveal_type(movie[key]) # Revealed type is 'str | int' + +.. _pep728-type-narrowing: - def fun(movie: MovieExtraStr) -> None: - reveal_type(movie.items()) # Revealed type is 'dict_items[str, str]' - reveal_type(movie.values()) # Revealed type is 'dict_values[str, str]' +Defining the type narrowing behavior for TypedDict is out-of-scope for this PEP. +This leaves flexibility for a type checker to be more/less restrictive about +indexed accesses with arbitrary str keys. For example, a type checker might opt +for more restriction by requiring an explicit ``'x' in d`` check. Interaction with dict[str, VT] ------------------------------ @@ -687,20 +698,32 @@ For example:: regular_dict: dict[str, int] = not_required_num_dict # OK f(not_required_num_dict) # OK -In this case, methods that are previously unavailable on a TypedDict are allowed:: +In this case, methods that are previously unavailable on a TypedDict are allowed, +with signatures matching ``dict[str, VT]`` +(e.g.: ``__setitem__(self, key: str, value: VT) -> None``):: - not_required_num.clear() # OK + not_required_num_dict.clear() # OK - reveal_type(not_required_num.popitem()) # OK. Revealed type is tuple[str, int] + reveal_type(not_required_num_dict.popitem()) # OK. Revealed type is 'tuple[str, int]' -However, ``dict[str, VT]`` is not necessarily assignable to a TypedDict type, + def f(not_required_num_dict: IntDictWithNum, key: str): + not_required_num_dict[key] = 42 # OK + del not_required_num_dict[key] # OK + +:ref:`Notes on indexed accesses ` from the previous section +still apply. + +``dict[str, VT]`` is not assignable to a TypedDict type, because such dict can be a subtype of dict:: class CustomDict(dict[str, int]): pass - not_a_regular_dict: CustomDict = {"num": 1} - int_dict: IntDict = not_a_regular_dict # Not OK + def f(might_not_be_a_builtin_dict: dict[str, int]): + int_dict: IntDict = might_not_be_a_builtin_dict # Not OK + + not_a_builtin_dict: CustomDict = {"num": 1} + f(not_a_builtin_dict) Runtime behavior ---------------- From 4ce20cae82f939e4f2b03862fafed185b6c69e18 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Tue, 29 Apr 2025 23:07:05 -0400 Subject: [PATCH 2/3] Address review feedback --- peps/pep-0728.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 4d93419d7f1..510c3cce971 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -508,10 +508,10 @@ checks:: details: MovieWithYear = {"name": "Kill Bill Vol. 1", "year": 2003} movie: Movie = details # Not OK. 'year' is not required in 'Movie', - # so it shouldn't be required in 'MovieWithYear' either + # but it is required in 'MovieWithYear' -Given that ``'year'`` is not-required and non-read-only in A (``Movie``), -it shouldn’t be required in B (``MovieWithYear``) either, according to this rule: +where ``MovieWithYear`` (B) is not assignable to ``Movie`` (A) +according to this rule: * For each non-required key in ``A``, if the item is not read-only in ``A``, the corresponding key is not required in ``B``. @@ -614,7 +614,7 @@ apply to extra items, following the same rationale from the `typing spec generate false positive errors for idiomatic code. Some operations, including indexed accesses and assignments with arbitrary str keys, -can be allowed due to the TypedDict being :term:`typing:assignable` to +may be allowed due to the TypedDict being :term:`typing:assignable` to ``Mapping[str, VT]`` or ``dict[str, VT]``. The two following sections will expand on that. @@ -644,15 +644,15 @@ For example:: int_mapping: Mapping[str, int] = extra_int # Not OK. 'int | str' is not assignable with 'int' int_str_mapping: Mapping[str, int | str] = extra_int # OK -Type checkers should be able to infer the precise signatures of ``values()``, -``items()``, etc. on such TypedDict types:: +Type checkers should infer the precise signatures of ``values()`` and ``items()`` +on such TypedDict types:: def foo(movie: MovieExtraInt) -> None: reveal_type(movie.items()) # Revealed type is 'dict_items[str, str | int]' reveal_type(movie.values()) # Revealed type is 'dict_values[str, str | int]' -By extension of this assignability rule, indexed accesses with arbitrary str keys -can be allowed as long as ``extra_items`` or ``closed=True`` is specified. +By extension of this assignability rule, type chekcers may allow indexed accesses +with arbitrary str keys when ``extra_items`` or ``closed=True`` is specified. For example:: def bar(movie: MovieExtraInt, key: str) -> None: @@ -662,7 +662,7 @@ For example:: Defining the type narrowing behavior for TypedDict is out-of-scope for this PEP. This leaves flexibility for a type checker to be more/less restrictive about -indexed accesses with arbitrary str keys. For example, a type checker might opt +indexed accesses with arbitrary str keys. For example, a type checker may opt for more restriction by requiring an explicit ``'x' in d`` check. Interaction with dict[str, VT] From 18eff6118efac09c16409065e3040b2ecc9e9946 Mon Sep 17 00:00:00 2001 From: Carl Meyer Date: Wed, 30 Apr 2025 07:47:55 -0700 Subject: [PATCH 3/3] Update peps/pep-0728.rst --- peps/pep-0728.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 510c3cce971..5d7e40fa9dd 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -651,7 +651,7 @@ on such TypedDict types:: reveal_type(movie.items()) # Revealed type is 'dict_items[str, str | int]' reveal_type(movie.values()) # Revealed type is 'dict_values[str, str | int]' -By extension of this assignability rule, type chekcers may allow indexed accesses +By extension of this assignability rule, type checkers may allow indexed accesses with arbitrary str keys when ``extra_items`` or ``closed=True`` is specified. For example::