From 142335df12147d85e2df76c9289f9ba026ac9398 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Thu, 14 Aug 2025 17:29:44 -0400 Subject: [PATCH 01/16] change awkward wording about reference implementation --- peps/pep-0798.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 3162d56fc94..aa6e3442b41 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -362,9 +362,9 @@ cases: Reference Implementation ======================== -A `reference implementation `_ -is available, which implements this functionality, including draft documentation and -additional test cases. +The `reference implementation `_ +implements this functionality, including draft documentation and additional +test cases. Backwards Compatibility ======================= From af559b7b93a08b267f575123c1f8e1c3859eae1c Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Thu, 14 Aug 2025 18:55:18 -0400 Subject: [PATCH 02/16] mention Reddit post --- peps/pep-0798.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index aa6e3442b41..c80db0e97eb 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -119,6 +119,11 @@ existing syntax ``[x for it in its for x in it]`` is one that students often get wrong, the natural impulse for many students being to reverse the order of the ``for`` clauses. +Additionally, the comment section of a `Reddit post +`__ +following the publication of this PEP shows substantial support for the +proposal and further suggests that the syntax proposed here is legible, +intuitive, and useful. Specification ============= From a054b421bb8d86a06d2ac4df9f65393eb2380191 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Thu, 14 Aug 2025 18:58:21 -0400 Subject: [PATCH 03/16] change recommended semantics of generator expressions, add alternatives to rejected ideas --- peps/pep-0798.rst | 91 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 79 insertions(+), 12 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index c80db0e97eb..5ecebb8c7b8 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -209,29 +209,36 @@ respectively:: for x in dicts: new_dict.update(expr) +.. _pep798-genexpsemantics: Semantics: Generator Expressions -------------------------------- -A generator expression ``(*expr for x in it)`` forms a generator producing -values from the concatenation of the iterables given by the expressions. -Specifically, the behavior is defined to be equivalent to the following -generator:: +Generator expressions using the unpacking syntax should form new generators +producing values from the concatenation of the iterables given by the +expressions. Specifically, the behavior is defined to be equivalent to the +following (though without defining for referencing the looping variable +``i``):: + # equivalent to generator = (*expr for x in it) def generator(): for x in it: - yield from expr - -Since ``yield from`` is not allowed inside of async generators (see the section -of :pep:`525` on Asynchronous ``yield from``), the equivalent for ``(*expr -async for x in ait())`` is more like the following (though of course this new -form should not define or reference the looping variable ``i``):: + for i in expr: + yield i + # equivalent to generator = (*expr for x in ait()) async def generator(): async for x in ait(): for i in expr: yield i + +The specifics of these semantics should be revisited in the future, +particularly if async generators receive support for ``yield from`` (in which +case both forms may wish to be changed to make use of ``yield from`` instead of +an explicit loop). See :ref:`pep798-alternativegenexpsemantics` for more +discussion. + Interaction with Assignment Expressions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -249,7 +256,8 @@ form, ``y`` will be bound in the containing scope instead of locally:: def generator(): for i in (0, 2, 4): - yield from (y := [i, i+1]) + for j in (y := [i, i+1]): + yield j In this example, the subexpression ``(y := [i, i+1])`` is evaluated exactly three times before the generator is exhausted: just after assigning ``i`` in @@ -577,7 +585,7 @@ expressions that involve unpacking:: yield expr g = generator() - # equivalent to g = (*expr for x in it) + # roughly equivalent to g = (*expr for x in it) def generator(): for x in it: yield from expr @@ -705,6 +713,65 @@ PEP. As such, these forms should continue to raise a ``SyntaxError``, but with a new error message as described above, though it should not be ruled out as a consideration for future proposals. +.. _pep798-alternativegenexpsemantics: + +Alternative Generator Expression Semantics +------------------------------------------ + +Another point of discussion centered around the semantics of unpacking in +generator expressions, particularly the relationship between the semantics of +synchronous and asynchronous generator expressions given that the latter do not +support ``yield from`` (see the section of :pep:`525` on Asynchronous ``yield +from``). + +Several reasonable options were considered, none of which was a clear winner in +a `poll in the Discourse thread +`__: + +1. Using ``yield from`` for unpacking in synchronous generator expressions but + not in asynchronous generator expressions (as proposed in the original draft + of this PEP). + + This strategy would have allowed unpacking in generator expressions to + closely mimic a popular way of writing generators that perform this + operation (using ``yield from``), but it would also have created an + asymmetry between synchronous and asynchronous versions, and also between + this new syntax and ``itertools.chain`` and the double-loop version. + +2. Using ``yield from`` for unpacking in synchronous generator expressions and + mimicking the behavior of ``yield from`` for unpacking in async generator + expressions. + + This strategy would also make unpacking in synchronous and asynchronous + generators behave similarly, but it would also be more complex, enough so + that the cost may not be worth the benefit, particularly in the absence of a + compelling practical use case for delegating to subgenerators during + unpacking. + +3. Disallowing unpacking in asynchronous generator expressions until they + support ``yield from``. + + This strategy could possibly reduce friction if asynchronous generator + expressions do gain support for ``yield from`` in the future, but in the + meantime, it would result in an even bigger discrepancy between synchronous + and asynchronous generator expressions than option 1. + +4. Disallowing unpacking in all generator expressions. + + This would retain symmetry between the two cases, but with the downside of + losing a very expressive form. + +Each of these options (including the one presented in this PEP) has its +benefits and drawbacks, with no option being clearly superior on all fronts; +but the semantics proposed in :ref:`pep798-genexpsemantics` represent a +reasonable compromise. + +As suggested above, this decision should be revisited in the event that +asynchronous generators receive support for ``yield from`` in the future, +in which case the ability to delegate to subgenerators during unpacking +could be added without significant cost. + + Concerns and Disadvantages ========================== From e840ceb0543e8b5cd1e58d3cdab07ccf891b67ca Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Thu, 14 Aug 2025 19:07:29 -0400 Subject: [PATCH 04/16] small wording tweaks --- peps/pep-0798.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 5ecebb8c7b8..81bf7df8e97 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -131,7 +131,7 @@ Specification Syntax ------ -The necessary grammatical changes are allowing the expression in list/set +The grammar should be changed to allow the expression in list/set comprehensions and generator expressions to be preceded by a ``*``, and allowing an alternative form of dictionary comprehension in which a double-starred expression can be used in place of a ``key: value`` pair. @@ -217,7 +217,7 @@ Semantics: Generator Expressions Generator expressions using the unpacking syntax should form new generators producing values from the concatenation of the iterables given by the expressions. Specifically, the behavior is defined to be equivalent to the -following (though without defining for referencing the looping variable +following (though without defining or referencing the looping variable ``i``):: # equivalent to generator = (*expr for x in it) @@ -655,7 +655,7 @@ resulting generator, but several alternatives were suggested in our discussion other aspects of this proposal are accepted. The reason to prefer this proposal over these alternatives is the preservation -of existent conventions for punctuation around generator expressions. +of existing conventions for punctuation around generator expressions. Currently, the general rule is that generator expressions must be wrapped in parentheses except when provided as the sole argument to a function, and this proposal suggests maintaining that rule even as we allow more kinds of @@ -794,7 +794,7 @@ were raised as well. This section aims to summarize those concerns. Complex uses of unpacking in comprehensions could obscure logic that would be clearer in an explicit loop. While this is already a concern with comprehensions more generally, the addition of ``*`` and ``**`` may make - particularly-complex uses even more difficult to read and understand at a + particularly complex uses even more difficult to read and understand at a glance. For example, while these situations are likely rare, comprehensions that use unpacking in multiple ways can make it difficult to know what's being unpacked and when: ``f(*(*x for *x, _ in list_of_lists))``. @@ -840,7 +840,7 @@ Many languages that support comprehensions support double loops: (for [xs [[1 2 3] [] [4 5]] x (concat xs xs)] x) Several other languages (even those without comprehensions) support these -operations via a built-in function/method to support flattening of nested +operations via a built-in function or method to support flattening of nested structures: .. code:: python @@ -850,7 +850,7 @@ structures: .. code:: javascript - // Javascript + // JavaScript [[1,2,3], [], [4,5]].flatMap(xs => [...xs, ...xs]) .. code:: haskell @@ -873,7 +873,7 @@ in Julia currently leads to a syntax error: As one counterexample, support for a similar syntax was recently added to `Civet `_. For example, the following is a valid comprehension in -Civet, making use of Javascript's ``...`` syntax for unpacking: +Civet, making use of JavaScript's ``...`` syntax for unpacking: .. code:: javascript From df14e2585b8f0724b8538fe90de4ab7f29c3139b Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Thu, 14 Aug 2025 22:19:16 -0400 Subject: [PATCH 05/16] fix typo in code Co-authored-by: Jelle Zijlstra --- peps/pep-0798.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 81bf7df8e97..2c4e872b842 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -226,7 +226,7 @@ following (though without defining or referencing the looping variable for i in expr: yield i - # equivalent to generator = (*expr for x in ait()) + # equivalent to generator = (*expr async for x in ait()) async def generator(): async for x in ait(): for i in expr: From 5c7ca79ab6c56e4f2b778d84b5c6d8edb3a3cd00 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Fri, 15 Aug 2025 18:04:48 -0400 Subject: [PATCH 06/16] wording and capitalization --- peps/pep-0798.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 2c4e872b842..d2f2fc9ba8c 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -343,7 +343,8 @@ cases: * The phrasing of some other existing error messages should similarly be adjusted to account for the presence of the new syntax, and/or to clarify ambiguous or confusing cases relating to unpacking more generally - (particularly those mentioned in :ref:`pep798-moregeneral`), for example:: + (particularly the cases mentioned in :ref:`pep798-moregeneral`), for + example:: >>> [*x if x else y] File "", line 1 @@ -850,7 +851,7 @@ structures: .. code:: javascript - // JavaScript + // javascript [[1,2,3], [], [4,5]].flatMap(xs => [...xs, ...xs]) .. code:: haskell From d8043725686e99d4f81320879613ed2fa2a4fe36 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 23 Aug 2025 17:05:56 -0400 Subject: [PATCH 07/16] update some more words --- peps/pep-0798.rst | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index d2f2fc9ba8c..10c427d002d 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -232,7 +232,6 @@ following (though without defining or referencing the looping variable for i in expr: yield i - The specifics of these semantics should be revisited in the future, particularly if async generators receive support for ``yield from`` (in which case both forms may wish to be changed to make use of ``yield from`` instead of @@ -391,6 +390,12 @@ in comprehensions would raise a ``SyntaxError``, or that relied on the particular phrasing of any of the old error messages being replaced, which we expect to be rare. +One related concern is that a hypotehtical future decision to change the +semantics of generator expressions to make use of ``yield from`` during +unpacking (delegating to generators that are being unpacked) would be not be +backwards-compatible because it would affect the behavior of the resulting +generators when used with ``.send()``, ``.throw()``, and ``.close()``. See +:ref:`pep798-alternativegenexpsemantics` for more discussion. .. _pep798-examples: @@ -721,9 +726,18 @@ Alternative Generator Expression Semantics Another point of discussion centered around the semantics of unpacking in generator expressions, particularly the relationship between the semantics of -synchronous and asynchronous generator expressions given that the latter do not -support ``yield from`` (see the section of :pep:`525` on Asynchronous ``yield -from``). +synchronous and asynchronous generator expressions given that async generators +do not support ``yield from`` (see the section of :pep:`525` on Asynchronous +``yield from``). + +The core question centered around whether sync and/or async generator +expressions should use ``yield from`` (or an equivalent) when unpacking, as +opposed to an explicit loop. The main difference between these options is +whether the resulting generator delegates to the objects being unpacked (see +:pep:`380`), which would affect the behavior of these generator expressions +when used with ``.send()``, ``.throw()``, and ``.close()``, in the case where +the objects being unpacked are themselves generators, which appears to be a +rare situation. Several reasonable options were considered, none of which was a clear winner in a `poll in the Discourse thread @@ -749,13 +763,15 @@ a `poll in the Discourse thread compelling practical use case for delegating to subgenerators during unpacking. -3. Disallowing unpacking in asynchronous generator expressions until they +3. Using ``yield from`` for unpacking in synchronous generator expressions, and + disallowing unpacking in asynchronous generator expressions until they support ``yield from``. This strategy could possibly reduce friction if asynchronous generator - expressions do gain support for ``yield from`` in the future, but in the - meantime, it would result in an even bigger discrepancy between synchronous - and asynchronous generator expressions than option 1. + expressions do gain support for ``yield from`` in the future by making sure + that any decision made at that point would be fully backwards-compatible, + but in the meantime, it would result in an even bigger discrepancy between + synchronous and asynchronous generator expressions than option 1. 4. Disallowing unpacking in all generator expressions. @@ -765,7 +781,9 @@ a `poll in the Discourse thread Each of these options (including the one presented in this PEP) has its benefits and drawbacks, with no option being clearly superior on all fronts; but the semantics proposed in :ref:`pep798-genexpsemantics` represent a -reasonable compromise. +reasonable compromise by allowing exactly the same kind of unpacking in +synchronous and asynchronous generator expressions and retaining an existing +property of generator expressions (that they do not As suggested above, this decision should be revisited in the event that asynchronous generators receive support for ``yield from`` in the future, From bb3706d9b293caa84444a4fd37cdb4feddaf093f Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 23 Aug 2025 18:19:32 -0400 Subject: [PATCH 08/16] revert to original proposal's semantics for genexps --- peps/pep-0798.rst | 96 +++++++++++++++++++++++++---------------------- 1 file changed, 52 insertions(+), 44 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 10c427d002d..af70e3726a4 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -217,26 +217,33 @@ Semantics: Generator Expressions Generator expressions using the unpacking syntax should form new generators producing values from the concatenation of the iterables given by the expressions. Specifically, the behavior is defined to be equivalent to the -following (though without defining or referencing the looping variable -``i``):: +following:: - # equivalent to generator = (*expr for x in it) + # equivalent to g = (*expr for x in it) def generator(): for x in it: - for i in expr: - yield i + yield from expr + + g = generator() + +Since ``yield from`` is not allowed inside of async generators (see the section +of :pep:`525` on Asynchronous ``yield from``), the equivalent for ``(*expr +async for x in ait())`` is more like the following (though of course this new +form should not define or reference the looping variable ``i``):: - # equivalent to generator = (*expr async for x in ait()) + # equivalent to g = (*expr async for x in ait()) async def generator(): async for x in ait(): for i in expr: yield i + g = generator() + The specifics of these semantics should be revisited in the future, particularly if async generators receive support for ``yield from`` (in which -case both forms may wish to be changed to make use of ``yield from`` instead of -an explicit loop). See :ref:`pep798-alternativegenexpsemantics` for more -discussion. +case the async variant may wish to be changed to make use of ``yield from`` +instead of an explicit loop). See :ref:`pep798-alternativegenexpsemantics` for +more discussion. Interaction with Assignment Expressions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -255,8 +262,7 @@ form, ``y`` will be bound in the containing scope instead of locally:: def generator(): for i in (0, 2, 4): - for j in (y := [i, i+1]): - yield j + yield from (y := [i, i+1]) In this example, the subexpression ``(y := [i, i+1])`` is evaluated exactly three times before the generator is exhausted: just after assigning ``i`` in @@ -390,11 +396,11 @@ in comprehensions would raise a ``SyntaxError``, or that relied on the particular phrasing of any of the old error messages being replaced, which we expect to be rare. -One related concern is that a hypotehtical future decision to change the -semantics of generator expressions to make use of ``yield from`` during +One related concern is that a hypothetical future decision to change the +semantics of async generator expressions to make use of ``yield from`` during unpacking (delegating to generators that are being unpacked) would be not be backwards-compatible because it would affect the behavior of the resulting -generators when used with ``.send()``, ``.throw()``, and ``.close()``. See +generators when used with ``.asend()``, ``.athrow()``, and ``.aclose()``. See :ref:`pep798-alternativegenexpsemantics` for more discussion. .. _pep798-examples: @@ -730,37 +736,38 @@ synchronous and asynchronous generator expressions given that async generators do not support ``yield from`` (see the section of :pep:`525` on Asynchronous ``yield from``). -The core question centered around whether sync and/or async generator -expressions should use ``yield from`` (or an equivalent) when unpacking, as -opposed to an explicit loop. The main difference between these options is -whether the resulting generator delegates to the objects being unpacked (see -:pep:`380`), which would affect the behavior of these generator expressions -when used with ``.send()``, ``.throw()``, and ``.close()``, in the case where -the objects being unpacked are themselves generators, which appears to be a -rare situation. +The core question centered around whether sync and async generator expressions +should use ``yield from`` (or an equivalent) when unpacking, as opposed to an +explicit loop. The main difference between these options is whether the +resulting generator delegates to the objects being unpacked (see :pep:`380`), +which would affect the behavior of these generator expressions when used with +``.send()/.asend()``, ``.throw()/.athrow()``, and ``.close()/.aclose()``, in +the case where the objects being unpacked are themselves generators, which +is unlikely to be a common use case. Several reasonable options were considered, none of which was a clear winner in a `poll in the Discourse thread -`__: +`__. +Beyond the proposal outlined above, the following were also considered: -1. Using ``yield from`` for unpacking in synchronous generator expressions but - not in asynchronous generator expressions (as proposed in the original draft - of this PEP). +1. Using explicit loops for both synchronous and asynchronous generator + expressions. - This strategy would have allowed unpacking in generator expressions to - closely mimic a popular way of writing generators that perform this - operation (using ``yield from``), but it would also have created an - asymmetry between synchronous and asynchronous versions, and also between - this new syntax and ``itertools.chain`` and the double-loop version. + This strategy would have resulted in a symmetry between synchronous and + asynchronous generator expressions but would have prevented a + potentially-useful tool by disallowing delegation in the case of synchronous + generator expressions. Moreover, the existing asymmetry between synchronous + and asynchronous generators mitigates concerns about asymmetry of the + unpacking operator within generator expressions. 2. Using ``yield from`` for unpacking in synchronous generator expressions and mimicking the behavior of ``yield from`` for unpacking in async generator expressions. This strategy would also make unpacking in synchronous and asynchronous - generators behave similarly, but it would also be more complex, enough so - that the cost may not be worth the benefit, particularly in the absence of a - compelling practical use case for delegating to subgenerators during + generators behave symmetrically, but it would also be more complex, enough + so that the cost may not be worth the benefit, particularly in the absence + of a compelling practical use case for delegating to subgenerators during unpacking. 3. Using ``yield from`` for unpacking in synchronous generator expressions, and @@ -770,25 +777,26 @@ a `poll in the Discourse thread This strategy could possibly reduce friction if asynchronous generator expressions do gain support for ``yield from`` in the future by making sure that any decision made at that point would be fully backwards-compatible, - but in the meantime, it would result in an even bigger discrepancy between - synchronous and asynchronous generator expressions than option 1. + but the utility of unpacking in that context seems to outweigh the potential + downside of a backwards-incompatible change in the future if aync generator + expressions do receive support for ``yield from``. 4. Disallowing unpacking in all generator expressions. This would retain symmetry between the two cases, but with the downside of losing a very expressive form. + Each of these options (including the one presented in this PEP) has its benefits and drawbacks, with no option being clearly superior on all fronts; but the semantics proposed in :ref:`pep798-genexpsemantics` represent a -reasonable compromise by allowing exactly the same kind of unpacking in -synchronous and asynchronous generator expressions and retaining an existing -property of generator expressions (that they do not - -As suggested above, this decision should be revisited in the event that -asynchronous generators receive support for ``yield from`` in the future, -in which case the ability to delegate to subgenerators during unpacking -could be added without significant cost. +reasonable compromise where unpacking in both synchronous and asynchronous +generator expressions mirrors common ways of writing equivalent generators. + +As suggested above, though, this decision should be revisited in the event that +asynchronous generators receive support for ``yield from`` in the future, in +which case adjusting the semantics of unpacking in async generator expressions +to use ``yield from`` should be considered. Concerns and Disadvantages From 53d9444c1101c295002ef3a0e3700577ca094ffd Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 23 Aug 2025 18:38:32 -0400 Subject: [PATCH 09/16] wording changes --- peps/pep-0798.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index af70e3726a4..556863ff65a 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -741,7 +741,7 @@ should use ``yield from`` (or an equivalent) when unpacking, as opposed to an explicit loop. The main difference between these options is whether the resulting generator delegates to the objects being unpacked (see :pep:`380`), which would affect the behavior of these generator expressions when used with -``.send()/.asend()``, ``.throw()/.athrow()``, and ``.close()/.aclose()``, in +``.send()/.asend()``, ``.throw()/.athrow()``, and ``.close()/.aclose()`` in the case where the objects being unpacked are themselves generators, which is unlikely to be a common use case. @@ -768,7 +768,9 @@ Beyond the proposal outlined above, the following were also considered: generators behave symmetrically, but it would also be more complex, enough so that the cost may not be worth the benefit, particularly in the absence of a compelling practical use case for delegating to subgenerators during - unpacking. + unpacking. Generator expressions using the unpacking operator should not + use semantics similar to ``yield from`` until ``yield from`` is supported + in asynchronous generators more generally. 3. Using ``yield from`` for unpacking in synchronous generator expressions, and disallowing unpacking in asynchronous generator expressions until they @@ -776,8 +778,8 @@ Beyond the proposal outlined above, the following were also considered: This strategy could possibly reduce friction if asynchronous generator expressions do gain support for ``yield from`` in the future by making sure - that any decision made at that point would be fully backwards-compatible, - but the utility of unpacking in that context seems to outweigh the potential + that any decision made at that point would be fully backwards-compatible. + But the utility of unpacking in that context seems to outweigh the potential downside of a backwards-incompatible change in the future if aync generator expressions do receive support for ``yield from``. @@ -791,7 +793,8 @@ Each of these options (including the one presented in this PEP) has its benefits and drawbacks, with no option being clearly superior on all fronts; but the semantics proposed in :ref:`pep798-genexpsemantics` represent a reasonable compromise where unpacking in both synchronous and asynchronous -generator expressions mirrors common ways of writing equivalent generators. +generator expressions mirrors common ways of writing equivalent generators +currently. As suggested above, though, this decision should be revisited in the event that asynchronous generators receive support for ``yield from`` in the future, in From 20e62fc94a0f8781d649134d2b5f87fd5866f061 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sun, 24 Aug 2025 11:44:16 -0400 Subject: [PATCH 10/16] fix a few typos, add a brief note to backwards compatibility section --- peps/pep-0798.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 556863ff65a..a8e1f0fc7b4 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -398,9 +398,12 @@ expect to be rare. One related concern is that a hypothetical future decision to change the semantics of async generator expressions to make use of ``yield from`` during -unpacking (delegating to generators that are being unpacked) would be not be +unpacking (delegating to generators that are being unpacked) would not be backwards-compatible because it would affect the behavior of the resulting -generators when used with ``.asend()``, ``.athrow()``, and ``.aclose()``. See +generators when used with ``.asend()``, ``.athrow()``, and ``.aclose()``. That +said, despite being backwards-incompatible, such a change would be unlikely to +have a large impact because it would only affect the behavior of structures +that, under this proposal, are not particularly useful. See :ref:`pep798-alternativegenexpsemantics` for more discussion. .. _pep798-examples: @@ -780,7 +783,7 @@ Beyond the proposal outlined above, the following were also considered: expressions do gain support for ``yield from`` in the future by making sure that any decision made at that point would be fully backwards-compatible. But the utility of unpacking in that context seems to outweigh the potential - downside of a backwards-incompatible change in the future if aync generator + downside of a backwards-incompatible change in the future if async generator expressions do receive support for ``yield from``. 4. Disallowing unpacking in all generator expressions. From 8a50f5c7659d7b6f2aa4c18e541e52c5ce1d8fc4 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 6 Sep 2025 22:47:21 -0400 Subject: [PATCH 11/16] add appendix about advanced generator usage, clean up wording about alternatives --- peps/pep-0798.rst | 195 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 171 insertions(+), 24 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index a8e1f0fc7b4..035ad0481bb 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -742,11 +742,12 @@ do not support ``yield from`` (see the section of :pep:`525` on Asynchronous The core question centered around whether sync and async generator expressions should use ``yield from`` (or an equivalent) when unpacking, as opposed to an explicit loop. The main difference between these options is whether the -resulting generator delegates to the objects being unpacked (see :pep:`380`), -which would affect the behavior of these generator expressions when used with -``.send()/.asend()``, ``.throw()/.athrow()``, and ``.close()/.aclose()`` in -the case where the objects being unpacked are themselves generators, which -is unlikely to be a common use case. +resulting generator delegates to the objects being unpacked, which would affect +the behavior of these generator expressions when used with +``.send()/.asend()``, ``.throw()/.athrow()``, and ``.close()/.aclose()`` in the +case where the objects being unpacked are themselves generators. The +differences between these options are summarized in +:ref:`pep798-appendix-yieldfrom`. Several reasonable options were considered, none of which was a clear winner in a `poll in the Discourse thread @@ -759,9 +760,11 @@ Beyond the proposal outlined above, the following were also considered: This strategy would have resulted in a symmetry between synchronous and asynchronous generator expressions but would have prevented a potentially-useful tool by disallowing delegation in the case of synchronous - generator expressions. Moreover, the existing asymmetry between synchronous - and asynchronous generators mitigates concerns about asymmetry of the - unpacking operator within generator expressions. + generator expressions. One specific concern with this approach is the + introduction of an asymmetry between synchronous and asynchronous + generators, but this concern is mitigated by the fact that these asymmetries + already exist between synchronous and asynchronous generators more + generally. 2. Using ``yield from`` for unpacking in synchronous generator expressions and mimicking the behavior of ``yield from`` for unpacking in async generator @@ -769,11 +772,10 @@ Beyond the proposal outlined above, the following were also considered: This strategy would also make unpacking in synchronous and asynchronous generators behave symmetrically, but it would also be more complex, enough - so that the cost may not be worth the benefit, particularly in the absence - of a compelling practical use case for delegating to subgenerators during - unpacking. Generator expressions using the unpacking operator should not - use semantics similar to ``yield from`` until ``yield from`` is supported - in asynchronous generators more generally. + so that the cost may not be worth the benefit. As such, this PEP proposes + that generator expressions using the unpacking operator should not use + semantics similar to ``yield from`` until ``yield from`` is supported in + asynchronous generators more generally. 3. Using ``yield from`` for unpacking in synchronous generator expressions, and disallowing unpacking in asynchronous generator expressions until they @@ -783,8 +785,8 @@ Beyond the proposal outlined above, the following were also considered: expressions do gain support for ``yield from`` in the future by making sure that any decision made at that point would be fully backwards-compatible. But the utility of unpacking in that context seems to outweigh the potential - downside of a backwards-incompatible change in the future if async generator - expressions do receive support for ``yield from``. + downside of a minimally-invasive backwards-incompatible change in the future + if async generator expressions do receive support for ``yield from``. 4. Disallowing unpacking in all generator expressions. @@ -793,13 +795,14 @@ Beyond the proposal outlined above, the following were also considered: Each of these options (including the one presented in this PEP) has its -benefits and drawbacks, with no option being clearly superior on all fronts; -but the semantics proposed in :ref:`pep798-genexpsemantics` represent a -reasonable compromise where unpacking in both synchronous and asynchronous -generator expressions mirrors common ways of writing equivalent generators -currently. - -As suggested above, though, this decision should be revisited in the event that +benefits and drawbacks, with no option being clearly superior on all fronts. +The semantics proposed in :ref:`pep798-genexpsemantics` represent a reasonable +compromise where unpacking in both synchronous and asynchronous generator +expressions mirrors common ways of writing equivalent generators currently. +Moreover, these subtle differences are unlikely to be impactful for the common +use case of combining simple collections. + +As suggested above, this decision should be revisited in the event that asynchronous generators receive support for ``yield from`` in the future, in which case adjusting the semantics of unpacking in async generator expressions to use ``yield from`` should be considered. @@ -842,8 +845,9 @@ were raised as well. This section aims to summarize those concerns. for maintainers of code formatters, linters, type checkers, etc., to make sure that the new syntax is supported. -Other Languages -=============== + +Appendix: Other Languages +========================= Quite a few other languages support this kind of flattening with syntax similar to what is already available in Python, but support for using unpacking syntax @@ -912,6 +916,149 @@ Civet, making use of JavaScript's ``...`` syntax for unpacking: for xs of [[1,2,3], [], [4,5]] then ...(xs++xs) +.. _pep798-appendix-yieldfrom: + +Appendix: Semantics of Generator Delegation +=========================================== + +One of the common questions about the semantics outlined above had to do with +the difference between using ``yield from`` when unpacking inside of a +generator expression, versus using an explicit loop. Because this is a +fairly-advanced feature of generators, this appendix attempts to summarize some +of the key differences between generators that use ``yield from`` versus those +that use explicit loops. + +Basic Behavior +-------------- + +For simple iteration over values, which we expect to be by far the most-common +use of unpacking in generator expressions, both approaches produce identical +results:: + + def yield_from(iterables): + for iterable in iterables: + yield from iterable + + def explicit_loop(iterables): + for iterable in iterables: + for item in iterable: + yield item + + # Both produce the same sequence of values + x = list(yield_from([[1, 2], [3, 4]])) + y = list(explicit_loop([[1, 2], [3, 4]])) + print(x == y) # prints True + +Advanced Generator Protocol Differences +--------------------------------------- + +The differences become apparent when using the advanced generator protocol +methods ``.send()``, ``.throw()``, and ``.close()``, and when the sub-iterables +are themselves generators rather than simple sequences. + +Delegation with ``.send()`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. code:: python + + def sub_generator(): + x = yield "first" + yield f"received: {x}" + yield "last" + + def yield_from(): + yield from sub_generator() + + def explicit_loop(): + for item in sub_generator(): + yield item + + # With yield from, values are passed through to sub-generator + gen1 = yield_from() + print(next(gen1)) # prints "first" + print(gen1.send("hello")) # prints "received: hello" + print(next(gen1)) # prints "last" + + # With explicit loop, .send() affects the outer generator; values don't reach the sub-generator + gen2 = explicit_loop() + print(next(gen2)) # prints "first" + print(gen2.send("hello")) # prints "received: None" (sub-generator receives None instead of "hello") + print(next(gen2)) # prints "last" + +Exception Handling with ``.throw()`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: python + + def sub_generator_with_exception_handling(): + try: + yield "first" + yield "second" + except ValueError as e: + yield f"caught: {e}" + + def yield_from(): + yield from sub_generator_with_exception_handling() + + def explicit_loop(): + for item in sub_generator_with_exception_handling(): + yield item + + # With yield from, exceptions are passed to sub-generator + gen1 = yield_from() + print(next(gen1)) # prints "first" + print(gen1.throw(ValueError("test"))) # prints "caught: test" + + # With explicit loop, exceptions affect the outer generator only + gen2 = explicit_loop() + print(next(gen2)) # prints "first" + print(gen2.throw(ValueError("test"))) # ValueError is raised; sub-generator doesn't see it + +Generator Cleanup with ``.close()`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: python + + # hold references to sub-generators so GC doesn't close the explicit loop version + references = [] + + def sub_generator_with_cleanup(): + try: + yield "first" + yield "second" + finally: + print("sub-generator received GeneratorExit") + + def yield_from(): + try: + g = sub_generator_with_cleanup() + references.append(g) + yield from g + finally: + print("outer generator received GeneratorExit") + + def explicit_loop(): + try: + g = sub_generator_with_cleanup() + references.append(g) + for item in g: + yield item + finally: + print("outer generator received GeneratorExit") + + # With yield from - GeneratorExit is passed through to sub-generator + gen1 = yield_from() + print(next(gen1)) # prints "first" + gen1.close() # closes sub-generator and then outer generator + + # With explicit loop - GeneratorExit goes to outer generator first + gen2 = explicit_loop() + print(next(gen2)) # prints "first" + gen2.close() #only closes outer generator + + print('program finished; GC will close the explicit loop subgenerator') + # second inner generator closes when GC closes it at the end + + References ========== From 86fe70e933a9e6faed26482345bf92f81128247c Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 6 Sep 2025 22:49:44 -0400 Subject: [PATCH 12/16] make punctuation consistent between generator examples --- peps/pep-0798.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 035ad0481bb..94019bb70a5 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -1045,15 +1045,15 @@ Generator Cleanup with ``.close()`` finally: print("outer generator received GeneratorExit") - # With yield from - GeneratorExit is passed through to sub-generator + # With yield from, GeneratorExit is passed through to sub-generator gen1 = yield_from() print(next(gen1)) # prints "first" gen1.close() # closes sub-generator and then outer generator - # With explicit loop - GeneratorExit goes to outer generator first + # With explicit loop, GeneratorExit goes to outer generator only gen2 = explicit_loop() print(next(gen2)) # prints "first" - gen2.close() #only closes outer generator + gen2.close() # only closes outer generator print('program finished; GC will close the explicit loop subgenerator') # second inner generator closes when GC closes it at the end From 87c3ef539219207cf3f8679f9b9e6d16fe09709c Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 6 Sep 2025 23:16:50 -0400 Subject: [PATCH 13/16] revert one more wording change related to genexp semantics --- peps/pep-0798.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 94019bb70a5..79cd786dc79 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -600,7 +600,7 @@ expressions that involve unpacking:: yield expr g = generator() - # roughly equivalent to g = (*expr for x in it) + # equivalent to g = (*expr for x in it) def generator(): for x in it: yield from expr From 9409aa5f81ea758136edca73cad1276658e0dc36 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 6 Sep 2025 23:19:29 -0400 Subject: [PATCH 14/16] another wording tweak --- peps/pep-0798.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 79cd786dc79..1b4236a1ca3 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -799,8 +799,9 @@ benefits and drawbacks, with no option being clearly superior on all fronts. The semantics proposed in :ref:`pep798-genexpsemantics` represent a reasonable compromise where unpacking in both synchronous and asynchronous generator expressions mirrors common ways of writing equivalent generators currently. -Moreover, these subtle differences are unlikely to be impactful for the common -use case of combining simple collections. +Moreover, these subtle differences are unlikely to be impactful for common use +cases (for example, there is no difference for the likely most-common use case +of combining simple collections). As suggested above, this decision should be revisited in the event that asynchronous generators receive support for ``yield from`` in the future, in From 4059f341397b9da1fb3e0122be424e4c1636d804 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sat, 6 Sep 2025 23:21:52 -0400 Subject: [PATCH 15/16] grammatical fix --- peps/pep-0798.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index 1b4236a1ca3..a8b9b9af30c 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -926,7 +926,7 @@ One of the common questions about the semantics outlined above had to do with the difference between using ``yield from`` when unpacking inside of a generator expression, versus using an explicit loop. Because this is a fairly-advanced feature of generators, this appendix attempts to summarize some -of the key differences between generators that use ``yield from`` versus those +of the key differences between generators that use ``yield from`` and those that use explicit loops. Basic Behavior From 8ec9dcdcae563cc3e55ae5346bcfef94200fd7e4 Mon Sep 17 00:00:00 2001 From: adam j hartz Date: Sun, 7 Sep 2025 08:51:47 -0400 Subject: [PATCH 16/16] one more clarifying sentence before the generator examples --- peps/pep-0798.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/peps/pep-0798.rst b/peps/pep-0798.rst index a8b9b9af30c..7fea147628a 100644 --- a/peps/pep-0798.rst +++ b/peps/pep-0798.rst @@ -955,7 +955,9 @@ Advanced Generator Protocol Differences The differences become apparent when using the advanced generator protocol methods ``.send()``, ``.throw()``, and ``.close()``, and when the sub-iterables -are themselves generators rather than simple sequences. +are themselves generators rather than simple sequences. In these cases, the +``yield from`` version results in the associated signal reaching the +subgenerator, but the version with the explicit loop does not. Delegation with ``.send()`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^