add appendix about advanced generator usage, clean up wording about alternatives

Author: adqm

peps/pep-0798.rst

@@ -742,11 +742,12 @@ do not support ``yield from`` (see the section of :pep:`525` on Asynchronous
 The core question centered around whether sync and async generator expressions
 should use ``yield from`` (or an equivalent) when unpacking, as opposed to an
 explicit loop.  The main difference between these options is whether the
-resulting generator delegates to the objects being unpacked (see :pep:`380`),
-which would affect the behavior of these generator expressions when used with
-``.send()/.asend()``, ``.throw()/.athrow()``, and ``.close()/.aclose()`` in
-the case where the objects being unpacked are themselves generators, which
-is unlikely to be a common use case.
+resulting generator delegates to the objects being unpacked, which would affect
+the behavior of these generator expressions when used with
+``.send()/.asend()``, ``.throw()/.athrow()``, and ``.close()/.aclose()`` in the
+case where the objects being unpacked are themselves generators.  The
+differences between these options are summarized in
+:ref:`pep798-appendix-yieldfrom`.
 
 Several reasonable options were considered, none of which was a clear winner in
 a `poll in the Discourse thread
@@ -759,21 +760,22 @@ Beyond the proposal outlined above, the following were also considered:
    This strategy would have resulted in a symmetry between synchronous and
    asynchronous generator expressions but would have prevented a
    potentially-useful tool by disallowing delegation in the case of synchronous
-   generator expressions.  Moreover, the existing asymmetry between synchronous
-   and asynchronous generators mitigates concerns about asymmetry of the
-   unpacking operator within generator expressions.
+   generator expressions.  One specific concern with this approach is the
+   introduction of an asymmetry between synchronous and asynchronous
+   generators, but this concern is mitigated by the fact that these asymmetries
+   already exist between synchronous and asynchronous generators more
+   generally.
 
 2. Using ``yield from`` for unpacking in synchronous generator expressions and
    mimicking the behavior of ``yield from`` for unpacking in async generator
    expressions.
 
    This strategy would also make unpacking in synchronous and asynchronous
    generators behave symmetrically, but it would also be more complex, enough
-   so that the cost may not be worth the benefit, particularly in the absence
-   of a compelling practical use case for delegating to subgenerators during
-   unpacking.  Generator expressions using the unpacking operator should not
-   use semantics similar to ``yield from`` until ``yield from`` is supported
-   in asynchronous generators more generally.
+   so that the cost may not be worth the benefit.  As such, this PEP proposes
+   that generator expressions using the unpacking operator should not use
+   semantics similar to ``yield from`` until ``yield from`` is supported in
+   asynchronous generators more generally.
 
 3. Using ``yield from`` for unpacking in synchronous generator expressions, and
    disallowing unpacking in asynchronous generator expressions until they
@@ -783,8 +785,8 @@ Beyond the proposal outlined above, the following were also considered:
    expressions do gain support for ``yield from`` in the future by making sure
    that any decision made at that point would be fully backwards-compatible.
    But the utility of unpacking in that context seems to outweigh the potential
-   downside of a backwards-incompatible change in the future if async generator
-   expressions do receive support for ``yield from``.
+   downside of a minimally-invasive backwards-incompatible change in the future
+   if async generator expressions do receive support for ``yield from``.
 
 4. Disallowing unpacking in all generator expressions.
 
@@ -793,13 +795,14 @@ Beyond the proposal outlined above, the following were also considered:
 
 
 Each of these options (including the one presented in this PEP) has its
-benefits and drawbacks, with no option being clearly superior on all fronts;
-but the semantics proposed in :ref:`pep798-genexpsemantics` represent a
-reasonable compromise where unpacking in both synchronous and asynchronous
-generator expressions mirrors common ways of writing equivalent generators
-currently.
-
-As suggested above, though, this decision should be revisited in the event that
+benefits and drawbacks, with no option being clearly superior on all fronts.
+The semantics proposed in :ref:`pep798-genexpsemantics` represent a reasonable
+compromise where unpacking in both synchronous and asynchronous generator
+expressions mirrors common ways of writing equivalent generators currently.
+Moreover, these subtle differences are unlikely to be impactful for the common
+use case of combining simple collections.
+
+As suggested above, this decision should be revisited in the event that
 asynchronous generators receive support for ``yield from`` in the future, in
 which case adjusting the semantics of unpacking in async generator expressions
 to use ``yield from`` should be considered.
@@ -842,8 +845,9 @@ were raised as well. This section aims to summarize those concerns.
   for maintainers of code formatters, linters, type checkers, etc., to make
   sure that the new syntax is supported.
 
-Other Languages
-===============
+
+Appendix: Other Languages
+=========================
 
 Quite a few other languages support this kind of flattening with syntax similar
 to what is already available in Python, but support for using unpacking syntax
@@ -912,6 +916,149 @@ Civet, making use of JavaScript's ``...`` syntax for unpacking:
 
     for xs of [[1,2,3], [], [4,5]] then ...(xs++xs)
 
+.. _pep798-appendix-yieldfrom:
+
+Appendix: Semantics of Generator Delegation
+===========================================
+
+One of the common questions about the semantics outlined above had to do with
+the difference between using ``yield from`` when unpacking inside of a
+generator expression, versus using an explicit loop.  Because this is a
+fairly-advanced feature of generators, this appendix attempts to summarize some
+of the key differences between generators that use ``yield from`` versus those
+that use explicit loops.
+
+Basic Behavior
+--------------
+
+For simple iteration over values, which we expect to be by far the most-common
+use of unpacking in generator expressions,  both approaches produce identical
+results::
+
+    def yield_from(iterables):
+        for iterable in iterables:
+            yield from iterable
+
+    def explicit_loop(iterables):
+        for iterable in iterables:
+            for item in iterable:
+                yield item
+
+    # Both produce the same sequence of values
+    x = list(yield_from([[1, 2], [3, 4]]))
+    y = list(explicit_loop([[1, 2], [3, 4]]))
+    print(x == y)  # prints True
+
+Advanced Generator Protocol Differences
+---------------------------------------
+
+The differences become apparent when using the advanced generator protocol
+methods ``.send()``, ``.throw()``, and ``.close()``, and when the sub-iterables
+are themselves generators rather than simple sequences.
+
+Delegation with ``.send()``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code:: python
+
+    def sub_generator():
+        x = yield "first"
+        yield f"received: {x}"
+        yield "last"
+
+    def yield_from():
+        yield from sub_generator()
+
+    def explicit_loop():
+        for item in sub_generator():
+            yield item
+
+    # With yield from, values are passed through to sub-generator
+    gen1 = yield_from()
+    print(next(gen1))  # prints "first"
+    print(gen1.send("hello"))  # prints "received: hello"
+    print(next(gen1))  # prints "last"
+
+    # With explicit loop, .send() affects the outer generator; values don't reach the sub-generator
+    gen2 = explicit_loop()
+    print(next(gen2))  # prints "first"
+    print(gen2.send("hello"))  # prints "received: None" (sub-generator receives None instead of "hello")
+    print(next(gen2))  # prints "last"
+
+Exception Handling with ``.throw()``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: python
+
+    def sub_generator_with_exception_handling():
+        try:
+            yield "first"
+            yield "second"
+        except ValueError as e:
+            yield f"caught: {e}"
+
+    def yield_from():
+        yield from sub_generator_with_exception_handling()
+
+    def explicit_loop():
+        for item in sub_generator_with_exception_handling():
+            yield item
+
+    # With yield from, exceptions are passed to sub-generator
+    gen1 = yield_from()
+    print(next(gen1))  # prints "first"
+    print(gen1.throw(ValueError("test")))  # prints "caught: test"
+
+    # With explicit loop, exceptions affect the outer generator only
+    gen2 = explicit_loop()
+    print(next(gen2))  # prints "first"
+    print(gen2.throw(ValueError("test")))  # ValueError is raised; sub-generator doesn't see it
+
+Generator Cleanup with ``.close()``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: python
+
+    # hold references to sub-generators so GC doesn't close the explicit loop version
+    references = []
+
+    def sub_generator_with_cleanup():
+        try:
+            yield "first"
+            yield "second"
+        finally:
+            print("sub-generator received GeneratorExit")
+
+    def yield_from():
+        try:
+            g = sub_generator_with_cleanup()
+            references.append(g)
+            yield from g
+        finally:
+            print("outer generator received GeneratorExit")
+
+    def explicit_loop():
+        try:
+            g = sub_generator_with_cleanup()
+            references.append(g)
+            for item in g:
+                yield item
+        finally:
+            print("outer generator received GeneratorExit")
+
+    # With yield from - GeneratorExit is passed through to sub-generator
+    gen1 = yield_from()
+    print(next(gen1))  # prints "first"
+    gen1.close()  # closes sub-generator and then outer generator
+
+    # With explicit loop - GeneratorExit goes to outer generator first
+    gen2 = explicit_loop()
+    print(next(gen2))  # prints "first"
+    gen2.close()  #only closes outer generator
+
+    print('program finished; GC will close the explicit loop subgenerator')
+    # second inner generator closes when GC closes it at the end
+
+
 References
 ==========