PEP 798: Add Discussion of Alternative GenExp Semantics

[adqm]

The main change here is adjusting the proposed semantics of unpacking in generator expressions and adding a section briefly outlining the pros and cons of the alternatives, but I also:

• added a link to a Reddit thread I stumbled upon, which was quite positive about the PEP (not sure if this is actually worth including or not), and
• made several small wording/grammar/spelling changes.

---

📚 Documentation preview 📚: https://pep-previews--4547.org.readthedocs.build/

[adqm]

Just a heads-up that I probably won't have my updates in tonight, since I'm still going back and forth in my head about what the right decision is here...

In case it's of interest, here's what I'm currently thinking:

• Even though we're likely talking about a niche use case, one thing I'd like to make sure is that we're setting things up for reasonable future upgrades to be backward-compatible with the details being proposed here.
• Right now, the version with yield from and the version with the explicit loop both potentially produce output even when we .send to them, but different outputs; so a future change to add this delegation functionality would change the behavior of any code using .send on one of these things. An error would be preferable from the perspective of future-proofing.

So I think I might actually be slowly convincing myself that the option you voted for in the poll might be the right choice 😄. That way we could use yield from in the sync case, add an explicit error message when someone tries to use unpacking in an async genexp, and let someone else decide what to do about the async situation later on (ideally, once we have proper delegation for async generators).

But, tl;dr: I'm going to need to think about this a bit more (and probably follow up in the Discourse thread, too).

[JelleZijlstra]

There's no hurry! The only sort-of-deadline is the Python 3.15 feature freeze next May, and we have plenty of time before that.

Here's some thoughts from me, but I don't have firm views either.
As you implied, the thing about picking one set of semantics is that backwards compatibility will mean that we'll have a hard time changing the behavior later. That's what makes it appealing to make cases where we're not sure of the right semantics into an error, since we can change our mind later about code that currently throws an error.

yield from vs. for ... in ...: yield matters here only if you call .send() or .throw() on the generator object created by the genexp. As far as I can tell, there's currently (pre-PEP 798) no good reason to call those methods on a generator created for a genexp: the value sent by .send() is ignored, and the exception thrown by .throw() is just thrown back into the calling scope.

With PEP 798 unpacking and yield from semantics, that changes and you could do theoretically useful things with .send() on genexps, like this:

>>> def make_gen(i):
...         val = yield "input:"
...         yield val * i
...         
>>> g = (*make_gen(i) for i in range(5, 10))
>>> 
>>> g.send(None)
'input:'
>>> g.send(1)
5
>>> g.send(None)
'input:'
>>> g.send(3)
18
>>> g.send(None)
'input:'
>>> g.send(4)
28

Well, not that practically useful, but maybe you'd be able to come up with a way to use this that is actually good for something useful.

This provides an argument that it's reasonable to go with for ... in: yield semantics now (either just for async genexps or for both), and switch to yield from later: the only behavior we'd change is around things that nobody has a good reason to do right now anyway. Still, people may do things even if they don't have a good reason to (or there might be a good reason that I'm not clever enough to think of), so compatibility might still be an issue.

[adqm]

Thanks for the input! I agree on all fronts.

And actually, the need to repeatedly send Nones at the right times to kickstart the internal generators (which I somehow hadn't thought of) seems to make the delegation even less useful in this context than I had thought it might be (and I already thought it wasn't likely to be terribly useful). So maybe it's the case the case that anyone wanting that kind of delegation should (and/or would want to) write it out long-hand instead of using a genexp anyway...

[JelleZijlstra]

the need to repeatedly send Nones at the right times to kickstart the internal generators

I think you don't actually need to send Nones for every one of them, e.g. this worked:

>>> g.send(4)
28
>>> g.send(42)
'input:'
>>> g.send(43)
344

But it's definitely a complicated interface to work with.

[adqm]

I think you don't actually need to send Nones for every one of them

Ah, sorry, yes, but every so often one thing that you send will effectively be ignored, right? That is, the output above would have been the same with any other object in place of 42 there.

[JelleZijlstra]

True, though I could have gotten it out if I had made the inner generator do something with the result of the second yield.

[adqm]

Oh, interesting. Yeah, you're right. Sorry; I was misunderstanding what was actually happening there!

[adqm]

After a lot of waffling back and forth, I decided to roll back the semantic changes here. I do think there's limited utility in delegating to subgenerators in this context, but I'm not sure that that's worth excluding it as a possibility.

No rush at all, but if you don't mind taking a look at the updated version sometime, @JelleZijlstra, feedback would be very welcome :)

[JelleZijlstra]

Thanks, spotted one typo.

[adqm]

Thanks! I fixed that one and another typo I found, and I added one more sentence to the Backwards Compatibility section trying to say that even though we're setting ourselves up for a backwards-incompatible change in the future, it's likely not a big deal.

[JelleZijlstra]

It might be worth adding some more detailed discussion of the actual change in behavior you get between using yield from and an implicit loop, something like the example we discussed above. I think it's not obvious what the actual difference is unless you are very into generators.

[adqm]

Good call. I'll see if I can add a small example today. I'm a little worried that the PEP is already quite long for proposing a relatively small change, but I agree that that would be a good addition.

[JelleZijlstra]

It might be a good fit for an appendix, like the two I put in PEP-800.

[adqm]

Sorry to be so slow on this, @JelleZijlstra! I just added an appendix with some examples of the differences between yield from and explicit loops, along with a few other small wording tweaks. As always, happy for more feedback if you have a chance to take a look!

[JelleZijlstra]

thank you!

[adqm]

Thanks, Jelle!