Migrate the full codebase to structured concurreny

[burnpanck]

This is my go at https://discuss.pylabrobot.org/t/structured-concurrency/475.
With 136 changed files and about 3k changed lines, certainly not a small refactor. Luckily, the majority of them are changes ofawait asyncio.sleep to await anyio.sleep. I have reviewed every single changed line in this refactor, and many of the more intricate refactors are hand-written and agent-verified, rather than the other way around.

All tests pass locally, though unfortunately, many of the more intricate changes are in the reader loops of the hardware interfaces, which are often untested or partially replaced by mocks, so a significant testing gap remains. An earlier commit from this refactor was battle-tested on a Hamilton STAR for a couple of days, and confirmed to significantly increase robustness of the code with respect to aborted protocols due to errors and cancellation. With this refactor, you should be able to expect from your PyLabRobot liquid handling system that it resumes into a new lifespan context without a hitch, no matter how you exited the previous one.

I encourage everyone to run their own LLMs/Agents over this refactor to look for issues or changes in behavior (other than improvements with respect to structured concurrency) and for course test on your favorite piece of hardware. Usually, it should be enough to ping-pong a few commands around the hardware, if that works, you are good. The many hardware-specific commands didn't change, just the low-level firmware command handling.

Since the main branch has moved forward in the meantime, I will try to periodically merge main (every couple of days). Once it's read to merge, we can also squash the commit history of this refactor.

Things that are still TODO:

• The visualizer
• Documentation changes, docstrings and error messages referencing setup.

[burnpanck]

Uh oh, I see you test again python 3.9. Is that a hard requirement? It's EOL since last year...

[rickwierenga]

sick PR, first glance is looking good but will respond properly in a bit

Uh oh, I see you test again python 3.9. Is that a hard requirement? It's EOL since last year...

any reason to not support it? I generally want to support as many versions and configurations as possible, but if there's something super nice that requires 3.10+ LMK

[burnpanck]

any reason to not support python 3.9?

I'm not aware of anything concrete at this point, I can give it a try.

IHMO, supporting outdated versions hurts code quality, because best practices change over time. The cognitive effort to properly phase-out old versions increases significantly the older these versions get. It's a compromise that projects need to balance when choosing their support policy. Personally, I'm used to developing for PSF active support versions (i.e. ±2 versions), and only guaranteeing extended support if an essential dependency requires it. In projects with OEM vendor dependencies (PySpin, harvesters, looking at you), unfortunately, that happens often.

[rickwierenga]

supporting outdated versions hurts code quality, because best practices change over time

agree in general, but what's the concrete example here? (for example 3.8 doesn't have asyncio.to_thread)

all else being equal, might as well support 3.9

PySpin

not for much longer! thanks to @vcjdeboer #985

[burnpanck]

agree in general, but what's the concrete example here?

Now that I think of it, PEP 654 (except* and exception groups) was made specifically to support structured concurrency. My refactor doesn't use any, because there is relatively little error-recovery built-in in the code-base; but it would be the modern technical basis of ChannelizedError. It's python 3.11 though.

[burnpanck]

Finally passes all the checks. The typechecker uncovered one or two real bugs, some of which were actually already in the pre-refactor codebase (one I recall: pylabrobot.io.ftdi.FTDI accidentally inherited from io.IOBase from the standard library instead of pylabrobot.io.io.IOBase).