-
Notifications
You must be signed in to change notification settings - Fork 21
Description
Summary
Once Python 3.10 support is dropped, several places in the codebase can benefit from asyncio.TaskGroup (introduced in Python 3.11) for proper structured concurrency.
High Priority
src/apify/request_loaders/_apify_request_list.py:145-161 — Remote URL fetching
Current code uses create_task + add_done_callback + gather. The callbacks spawn fire-and-forget asyncio.create_task calls inside lambdas — those secondary tasks are never awaited. If create_requests_from_response fails, the error is silently lost.
Wins:
- Eliminates fire-and-forget tasks that can silently swallow errors.
- All tasks (fetch + process) are managed under one scope.
- Errors propagate properly via
ExceptionGroup.
Medium Priority
src/apify/_actor.py:1187 — Reboot listener dispatch
asyncio.gather fires persist-state and migrating listeners concurrently. If one listener raises, the others are left running unmanaged.
Wins:
- All listener tasks are cancelled on first failure instead of being left dangling.
- Produces an
ExceptionGroupwith all errors, improving debuggability.
Low Priority
src/apify/events/_apify_event_manager.py:76 — Background WebSocket task
A long-running create_task with manual cancel + suppress(CancelledError) in __aexit__. Could be replaced by a TaskGroup owned by the context manager, but the task must outlive __aenter__, so it would require restructuring the class.
Wins:
- Cleaner lifecycle management (no manual cancel/suppress boilerplate).
- Requires class restructuring, so benefit-to-effort ratio is lower.
Not Applicable
src/apify/scrapy/_async_thread.py:103-113 — Shutdown tasks
Cancels pre-existing tasks and awaits them with gather(*tasks, return_exceptions=True). TaskGroup is for spawning new tasks, not for wrangling already-running ones. Leave as-is.