RuntimeError on async generator cleanup due to task group context mismatch

[deleon626]

Bug Report: RuntimeError on async generator cleanup due to task group context mismatch

Summary

The Claude Agent SDK throws a RuntimeError: Attempted to exit cancel scope in a different task than it was entered in during cleanup when using the query() async generator. The functionality works correctly, but the error appears during shutdown.

Environment

• Python Version: 3.13
• claude-agent-sdk Version: (latest from pip)
• anyio Version: (dependency of SDK)
• OS: macOS Darwin 24.6.0

Reproduction

Minimal Example

import asyncio
from claude_agent_sdk import query, ClaudeAgentOptions, ResultMessage

async def main():
    options = ClaudeAgentOptions(
        model="claude-sonnet-4-20250514",
        output_format={"type": "json_schema", "schema": {"type": "object"}}
    )

    async for message in query(prompt="Say hello", options=options):
        if isinstance(message, ResultMessage):
            print(message.result)
            break  # Exit early after getting result

if __name__ == "__main__":
    asyncio.run(main())

Error Output

Task exception was never retrieved
future: <Task finished name='Task-5' coro=<<async_generator_athrow without __name__>()> exception=RuntimeError('Attempted to exit cancel scope in a different task than it was entered in')>
Traceback (most recent call last):
  File ".../claude_agent_sdk/_internal/client.py", line 121, in process_query
    yield parse_message(data)
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../claude_agent_sdk/_internal/client.py", line 124, in process_query
    await query.close()
  File ".../claude_agent_sdk/_internal/query.py", line 609, in close
    await self._tg.__aexit__(None, None, None)
  File ".../anyio/_backends/_asyncio.py", line 783, in __aexit__
    return self.cancel_scope.__exit__(exc_type, exc_val, exc_tb)
  File ".../anyio/_backends/_asyncio.py", line 457, in __exit__
    raise RuntimeError(
        "Attempted to exit cancel scope in a different task than it was entered in"
    )
RuntimeError: Attempted to exit cancel scope in a different task than it was entered in

Root Cause Analysis

The issue is in the manual management of the anyio task group lifecycle in query.py:

Current Implementation

In query.py:160-165 (start method):

async def start(self) -> None:
    """Start reading messages from transport."""
    if self._tg is None:
        self._tg = anyio.create_task_group()
        await self._tg.__aenter__()  # Entered in context A
        self._tg.start_soon(self._read_messages)

In query.py:602-610 (close method):

async def close(self) -> None:
    """Close the query and transport."""
    self._closed = True
    if self._tg:
        self._tg.cancel_scope.cancel()
        with suppress(anyio.get_cancelled_exc_class()):
            await self._tg.__aexit__(None, None, None)  # Exited in context B
    await self.transport.close()

Why It Fails

1. __aenter__() is called in the process_query() generator context when query.start() is invoked
2. When the user breaks out of the async for loop early (or when asyncio.run() shuts down), Python cleans up the generator
3. The finally block in client.py:123-124 calls query.close()
4. This cleanup happens in a different async task context than where __aenter__() was called
5. AnyIO's cancel scopes require enter/exit to happen in the same task, hence the RuntimeError

Execution Flow

client.py:106  →  query.start()
                      ↓
query.py:164   →  self._tg.__aenter__()  [Task Context A]
                      ↓
client.py:120  →  yield messages (generator running)
                      ↓
                  [User breaks loop or asyncio.run() exits]
                      ↓
client.py:124  →  query.close()  [Task Context B - cleanup task]
                      ↓
query.py:609   →  self._tg.__aexit__()  [FAILS - wrong context]

Suggested Fix

Option 1: Use proper async context manager pattern

Restructure the code to use async with instead of manual __aenter__/__aexit__:

async def start(self) -> None:
    """Start reading messages from transport."""
    if self._tg is None:
        # Don't manually call __aenter__, instead manage lifecycle differently
        pass

# Use async with in the calling code
async with anyio.create_task_group() as tg:
    query._tg = tg
    tg.start_soon(query._read_messages)
    # ... rest of logic

Option 2: Handle cleanup in the same context

Ensure close() is called within the same async context as start():

async def close(self) -> None:
    """Close the query and transport."""
    self._closed = True
    if self._tg:
        self._tg.cancel_scope.cancel()
        # Don't call __aexit__ manually - let the context manager handle it
        # Or use a flag to signal the task group to exit gracefully
    await self.transport.close()

Option 3: Use anyio.from_thread or task-safe cleanup

async def close(self) -> None:
    """Close the query and transport."""
    self._closed = True
    if self._tg:
        self._tg.cancel_scope.cancel()
        # Skip __aexit__ if we're in a different context
        try:
            await self._tg.__aexit__(None, None, None)
        except RuntimeError as e:
            if "different task" in str(e):
                pass  # Expected during generator cleanup
            else:
                raise
    await self.transport.close()

Impact

• Severity: Low (functionality works, just produces warning)
• Frequency: Every time an async generator exits before natural completion
• User Impact: Confusing error messages, potential for masking real errors

Additional Context

This issue is specific to:

• Using the query() function as an async generator
• Breaking out of the loop early (e.g., after getting ResultMessage)
• Python 3.13 with anyio's asyncio backend

The error does not occur when:

• The generator naturally exhausts all messages
• Using synchronous wrappers that handle cleanup differently

Related Issues

• Similar pattern issues in other async libraries: https://github.com/agronholm/anyio/issues/XXX
• Python async generator cleanup behavior: https://docs.python.org/3/reference/expressions.html#asynchronous-generator-functions

[brianantonelli]

Additional case: Parallel queries still trigger this bug

PR #527 fixed the basic single-query case, but we're still hitting this error when running multiple queries in parallel with asyncio.gather().

Reproduction

async def analyze_multiple(): 
   queries = [ 
       query(prompt=f"Analyze component {i}", options=...) 
       for i in range(10) 
   ] 

async def run_query(q): 
   async for message in q: 
   # Any work here, even just checking message type 
   if hasattr(message, 'structured_output'): 
       return message 
   
   return None 

This triggers the cancel scope bug

results = await asyncio.gather(*[run_query(q) for q in queries])

Error

 ERROR: Task exception was never retrieved 
 future: <Task finished name='Task-40' coro=<<async_generator_athrow without __name__>()> exception=RuntimeError('Attempted to exit cancel scope in a different 
 task than it was entered in')> 

Analysis

The issue seems to be that when multiple queries run in parallel via asyncio.gather(), each query's generator cleanup happens in a different task context than
where its cancel scope was entered. Even with PR #527 applied, the parallel execution case still triggers the anyio cancel scope error.

Workaround

We've had to avoid doing any work inside the async for loop that consumes the generator. Instead, we only interact with the query result after fully consuming the generator, which loses the streaming/real-time aspect.

A proper fix might require restructuring how the SDK manages query lifecycles, or providing a non-generator API for cases where streaming isn't needed.

[AmanKishore]

Additional findings from production harness testing

We're hitting this same bug in our live runtime reliability harness (pytest-based, subprocess-isolated per scenario). Adding context from our investigation:

Impact is worse than "Low" severity

The issue description says severity is Low because "functionality works, just produces warning." In our experience, the cancel scope leak poisons the entire event loop — once the RuntimeError fires, every subsequent await in that event loop gets CancelledError. This is not just a warning; it makes the SDK completely unusable within that process after the first occurrence.

Key observations:

• task.uncancel() only clears asyncio-level cancellations, NOT AnyIO cancel scopes
• Once leaked, the cancel scope infects all coroutines in the event loop
• Internal retry loops (e.g., re-calling run_query()) fail immediately because the event loop is poisoned
• Only process-level isolation (spawning a new subprocess) recovers from this state

Reproduction in production workloads

We see this reliably on longer-running queries (60-120s) that use MCP tool servers with 30+ tools. The pattern:

1. process_query() async generator streams events normally
2. During MCP tool execution, the CLI transport dies (ProcessTransport is not ready for writing)
3. Generator cleanup triggers the cross-task cancel scope exit
4. All subsequent retries within the same process fail with CancelledError

Workaround

Subprocess isolation per query invocation. Each retry spawns a fresh Python process with a clean event loop. This is expensive but the only reliable workaround until the SDK fix lands.

Environment

• Python 3.12.7
• claude-agent-sdk 0.1.39
• anyio 4.12.1
• macOS Darwin 25.2.0