Skip to content

Python: Testing agent-framework WorkflowAgent with Checkpoint and HITL #3245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lusu-msft wants to merge 2 commits into from
Closed

Python: Testing agent-framework WorkflowAgent with Checkpoint and HITL #3245

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
495e4a3
test workflow agent with checkpoint and hitl
lusu-msft Jan 15, 2026
2bdfaf6
remove unrelated change
lusu-msft Jan 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 53 additions & 4 deletions python/samples/getting_started/workflows/agents/workflow_as_agent_human_in_the_loop.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
WorkflowContext,
handler,
response_handler,
InMemoryCheckpointStorage,
)
from getting_started.workflows.agents.workflow_as_agent_reflection_pattern import ( # noqa: E402
ReviewRequest,
Expand Down Expand Up @@ -124,14 +125,46 @@ async def main() -> None:
print("Query: 'Write code for parallel reading 1 million files on disk and write to a sorted output file.'")
print("-" * 50)

checkpoint_repo = {} # cache for checkpoints

checkpoint_storage_1 = InMemoryCheckpointStorage()
checkpoint_repo["conv1"] = checkpoint_storage_1
Comment on lines +128 to +131
Copy link

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The checkpoint_repo dictionary seems to be introduced for managing multiple conversations, but given that the second conversation fails and should be removed, this pattern adds unnecessary complexity to the sample. If only one conversation is being demonstrated, the dictionary wrapper is not needed - just use checkpoint_storage_1 directly.

Copilot uses AI. Check for mistakes.

# Run the agent with an initial query.
response = await agent.run(
"Write code for parallel reading 1 million Files on disk and write to a sorted output file."
response1 = await agent.run(
"Write code for parallel reading 1 million Files on disk and write to a sorted output file.",
checkpoint_storage=checkpoint_storage_1,
)
print("\n\nconversation 1 completed, waiting for human in the loop response...\n\n")

# A new conversation with blank checkpoint storage
checkpoint_storage_2 = InMemoryCheckpointStorage()
checkpoint_repo["conv2"] = checkpoint_storage_2
response2 = await agent.run(
"A new conversation after human in the loop",
checkpoint_storage=checkpoint_storage_2,
)

"""
the workflow fails here with the following error:
Traceback (most recent call last):
.........
File "....\site-packages\agent_framework\_workflows\_agent.py", line 157, in run
async for update in self._run_stream_impl(
File "....\site-packages\agent_framework\_workflows\_agent.py", line 246, in _run_stream_impl
function_responses = self._extract_function_responses(input_messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "....\site-packages\agent_framework\_workflows\_agent.py", line 426, in _extract_function_responses
raise AgentExecutionException("Unexpected content type while awaiting request info responses.")
agent_framework.exceptions.AgentExecutionException: Unexpected content type while awaiting request info responses.
"""
Comment on lines +140 to +160
Copy link

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code attempts to start a second conversation with a separate checkpoint storage while the first conversation has pending request_info responses. According to the documented error and the code logic in _agent.py, this is not supported - the agent expects only function responses when there are pending requests. This test case appears to be testing an unsupported scenario. Consider removing this test case or documenting that starting new conversations while pending requests exist is not currently supported.

Suggested change
# A new conversation with blank checkpoint storage
checkpoint_storage_2 = InMemoryCheckpointStorage()
checkpoint_repo["conv2"] = checkpoint_storage_2
response2 = await agent.run(
"A new conversation after human in the loop",
checkpoint_storage=checkpoint_storage_2,
)
"""
the workflow fails here with the following error:
Traceback (most recent call last):
.........
File "....\site-packages\agent_framework\_workflows\_agent.py", line 157, in run
async for update in self._run_stream_impl(
File "....\site-packages\agent_framework\_workflows\_agent.py", line 246, in _run_stream_impl
function_responses = self._extract_function_responses(input_messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "....\site-packages\agent_framework\_workflows\_agent.py", line 426, in _extract_function_responses
raise AgentExecutionException("Unexpected content type while awaiting request info responses.")
agent_framework.exceptions.AgentExecutionException: Unexpected content type while awaiting request info responses.
"""
# NOTE:
# Starting a new conversation with a separate checkpoint storage while the
# first conversation has pending `request_info` (human-in-the-loop) responses
# is not currently supported by the agent framework.
#
# Attempting to call `agent.run` again at this point with a different
# `checkpoint_storage` instance will result in:
# AgentExecutionException("Unexpected content type while awaiting request info responses.")
#
# If you need to run multiple conversations, ensure that all human-in-the-loop
# interactions for the first conversation are fully completed before starting
# another conversation, or serialize the conversations so that only one has
# pending `request_info` responses at any given time.

Copilot uses AI. Check for mistakes.
Comment on lines +147 to +160
Copy link

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This multi-line comment documenting a known failure should not be included in a sample file. Sample code should demonstrate working functionality, not document failing test cases. If this is intended as a bug report or investigation, it should be tracked in an issue, not committed to a sample file. Consider removing the second conversation test entirely or fixing the underlying issue before including it in the samples.

Suggested change
"""
the workflow fails here with the following error:
Traceback (most recent call last):
.........
File "....\site-packages\agent_framework\_workflows\_agent.py", line 157, in run
async for update in self._run_stream_impl(
File "....\site-packages\agent_framework\_workflows\_agent.py", line 246, in _run_stream_impl
function_responses = self._extract_function_responses(input_messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "....\site-packages\agent_framework\_workflows\_agent.py", line 426, in _extract_function_responses
raise AgentExecutionException("Unexpected content type while awaiting request info responses.")
agent_framework.exceptions.AgentExecutionException: Unexpected content type while awaiting request info responses.
"""

Copilot uses AI. Check for mistakes.

print("\n\nconversation 2 completed")
print(f"📤 Agent Response: {response2}")

Comment on lines +141 to 164
Copy link

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is unreachable because the execution fails at line 143-146 as documented in the comment above. This unreachable code should be removed along with the failed test case.

Suggested change
checkpoint_storage_2 = InMemoryCheckpointStorage()
checkpoint_repo["conv2"] = checkpoint_storage_2
response2 = await agent.run(
"A new conversation after human in the loop",
checkpoint_storage=checkpoint_storage_2,
)
"""
the workflow fails here with the following error:
Traceback (most recent call last):
.........
File "....\site-packages\agent_framework\_workflows\_agent.py", line 157, in run
async for update in self._run_stream_impl(
File "....\site-packages\agent_framework\_workflows\_agent.py", line 246, in _run_stream_impl
function_responses = self._extract_function_responses(input_messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "....\site-packages\agent_framework\_workflows\_agent.py", line 426, in _extract_function_responses
raise AgentExecutionException("Unexpected content type while awaiting request info responses.")
agent_framework.exceptions.AgentExecutionException: Unexpected content type while awaiting request info responses.
"""
print("\n\nconversation 2 completed")
print(f"📤 Agent Response: {response2}")
# Note: a second conversation example that previously attempted to continue
# after human-in-the-loop has been removed due to a known AgentExecutionException
# in the underlying agent framework, which caused the code below to be unreachable.

Copilot uses AI. Check for mistakes.
# Locate the human review function call in the response messages.
human_review_function_call: FunctionCallContent | None = None
for message in response.messages:
for message in response1.messages:
for content in message.contents:
if isinstance(content, FunctionCallContent) and content.name == WorkflowAgent.REQUEST_INFO_FUNCTION_NAME:
human_review_function_call = content
Expand Down Expand Up @@ -166,8 +199,24 @@ async def main() -> None:
call_id=human_review_function_call.call_id,
result=human_response,
)

checkpoint_storage = checkpoint_repo.get("conv1")
checkpoints = await checkpoint_storage.list_checkpoints()
print(f"Available checkpoints: {checkpoints}")
last_checkpoint = checkpoints[-1] if checkpoints else None

if last_checkpoint:
# load checkpoint before sending back human review
response = await agent.run(
Copy link

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assignment to 'response' is unnecessary as it is redefined before this value is used.

Suggested change
response = await agent.run(
await agent.run(

Copilot uses AI. Check for mistakes.
checkpoint_id=last_checkpoint.checkpoint_id,
checkpoint_storage=checkpoint_storage,
)

Comment on lines +208 to +214
Copy link

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading the checkpoint before sending the human review response appears unnecessary. The checkpoint should already contain the pending request state. This extra call to agent.run() doesn't pass any new messages or responses, so it may not serve a clear purpose. Consider clarifying why this step is needed or removing it if it's redundant.

Suggested change
if last_checkpoint:
# load checkpoint before sending back human review
response = await agent.run(
checkpoint_id=last_checkpoint.checkpoint_id,
checkpoint_storage=checkpoint_storage,
)

Copilot uses AI. Check for mistakes.
# Send the human review result back to the agent.
response = await agent.run(ChatMessage(role=Role.TOOL, contents=[human_review_function_result]))
response = await agent.run(
ChatMessage(role=Role.TOOL, contents=[human_review_function_result]),
checkpoint_storage=checkpoint_storage,
)
print(f"📤 Agent Response: {response.messages[-1].text}")

print("=" * 50)
Expand Down
Loading