Skip to content

[Eval SDK] Fix argument type in agent response reformatting for evaluators #44897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
salma-elshafey wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[Eval SDK] Fix argument type in agent response reformatting for evaluators #44897

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7c75991
Fix argument type in agent response reformatting
Jan 28, 2026
e7291a5
Black
Jan 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -710,6 +710,15 @@ def reformat_conversation_history(query, logger=None, include_system_messages=Fa
return query


def _format_value(v):
if isinstance(v, str):
return f'"{v}"'
elif v is None:
return "None"
else:
return str(v)
Comment on lines 713 to 719
Copy link

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _format_value function uses str() for complex types (dicts, lists), which produces Python-style representations (e.g., {'key': 'value'} instead of JSON-style {"key": "value"}). For tool call arguments that may contain nested structures, consider whether json.dumps() would be more appropriate to ensure consistent JSON-style formatting. This would produce output like ranking_options={"ranker": "default_2024_08_21", "score_threshold": 0.0} instead of ranking_options={'ranker': 'default_2024_08_21', 'score_threshold': 0.0}.

Copilot uses AI. Check for mistakes.
Comment on lines 713 to 719
Copy link

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bug fix improves argument formatting in tool call messages, but test coverage should be expanded to verify the behavior with different argument types. Consider adding test cases to test__get_agent_response_with_tool_messages that verify formatting with:

  • Numeric arguments (integers and floats like score_threshold: 0.0)
  • Boolean arguments (e.g., enabled: true)
  • None/null values
  • Nested dictionaries (e.g., ranking_options: {"ranker": "default", "score_threshold": 0.0})

This would ensure the fix works correctly for all the data types that can appear in tool call arguments.

Copilot uses AI. Check for mistakes.
Comment on lines 713 to 719
Copy link

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _format_value helper function lacks a docstring explaining its purpose and behavior. Consider adding a docstring that describes:

  • The purpose: Format tool call argument values for display in agent response text
  • Parameter type and meaning
  • Return value and format
  • Examples of how different types are formatted (strings, numbers, None, booleans, complex types)

This would improve code maintainability and help other developers understand the formatting rules.

Copilot uses AI. Check for mistakes.
Comment on lines 713 to 719
Copy link

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _format_value function handles booleans using str(), which produces Python-style capitalized boolean strings ("True" or "False"). Consider whether lowercase booleans ("true" or "false") would be more appropriate for consistency with JSON conventions, especially since tool call arguments often originate from JSON data. If the current behavior is intentional for Python-style output, this is acceptable.

Copilot uses AI. Check for mistakes.


def _get_agent_response(agent_response_msgs, include_tool_messages=False):
"""Extracts formatted agent response including text, and optionally tool calls/results."""
agent_response_text = []
Expand Down Expand Up @@ -743,7 +752,7 @@ def _get_agent_response(agent_response_msgs, include_tool_messages=False):
tool_call_id = content.get("tool_call_id")
func_name = content.get("name", "")
args = content.get("arguments", {})
args_str = ", ".join(f'{k}="{v}"' for k, v in args.items())
args_str = ", ".join(f"{k}={_format_value(v)}" for k, v in args.items())
call_line = f"[TOOL_CALL] {func_name}({args_str})"
agent_response_text.append(call_line)
if tool_call_id in tool_results:
Expand Down