Python: Tuning auto sample validation workflow

[TaoChenOSU]

Motivation and Context

The current sample auto validation workflow is just a placeholder so that we can test the workflow in GH actions.

Description

This PR makes the workflow works and improves the sample validation scripts.

We have many failing samples. Will fix in another PR.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

[Copilot]

Pull request overview

This PR tunes the Python sample validation workflow by consolidating setup steps into a reusable composite action, standardizing configuration across jobs, and improving report output visibility. The changes aim to streamline CI/CD for automated sample testing using GitHub Copilot agents. Two complex sample categories (04-hosting and 05-end-to-end) are temporarily disabled due to their complexity.

Changes:

• Created a new composite action (sample-validation-setup) that consolidates Node.js setup, Copilot CLI installation, Azure login, and Python setup into a single reusable step
• Standardized environment variable configuration across all validation jobs, moving from secrets to vars for most configuration values
• Enhanced report output by adding result sorting (failures first) and JSON output for better GitHub Actions visibility

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
.github/workflows/python-sample-validation.yml	Updated all validation jobs to use the new composite action, standardized environment variables from secrets to vars, added global Copilot configuration, and disabled complex sample jobs
.github/actions/sample-validation-setup/action.yml	New composite action that consolidates environment setup including Node.js, Copilot CLI, Azure login, and Python installation
python/samples/_sample_validation/report.py	Added sorting to prioritize failures/errors in reports and JSON output for GitHub Actions visibility
python/samples/_sample_validation/create_dynamic_workflow_executor.py	Improved agent instructions for clarity and corrected punctuation

Comments suppressed due to low confidence (1)

.github/workflows/python-sample-validation.yml:273

• Inconsistent GitHub variable references for OpenAI configuration. This job uses vars.OPENAI_CHAT_MODEL_ID and vars.OPENAI_RESPONSES_MODEL_ID (lines 272-273), while the validate-autogen-migration job uses vars.OPENAI_CHAT_MODEL_NAME and vars.OPENAI_REASONING_MODEL_NAME (lines 231-232). Both are setting the same environment variables (OPENAI_CHAT_MODEL_ID and OPENAI_RESPONSES_MODEL_ID), so they should reference the same GitHub variables. Additionally, other integration test workflows in this repository consistently use vars.OPENAI__CHATMODELID and vars.OPENAI__RESPONSESMODELID. All jobs should use the same variable names from GitHub vars.

      OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI_CHAT_MODEL_ID }}
      OPENAI_RESPONSES_MODEL_ID: ${{ vars.OPENAI_RESPONSES_MODEL_ID }}

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/azurefunctions/agent_framework_azurefunctions
_app.py	500	172	65%	261–262, 265, 267–269, 275, 277–280, 282–283, 285–287, 291, 294, 296, 298–299, 302–304, 306, 308, 316, 324–327, 330, 333, 338–339, 342–345, 348, 351–353, 364–367, 373, 375, 383–384, 387, 392–393, 395–396, 398, 401, 404, 406, 408, 410–412, 416–419, 421, 423–424, 426, 437–439, 443–444, 446–447, 449, 460–465, 473, 475, 481–483, 489–490, 492–493, 495–498, 502, 508, 527, 626–627, 735, 743–744, 764–766, 772–774, 780–782, 815–816, 876–877, 926–927, 932, 1014, 1017, 1026–1028, 1030–1032, 1034, 1036, 1047, 1049–1052, 1054, 1056–1057, 1059, 1066–1067, 1069–1070, 1072–1073, 1075, 1079, 1089–1091, 1093–1094, 1096–1098, 1105, 1107–1108, 1110, 1131, 1136, 1148, 1220, 1310, 1325–1328, 1353
packages/core/agent_framework
_types.py	1024	86	91%	59, 68–69, 123, 128, 147, 149, 153, 157, 159, 161, 163, 181, 185, 211, 233, 238, 243, 247, 273, 277, 632–633, 1004, 1066, 1083, 1101, 1106, 1124, 1134, 1151–1152, 1154, 1172–1173, 1175, 1182–1183, 1185, 1220, 1231–1232, 1234, 1272, 1499, 1551, 1642–1647, 1669, 1674, 1840, 1852, 2104, 2125, 2220, 2449, 2656, 2726, 2738, 2756, 2954–2956, 2959–2961, 2965, 2970, 2974, 3058–3060, 3089, 3143, 3162–3163, 3166–3170, 3176
packages/core/agent_framework/_workflows
_workflow.py	267	19	92%	87, 268–270, 272–273, 291, 295, 429, 617, 638, 694, 706, 712, 717, 737–739, 752
TOTAL	22215	2760	87%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
4707	247 💤	0 ❌	0 🔥	1m 20s ⏱️