Skip to content

Commit c45d47d

Browse files
TaoChenOSUTaoChenOSU
authored
Python: Tuning auto sample validation workflow (#4218)
* Tuning validate-01-get-started * Add gh token * Add model * enable debug log * bump up timeout for testing purposes * Test cli is working * Fix end quote * Run gh auth * Run gh auth trail 2 * Run gh auth trail 3 * Test token * Add zcure login * Add zcure login 2 * Add zcure login 3 * Add zcure login 4 * Extract common actions * Extract common actions 2 * Correct env vars * Print outputs to action console * Disable end-to-end samples * Fix ruff errors * Fix ruff errors 2 * Revert workflow changes to fix tests * Revert workflow changes to fix tests 2 * Revert workflow changes to fix tests 3 * Revert workflow changes to fix tests 4
1 parent 54c0bea commit c45d47d

9 files changed

Lines changed: 171 additions & 112 deletions

File tree

‎.github/actions/sample-validation-setup/action.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: Sample Validation Setup
2+
description: Sets up the environment for sample validation (checkout, Node.js, Copilot CLI, Azure login, Python)
3+
4+
inputs:
5+
azure-client-id:
6+
description: Azure Client ID for OIDC login
7+
required: true
8+
azure-tenant-id:
9+
description: Azure Tenant ID for OIDC login
10+
required: true
11+
azure-subscription-id:
12+
description: Azure Subscription ID for OIDC login
13+
required: true
14+
python-version:
15+
description: The Python version to set up
16+
required: false
17+
default: "3.12"
18+
os:
19+
description: The operating system to set up
20+
required: false
21+
default: "Linux"
22+
23+
runs:
24+
using: "composite"
25+
steps:
26+
- name: Set up Node.js environment
27+
uses: actions/setup-node@v4
28+
29+
- name: Install Copilot CLI
30+
shell: bash
31+
run: npm install -g @github/copilot
32+
33+
- name: Test Copilot CLI
34+
shell: bash
35+
run: copilot -p "What can you do in one sentence?"
36+
37+
- name: Azure CLI Login
38+
uses: azure/login@v2
39+
with:
40+
client-id: ${{ inputs.azure-client-id }}
41+
tenant-id: ${{ inputs.azure-tenant-id }}
42+
subscription-id: ${{ inputs.azure-subscription-id }}
43+
44+
- name: Set up python and install the project
45+
uses: ./.github/actions/python-setup
46+
with:
47+
python-version: ${{ inputs.python-version }}
48+
os: ${{ inputs.os }}

‎.github/workflows/python-sample-validation.yml‎

Lines changed: 94 additions & 96 deletions
Original file line numberDiff line numberDiff line change
@@ -8,32 +8,38 @@ on:
88
env:
99
# Configure a constant location for the uv cache
1010
UV_CACHE_DIR: /tmp/.uv-cache
11+
# GitHub Copilot configuration
12+
GITHUB_COPILOT_MODEL: claude-opus-4.6
13+
COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }}
14+
15+
permissions:
16+
contents: read
17+
id-token: write
1118

1219
jobs:
1320
validate-01-get-started:
1421
name: Validate 01-get-started
1522
runs-on: ubuntu-latest
16-
permissions:
17-
contents: read
23+
environment: integration
1824
env:
19-
# Azure AI configuration for get-started samples
20-
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZURE_AI_PROJECT_ENDPOINT }}
21-
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ secrets.AZURE_AI_MODEL_DEPLOYMENT_NAME }}
22-
# GitHub Copilot configuration
23-
GITHUB_COPILOT_MODEL: ${{ vars.GITHUB_COPILOT_MODEL }}
25+
# Required configuration for get-started samples
26+
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
27+
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
28+
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
29+
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
2430
defaults:
2531
run:
2632
working-directory: python
2733
steps:
2834
- uses: actions/checkout@v6
2935

30-
- name: Set up python and install the project
31-
uses: ./.github/actions/python-setup
36+
- name: Setup environment
37+
uses: ./.github/actions/sample-validation-setup
3238
with:
33-
python-version: "3.12"
39+
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
40+
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
41+
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
3442
os: ${{ runner.os }}
35-
env:
36-
UV_CACHE_DIR: /tmp/.uv-cache
3743

3844
- name: Run sample validation
3945
run: |
@@ -49,37 +55,34 @@ jobs:
4955
validate-02-agents:
5056
name: Validate 02-agents
5157
runs-on: ubuntu-latest
52-
permissions:
53-
contents: read
58+
environment: integration
5459
env:
5560
# Azure AI configuration
56-
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZURE_AI_PROJECT_ENDPOINT }}
57-
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ secrets.AZURE_AI_MODEL_DEPLOYMENT_NAME }}
61+
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
62+
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
5863
# Azure OpenAI configuration
59-
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
60-
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }}
61-
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME }}
64+
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
65+
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
66+
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
6267
# OpenAI configuration
6368
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
64-
OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI_CHAT_MODEL_ID }}
65-
OPENAI_RESPONSES_MODEL_ID: ${{ vars.OPENAI_RESPONSES_MODEL_ID }}
69+
OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI_CHAT_MODEL_NAME }}
70+
OPENAI_RESPONSES_MODEL_ID: ${{ vars.OPENAI_REASONING_MODEL_NAME }}
6671
# Observability
6772
ENABLE_INSTRUMENTATION: "true"
68-
# GitHub Copilot configuration
69-
GITHUB_COPILOT_MODEL: ${{ vars.GITHUB_COPILOT_MODEL }}
7073
defaults:
7174
run:
7275
working-directory: python
7376
steps:
7477
- uses: actions/checkout@v6
7578

76-
- name: Set up python and install the project
77-
uses: ./.github/actions/python-setup
79+
- name: Setup environment
80+
uses: ./.github/actions/sample-validation-setup
7881
with:
79-
python-version: "3.12"
82+
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
83+
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
84+
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
8085
os: ${{ runner.os }}
81-
env:
82-
UV_CACHE_DIR: /tmp/.uv-cache
8386

8487
- name: Run sample validation
8588
run: |
@@ -95,31 +98,28 @@ jobs:
9598
validate-03-workflows:
9699
name: Validate 03-workflows
97100
runs-on: ubuntu-latest
98-
permissions:
99-
contents: read
101+
environment: integration
100102
env:
101103
# Azure AI configuration
102-
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZURE_AI_PROJECT_ENDPOINT }}
103-
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ secrets.AZURE_AI_MODEL_DEPLOYMENT_NAME }}
104+
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
105+
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
104106
# Azure OpenAI configuration
105-
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
106-
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }}
107-
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME }}
108-
# GitHub Copilot configuration
109-
GITHUB_COPILOT_MODEL: ${{ vars.GITHUB_COPILOT_MODEL }}
107+
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
108+
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
109+
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
110110
defaults:
111111
run:
112112
working-directory: python
113113
steps:
114114
- uses: actions/checkout@v6
115115

116-
- name: Set up python and install the project
117-
uses: ./.github/actions/python-setup
116+
- name: Setup environment
117+
uses: ./.github/actions/sample-validation-setup
118118
with:
119-
python-version: "3.12"
119+
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
120+
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
121+
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
120122
os: ${{ runner.os }}
121-
env:
122-
UV_CACHE_DIR: /tmp/.uv-cache
123123

124124
- name: Run sample validation
125125
run: |
@@ -134,31 +134,31 @@ jobs:
134134

135135
validate-04-hosting:
136136
name: Validate 04-hosting
137+
if: false # Temporarily disabled because of sample complexity
137138
runs-on: ubuntu-latest
138-
permissions:
139-
contents: read
139+
environment: integration
140140
env:
141141
# Azure AI configuration
142-
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZURE_AI_PROJECT_ENDPOINT }}
143-
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ secrets.AZURE_AI_MODEL_DEPLOYMENT_NAME }}
142+
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
143+
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
144144
# Azure OpenAI configuration
145-
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
146-
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME }}
147-
# GitHub Copilot configuration
148-
GITHUB_COPILOT_MODEL: ${{ vars.GITHUB_COPILOT_MODEL }}
145+
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
146+
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
147+
# A2A configuration
148+
A2A_AGENT_HOST: http://localhost:5001/
149149
defaults:
150150
run:
151151
working-directory: python
152152
steps:
153153
- uses: actions/checkout@v6
154154

155-
- name: Set up python and install the project
156-
uses: ./.github/actions/python-setup
155+
- name: Setup environment
156+
uses: ./.github/actions/sample-validation-setup
157157
with:
158-
python-version: "3.12"
158+
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
159+
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
160+
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
159161
os: ${{ runner.os }}
160-
env:
161-
UV_CACHE_DIR: /tmp/.uv-cache
162162

163163
- name: Run sample validation
164164
run: |
@@ -173,36 +173,36 @@ jobs:
173173

174174
validate-05-end-to-end:
175175
name: Validate 05-end-to-end
176+
if: false # Temporarily disabled because of sample complexity
176177
runs-on: ubuntu-latest
177-
permissions:
178-
contents: read
178+
environment: integration
179179
env:
180180
# Azure AI configuration
181-
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZURE_AI_PROJECT_ENDPOINT }}
182-
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ secrets.AZURE_AI_MODEL_DEPLOYMENT_NAME }}
181+
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
182+
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
183183
# Azure OpenAI configuration
184-
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
185-
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }}
186-
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME }}
184+
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
185+
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
186+
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
187187
# Azure AI Search (for evaluation samples)
188188
AZURE_SEARCH_ENDPOINT: ${{ secrets.AZURE_SEARCH_ENDPOINT }}
189189
AZURE_SEARCH_API_KEY: ${{ secrets.AZURE_SEARCH_API_KEY }}
190190
AZURE_SEARCH_INDEX_NAME: ${{ secrets.AZURE_SEARCH_INDEX_NAME }}
191-
# GitHub Copilot configuration
192-
GITHUB_COPILOT_MODEL: ${{ vars.GITHUB_COPILOT_MODEL }}
191+
# Evaluation sample
192+
AZURE_AI_MODEL_DEPLOYMENT_NAME_WORKFLOW: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
193193
defaults:
194194
run:
195195
working-directory: python
196196
steps:
197197
- uses: actions/checkout@v6
198198

199-
- name: Set up python and install the project
200-
uses: ./.github/actions/python-setup
199+
- name: Setup environment
200+
uses: ./.github/actions/sample-validation-setup
201201
with:
202-
python-version: "3.12"
202+
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
203+
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
204+
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
203205
os: ${{ runner.os }}
204-
env:
205-
UV_CACHE_DIR: /tmp/.uv-cache
206206

207207
- name: Run sample validation
208208
run: |
@@ -218,30 +218,31 @@ jobs:
218218
validate-autogen-migration:
219219
name: Validate autogen-migration
220220
runs-on: ubuntu-latest
221-
permissions:
222-
contents: read
221+
environment: integration
223222
env:
224223
# Azure AI configuration
225-
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZURE_AI_PROJECT_ENDPOINT }}
226-
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ secrets.AZURE_AI_MODEL_DEPLOYMENT_NAME }}
224+
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
225+
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
227226
# Azure OpenAI configuration
228-
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
229-
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }}
230-
# GitHub Copilot configuration
231-
GITHUB_COPILOT_MODEL: ${{ vars.GITHUB_COPILOT_MODEL }}
227+
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
228+
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
229+
# OpenAI configuration
230+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
231+
OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI_CHAT_MODEL_NAME }}
232+
OPENAI_RESPONSES_MODEL_ID: ${{ vars.OPENAI_REASONING_MODEL_NAME }}
232233
defaults:
233234
run:
234235
working-directory: python
235236
steps:
236237
- uses: actions/checkout@v6
237238

238-
- name: Set up python and install the project
239-
uses: ./.github/actions/python-setup
239+
- name: Setup environment
240+
uses: ./.github/actions/sample-validation-setup
240241
with:
241-
python-version: "3.12"
242+
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
243+
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
244+
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
242245
os: ${{ runner.os }}
243-
env:
244-
UV_CACHE_DIR: /tmp/.uv-cache
245246

246247
- name: Run sample validation
247248
run: |
@@ -257,16 +258,15 @@ jobs:
257258
validate-semantic-kernel-migration:
258259
name: Validate semantic-kernel-migration
259260
runs-on: ubuntu-latest
260-
permissions:
261-
contents: read
261+
environment: integration
262262
env:
263263
# Azure AI configuration
264-
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZURE_AI_PROJECT_ENDPOINT }}
265-
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ secrets.AZURE_AI_MODEL_DEPLOYMENT_NAME }}
264+
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
265+
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
266266
# Azure OpenAI configuration
267-
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
268-
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }}
269-
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME }}
267+
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
268+
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
269+
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
270270
# OpenAI configuration
271271
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
272272
OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI_CHAT_MODEL_ID }}
@@ -276,21 +276,19 @@ jobs:
276276
COPILOTSTUDIOAGENT__SCHEMANAME: ${{ secrets.COPILOTSTUDIOAGENT__SCHEMANAME }}
277277
COPILOTSTUDIOAGENT__TENANTID: ${{ secrets.COPILOTSTUDIOAGENT__TENANTID }}
278278
COPILOTSTUDIOAGENT__AGENTAPPID: ${{ secrets.COPILOTSTUDIOAGENT__AGENTAPPID }}
279-
# GitHub Copilot configuration
280-
GITHUB_COPILOT_MODEL: ${{ vars.GITHUB_COPILOT_MODEL }}
281279
defaults:
282280
run:
283281
working-directory: python
284282
steps:
285283
- uses: actions/checkout@v6
286284

287-
- name: Set up python and install the project
288-
uses: ./.github/actions/python-setup
285+
- name: Setup environment
286+
uses: ./.github/actions/sample-validation-setup
289287
with:
290-
python-version: "3.12"
288+
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
289+
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
290+
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
291291
os: ${{ runner.os }}
292-
env:
293-
UV_CACHE_DIR: /tmp/.uv-cache
294292

295293
- name: Run sample validation
296294
run: |

‎python/packages/azurefunctions/agent_framework_azurefunctions/_app.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -612,11 +612,11 @@ def get_agent(
612612
context: Durable Functions orchestration context invoking the agent.
613613
agent_name: Name of the agent registered on this app.
614614
615-
Raises:
616-
ValueError: If the requested agent has not been registered.
617-
618615
Returns:
619616
DurableAIAgent[AgentTask] wrapper bound to the orchestration context.
617+
618+
Raises:
619+
ValueError: If the requested agent has not been registered.
620620
"""
621621
normalized_name = str(agent_name)
622622

0 commit comments

Comments
 (0)