[azure-ai-ml] Add generated integration tests and test infrastructure for coverage improvement by deyaaeldeen · Pull Request #45967 · Azure/azure-sdk-for-python

deyaaeldeen · 2026-03-27T16:53:30Z

Summary

This PR adds auto-generated live integration tests for the azure-ai-ml SDK to improve test coverage of the hand-written operations layer and public API.

Test Files (22 files, ~140 tests)

Generated using test-gen tool targeting uncovered code paths in azure.ai.ml.operations and _ml_client.py
All tests follow AzureRecordedTestCase pattern with proper fixtures
120 passed, 0 failed, 21 skipped in playback mode
Recordings pushed to azure-sdk-assets (tag: python/ml/azure-ai-ml_0f205ad0cc)

Coverage Impact (operations layer)

Before: 51.7% branch coverage
After: 56.2% branch coverage (+4.6pp, +258 lines, +71 branches)

Infrastructure Updates (`test-resources.json`)

Added resources needed for comprehensive test coverage:

Azure ML Registry (Microsoft.MachineLearningServices/registries)
ADLS Gen2 storage account (HNS enabled) for feature store offline store
Azure Cache for Redis for feature store online store
User-assigned managed identity for test operations
Corresponding parameters and outputs

Commits

Add auto-generated live integration tests (22 files)
Fix 5 failing generated tests
Improve test quality: dedup, remove dead tests, fix exception types
Fix playback compatibility for recorded tests
Add registry, ADLS Gen2, Redis, and identity to test-resources template
Update assets.json tag with gap test recordings

Generated 151 live integration tests across 22 files targeting coverage gaps in the azure-ai-ml operations layer. Tests use AzureRecordedTestCase, @pytest.mark.e2etest markers, and make real Azure service calls. Generated by test-gen tool with gpt-5-mini model covering 20 source files: - Workspace, Job, Model, Component, Environment operations - Data, Datastore, Schedule, Deployment operations - Endpoint, Feature Store, MLClient operations Results: 128 passed, 5 failed, 14 skipped (87% pass rate) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- test_model_operations_gaps: Use correct evaluator properties ('is-evaluator'/'is-promptflow' == 'true') instead of '__is_evaluator' - test_schedule_gaps: Remove Z-suffix from datetime strings (service rejects it) and update start_time to recent past - test_workspace_operations_base_gaps_additional: Replace hub/project creation (>120s timeout) with get() on existing workspace All 5 previously-failing tests now pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Quality improvements across 12 generated test files: - Remove 17 duplicate method names (renamed with descriptive suffixes) - Delete 6 always-skipped tests in batch_deployment_ops (zero value) - Delete 1 always-skip test in capability_hosts_ops - Remove broken test (create_or_update doesn't raise for validation) - Replace 12 broad pytest.raises(Exception) with specific types: ValidationException, ResourceNotFoundError, HttpResponseError, UserErrorException, AssertionError, MlException - Clean up unused imports and duplicate class definitions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Replace uuid.uuid4() with rand_online_name/rand_batch_name fixtures for deterministic name generation via VariableRecorder - Replace datetime.now() with hardcoded far-future dates in schedule tests to avoid timestamp mismatches between recording and playback - Add is_live() skip guards for tests that require real credentials: JWT token decoding, credential type checks, key regeneration - Fix experiment_name to be deterministic in pipeline job tests Playback results: 120 passed, 0 failed, 21 skipped Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add infrastructure resources needed for comprehensive test coverage: - Azure ML Registry for model/component sharing tests - ADLS Gen2 storage account (HNS enabled) for feature store offline store - Azure Cache for Redis for feature store online store - User-assigned managed identity for test operations - Corresponding parameters and outputs for all new resources Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Tag: python/ml/azure-ai-ml_0f205ad0cc 167 sanitized recordings for generated integration tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR adds auto-generated live integration tests for azure-ai-ml to increase coverage across the operations layer/public MLClient surface, and updates the shared ARM test-resources template plus the assets tag to support the new scenarios and recordings.

Changes:

Add a set of new AzureRecordedTestCase-style gap tests across multiple operations (jobs, endpoints, workspaces, data/assets, etc.).
Extend sdk/ml/test-resources.json with additional Azure resources (Registry, ADLS Gen2, Redis, user-assigned identity) and expose new outputs for tests.
Update sdk/ml/azure-ai-ml/assets.json tag to point to the new recordings.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
sdk/ml/test-resources.json	Adds Registry/ADLS/Redis/UAI resources + outputs to support broader live test coverage.
sdk/ml/azure-ai-ml/tests/test_workspace_outbound_rule_operations_gaps.py	Adds outbound rule operation gap tests (validation + list behavior).
sdk/ml/azure-ai-ml/tests/test_workspace_operations_gaps.py	Adds workspace list/create/join/diagnose gap tests.
sdk/ml/azure-ai-ml/tests/test_workspace_operations_base_gaps_additional.py	Adds additional workspace get() type/branch coverage.
sdk/ml/azure-ai-ml/tests/test_workspace_operations_base_gaps.py	Adds a live-only placeholder test to validate scaffolding.
sdk/ml/azure-ai-ml/tests/test_schedule_gaps.py	Adds schedule lifecycle tests (create/list/trigger/enable/disable/delete).
sdk/ml/azure-ai-ml/tests/test_operation_orchestrator_gaps.py	Adds model listing smoke tests intended to exercise orchestrator paths.
sdk/ml/azure-ai-ml/tests/test_online_endpoint_operations_gaps.py	Adds online endpoint validation/credentials/keys gap tests.
sdk/ml/azure-ai-ml/tests/test_online_deployment_operations_gaps.py	Adds online deployment validation gap tests (vscode debug/local GPU/logs container).
sdk/ml/azure-ai-ml/tests/test_model_operations_gaps.py	Adds model create_or_update evaluator-validation gap tests.
sdk/ml/azure-ai-ml/tests/test_ml_client_gaps.py	Adds MLClient singledispatch/from_config/_ml_client_cli gap tests.
sdk/ml/azure-ai-ml/tests/test_job_ops_helper_gaps.py	Adds unit-style tests for internal job ops helper utilities.
sdk/ml/azure-ai-ml/tests/test_job_operations_gaps_basic_props.py	Adds job operations property/lazy-client coverage tests plus additional gap tests.
sdk/ml/azure-ai-ml/tests/test_job_operations_gaps.py	Adds job operations gap tests (download/get/validate/token/header helpers).
sdk/ml/azure-ai-ml/tests/test_feature_store_operations_gaps.py	Adds feature store validation gap tests (offline/online store types, update/delete paths).
sdk/ml/azure-ai-ml/tests/test_environment_operations_gaps.py	Adds environment get validation + share failure restoration tests.
sdk/ml/azure-ai-ml/tests/test_deployment_template_operations_gaps.py	Adds deployment template validation/error-path tests.
sdk/ml/azure-ai-ml/tests/test_datastore_operations_gaps.py	Adds datastore mount validation + live-only persistent mount polling coverage tests.
sdk/ml/azure-ai-ml/tests/test_data_operations_gaps.py	Adds data get/create/mount validation gap tests.
sdk/ml/azure-ai-ml/tests/test_component_operations_gaps.py	Adds component refine/validate error-path tests for non-DSL/non-mldesigner functions.
sdk/ml/azure-ai-ml/tests/test_capability_hosts_operations_gaps.py	Adds capability host tests for Hub/Project workspace behaviors and validation.
sdk/ml/azure-ai-ml/tests/test_batch_endpoint_operations_gaps.py	Adds batch endpoint invoke/list-jobs validation gap tests.
sdk/ml/azure-ai-ml/tests/test_batch_deployment_operations_gaps.py	Adds batch deployment validation gap tests (scoring script + component validation behavior).
sdk/ml/azure-ai-ml/assets.json	Updates assets tag to reference the new recordings.

Comments suppressed due to low confidence (1)

sdk/ml/azure-ai-ml/tests/test_batch_deployment_operations_gaps.py:76

The cleanup path starts deleting the batch endpoint but does not wait for the delete poller to finish. This can leak resources and make subsequent live test runs flaky; call .result() (and consider cleanup even if endpoint creation fails).

            assert isinstance(err, (HttpResponseError, Exception))
        finally:
            # Cleanup endpoint
            client.batch_endpoints.begin_delete(name=endpoint.name)

sdk/ml/azure-ai-ml/tests/test_data_operations_gaps.py

sdk/ml/azure-ai-ml/tests/test_online_deployment_operations_gaps.py

sdk/ml/azure-ai-ml/tests/test_online_endpoint_operations_gaps.py

sdk/ml/azure-ai-ml/tests/test_job_ops_helper_gaps.py

sdk/ml/test-resources.json

sdk/ml/azure-ai-ml/tests/test_job_operations_gaps.py

sdk/ml/azure-ai-ml/tests/test_batch_deployment_operations_gaps.py

Replace nested double quotes in f-string on line 62 of test_job_operations_gaps.py with a plain string literal. Python 3.10 does not support nested quotes in f-strings (PEP 701 was added in 3.12), causing a SyntaxError at collection time that blocks all tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Use tmp_path fixture for data_missing_path.yaml instead of CWD (test_data_operations_gaps.py) - Await .result() on delete LRO pollers to prevent resource leaks (test_online_deployment_operations_gaps.py, test_batch_deployment_operations_gaps.py) - Import MLClient from public azure.ai.ml instead of private _ml_client (test_online_endpoint_operations_gaps.py) - Move mid-file imports to top-level import section (test_job_ops_helper_gaps.py) - Narrow meaningless isinstance(err, (HttpResponseError, Exception)) assertion to just HttpResponseError (test_batch_deployment_operations_gaps.py) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-27T19:36:57Z

API Change Check

APIView identified API level changes in this PR and created the following API reviews

azure-ai-ml

# Conflicts: # sdk/ml/azure-ai-ml/assets.json

Merged assets tags python/ml/azure-ai-ml_0f205ad0cc (gap test recordings) and python/ml/azure-ai-ml_1e2cb117b2 (latest main) into new combined tag python/ml/azure-ai-ml_d0dbceadc6 using test-proxy tag-merge tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ties - Re-recorded all 22 gap test files against live TME resources - Added body key sanitizers for keyValue, primaryKey, secondaryKey to prevent secrets from leaking into recordings - Fixed TestJobOperationsBasicProperties: added recorded_test fixture and AzureRecordedTestCase base class so tests go through test proxy in playback mode - All 120 tests pass in playback, 134 pass live, 0 failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The test_validate_git_code_path_rejected_when_private_preview_disabled test was failing in CI because prior tests in the session enable the AZURE_ML_CLI_PRIVATE_FEATURES_ENABLED env var, which causes is_private_preview_enabled() to return True and skip the git-code validation. Fix by explicitly patching the env var to 'False' within the test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ording The original live recording session re-recorded not just the new gap tests but also 106 pre-existing tests against a TME workspace that lacks Singularity clusters, managed datastores, and pre-registered components. This caused playback failures for tests like test_command_job_with_singularity, test_data_auto_delete_setting, test_distribution_components, etc. Fix: restore those 106 recordings from the main branch tag. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…2e-coverage-v2

deyaaeldeen and others added 6 commits March 27, 2026 16:20

Update assets.json tag with gap test recordings

63f79af

Tag: python/ml/azure-ai-ml_0f205ad0cc 167 sanitized recordings for generated integration tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 27, 2026 16:53

deyaaeldeen requested review from JustinFirsching, NonStatic2014, achauhan-scc, arunsu, jayesh-tanna, kingernupur, nick863, novaturient95, raduk, rtanase, sharma-riti and vivram as code owners March 27, 2026 16:53

github-actions bot added the Machine Learning label Mar 27, 2026

Copilot started reviewing on behalf of deyaaeldeen March 27, 2026 16:56 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

deyaaeldeen and others added 2 commits March 27, 2026 18:02

deyaaeldeen and others added 5 commits March 27, 2026 20:13

Merge remote-tracking branch 'origin/main' into pr-45967

97fac2f

# Conflicts: # sdk/ml/azure-ai-ml/assets.json

removed recorded tag for tests that are purely unit tests

beb079b

deyaaeldeen and others added 3 commits March 28, 2026 01:32

Run black on gap test files

f507443

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into feature/azure-ai-ml-e…

53e392e

…2e-coverage-v2

deyaaeldeen force-pushed the feature/azure-ai-ml-e2e-coverage-v2 branch from 50fd6a3 to 53e392e Compare March 28, 2026 04:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[azure-ai-ml] Add generated integration tests and test infrastructure for coverage improvement#45967

[azure-ai-ml] Add generated integration tests and test infrastructure for coverage improvement#45967
deyaaeldeen wants to merge 16 commits intomainfrom
feature/azure-ai-ml-e2e-coverage-v2

deyaaeldeen commented Mar 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

deyaaeldeen commented Mar 27, 2026

Summary

Test Files (22 files, ~140 tests)

Coverage Impact (operations layer)

Infrastructure Updates (test-resources.json)

Commits

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

API Change Check

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Infrastructure Updates (`test-resources.json`)

github-actions bot commented Mar 27, 2026 •

edited

Loading