Skip to content

Add validate command for custom training#7407

Open
saanikaguptamicrosoft wants to merge 27 commits intoAzure:foundry-training-devfrom
saanikaguptamicrosoft:saanika/validate
Open

Add validate command for custom training#7407
saanikaguptamicrosoft wants to merge 27 commits intoAzure:foundry-training-devfrom
saanikaguptamicrosoft:saanika/validate

Conversation

@saanikaguptamicrosoft
Copy link
Copy Markdown
Collaborator

@saanikaguptamicrosoft saanikaguptamicrosoft commented Mar 31, 2026

Notes

Added validation for-

  1. command field is required
  2. environment field is required
  3. compute field is required
  4. code if present, it should not start with git:// or git+
  5. The local path mentioned in code and inputs.<key>.path are existent
  6. Placeholder mapping is correct. Different cases:
    • The command field contains placeholder in the form of ${{inputs.xxx}}, the xxx should be present as key in inputs.
      • For inputs -> if key is present but its value is {} or None, throw error.
    • The command field contains placeholder in the form ${{outputs.xxx}}, the xxx may or may not be present as key in inputs. If it's not present, means user wants to go with default output values. If it's present means user wants to override output configs.
      • For outputs -> if key is present but its value is {} or None, show warning. Because user might have forgotten to override the value.
    • If ${{inputs.xxx}} is wrapped inside [ ], it's optional, so we don't validate its presence in inputs key.
    • Single bracket i.e. {inputs.xxx} or {outputs.xxx} or ${inputs.xxx} or ${outputs.xxx} should throw error

Added unit tests.

Testing

  • Local testing
image image image image image
  • Build ~ Successful
azd x build
  • UTs ~ Successful
go test ./internal/utils/ -v -count=1

@saanikaguptamicrosoft saanikaguptamicrosoft changed the title Saanika/validate Add validate command for custom training Mar 31, 2026
Copy link
Copy Markdown
Member

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review - #7407

Add validate command for custom training by @saanikaguptamicrosoft

Summary

What: Adds an offline job validate command that checks a YAML job definition for required fields, invalid git paths, local path existence, placeholder correctness, and empty input/output definitions - collecting all findings before reporting instead of failing on the first error.

Why: Lets users catch YAML mistakes locally before submitting to the backend, saving a round-trip.

Assessment: The validation logic is solid and well-structured, but the command can't actually run offline due to the parent command's PersistentPreRunE hook. This should be addressed before merge. A few edge cases in the validator produce confusing output (duplicate errors on git paths, silent pass on ${inputs.xxx} typos).

Findings

Category High Medium Low
Design 1 0 0
Logic 0 2 0
Error Handling 0 1 0
Consistency 0 1 1
Tests 0 0 1
Total 1 4 2

Key Findings

  1. [HIGH] Design: Parent job command's PersistentPreRunE requires Azure env setup, preventing offline use
  2. [MEDIUM] Logic: git:// code paths produce duplicate errors (git-not-supported + local-path-not-found)
  3. [MEDIUM] Logic: ${inputs.xxx} typos silently bypass single-brace detection
  4. [MEDIUM] Error Handling: os.Stat permission/IO errors silently ignored in path checks
  5. [MEDIUM] Consistency: Missing -f shorthand for --file flag (inconsistent with job submit)
  6. [LOW] Consistency: Missing Long description on cobra command (all sibling commands have one)
  7. [LOW] Tests: No test for the git:// double-error scenario (assert only 1 finding, not 2)

Test Coverage Estimate

  • Well covered: required fields, git path detection, local path checks, placeholder mapping, single-brace detection, empty definitions, multiline commands
  • Missing: command-layer test (cobra handler), ${inputs.xxx} edge case, git:// double-error assertion, os.Stat permission errors, empty/zero-byte YAML input

What's Done Well

  • Collects all findings instead of failing on the first error - much better UX than the existing ValidateJobDefinition
  • Good separation between the command handler and validation logic
  • Optional [...] block handling for input placeholders is a nice touch
  • Test file has clear YAML comments showing what each test case maps to

5 inline comments below.

Comment thread cli/azd/extensions/azure.ai.customtraining/internal/cmd/job.go
Copy link
Copy Markdown
Member

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous 5 findings addressed. New issues found:

  • job_validator_test.go:223 - single-brace test assertions search for "single-brace '...'" but the actual message is "Incorrect placeholder format...". Three assertions will fail (223, 226, 236)
  • job_validator.go:201 - duplicate findings when the same placeholder key appears twice in command
  • job_validator.go:106 - fmt.Sprintf with no format verbs (staticcheck SA1006)

Comment thread cli/azd/extensions/azure.ai.customtraining/internal/utils/job_validator_test.go Outdated
Comment thread cli/azd/extensions/azure.ai.customtraining/internal/utils/job_validator.go Outdated
Copy link
Copy Markdown
Member

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All 8 findings from my previous reviews are addressed - nice work. A few new items from this pass, mostly edge cases in the placeholder validation. Nothing blocking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants