docs: clarify baseline deployment name must match pricing config key#3
Open
jpsantoscosta wants to merge 1 commit into
Open
Conversation
Add a comment in .env.example and a callout block in QUICKSTART.md explaining that AZURE_BASELINE_DEPLOYMENT must match a key in the pricing section of the config file. If a custom deployment name is used (e.g. gpt-4o-baseline) without a matching pricing entry, baseline costs are silently recorded as $0.0000 with no warning, making cost comparison inaccurate. Discovered while running evaluations against a Quick Deploy setup. Applies to both Quick Deploy and Custom Deploy modes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.env.exampleaboveAZURE_BASELINE_DEPLOYMENTexplaining that the value must match a key in thepricingsection of the config fileQUICKSTART.md(Part 2, credentials step) with the same guidance and an example of how to add a custom entryDiscovered while running evaluations against a Quick Deploy setup. If you use a descriptive deployment name (e.g.
gpt-4o-baseline) that doesn't match anypricing:key in your config, baseline costs are recorded as $0.0000 with no warning — making the cost comparison inaccurate. The underlying silent-failure is tracked in #2.Changes
.env.example: three-line comment aboveAZURE_BASELINE_DEPLOYMENTQUICKSTART.md: one-sentence callout block after the credentials setup stepTest plan
pytest tests/ -v -m "not integration"— 165 passed, 2 pre-existing failures (optional[foundry]extras not installed, unrelated to this change)ruff check src/ tests/ scripts/— all checks passed