Description
If AZURE_BASELINE_DEPLOYMENT is set to a deployment name that does not exactly match a key in the pricing section of the config file, baseline costs are silently recorded as $0.0000 throughout the run. No warning or error is shown.
This makes the cost comparison between Model Router and the baseline meaningless — the router will always appear to cost more.
Steps to reproduce
- Create a deployment in Azure named something like
gpt-4o-baseline (a custom name rather than the canonical model name).
- Set
AZURE_BASELINE_DEPLOYMENT=gpt-4o-baseline in .env.
- Leave
configs/default.yaml with only gpt-4o under pricing: (no gpt-4o-baseline entry).
- Run the evaluation:
python scripts/run_eval.py
- Open the report — baseline Estimated cost shows
$0.0000 and Cost savings shows an inflated figure.
Expected behaviour
The tool should either:
- Warn at startup that the baseline deployment name has no matching pricing entry and costs will be $0.00, or
- Fail fast with a clear message pointing to the relevant config key so the user can fix it before wasting a full run.
Actual behaviour
The run completes silently. Pricing is calculated as $0.00 for every baseline prompt. The cost-savings percentage in the report is wildly inflated with no indication that the baseline pricing lookup failed.
Suggested fix
In the cost-calculation path, add a check: if the resolved pricing for the baseline model is None or zero, emit a warning at the start of the run and/or in the final report. Ideally also check at config-load time if the deployment name resolves to a known pricing key.
Notes
This is easy to hit in practice. Deployment names in Azure are set by the user and do not have to match the underlying model name. Many users, especially those comparing multiple baselines, will use descriptive names like gpt-4o-baseline, gpt5-prod, etc.
A one-line note has been added to .env.example and QUICKSTART.md as a short-term mitigation (see PR), but a runtime warning would catch the case for existing users who don't re-read the docs.
Description
If
AZURE_BASELINE_DEPLOYMENTis set to a deployment name that does not exactly match a key in thepricingsection of the config file, baseline costs are silently recorded as $0.0000 throughout the run. No warning or error is shown.This makes the cost comparison between Model Router and the baseline meaningless — the router will always appear to cost more.
Steps to reproduce
gpt-4o-baseline(a custom name rather than the canonical model name).AZURE_BASELINE_DEPLOYMENT=gpt-4o-baselinein.env.configs/default.yamlwith onlygpt-4ounderpricing:(nogpt-4o-baselineentry).$0.0000and Cost savings shows an inflated figure.Expected behaviour
The tool should either:
Actual behaviour
The run completes silently. Pricing is calculated as $0.00 for every baseline prompt. The cost-savings percentage in the report is wildly inflated with no indication that the baseline pricing lookup failed.
Suggested fix
In the cost-calculation path, add a check: if the resolved pricing for the baseline model is
Noneor zero, emit a warning at the start of the run and/or in the final report. Ideally also check at config-load time if the deployment name resolves to a known pricing key.Notes
This is easy to hit in practice. Deployment names in Azure are set by the user and do not have to match the underlying model name. Many users, especially those comparing multiple baselines, will use descriptive names like
gpt-4o-baseline,gpt5-prod, etc.A one-line note has been added to
.env.exampleandQUICKSTART.mdas a short-term mitigation (see PR), but a runtime warning would catch the case for existing users who don't re-read the docs.