From c8d971c8ba92b98be760f5b3d85c88a8fb4de862 Mon Sep 17 00:00:00 2001 From: Joao Date: Fri, 22 May 2026 16:29:26 +0100 Subject: [PATCH] docs: clarify baseline deployment name must match pricing config key Add a comment in .env.example and a callout block in QUICKSTART.md explaining that AZURE_BASELINE_DEPLOYMENT must match a key in the pricing section of the config file. If a custom deployment name is used (e.g. gpt-4o-baseline) without a matching pricing entry, baseline costs are silently recorded as $0.0000 with no warning, making cost comparison inaccurate. Discovered while running evaluations against a Quick Deploy setup. Applies to both Quick Deploy and Custom Deploy modes. Co-Authored-By: Claude Sonnet 4.6 --- .env.example | 3 +++ QUICKSTART.md | 2 ++ 2 files changed, 5 insertions(+) diff --git a/.env.example b/.env.example index 7b868bd..afa248b 100644 --- a/.env.example +++ b/.env.example @@ -6,6 +6,9 @@ AZURE_MODEL_ROUTER_DEPLOYMENT=model-router # Azure OpenAI endpoint (for baseline model) AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com AZURE_OPENAI_KEY=your-azure-openai-api-key +# Must match a key in the pricing section of your config file (e.g. gpt-4o, gpt-5). +# If you use a custom deployment name, add a matching entry to configs/default.yaml +# under pricing: using your exact deployment name. Otherwise baseline costs show as $0.00. AZURE_BASELINE_DEPLOYMENT=gpt-5 # Judge model (defaults to same endpoint as baseline; override to use a different model) diff --git a/QUICKSTART.md b/QUICKSTART.md index 2cd6121..76644ad 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -139,6 +139,8 @@ copy .env.example .env # Windows ``` Open `.env` and set the four endpoint URLs and API keys (router, baseline, judge, optional Foundry project). The file is in `.gitignore`, so your secrets won't be committed. +> **Note:** The value you set for `AZURE_BASELINE_DEPLOYMENT` must match a key in the `pricing` section of your config file. If you use a custom deployment name such as `gpt-4o-baseline`, add a matching entry under `pricing:` in `configs/default.yaml` with the same input/output rates. Otherwise baseline costs will show as $0.00 and the cost comparison will be inaccurate. This applies whether you are using Quick Deploy or Custom Deploy. + ### 2. Pick or edit a config - `configs/quick_test.yaml` — small, fast (~10 prompts) — good first run. - `configs/default.yaml` — full benchmark (100 prompts).