Skip to content

Commit c8d971c

Browse files
jpsantoscostaclaude
andcommitted
docs: clarify baseline deployment name must match pricing config key
Add a comment in .env.example and a callout block in QUICKSTART.md explaining that AZURE_BASELINE_DEPLOYMENT must match a key in the pricing section of the config file. If a custom deployment name is used (e.g. gpt-4o-baseline) without a matching pricing entry, baseline costs are silently recorded as $0.0000 with no warning, making cost comparison inaccurate. Discovered while running evaluations against a Quick Deploy setup. Applies to both Quick Deploy and Custom Deploy modes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 78720ac commit c8d971c

2 files changed

Lines changed: 5 additions & 0 deletions

File tree

.env.example

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,9 @@ AZURE_MODEL_ROUTER_DEPLOYMENT=model-router
66
# Azure OpenAI endpoint (for baseline model)
77
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
88
AZURE_OPENAI_KEY=your-azure-openai-api-key
9+
# Must match a key in the pricing section of your config file (e.g. gpt-4o, gpt-5).
10+
# If you use a custom deployment name, add a matching entry to configs/default.yaml
11+
# under pricing: using your exact deployment name. Otherwise baseline costs show as $0.00.
912
AZURE_BASELINE_DEPLOYMENT=gpt-5
1013

1114
# Judge model (defaults to same endpoint as baseline; override to use a different model)

QUICKSTART.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,8 @@ copy .env.example .env # Windows
139139
```
140140
Open `.env` and set the four endpoint URLs and API keys (router, baseline, judge, optional Foundry project). The file is in `.gitignore`, so your secrets won't be committed.
141141

142+
> **Note:** The value you set for `AZURE_BASELINE_DEPLOYMENT` must match a key in the `pricing` section of your config file. If you use a custom deployment name such as `gpt-4o-baseline`, add a matching entry under `pricing:` in `configs/default.yaml` with the same input/output rates. Otherwise baseline costs will show as $0.00 and the cost comparison will be inaccurate. This applies whether you are using Quick Deploy or Custom Deploy.
143+
142144
### 2. Pick or edit a config
143145
- `configs/quick_test.yaml` — small, fast (~10 prompts) — good first run.
144146
- `configs/default.yaml` — full benchmark (100 prompts).

0 commit comments

Comments
 (0)