Commit ec708e9
fix: spell swe-bench-pro's models the way the gateway matches them
The agents strip only the `openai/` prefix before calling the gateway
(`removeprefix("openai/")`), and the gateway matches the allow-list exactly
(`model not in scope.allowed_models` -> 403 model_denied). So `model:
openai/gpt-4o` puts `gpt-4o` on the wire, and an allow-list carrying the
prefixed spelling never matches. Listing both spellings papered over that
and then fails the config invariant that the evaluation scope allow exactly
the target model.
Drop the prefix on both sides, and template the producer list the way every
other benchmark does so `--model` / `--param optimizer_model` keeps working
without a config edit.
This config still cannot be swept by tests/test_v05_benchmark_configs.py:
its partitions are deliberately empty pending real task ids, so it fails to
load. It should join BENCHMARKS there once the partitions and the task
source digest are pinned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>1 parent 5e6c101 commit ec708e9
1 file changed
Lines changed: 3 additions & 3 deletions
Lines changed: 3 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
94 | 94 | | |
95 | 95 | | |
96 | | - | |
| 96 | + | |
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
| |||
0 commit comments