Skip to content

Commit ec708e9

Browse files
fix: spell swe-bench-pro's models the way the gateway matches them
The agents strip only the `openai/` prefix before calling the gateway (`removeprefix("openai/")`), and the gateway matches the allow-list exactly (`model not in scope.allowed_models` -> 403 model_denied). So `model: openai/gpt-4o` puts `gpt-4o` on the wire, and an allow-list carrying the prefixed spelling never matches. Listing both spellings papered over that and then fails the config invariant that the evaluation scope allow exactly the target model. Drop the prefix on both sides, and template the producer list the way every other benchmark does so `--model` / `--param optimizer_model` keeps working without a config edit. This config still cannot be swept by tests/test_v05_benchmark_configs.py: its partitions are deliberately empty pending real task ids, so it fails to load. It should join BENCHMARKS there once the partitions and the task source digest are pinned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 5e6c101 commit ec708e9

1 file changed

Lines changed: 3 additions & 3 deletions

File tree

  • harness-engineering-bench/swe-bench-pro/baseline

harness-engineering-bench/swe-bench-pro/baseline/build.yaml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ score_baseline: true
4949
rescore_top_k: 3
5050
rescore_attempts: 1
5151

52-
model: openai/gpt-4o
52+
model: gpt-4o
5353
environment_name: ${inner_env:-modal}
5454
# inner eval sandboxes share a dedicated Modal app instead of the __harbor__ default
5555
extra_harbor_args: ["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]
@@ -90,10 +90,10 @@ inference_gateway:
9090
upstream_base_url_env: OPENAI_BASE_URL
9191
producer:
9292
# Both prefixed and bare forms: the gateway model match is prefix-sensitive.
93-
allowed_models: ["gpt-5.3-codex", "openai/gpt-5.3-codex"]
93+
allowed_models: ["${optimizer_model:-gpt-5.3-codex}"]
9494
max_concurrency: 8
9595
evaluation:
96-
allowed_models: ["gpt-4o", "openai/gpt-4o"]
96+
allowed_models: [gpt-4o]
9797
max_requests: 15000
9898
max_tokens: 100000000
9999
max_concurrency: 64

0 commit comments

Comments
 (0)