Skip to content

Commit 92d20ff

Browse files
varunursekarclaude
andcommitted
Drop the ALE-Bench benchmark
It was the suite's only Family B task and the only one that bypassed the compiler: a hand-written task directory with no build.yaml, so it got none of the build-time validation the other five benchmarks get. Its optimizer was also unmetered -- no gateway container, raw upstream credentials in main, no model allow-list -- which is a fair-play hole in a benchmark whose whole point is controlled measurement. Its sidecar factory had additionally gone stale against SidecarEvaluationPolicy and would have died at startup. The nested-backend refactor now supports `evaluation_backend: command`, so a replacement Family B task can be built through the compiler and get the gateway for free. examples/harbor-circle-packing carries a worked version of exactly that shape, verified end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 3aaecdd commit 92d20ff

17 files changed

Lines changed: 0 additions & 1007 deletions

File tree

harness-engineering-bench/README.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,4 +36,3 @@ promote a task set to the top level once it is ready.
3636
| [SWE-Atlas-QnA baseline](swe-atlas-qna/baseline/) | Codebase investigation agent | Harbor `scale-ai/swe-atlas-qna` | 20% / 40% / 40% |
3737
| [tau3 baseline](tau3/baseline/) | MCP customer-service agent | Harbor `sierra-research/tau3-bench` | 20% / 40% / 40% |
3838
| [BrowseComp-Plus baseline](browsecomp-plus/baseline/) | Fixed-corpus deep-research agent | Pinned local Harbor tasks | 20% / 40% / 40% |
39-
| [ALE-Bench (ahc011)](ale-bench/) | C++ solver program (component, not an agent) | ALE-Bench AHC problem | public / private seeds |

harness-engineering-bench/ale-bench/.gitignore

Lines changed: 0 additions & 2 deletions
This file was deleted.

harness-engineering-bench/ale-bench/environment/Dockerfile

Lines changed: 0 additions & 19 deletions
This file was deleted.

harness-engineering-bench/ale-bench/environment/agent-baseline/solution.cpp

Lines changed: 0 additions & 8 deletions
This file was deleted.

harness-engineering-bench/ale-bench/environment/agent-seed/solution.cpp

Lines changed: 0 additions & 8 deletions
This file was deleted.

harness-engineering-bench/ale-bench/environment/docker-compose.yaml

Lines changed: 0 additions & 60 deletions
This file was deleted.

harness-engineering-bench/ale-bench/environment/main/seed.sh

Lines changed: 0 additions & 12 deletions
This file was deleted.

harness-engineering-bench/ale-bench/environment/sidecar/Dockerfile

Lines changed: 0 additions & 42 deletions
This file was deleted.

harness-engineering-bench/ale-bench/environment/sidecar/ale_factory.py

Lines changed: 0 additions & 153 deletions
This file was deleted.

harness-engineering-bench/ale-bench/environment/sidecar/dind-entrypoint.sh

Lines changed: 0 additions & 27 deletions
This file was deleted.

0 commit comments

Comments
 (0)