Benchmark validity: de-leak ranked phases, fail-closed budget cap, fix closed-book provenance#116
Open
anantham wants to merge 3 commits into
Open
Benchmark validity: de-leak ranked phases, fail-closed budget cap, fix closed-book provenance#116anantham wants to merge 3 commits into
anantham wants to merge 3 commits into
Conversation
…ance The published leaderboard hardcoded `closedBook: true` with the note "Models receive ONLY the raw Pāli phrase — no dictionary or retrieval context." That contradicted reality AND its own source list: the benchmark feeds DPD attestations (roots/POS/attested senses) to both the Anatomist and Lexicographer passes (the DPD source is even listed right below as "factual authority"). The P2.1 anatomist grounding sharpened the contradiction — the benchmark was never closed-book (the lexicographer was always DPD-grounded). Set `closedBook: false` and rewrote the note to state accurately what models receive: raw Pāli PLUS DPD attestations at the Anatomist/Lexicographer passes, while still withholding the SuttaCentral dictionary, retrieval context, and prior-phase window that production adds (the SUTTA-014 parity gap) — so benchmark output is LESS grounded than production, not un-grounded. This is the code that stamps the field on the next generated board; the current published JSON is a stale July-3 preview that the v2.2 rerun replaces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L
…budget cap #2 — held-out leakage. The 30-phase ranked set claimed to be "honest, uncontaminated," but phase-ad, phase-ag and phase-aj each grade the exact `ātāpī sampajāno satimā` sequence that phase-aa teaches the model as a worked example (config/suttaStudioExamples) — the satipaṭṭhāna refrain recurs verbatim, so those three phases were the answer key in the prompt. Removed them from phasesToTest (now 27); re-goldening can't help, the sequence IS the example, and the refrain is still exercised once as phase-aa. Added tests/scripts/sutta-studio/phase-contract.test.ts, which derives the example sequences from the actual prompt example objects and fails if any ranked phase grades one — verified RED against the pre-fix config (it lists ad/ag/aj) and green after removal. Nothing hardcodes a 30-phase count; the survivorship logic reads phasesToTest.length. #6 — no spend cap. The runner estimated cost after each response, let an unknown price become null (counting as $0), and had no cumulative stop, so a mispriced model or runaway loop could spend unbounded. New SpendGuard (scripts/sutta-studio/spend-guard.ts) tracks cumulative spend; every LLM call accrues, and the run ABORTS at the next loop boundary once BENCHMARK_CONFIG.maxSpendUsd (default $50) is reached. Enforcement is at the model- and phase-loop tops, NOT inside a pass runner — a pass runner's try/catch would swallow the throw into a per-phase failure and let the run keep spending; a loop-top throw propagates to runBenchmark().catch and aborts. A null-cost call is charged a conservative $0.25 estimate so unpriced spend drives toward the cap instead of slipping through as $0. Unit-tested. NOT addressed here (still gates a paid run, per the review): the roster is still 6 models not the intended 12, the 40/30/30 formula decision, and the full 30-phase golden-contract repair (#1) — those are operator-directed and larger. tsc: 17 pre-existing errors, none in touched files. Full sutta-studio suite green (81). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Jul 16, 2026
anantham
marked this pull request as ready for review
July 17, 2026 13:48
This was referenced Jul 17, 2026
assertUnderBudget ran only at the model and phase boundaries, so a chunked skeleton (the biggest pass) or a repeat starting after the prior one exhausted the budget could overshoot the cap by that work. Add a fail-closed check inside the repeat loop, before the pass loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three well-scoped fixes from the "don't start the paid fleet yet" review — the ones that make a future run safe and honest without pre-empting the operator's bigger calls (formula, roster, full golden repair).
#6 — fail-closed cumulative budget cap
The runner estimated cost after each response, let an unknown price become
null(counted as $0), and had no cumulative stop — a mispriced model or runaway loop could spend unbounded. NewSpendGuardtracks spend; the run aborts at the next loop boundary onceBENCHMARK_CONFIG.maxSpendUsd(default $50) is hit. Enforced at the model/phase loop tops (a throw inside a pass runner would be swallowed into a per-phase failure and keep spending). A null-cost call is charged a conservative $0.25 so unpriced spend drives toward the cap instead of slipping through. Unit-tested.#2 — de-leak the ranked phase set
phase-ad/ag/aj each grade the exact
ātāpī sampajāno satimāthe prompt teaches as the phase-aa worked example (the refrain recurs) — the answer key was in the prompt, while the config called all 30 "honest, uncontaminated." Removed those three (ranked set now 27); the refrain is still tested once as phase-aa. Newphase-contract.test.tsderives the example sequences from the actual prompt objects and fails on any leak — verified RED against the pre-fix config (lists exactly ad/ag/aj), green after.#5 — fix the closed-book provenance
The board hardcoded
closedBook: true/ "raw Pāli only," contradicting its own DPD source row and reality (DPD is fed to Anatomist + Lexicographer; my P2.1 anatomist grounding sharpened the contradiction). SetclosedBook: falsewith an accurate note: raw Pāli plus DPD attestations, still short of production's dictionary/retrieval/prior-phase grounding.Explicitly NOT done here (still gate a paid run — operator calls)
gemini-3-flash).So: this closes the "unsafe to run" blockers I can close mechanically. It does not on its own make the board ready to rank — the formula, roster, and golden repair are still needed before the twelve-model retake.
Verification
🤖 Generated with Claude Code
https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L