Calibration round 2: rubric v2, agent-diff judge input, per-task dimensions, judge-matrix run plan#75
Open
szjanikowski wants to merge 56 commits into
Open
Calibration round 2: rubric v2, agent-diff judge input, per-task dimensions, judge-matrix run plan#75szjanikowski wants to merge 56 commits into
szjanikowski wants to merge 56 commits into
Conversation
Round-2 calibration of the weather-discount task (sink PRs #9-#21, 60 inline comments, human-approved reference buckets A/B/C/D): - assessment_criteria.md v2: decidable FULL/PARTIAL/NONE checks with real-trial anchors, one failure mode = one check = one dimension, base-model intent preamble ([Pure], [DddFactory], ExchangeRate is denomination not policy). Dimension names/scales unchanged. - ground_truth_decisions.json: seven reference decisions (closure in factory, no phantom discounts, canonical OfferLevelDiscount reuse, explicit exclusivity, failure != measurement, resemble existing code, restraint) - auto-injected into the judge prompt. - precheck.sh: deterministic restraint signals from git (judge has no git); advisory AC1-AC4 scores + hard-fail flag mapping to bucket-D score cap. Smoke-tested: #21 -> 10/20, #18 -> 1/20 hard-fail. - CALIBRATION_ROUND2_2026-07-07.md: reference ranking, v1 failure evidence, acceptance criteria (std/gap/Spearman/disjointness + regression assertions) and the orchestrator procedure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The weather-discount calibration round exposed two structural limits and this change removes them (loop-first: the toolkit adapts to calibration practice, not the other way around): - Per-task dimensions: assessment_dimensions.json next to a task's assessment_criteria.md now overrides the benchmark-wide file for that task only (resolve_dimensions_path; used by the evaluator and by calibrate publish). Different file -> different fingerprint, so old and new evaluations are never mixed in one summary group. - Deterministic precheck: an optional task-level precheck.sh runs on the host before judging (the judge has Read/Glob/Grep, no git). Its JSON is injected into the judge prompt as ground facts, recorded in assessment_eval_*.json, and its optional normalized_score_cap is enforced (with the cap application recorded for explainability). - calibrate publish now bundles ground_truth_decisions.json for the reviewer. Applied to ddd-weather-discount: three disjoint task-level dimensions (model_fit 50 / restraint 25 / test_quality 25), criteria v2 regrouped into M1-M7/R1-R6/T1-T5 decidable checks, dual-mode precheck.sh (evaluator workspace / calibration-sink refs) smoke-tested against the sink trials, round-2 doc updated with the new acceptance criteria. Skill nasde-benchmark-calibration and ARCHITECTURE.md document the new mechanisms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The judge sees only the final workspace state and - like a human reviewer without a diff - cannot see removals or out-of-feature edits. Instead of per-task precheck greps (which overfit to known failure modes) or free-form git access for the judge (which breeds variance - the codex judge always had shell and demonstrably didn't use it), the evaluator now materializes the agent's complete diff once, deterministically, for every task: - new workspace_diff module: capture_patch (moved from results_exporter, re-exported for compatibility) + capture_diffstat; shared by the exporter, the calibration publisher and the evaluator. - evaluator writes <trial>/agent_changes.diff and injects an "Agent diff" prompt section: diffstat inline, full diff as a file the judge can Read (paginated) and Grep; skipped gracefully without git or changes. - claude backend grants --add-dir on the trial dir when the diff exists (mirrors the trajectory plumbing). - weather-discount rubric v2: restraint checks R1-R4 now answered from the agent diff (universal), with precheck.sh narrowed to its real role - the mechanical bucket-D disqualification cap; docs and calibration skill updated accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ompt The "Agent diff" section alone only worked when a task's criteria told the judge to use it. The base "How to evaluate" procedure now starts from the diff whenever it exists - establish WHAT changed before judging HOW well - independent of what any task's assessment criteria say, so every rubric (current and future) benefits from the reference point. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- nasde eval gains --eval-model / --eval-backend per-run overrides, so a judge-model comparison matrix needs no nasde.toml edits between runs. - calibration_round2_run.sh: one-command matrix (claude-fable-5, claude-opus-4-8, gpt-5.5-codex; 13 trials x 3 judges x 3 reps), followed by the acceptance check and a results-export + commit into the nasde-results repo (push opt-in via NASDE_RESULTS_PUSH=1). - calibration_round2_check.py: the five stop conditions as executable assertions (repeatability std, cross-judge gap, Spearman + strict bucket separation vs the human reference ranking, dimension disjointness, pinned regression assertions incl. the #18/#20 cap and a WARN-level anti-gaming guard on high test_quality). - calibration_round2_restore.sh: rebuilds a trial dir from the calibration sink for trials whose local jobs/ artifacts are gone (URtZnzf, FjYQ3XQ) - workspace HEAD = base snapshot, agent diff applied uncommitted, result/config synthesized from metrics.json; both restores verified against workspace_diff.capture_patch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bsent validate_auth demanded ANTHROPIC_API_KEY/CLAUDE_CODE_OAUTH_TOKEN even though the backend deliberately omits --bare precisely so the claude CLI can read OAuth from the keychain (subscription accounts) - the preflight contradicted the documented design and blocked keychain-authenticated runs. Downgraded to an informational note; a truly unauthenticated CLI still fails loudly on the first evaluation call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Durable record of the measured v2.0 ordering defects (top tie #21/#16, B/C inversion #13>#14 confirmed on n=2 means), the check-level cause matrix from Fable's reasoning, three surgical M1/M4/M5 edit proposals with a simulation restoring the exact human reference order, and the application protocol (corroborate with cheaper judges first, fingerprint bump on apply). NOT applied - owner gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The three scripts actually used to drive the budget-constrained Fable subset (previously session-scratchpad only, promoted so they survive): - calibration_round2_eval_one.sh <suffix>: exactly ONE evaluation of one trial (per-eval manual gating: run -> discuss -> owner decides). - calibration_round2_fable_subset.sh: the 5-trial x 2-rep subset driver (counts existing v2 fable evals, tops up at concurrency 1, sleeps out exhausted subscription windows). - calibration_round2_stop_after_one.sh: watchdog that kills the driver the moment one new eval lands (converts a running loop into a single step without wasting the in-flight evaluation). Paths made repo-relative; see CALIBRATION_ROUND2_PENDING_RUBRIC_FIXES.md for the live round state these operate on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…S / 2 FAIL, both FAILs = the two defects the pending v2.1 edits target Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…from the Fable subset Three surgical edits, each fixing a measured v2.0 ordering defect (10/10 subset evals, check-level verdicts stable across reps): - M1: sequential-instead-of-parallel awaiting no longer degrades to PARTIAL (style note at FULL); PARTIAL reserved for closures capturing unresolved dependencies. v2.0 priced a style nuance at half the heaviest check, dragging solid trials (#14, #16) below flawed ones. - M4: factory-filtered rules with an always-present but possibly-empty SHARED aggregate now earn FULL - that shape IS the owner's recipe ("add to the aggregation only when the weather qualifies"); PARTIAL reserved for self-disabling modifiers deciding inside ApplyOn. - M5: middle tier added - PARTIAL for stacking AFTER the existing chain (silent or comment-stated), NONE reserved for applying BEFORE the chain (feeding min()) or tests canonizing compounding. v2.0's floor equated two qualitatively different sins, hiding #13's worst-variant ordering behind #14's conservative default. assessment_dimensions.json carries a rubric_version marker so the fingerprint changes: v2.1 evaluations form their own summary groups, never mixed with v2.0 (same confound-isolation rule as for evaluator params). Simulation on collected verdicts: order becomes 0.66/0.64/ 0.58/0.55/0.445 = exactly the human reference buckets (Spearman 1.0); live v2.1 confirmation run pending (Fable after window reset, Opus back-to-back before/after comparison planned). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ynamically Hardcoded v2.0 fingerprint would silently miscount after any rubric version bump; the scripts now derive it from the task's assessment_dimensions.json, so they always target the current rubric version's evaluation group. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… Opus back-to-back procedure Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- variants/claude-supple: the "supple design" (Evans) expectation as a variant CLAUDE.md, NOT an instruction change - the task stays canonical so past trials remain comparable; the coaching lives where engineering culture lives. - Round doc: live v2.1 verification (4 anchors, all simulation predictions confirmed; negative control exact; reference order A>B>C restored), both Fable-coder trials (vanilla 0.66, deeper- instruction probe 0.70 - highest ever: first composition test in 15 trials, dedicated WeatherDiscounts module, and a discovered+fixed latent bug in base Discount.Value(isPercentage:true)), and the open round-3 question: should a justified, tested bug fix in a pre-existing file cost restraint points? Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ings Four edits from the round-2 live verification (owner-approved): - R1: a justified, tested bug fix in a pre-existing file is restraint-neutral (defect demonstrable + minimal fix + covering test); sharply distinguished from behavior fabrication, which stays NONE/hard-fail. Trigger: the deeper-instruction Fable trial found and fixed a real latent bug (base Discount.Value passes isPercentage: true) and was the only one of 15 trials to notice it. - R4 + precheck: root CLAUDE.md/AGENTS.md and .claude/ /.codex/ are injected by the harness (variant sandbox_files), not created by the agent - excluded from the artifact check, which previously docked every trial uniformly for harness files. Agent-created junk (decompile.csx) is still caught (verified offline on sink trials). - M4: construction-time qualification resolution codified as factory-time FULL (aligns the text with the judge's validated reading on #14, removing cross-judge interpretation room before the Opus comparison). - precheck R3 mapping softened (1-2 changed declaration lines -> PARTIAL) to match the rubric's intent for single cosmetic rewrites. - Hard rule reworded: precheck signals are facts, verdicts are the judge's; suggested_scores are advisory. Fingerprint: 8e0d00bfd8df (v2.0) -> 42d6a9e593cf (v2.1) -> 25e9d07f8b31 (v2.2). Offline checks: #18 hard-fail intact, #17 R4 no longer dinged for harness files, #10 decompile.csx still flagged, #21 R3 PARTIAL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…next actions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The spec is silent on discount interaction and the agent has no channel to ask, so the direction of the decision is free: a tested accumulation assumption and a tested exclusivity decision score identically. M5 now prices VISIBILITY of the decision only; the 'tests with no spec basis' deduction is removed (a test introducing a hypothetical second weather rule to pin combination semantics is the strongest form of visibility, not a fabrication). Applying weather BEFORE the existing chain stays NONE. Base-intent note corrected after auditing the start-state code: the base has NO Pricing tests; every implemented interaction idiom chooses (ClientLevelDiscounts override, IndividualSalesConditions min(), SpecialOffer.Or fallback with unimplemented bodies), but the unused AggregatedModifier is a base-provided sequential-composition idiom - the base prescribes visibility, not a direction. Ground truth entry renamed explicit-exclusivity -> explicit-interaction accordingly. Fingerprint: 25e9d07f8b31 (v2.2) -> fdb83b1535a6 (v2.3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…ction policy is the M5 ceiling A tested single-direction decision (accumulation or exclusivity) scores FULL at 6/7. The 7th point is reserved for reifying the choice itself: exclusive-vs-accumulate modeled as a configuration decision, both policies expressible through composition idioms (the way .Or and AggregatedModifier already embody the two directions) and demonstrated both ways in tests - reward any genuine movement in this direction. Ground truth canonical example updated accordingly. Amended pre-first-run: fp fdb83b1535a6 -> 37ffc5f460d2, no evaluations existed under the earlier fingerprint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
Replace the prose 'exemplary ceiling' with an explicit verdict ladder: MAX (7) / FULL (6) / PARTIAL / NONE, and state that M5 is the one check with a verdict above FULL. Semantics unchanged from the previous commit; dimensions JSON untouched, so the v2.3 fingerprint stays 37ffc5f460d2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
… bar Replace 'reward any genuine movement in this direction' (a sentiment that licenses generosity for extensible-looking structure) with a decidable gate: MAX requires (a) the composition point where a configurer selects the interaction policy without touching discount logic (file + line) and (b) executable proof the second policy is real - a test exercising it or a working alternative combinator. Prose is not movement: extensibility comments, unused strategy parameters and TODOs stay FULL. Dimensions JSON untouched; v2.3 fingerprint remains 37ffc5f460d2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…niversal proof discipline Rewrites the supple instruction in its universal form: study the model until the deeper idea is understood (what concepts mean / how they compose / why the author shaped them), fit as if the original author had extended it, only changes the concept requires. Drops 'maintaining extension points for the future' - the one phrase that biased toward standing always-in-chain structures (M4 PARTIAL in the BAzkEPJ probe). Adds two standing universal hints (user's design values, mechanism-free): deliberate edge-case thinking (boundary values, missing data, failure paths) with honest tests, and tests-as-documentation - every property the design claims, including its extension points, must be demonstrated by an executable test, not a comment. Phrased as 'properties your design claims' to avoid inviting speculative extension points built for the sake of testing them. Variant has never been run - redefinition invalidates nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…nti-abstraction restraint Per user direction: 'supple design' now carries a compact explanation of what it is (a design that invites change; client code as a statement of intent) and its six Evans elements - intention-revealing interfaces, side-effect-free functions, assertions, conceptual contours, standalone classes, closure of operations - framed explicitly as inspiration, not a checklist. Textbook doctrine, mechanism-free: it names design values, not this task's solution. Paired with the new restraint clause: avoid interfaces nothing needs yet, model only the abstractions the concept truly requires - a supple model stays small and concrete wherever the domain is concrete. Variant still never run - redefinition invalidates nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…ive, supple-v3 probe measured Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…silent boundary Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…efore-chain NONE held Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…e 2026-07-09 Also note the claude-opus-4-8 output-rate discrepancy (toml 15 vs official 25) - left unchanged pending owner decision on restating June figures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…ved restatement) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
… variant A file at variants/<v>/tasks/<task>/instruction.md replaces that task's instruction for the variant's runs only: the task dir is staged to a temp copy with instruction.md swapped, Harbor gets the staged path, and after the job each override trial's task_id.path is normalized back to the canonical task dir with the swap recorded as instruction_override (project-relative). Evaluations - including future re-evaluations - keep running against the live rubric and canonical instruction, matching the precedent set by the BAzkEPJ probe. variants/claude-supple-hint reifies that probe: canonical ddd-weather-discount instruction with the 'Fit into the existing DDD architecture' bullet replaced by the original supple-design paragraph (text extracted verbatim from the BAzkEPJ trajectory), so the 0.85 arm of the article experiment is a committed, reproducible artifact. Tests: staging swaps only instruction.md, respects task filter, registry override plumbing, task_id normalization + provenance; pricing test set updated for claude-fable-5. Full suite 466 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…riant) The instruction is part of the benchmark's definition, not of a variant: variants may differ in what the agent brings (CLAUDE.md, skills), never in what it is asked to do - otherwise scores are not comparable. Changing the instruction means a new benchmark version and re-running ALL variants against it. The BAzkEPJ instruction-swap probe stays an exploratory side-note, not a publishable arm. Keeps the unrelated test_pricing claude-fable-5 fix from the reverted commit. Suite: 461 passed. This reverts commit f79645a. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…instruction untouched The owner-approved minimal-hint arm for the Fable article: vanilla CLAUDE.md plus one explicit sentence to think deeply about the existing domain model before coding. The task instruction stays canonical - the only manipulated variable across arms is the CLAUDE.md content (vanilla / minimal hint / rich supple v3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…ults Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…E null-object Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
… variance real Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…, mean ~0.78 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…notated, arm means 0.75 vs 0.78 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
….03/+0.05 across judges Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…-coder arm n=2 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
Root cause of the 137/SIGKILL trial failures (A5TDdvJ, tXRxnyj): kernel memcg OOM at the task's 6 GiB limit. The OOM process table shows the claude agent at ~71 MB and 45+ concurrent dotnet processes at ~120-150 MB each - three generations of MSBuild worker fleets (~16 nodes per build on a 16-core host, default nodeReuse keeps them alive 15 min) accumulated across the agent's repeated build/test cycles. Heavy arms (skill, some hint runs) iterate builds most and crossed the ceiling at ~23 min; light runs finished before the third generation spawned. MSBUILDDISABLENODEREUSE=1 + DOTNET_CLI_USE_MSBUILD_SERVER=0 make build workers exit with each build: steady-state ~1-1.5 GiB, comfortably inside the existing 6 GiB limit. Cost: ~1-2 s slower build startup, equal for all arms; task semantics, instruction and rubric untouched. Raising memory_mb instead would only postpone the same accumulation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…e) + OOM fix verified live Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…straint 25, tests 11) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…hapes finale Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…ape recorded Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…rid recorded Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
… +0.10 -> +0.03 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…/EN) 96 grid evaluations as raw dots (shade=trial, connector=judge pair, larger dot=identical pair) and the judge repeatability diagonal (mean |d| 0.017 Fable / 0.022 Opus, ties 8/24 and 6/24, max 0.05-0.06). Publication-ready EN variants included; owner decision: raw clouds carry the article's conclusions, no bootstrap CI at n=4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…pup scripts trial_probe.sh parameterizes the pattern used for all 24 grid trials (token preflight with stale-token refresh, canonical-instruction guard, coding run at xhigh, workspace smoke check, 2x Fable + 2x Opus evals, results digest). eval_topup.sh completes judge pairs on existing jobs after window limits or interruptions. Both previously lived only in the session scratchpad and were lost once to a reboot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…kill Future sessions (any working directory, any agent) now discover trial_probe.sh / eval_topup.sh / the handover doc / the owner protocol without relying on one session's memory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…erified 2026-07-13 Rates confirmed unchanged on the official pricing page (Fable 5 10/50, Opus 4.8 5/25 per MTok); as_of dates bumped. Chart reads rates live from pricing.toml and computes house-formula costs from trial trajectories. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…asoning Per-check verdicts recovered from free-text dimension reasoning with a score-consistency audit; three known Opus-judge prose/score mismatches are flagged on stdout and kept as prose says. M5 FULL=6/7 handled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…/ R teal / T amber) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
cost_usd now bills fresh input at the full rate, cache writes at the 1h write rate, cache reads at the cached rate - matching Harbor's own per-step accounting to the cent on 20/24 grid trials. The full-rate ceiling is no longer stored (derivable from token_usage). TokenUsage gains cache_write_tokens; pricing.toml gains cache_write_per_1m. Grid reality check: campaign $958 as-if -> $220 real; cache reads are 92.6-97.7% of input on all 24 trials. Charts repriced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…w are pre-change Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
ARCHITECTURE.md (also dropped the stale efficiency-ratio claim), the twin AGENTS/CLAUDE knowledge blocks, the website token-cost page (user-facing), the scaffolded pricing.toml.example (now shows cache rate fields), and a new CHANGELOG entry. ADR-011 text and the 0.5.0 changelog entry stay as historical records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
…repo
The round-2 campaign's working artifacts are not toolkit deliverables:
- CALIBRATION_ROUND2_PENDING_RUBRIC_FIXES.md (session day log / handover)
- trial_probe.sh, eval_topup.sh, calibration_round2_*.sh (machine-local drivers)
- grid_raw_scores_plot{,_en}.py + raw_scores_grid_{v3,en}.png (chart iterations)
They now live in NoesisVision/nasde-calibration (round2-fable-grid-ops/).
References repointed to the public methodology record
(CALIBRATION_ROUND2_2026-07-07.md). Also drop repo-root AGENTS.md (broken
Claude->Codex text substitution; untested).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0112rjaT9dWVPXhqm2FCQQPh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the first full human-in-the-loop calibration iteration for
ddd-weather-discount(13 sink PRs reviewed, 60 inline comments, human-approved reference ranking A/B/C/D) and extends the toolkit where the loop needed it.Engine (generic, all tasks)
workspace_diff.capture_patch, shared withchanges.patchexport) into<trial>/agent_changes.diff; judge prompt gets diffstat inline + the file for Read/Grep; base "How to evaluate" procedure is now diff-first whenever a diff exists. Claude backend grants--add-diron the trial dir.assessment_dimensions.jsonoverrides the challenge-wide file (resolve_dimensions_path); different fingerprint → never mixed with old evals.precheck.shruns on the host; JSON injected into the prompt as ground facts, recorded in eval json; optionalnormalized_score_capenforced (cap application recorded).nasde eval --eval-model/--eval-backendper-run judge overrides;calibrate publishbundles per-task dims +ground_truth_decisions.json.Task: ddd-weather-discount (calibrated assets)
model_fit50 /restraint25 /test_quality25.ground_truth_decisions.json(7 reference decisions),precheck.sh(bucket-D cap; smoke-tested: docs: add Noesis website link to README #18 → cap 0.45, Support claude_config.json for MCP server configuration #21 → no cap).Round-2 run plan
CALIBRATION_ROUND2_2026-07-07.md: reference ranking, v1 failure evidence, acceptance criteria (stop conditions).calibration_round2_check.py: the stop conditions as executable assertions (repeatability std ≤ 8%, cross-judge gap ≤ 12%, Spearman ≥ 0.8 + strict bucket separation, dimension disjointness |r| ≤ 0.6, pinned regressions).calibration_round2_run.sh/calibration_round2_restore.sh: judge matrix (fable-5 / opus-4.8 / gpt-5.5-codex) + sink-restore for the two trials whose local job dirs are gone; export + commit to nasde-results.461 tests green, ruff clean. Skill
nasde-benchmark-calibrationandARCHITECTURE.mdupdated.🤖 Generated with Claude Code