Benchmark v2.2: ranked formula = gate × (0.40 seg + 0.30 facts + 0.30 sense)#119
Open
anantham wants to merge 1 commit into
Open
Benchmark v2.2: ranked formula = gate × (0.40 seg + 0.30 facts + 0.30 sense)#119anantham wants to merge 1 commit into
anantham wants to merge 1 commit into
Conversation
…re weights) Operator resolved the review's finding #3: the 40/30/30 weights are the FINAL score, not the fidelity-internal weights nested inside the old 60/25/15. So the ranked score becomes: overall = gateFactor × (0.40·segmentationF1 + 0.30·factsCore + 0.30·senseF1) quality-scorer.ts (RUBRIC_VERSION 2.1 → 2.2; leaderboard RANKED_RUBRIC_VERSION → 2.2): - segmentationF1 — morpheme-boundary micro-F1 vs golden (unchanged). - factsCore — root + word-class macro from scoreFactsDetail. Morph is EXCLUDED (advisory this cycle per the reviewer); its gaming + vocabulary were fixed separately (review #4) so it can be promoted later. Scored against the golden's DPD-verified √tooltips (no external DPD needed). - senseF1 — strict sense-english micro-F1 (scoreSenseFidelityDetail, SUTTA-012 drop-penalised). - Usability and richness are RETIRED from rank (still computed for display); alignment, the LLM judge and the reader probe stay advisory. - Components a phase's golden doesn't provide (e.g. a function-only phase with no senses) renormalise over the present weights rather than being charged 0. - fidelityScore stays the v2.1 value, still used as the leaderboard's "has a golden → rankable" gate. The board's methodology string now discloses the v2.2 formula; ADR SUTTA-013 amended with the decision + implementation. Note the facts-scorer import creates a call-time cycle with quality-scorer (which exports alignWords/tokenize back) — safe because both sides use the other's exports only inside functions. Tests: 4 new guards (the 40/30/30 weights, the retirement of usability/richness, renormalisation), verified RED against the v2.1 scorer (it returns 0 where v2.2 returns 0.55 — it never read factsCore/senseF1). Full sutta-studio + scorer suite green. tsc 17 (baseline). This is a rubric bump — v2.1/v2.2 scores are not comparable (version gate enforces it), so it needs a FLEET RE-RUN before the board is republished. Remaining follow-ups: board columns for factsCore/senseF1, and the twelve model IDs for the run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
anantham
marked this pull request as ready for review
July 17, 2026 13:48
This was referenced Jul 18, 2026
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the v2.2 ranked formula. Your call resolved the review's finding #3 — the 40/30/30 are final-score weights, not fidelity-internal weights nested in the old 60/25/15. So:
What changed
RUBRIC_VERSION/RANKED_RUBRIC_VERSION→ 2.2; the board's methodology string discloses the new formula; ADR SUTTA-013 amended with the decision + implementation.Verification
Important: this is a rubric bump
v2.1 and v2.2 scores are not comparable (the version gate enforces it), so the board needs a full fleet re-run before republishing.
The whole benchmark-validity arc — where it lands
Every review finding is now closed across the PRs: #114 (P1/P2 code bugs), #116 (de-leak + budget cap + provenance), #117 (facts scorer #4), #118 (golden contract #1), and this (the v2.2 formula, #3). The one thing left to actually run the paid retake is the twelve model IDs (and whether
gemini-3-flashis in). Give me those and I can kick off the run — the fail-closed $50 cap and the 30/30 contract gate will keep it safe.Two small display follow-ups remain (not blockers): board columns for factsCore/senseF1.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L