Benchmark validity #4: make the facts scorer un-gameable (morph denominator + vocabulary)#117
Open
anantham wants to merge 2 commits into
Open
Benchmark validity #4: make the facts scorer un-gameable (morph denominator + vocabulary)#117anantham wants to merge 2 commits into
anantham wants to merge 2 commits into
Conversation
…ph + vocab
The proposed facts score (advisory today, ranked under the v2.2 40/30/30 proposal) had two
confirmed defects the review flagged:
1. A bogus assertion could RAISE the macro. Morph was graded per-word, gated on the model
having asserted *something*, and fitsReading passed VACUOUSLY when the DPD reading was
silent on the asserted key. So `{note: "nonsense"}` made the word "assert morph", the
junk key vacuously "fit", and morph appeared as a free 1.0 category — the reviewer
measured a weak macro rise 0.500 → 0.667.
2. A correct assertion could score WRONG on vocabulary. `case: "ins"` vs DPD's `instr` are
the same case, but the exact-string compare called it a miss (macro → 0.333).
Fix:
- Grade morph per AUTHORITY-KNOWN key, not per model assertion. For each eligible word,
the gradeable keys are the gender/case/number a reading actually specifies; the model's
value on such a key is scored (correct iff it matches some reading), and keys no reading
covers — like an invented `note` — are IGNORED. The denominator is authority-set, so extra
assertions can't game it; omission of a known key stays uncharged (silence ≠ wrong), but an
unconfirmable assertion is no longer free.
- Canonicalise morph vocabulary before comparison (`ins`≡`instr`, `m`≡`masc`, `sg`≡`sing`, …),
so equivalent abbreviations match.
- `pos` is excluded from morph (open-ended vocabulary; the word-class check covers it) and its
docstring now says honestly that the `pos` breakdown is a word-class agreement check, not
morphological part-of-speech — matching the reviewer's accepted "root/word-class macro".
Tests: two new guards reproduce the reviewer's exact repros, verified RED against the pre-fix
scorer (junk inflates morph to 1/1; `ins` scores 0/1). One existing morph test updated for
the deliberate per-word → per-key change (kāye 2/2, kurūnaṁ 0/2). Facts + quality scorer
suites green (28).
NOT addressed (still gates a paid run, operator-directed): the 12-model roster, the 40/30/30
formula decision, and the full 30-phase golden-contract repair (#1).
tsc: 17 pre-existing errors, none in touched files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Jul 16, 2026
anantham
marked this pull request as ready for review
July 17, 2026 13:48
…view) The morph consistency check unioned acceptable values per key across all ambiguous readings, so a model could assemble a valid-looking analysis from keys drawn from different, mutually-exclusive readings (case from one, number from another) and be fully accepted. Now score against the single reading that matches the most of the model's asserted keys. Morph is advisory (excluded from ranked factsCore) so this is latent, but real. New test verified RED against the union code; the existing morph test is unchanged (its keys all match one reading). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two confirmed facts-scorer defects from the "don't run the paid fleet yet" review. The facts score is advisory today; this makes it trustworthy for when it's promoted to rank under the proposed v2.2 40/30/30 formula.
The two bugs (both reproduced, both fixed)
1. A bogus assertion could raise the macro (0.500 → 0.667). Morph was graded per word, gated on the model asserting anything, and
fitsReadingpassed vacuously on a key the DPD reading was silent about. So{note: "nonsense"}made a word "assert morph", vacuously "fit", and appear as a free 1.0 category.Fix: grade morph per authority-known key (the gender/case/number a reading actually specifies), over adjudicable assertions only. Keys no reading covers — like an invented
note— are ignored, so the denominator is authority-set and extra assertions can't game it. Omission of a known key stays uncharged (silence ≠ wrong); an unconfirmable assertion is no longer free.2. A correct assertion could score wrong on vocabulary (
insvsinstr→ 0.333). Exact-string compare.Fix: canonicalise morph values before comparison —
ins≡instr,m≡masc,sg≡sing, etc.Also:
posis excluded from morph (open-ended vocabulary; covered by the word-class check) and its docstring now says honestly it's a word-class agreement check, not morphological part-of-speech — matching the reviewer's accepted "root/word-class macro" framing.Verification
insscores 0/1).Still gating a paid run (operator-directed, not in this PR)
The 12-model roster, the 40/30/30 formula decision, and the full 30-phase golden-contract repair (#1).
🤖 Generated with Claude Code
https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L