Rebuild the YAML scope-gap metric on an independent CST oracle (#18)

johnsoncodehk · web-flow · commit 4e5e656ac826 · 2026-06-07T13:52:45.000Z
RedCMD's monogram#12 comment (issuecomment-4640869820) listed ten YAML highlighting bugs. The scope-gap metric reported zero of them and claimed 100% / 0 Monogram-wrong: it was measuring agreement with a co-biased oracle, not correctness. Five blind spots let the bugs through: A. the oracle hand-rolled comment/directive/flow/tag with regexes that reproduced the same mistakes, so it graded the buggy scope as correct; B. isGradable skipped every invalid input (much of the report); C. the corpus was the yaml-test-suite only (the repros weren't in it); D. each oracle token was graded only at its START char, so a bug mid-span (a %YAML folded into a plain scalar, a # line in a |5 body) was invisible; E. the oracle was silent on whole constructs, so a metric that only grades where its oracle speaks could never see those bugs. This reworks the metric only; the grammars are unchanged and the other six languages are byte-identical. - yaml-oracle.ts is rebuilt on the yaml package's low-level CST (new Parser().parse()) for presentation tokens (comment, directive, anchor, alias, tag, flow punctuation, block-scalar, doc markers) plus the AST for scalar value-typing, key detection and escapes. The CST is a different author than Monogram's grammar and is error-tolerant, so it can't share the blind spots and it grades malformed input too. (A,B,E) - scope-gap.ts grades the FULL span of each oracle token, not just its start (opt-in per adapter; on for YAML, off elsewhere so the fine-grained TS/HTML oracles are unaffected). (D) - scope-gap.ts adds a differential pass: every position where Monogram and the official grammar paint different visual classes AND the oracle is silent is flagged, separating genuine class-confusion from the error-overlay design difference. This oracle-independent net keeps an unmodeled construct from becoming a silent blind spot. (E) - yaml-issue12-regressions.ts pins all ten repros as asserted should-be scopes (xfail for the ones still open) and runs in CI. (C) The metric now exposes seven real Monogram YAML bugs (previously zero): items #6, #7, #8, #9, #10 from the report, plus two it did not list (an anchor mis-detected inside a plain scalar, and a multi-line key typed as a constant). The grammar fixes for those are a follow-up; this is only the measurement. Refs #12
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -55,6 +55,7 @@ jobs:
           node test/vue-directives.ts
           node test/vue-embed-boundary.ts
           node test/vue-interp-expr.ts
+          node test/yaml-issue12-regressions.ts
 
   # The derived tree-sitter highlighter is the strongest thesis proof (a real GLR
   # parser from the same grammar, beating the official hand-written one). Build its
diff --git a/package.json b/package.json
@@ -12,6 +12,7 @@
     "conformance:html": "node test/html-conformance.ts",
     "test:tm-diagnostics": "node test/redcmd-tm-diagnostics.ts",
     "test:tm-guards": "node test/tm-highlight-guards.ts",
+    "test:yaml-issues": "node test/yaml-issue12-regressions.ts",
     "spike:html-lexer": "node test/html-lexer-spike.ts",
     "bench:html-official": "node test/html-bench.ts",
     "bench:html-embed": "node test/html-embed-js.ts",
diff --git a/test/scope-gap-yaml.ts b/test/scope-gap-yaml.ts
@@ -12,6 +12,7 @@ import { dirname, join } from 'node:path';
 import { parse as yamlParse, parseAllDocuments } from 'yaml';
 import { run } from './scope-gap.ts';
 import { yamlOracle } from './yaml-oracle.ts';
+import { cases as issue12 } from './yaml-issue12-regressions.ts';
 
 const OFFICIAL = process.env.MONOGRAM_OFFICIAL_YAML ?? '/tmp/redcmd-yaml/syntaxes/yaml.tmLanguage.json';
 // The RedCMD/VS Code YAML grammar is a dispatcher stub that include()s version-specific
@@ -37,6 +38,10 @@ for (const f of readdirSync(SUITE).filter((n) => n.endsWith('.yaml'))) {
     }
   } catch { /* skip */ }
 }
+// Plus the RedCMD monogram#12 repros (many are tiny edge/error inputs absent from the suite) so the
+// metric actually SEES the constructs the comment flagged. Asserted should-be scopes live in their
+// own gate (yaml-issue12-regressions.ts); here they just widen what the gap/differential pass covers.
+for (const c of issue12) corpus.push({ name: `monogram#12 ${c.id}`, text: c.src });
 
 await run({
   name: 'YAML',
@@ -46,6 +51,18 @@ await run({
   monogramPath: 'yaml.tmLanguage.json',
   loadCorpus: () => corpus,
   roleOracle: yamlOracle,
-  // grade only inputs the parser fully accepts (the oracle is meaningful there).
+  // The GRADED headline stays valid-only: on malformed YAML the AST's key/value resolution is itself
+  // unreliable, so grading it would inject false "Monogram-wrong" tokens and poison the very signal
+  // we're making trustworthy. The invalid-input blind spot is instead closed by TWO mechanisms that
+  // stay honest there: (1) the asserted regression gate (yaml-issue12-regressions.ts) pins the
+  // should-be scope of the specific malformed repros (#4/#5/#8); (2) the differential pass below runs
+  // on ALL inputs and FLAGS invalid-input divergences for human review without auto-judging them.
   isGradable: (text) => { try { return parseAllDocuments(text).every((d: any) => d.errors.length === 0); } catch { return false; } },
+  // YAML's oracle emits COARSE, role-homogeneous spans (a whole plain scalar, a block-scalar body, a
+  // directive line); grade every char so a bug mid-span (a `%YAML` folded into a scalar, a block line
+  // bailing to a comment) is caught instead of hidden behind a correct start. See scope-gap.ts.
+  fullSpan: true,
+  // Also report oracle-INDEPENDENT divergences (Monogram vs official, where the oracle is silent) so a
+  // construct the CST oracle doesn't model can't become a silent blind spot. See scope-gap.ts.
+  differential: true,
 });
diff --git a/test/scope-gap.ts b/test/scope-gap.ts
@@ -16,8 +16,8 @@ import { readFileSync, existsSync } from 'node:fs';
 import { createRequire } from 'node:module';
 import vsctm from 'vscode-textmate';
 import onig from 'vscode-oniguruma';
-import { gradeScopeStack, isCorrect, ROLE_SPEC } from './scope-roles.ts';
-import type { RoleName } from './scope-roles.ts';
+import { gradeScopeStack, isCorrect, ROLE_SPEC, normScope } from './scope-roles.ts';
+import type { RoleName, Verdict } from './scope-roles.ts';
 
 // A parser-assigned token role over a span. The per-language oracle returns these (e.g.
 // oracle.ts for TS/JS). Only start/end/text/role are required by the harness.
@@ -35,6 +35,21 @@ export interface ScopeGapAdapter {
   loadCorpus: () => { name: string; text: string }[];
   roleOracle: (text: string) => GoldToken[];   // parser → per-token role (the neutral oracle)
   isGradable?: (text: string) => boolean;       // skip inputs the oracle can't judge (default: all)
+  // Grade every non-whitespace codepoint of an oracle token, not just its start offset. Needed when
+  // the oracle emits COARSE spans (a multi-line YAML plain scalar, a block-scalar body) whose role
+  // must hold across the whole span: a token correct at its start but wrong mid-span (a `%YAML`
+  // folded into a plain scalar, a block-scalar line bailing out to a comment) is otherwise invisible.
+  // Default false (start-only) — opt in only where the oracle's spans are role-HOMOGENEOUS, i.e. the
+  // role applies to every char (NOT e.g. a TS template literal whose `${…}` holes are expressions).
+  fullSpan?: boolean;
+  // Run the DIFFERENTIAL pass: report every position where Monogram and the official grammar paint
+  // DIFFERENT visual token classes (comment/string/number/keyword/name) AND the oracle has NO opinion
+  // there (no non-lexical gold token covers it). These are UNADJUDICATED divergences — the metric
+  // cannot say who is right, so they are flagged for human review. This is the structural fix for the
+  // "oracle is silent → blind spot" failure mode: a metric that only grades where its oracle speaks
+  // can never see a bug in a construct the oracle does not model; the differential pass catches the
+  // disagreement regardless, so a clean run means "no Monogram-wrong AND no unexamined divergence".
+  differential?: boolean;
 }
 
 const { INITIAL, Registry, parseRawGrammar } = vsctm;
@@ -89,6 +104,70 @@ function scopeAt(toks: TmToken[], pos: number): string[] {
   while (lo <= hi) { const mid = (lo + hi) >> 1; if (toks[mid].start <= pos) { ans = mid; lo = mid + 1; } else hi = mid - 1; }
   return ans >= 0 && toks[ans].end > pos ? toks[ans].scopes : [];
 }
+const innerOf = (s: string[]): string => (s.length ? s[s.length - 1] : '(none)');
+const isWs = (c: number): boolean => c === 32 || c === 9 || c === 10 || c === 13;
+// Grade an oracle token. start-only (default) reproduces the historical single-point sampling; full
+// grades every non-whitespace codepoint in [start,end) and returns the WORST verdict (first WRONG
+// position wins, with its scope for reporting) — so a coarse span that is right at its start but
+// wrong mid-span reads WRONG. Whitespace is skipped (continuation indent is colourless either way).
+interface SpanGrade { v: Verdict; scope: string; wrongAt?: number }
+function gradeSpan(role: RoleName, toks: TmToken[], start: number, end: number, text: string, full: boolean): SpanGrade {
+  if (!full) { const s = scopeAt(toks, start); return { v: gradeScopeStack(role, s), scope: innerOf(s) }; }
+  let worst: Verdict = 'exact';
+  for (let p = start; p < end; p++) {
+    if (isWs(text.charCodeAt(p))) continue;
+    const s = scopeAt(toks, p);
+    const v = gradeScopeStack(role, s);
+    if (v === 'wrong') return { v: 'wrong', scope: innerOf(s), wrongAt: p };
+    if (v === 'family' && worst === 'exact') worst = 'family';
+  }
+  return { v: worst, scope: innerOf(scopeAt(toks, start)) };
+}
+
+// ─── differential pass: oracle-INDEPENDENT divergence detector ───────────────────────────────────
+// Map a TM scope chain to a coarse VISUAL bucket — the level at which a highlight difference is
+// actually visible (a comment vs a string vs a number vs a keyword). `name` (identifiers/entities) and
+// `punct`/`none` are convention noise we don't flag on their own; a divergence is only interesting if
+// at least one side is a visually-distinct class. Innermost wins; scan inner→outer for the first hit.
+type Bucket = 'invalid' | 'comment' | 'string' | 'number' | 'keyword' | 'name' | 'punct' | 'none';
+function scopeBucket(chain: string[]): Bucket {
+  for (let i = chain.length - 1; i >= 0; i--) {
+    const s = normScope(chain[i]);
+    if (/^invalid/.test(s)) return 'invalid';   // an error overlay (official marks errors; a highlighter may legitimately highlight-normally instead)
+    if (/^comment/.test(s)) return 'comment';
+    if (/^constant\.numeric/.test(s)) return 'number';
+    if (/^(string|constant\.character|constant\.other\.symbol)/.test(s)) return 'string';
+    if (/^(keyword|storage|constant\.language|support\.constant|variable\.language)/.test(s)) return 'keyword';
+    if (/^(entity|variable|support|constant)/.test(s)) return 'name';
+    if (/^punctuation/.test(s)) return 'punct';
+  }
+  return 'none';
+}
+// Visually-distinct classes whose confusion is a real (not convention-noise) difference. `invalid`
+// is reported but flagged separately below — official-marks-error vs Monogram-highlights-normally is a
+// design stance (cf. monogram#12 #3 "should still be highlighted normally"), not necessarily a bug.
+const DISTINCT = new Set<Bucket>(['invalid', 'comment', 'string', 'number', 'keyword']);
+const involvesInvalid = (a: Bucket, b: Bucket): boolean => a === 'invalid' || b === 'invalid';
+// a divergence matters when the two buckets differ AND at least one is a visually-distinct class
+const interestingDivergence = (a: Bucket, b: Bucket): boolean => a !== b && (DISTINCT.has(a) || DISTINCT.has(b));
+
+export interface Divergence { pos: number; text: string; mono: string; off: string; bM: Bucket; bO: Bucket }
+// Positions where the two grammars disagree visually AND no non-lexical oracle token adjudicates.
+function divergences(off: TmToken[], mono: TmToken[], gold: GoldToken[], text: string): Divergence[] {
+  const cov = gold.filter((g) => { const t = ROLE_SPEC[g.role]?.tier; return t && t !== 'lexical'; }).map((g) => [g.start, g.end] as const);
+  const isCovered = (p: number) => cov.some(([a, b]) => p >= a && p < b);
+  const positions = [...new Set([...mono.map((t) => t.start), ...off.map((t) => t.start)])].sort((a, b) => a - b);
+  const out: Divergence[] = [];
+  for (const pos of positions) {
+    if (pos >= text.length || isWs(text.charCodeAt(pos))) continue;
+    if (isCovered(pos)) continue;                       // oracle already has an opinion here → graded above
+    const cm = scopeAt(mono, pos), co = scopeAt(off, pos);
+    const bM = scopeBucket(cm), bO = scopeBucket(co);
+    if (!interestingDivergence(bM, bO)) continue;
+    out.push({ pos, text: text.slice(pos, pos + 12), mono: innerOf(cm), off: innerOf(co), bM, bO });
+  }
+  return out;
+}
 
 export async function run(adapter: ScopeGapAdapter): Promise<void> {
   if (!existsSync(adapter.officialPath)) { console.error(`Official grammar not found:\n  ${adapter.officialPath}`); process.exit(1); }
@@ -100,6 +179,7 @@ export async function run(adapter: ScopeGapAdapter): Promise<void> {
 
   const corpus = adapter.loadCorpus();
   const gradable = adapter.isGradable ?? (() => true);
+  const fullSpan = adapter.fullSpan ?? false;
 
   let nFiles = 0;
   const tally = { oCorrect: 0, oExact: 0, mCorrect: 0, mExact: 0, total: 0 };
@@ -118,17 +198,17 @@ export async function run(adapter: ScopeGapAdapter): Promise<void> {
     for (const t of gold) {
       const tier = ROLE_SPEC[t.role]?.tier;
       if (!tier || tier === 'lexical') continue;   // lexical floor: excluded from the headline
-      const so = scopeAt(tmO, t.start), sm = scopeAt(tmM, t.start);   // full scope CHAINS
-      const vo = gradeScopeStack(t.role, so), vm = gradeScopeStack(t.role, sm);
+      const go = gradeSpan(t.role, tmO, t.start, t.end, text, fullSpan);   // full scope CHAINS
+      const gm = gradeSpan(t.role, tmM, t.start, t.end, text, fullSpan);
+      const vo = go.v, vm = gm.v;
       const oc = isCorrect(vo), mc = isCorrect(vm);
       tally.total++; gradedHere++;
       if (oc) tally.oCorrect++; if (vo === 'exact') tally.oExact++;
       if (mc) tally.mCorrect++; if (vm === 'exact') tally.mExact++;
       const pr = perRole.get(t.role) ?? { n: 0, oC: 0, mC: 0 }; pr.n++; if (oc) pr.oC++; if (mc) pr.mC++; perRole.set(t.role, pr);
       if (!oc) okO = false; if (!mc) okM = false;
-      const inner = (s: string[]) => s.length ? s[s.length - 1] : '(none)';
-      if (mc && !oc && onlyMono.length < 40) onlyMono.push({ text: t.text, role: t.role, o: inner(so), m: inner(sm) });
-      if (oc && !mc && onlyOff.length < 40) onlyOff.push({ text: t.text, role: t.role, o: inner(so), m: inner(sm) });
+      if (mc && !oc && onlyMono.length < 40) onlyMono.push({ text: t.text, role: t.role, o: go.scope, m: gm.scope });
+      if (oc && !mc && onlyOff.length < 40) onlyOff.push({ text: t.text, role: t.role, o: go.scope, m: gm.scope });
     }
     if (gradedHere) { snip.n++; if (okO) snip.o++; if (okM) snip.m++; }
   }
@@ -158,11 +238,40 @@ export async function run(adapter: ScopeGapAdapter): Promise<void> {
     console.log(`\n  only-official-correct tokens (Monogram wrong) — ${onlyOff.length} shown:`);
     for (const x of onlyOff.slice(0, 12)) console.log(`    «${x.text.slice(0, 18)}» ${x.role}: official «${x.o}» → monogram «${x.m}»`);
   }
+
+  // ── DIFFERENTIAL pass: oracle-INDEPENDENT divergences (the blind-spot net) ───────────────────────
+  let divTotal = 0;
+  if (adapter.differential) {
+    const all: Divergence[] = [];
+    let divFiles = 0;
+    for (const { text } of corpus) {   // ALL inputs, incl. ones the oracle/grader skip
+      let gold: GoldToken[], tmO: TmToken[], tmM: TmToken[];
+      try { gold = adapter.roleOracle(text); tmO = tmTokenize(official, text); tmM = tmTokenize(monogram, text); } catch { continue; }
+      const ds = divergences(tmO, tmM, gold, text);
+      if (ds.length) divFiles++;
+      all.push(...ds);
+    }
+    divTotal = all.length;
+    const genuine = all.filter((d) => !involvesInvalid(d.bM, d.bO));   // real class confusion (not error-overlay)
+    const overlay = divTotal - genuine.length;                         // official-marks-error vs Monogram-normal
+    const byPair = new Map<string, { n: number; sample: Divergence }>();
+    for (const d of all) { const k = `${d.bM}≠${d.bO}`; const e = byPair.get(k); if (e) e.n++; else byPair.set(k, { n: 1, sample: d }); }
+    console.log(`\n  ── DIFFERENTIAL (oracle-independent) — UNADJUDICATED divergences over ${divFiles} files ──`);
+    console.log(`     positions where Monogram and official paint different VISUAL classes and the oracle is`);
+    console.log(`     silent → the metric cannot adjudicate; each is a candidate bug for human review.`);
+    console.log(`     ${genuine.length} genuine class-confusion  +  ${overlay} error-overlay (official invalid.illegal vs Monogram highlight-normally)`);
+    for (const [k, e] of [...byPair.entries()].sort((a, b) => b[1].n - a[1].n).slice(0, 12)) {
+      const s = e.sample;
+      console.log(`     ${k.padEnd(18)} ×${String(e.n).padStart(4)}   e.g. «${s.text.replace(/\n/g, '\\n')}» mono«${s.mono}» off«${s.off}»`);
+    }
+  }
+
   // Machine-readable summary for the README coverage-table generator (test/coverage-table.ts).
   console.log('##SCOPEGAP## ' + JSON.stringify({
     name: adapter.name, official: adapter.officialPath.replace(/^.*\//, ''), tokens: tally.total,
     officialPct: tally.total ? (100 * tally.oCorrect) / tally.total : null,
     monogramPct: tally.total ? (100 * tally.mCorrect) / tally.total : null,
+    monogramWrong: onlyOff.length, unadjudicated: divTotal,
   }));
   console.log('\nDone.');
 }
diff --git a/test/scope-roles.ts b/test/scope-roles.ts
@@ -270,7 +270,19 @@ export const ROLE_SPEC: Record<RoleName, RoleSpec> = {
   // (e.g. a coarse generic `punctuation` for flow, the split punctuation.definition.tag for a tag).
   [R.blockIndicator]: { tier: 'strict', desc: 'YAML block-scalar indicator (|/>, chomping, indent)', exact: ['keyword.control.flow.block-scalar'], family: ['keyword', 'storage.modifier', 'constant.numeric', 'punctuation.definition'] },
   [R.tagType]: { tier: 'strict', desc: 'YAML node tag (!!str / !foo / !<verbatim>)', exact: ['storage.type.tag'], family: ['storage.type', 'storage', 'entity.name.type', 'support.type', 'keyword.other', 'punctuation.definition.tag'] },
-  [R.directive]: { tier: 'strict', desc: 'YAML directive name (%YAML / %TAG / %…)', exact: ['keyword.other.directive'], family: ['keyword.other', 'keyword', 'punctuation.definition.directive'] },
+  // A directive is graded over its WHOLE line (full-span): the name is keyword.other.directive, but
+  // the interior legitimately carries finer scopes a maintained grammar splits out — the version
+  // (constant.numeric), the tag handle (punctuation.definition.tag), the tag prefix (support.type),
+  // directive parameters (string.unquoted.directive-*). family accepts all of those so a correct
+  // fine-grained directive is not penalised, while STILL rejecting the two readings that are the
+  // monogram#12 bugs: a `#…` mis-read as a comment (#8) and a `,` mis-read as a flow separator (#1)
+  // — neither `comment` nor `punctuation.separator` is in the family.
+  // NB family carries `string.unquoted` (a maintained grammar scopes a directive's name/params as
+  // string.unquoted.directive-name / -parameter): the comparative bench grades by visual CLASS, and a
+  // string-coloured param is a defensible rendering, so it must not read WRONG. The PRECISE should-be
+  // (a param belongs to the directive, not a stray plain scalar — monogram#12 #4) is pinned by the
+  // strict regression gate yaml-issue12-regressions.ts, not relaxed here.
+  [R.directive]: { tier: 'strict', desc: 'YAML directive line (%YAML / %TAG / %…) — name + handle/prefix/version', exact: ['keyword.other.directive'], family: ['keyword', 'punctuation.definition', 'support', 'storage', 'constant.numeric', 'string.unquoted', 'entity.name', 'variable.other'] },
   [R.flowPunct]: { tier: 'strict', desc: 'YAML flow punctuation ([ ] { } ,)', exact: ['punctuation.definition', 'punctuation.separator'], family: ['punctuation'] },
 
   // ── lexical floor: reported, excluded from the headline ───────────────────────
@@ -461,6 +473,12 @@ if (import.meta.url === `file://${process.argv[1]}`) {
     [R.tagType, 'storage.type.tag.yaml', 'exact'],
     [R.tagType, 'punctuation.definition.tag.begin.yaml', 'family'],
     [R.directive, 'keyword.other.directive.yaml', 'exact'],
+    // directive interior: the version/handle/prefix sub-scopes a maintained grammar splits out are
+    // FAMILY-correct; the two monogram#12 mis-readings (comment / flow-separator) stay WRONG.
+    [R.directive, 'constant.numeric.yaml-version.yaml', 'family'],
+    [R.directive, 'punctuation.definition.tag.begin.yaml', 'family'],
+    [R.directive, 'comment.line.number-sign.yaml', 'wrong'],
+    [R.directive, 'punctuation.separator.sequence.yaml', 'wrong'],
     [R.flowPunct, 'punctuation.definition.sequence.begin.yaml', 'exact'],
     [R.flowPunct, 'punctuation.yaml', 'family'],
   ];
diff --git a/test/yaml-issue12-regressions.ts b/test/yaml-issue12-regressions.ts
diff --git a/test/yaml-oracle.ts b/test/yaml-oracle.ts
