Mutation-test the completeness detector (#51)

A passing checker means nothing if the checker is blind. Add
test/tm-mutation.ts: it injects a catalogue of known gaps into the emitted
grammar (drop a key / all of a token's includes, neuter a scope to the bare
root, add a dead key, a dangling include, mis-scope a token to a wrong role,
reorder two disambiguation patterns) and records which detector layer kills
each — measuring the detector's power instead of asserting it.

Measured: every PRESENCE / REACHABILITY gap is killed corpus-free (16/16,
12/16 by reachability / token-census / the new flat-token neuter check);
WRONG-ROLE gaps are caught only by a differential witness (presence ≠
correctness); ORDERING gaps are a measured blind spot — TextMate is
order-sensitive and pattern rank lives in the emitted artifact, not the
grammar algebra, so no corpus-free structural check reaches it. This is the
honest boundary, now empirical: the structural proof covers presence +
reachability; ordering / correctness are the soundness axis, reached only by
evaluation. The over-claim of an a-priori "no gap can hide" over the whole
gap space is dropped — COMPLETENESS.md states the bounded, measured claim.

The harness motivated one detector strengthening: tokenCensus now flags a
flat token NEUTERED to the bare root scope (an entry that exists but paints
inert document text), moving that gap class from differential-only to
corpus-free. Wired as a meta-gate in `npm run check`.

Author: johnsoncodehk

COMPLETENESS.md

@@ -111,8 +111,35 @@ The empirical witness that all of the above actually paint is **leaf coverage**:
 deterministic grammar-derived corpus (`test/grammar-gen.ts`), every parsed leaf whose
 by-construction role (`buildRoleMap`) is a content/keyword role (keyword / string / number /
 comment) is confirmed to receive a non-root scope. The denominator is fixed (the obligation
-leaves); the metric is non-vacuous (deleting a discharging repository key drops it below 100%).
-Result: **2433/2433 across all six grammars.**
+leaves). Result: **2433/2433 across all six grammars.**
+
+## Measuring the detector — mutation testing
+
+A passing checker is worthless if the checker is *blind* — the corpus-trap this project has been
+bitten by. So the guarantee is not asserted, it is **measured**: `test/tm-mutation.ts` injects a
+catalogue of known gaps into the emitted grammar (drop a key, drop all of a token's includes,
+neuter a scope to the bare root, add a dead key, a dangling include, mis-scope a token to a wrong
+role, reorder two disambiguation patterns) and records which detector layer — if any — kills each.
+The honest, measured result:
+
+- **Presence gaps** (a token / scope / key dropped or neutered): **16/16 killed · 12/16 by a
+  CORPUS-FREE structural detector** (reachability dead/dangling · the token census · the flat-token
+  neuter check). The remaining four — a *region* token neutered — are caught by a targeted
+  differential witness, not corpus-free. **No presence gap survives; this is the gate.**
+- **Wrong-role gaps** (a token still painted, but the wrong role): caught by the differential
+  (a bucket change at the witness), *not* by the structural detector — a token that *is* painted
+  satisfies presence. This is the completeness/soundness seam: presence ≠ correctness.
+- **Ordering gaps** (two patterns reordered so a looser rule shadows a tighter one): a **measured
+  blind spot**. TextMate is order-sensitive, and which pattern wins is a property of the emitted
+  artifact's *sequence*, not the grammar's algebra — so no corpus-free structural check reaches it,
+  and a scope-preserving reorder slips even the bucket-level differential.
+
+So the claim this document makes is bounded and measured: **every presence / reachability gap is
+caught corpus-free** (mutation-proven, the gate); **wrong-role and ordering gaps are the soundness /
+interaction axis**, reached only by evaluation (the differential, or `test/gap-ledger.ts`), never by
+a grammar-algebraic proof. An a-priori "no gap can hide" over the *whole* gap space is not available
+— ordering and correctness obligations live in the emitted artifact and slide toward regex-vs-CFG
+undecidability — and this document does not claim it.
 
 ## Reachability — root ∪ export surfaces
 
@@ -179,8 +206,10 @@ Auto-generated by `node test/tm-completeness.ts --write`; `--check` fails CI if
 ## The gates that hold this exact
 
 - `test/tm-completeness.ts` — Layer A closure (RuleExpr / TokenPattern / `collectLiterals`), the
-  `sep`-recursion regression guard, reachability, the token census, and leaf coverage with a fixed
-  denominator. `npm run completeness` prints it; `npm run completeness:check` gates the ledger.
+  `sep`-recursion regression guard, reachability, the token census (orphans + neuter), and leaf
+  coverage with a fixed denominator. `npm run completeness`; `npm run completeness:check` gates the ledger.
+- `test/tm-mutation.ts` — the **meta-gate**: injects known gaps and asserts every presence gap is
+  killed with no false alarms, measuring (not asserting) the detector's power. `npm run completeness:mutation`.
 - `test/agnostic.ts` — detector shape-completeness: the detectors fire on structure, not on TS
   names, so "every shape that bears the obligation is detected" holds for any grammar.
 - `test/scope-gap.ts`, `test/gap-ledger.ts` — the **soundness** axis (is each painted scope

package.json

@@ -39,6 +39,7 @@
     "completeness": "node test/tm-completeness.ts",
     "completeness:check": "node test/tm-completeness.ts --check",
     "completeness:write": "node test/tm-completeness.ts --write",
+    "completeness:mutation": "node test/tm-mutation.ts",
     "ledger:selftest": "node test/gap-ledger-selftest.ts",
     "ledger:issues": "node test/gap-issues.ts",
     "ledger:issues:dry": "node test/gap-issues.ts --dry-run",

test/check.ts

@@ -37,6 +37,7 @@ const GATES: Gate[] = [
   { group: 'conformance', name: 'html', args: ['test/html-conformance.ts'] },
   { group: 'highlighter', name: 'tm-guards', args: ['test/tm-highlight-guards.ts'] },
   { group: 'highlighter', name: 'tm-completeness', args: ['test/tm-completeness.ts', '--check'] },
+  { group: 'highlighter', name: 'tm-mutation', args: ['test/tm-mutation.ts'] },
   { group: 'highlighter', name: 'tm-diagnostics', args: ['test/redcmd-tm-diagnostics.ts'] },
   { group: 'highlighter', name: 'angle-depth', args: ['test/angle-depth-probe.ts'] },
   { group: 'highlighter', name: 'html-monarch', args: ['test/html-monarch.ts'] },

test/tm-completeness.ts

@@ -180,7 +180,7 @@ function checkTokenPatternClosure(): void {
 //  REACHABILITY — every emitted repo key reachable from root ∪ export surfaces
 // ════════════════════════════════════════════════════════════════════════════
 
-interface TmGrammarJson { patterns?: unknown[]; repository?: Record<string, unknown>; scopeName?: string }
+export interface TmGrammarJson { patterns?: unknown[]; repository?: Record<string, unknown>; scopeName?: string }
 
 // The DECLARED export surfaces of a grammar — repository keys an external embedder reaches
 // not from the root but by an explicit `<scope>#<key>` include: the #expression sub-grammar
@@ -193,9 +193,9 @@ function exportSurfaceKeys(g: CstGrammar): string[] {
   return out;
 }
 
-interface ReachResult { repoKeys: number; reached: number; dead: string[]; danglingWithSource: string[] }
+export interface ReachResult { repoKeys: number; reached: number; dead: string[]; danglingWithSource: string[] }
 
-function checkReachability(g: CstGrammar, tm: TmGrammarJson): ReachResult {
+export function checkReachability(g: CstGrammar, tm: TmGrammarJson): ReachResult {
   const scope = tm.scopeName ?? g.scopeName ?? `source.${g.name}`;
   const repo = tm.repository ?? {};
   const reached = new Set<string>();
@@ -242,25 +242,33 @@ function checkReachability(g: CstGrammar, tm: TmGrammarJson): ReachResult {
 //  grammar emits no per-token keys — generateMarkupTm owns text/tag/attr), or a region that
 //  owns the token's delimiter (the JSX `/>` / `</` punctuation, scoped inside the JSX patterns).
 //  An ORPHAN — a non-skip token with no discharge path — is an emitter-completeness gap.
-interface TokenCensus { total: number; skip: number; byPath: Record<string, number>; orphans: string[] }
-function tokenCensus(g: CstGrammar, tmJson: TmGrammarJson): TokenCensus {
+export interface TokenCensus { total: number; skip: number; byPath: Record<string, number>; orphans: string[]; neutered: string[] }
+export function tokenCensus(g: CstGrammar, tmJson: TmGrammarJson): TokenCensus {
   const repo = tmJson.repository ?? {};
+  const root = tmJson.scopeName ?? `source.${g.name}`;
   const full = JSON.stringify(tmJson);
   const byPath: Record<string, number> = {};
   const orphans: string[] = [];
+  const neutered: string[] = [];   // a flat token whose entry exists but paints only the bare root (no visual scope)
   let skip = 0;
   const bump = (p: string) => byPath[p] = (byPath[p] ?? 0) + 1;
+  // a flat `{name, match}` entry discharges its scope obligation only if `name` is a real
+  // visual scope — not the bare document root and not empty. An entry whose name was reduced
+  // to the root scope is a "neuter" gap (the token tokenises but reads as inert document text),
+  // structurally visible without any corpus.
+  const flatNeutered = (e: any): boolean => !e.begin && !e.patterns && (!e.name || String(e.name).split(' ').every((s: string) => s === root || !s));
   for (const t of g.tokens) {
     if (t.flags.includes('skip')) { skip++; continue; }
-    if (repo[t.name.toLowerCase()]) { bump('flat'); continue; }
+    const flat = repo[t.name.toLowerCase()];
+    if (flat) { if (flatNeutered(flat)) neutered.push(`${t.name}→${(flat as any).name ?? '∅'}`); else bump('flat'); continue; }
     if (t.flags.includes('regex')) { bump('regex-family'); continue; }
     if (tokenPatternIsNever(t)) { bump('engine-emitted'); continue; }
     if (g.markup) { bump('markup-region'); continue; }                 // generateMarkupTm owns it
     const delim = tokenPatternLiteralText(t);                          // a region owns this token's delimiter?
     if (delim && full.includes(JSON.stringify(delim).slice(1, -1))) { bump('region-owned'); continue; }
     orphans.push(`${t.name}[${t.flags.join(',') || '-'}]`);
   }
-  return { total: g.tokens.length, skip, byPath, orphans };
+  return { total: g.tokens.length, skip, byPath, orphans, neutered };
 }
 
 // ════════════════════════════════════════════════════════════════════════════
@@ -272,7 +280,7 @@ const require = createRequire(import.meta.url);
 const wasmBin = readFileSync(require.resolve('vscode-oniguruma/release/onig.wasm'));
 await loadWASM(wasmBin.buffer.slice(wasmBin.byteOffset, wasmBin.byteOffset + wasmBin.byteLength));
 
-async function loadTmFromObject(scopeName: string, grammars: Record<string, object>): Promise<vsctm.IGrammar | null> {
+export async function loadTmFromObject(scopeName: string, grammars: Record<string, object>): Promise<vsctm.IGrammar | null> {
   const reg = new Registry({
     onigLib: Promise.resolve({ createOnigScanner: (p: string[]) => new OnigScanner(p), createOnigString: (s: string) => new OnigString(s) }),
     loadGrammar: async (sn: string) => grammars[sn] ? parseRawGrammar(JSON.stringify(grammars[sn]), sn + '.json') : null,
@@ -287,7 +295,7 @@ async function loadTmFromFiles(scopeName: string, files: Record<string, string>)
   });
   return reg.loadGrammar(scopeName);
 }
-function tmTokenize(grammar: vsctm.IGrammar, text: string): TmTok[] {
+export function tmTokenize(grammar: vsctm.IGrammar, text: string): TmTok[] {
   const toks: TmTok[] = []; let rs = INITIAL, off = 0;
   for (const line of text.split('\n')) { const r = grammar.tokenizeLine(line, rs); for (const t of r.tokens) toks.push({ start: off + t.startIndex, end: off + t.endIndex, scopes: t.scopes }); rs = r.ruleStack; off += line.length + 1; }
   return toks;
@@ -307,9 +315,9 @@ function tmTokenize(grammar: vsctm.IGrammar, text: string): TmTok[] {
 // ════════════════════════════════════════════════════════════════════════════
 const CONTENT_OBLIGATION = new Set<Bucket>(['keyword', 'string', 'number', 'comment']);
 
-interface CoverageResult { den: number; painted: number; uncovered: { text: string; want: string; ctx: string }[] }
+export interface CoverageResult { den: number; painted: number; uncovered: { text: string; want: string; ctx: string }[] }
 
-function leafCoverage(grammar: CstGrammar, tm: vsctm.IGrammar, opts = GEN_OPTS): CoverageResult {
+export function leafCoverage(grammar: CstGrammar, tm: vsctm.IGrammar, opts = GEN_OPTS): CoverageResult {
   const { parse } = createParser(grammar);
   const roleOf = buildRoleMap(grammar);
   const inputs = generateInputs(grammar, opts);
@@ -510,6 +518,7 @@ async function main(): Promise<void> {
     check(`reachability(${cfg.name}): no dangling self-#refs with present source`, r.danglingWithSource.length === 0, r.danglingWithSource.join(', '));
     const tc = tokenCensus(g, tmJson);
     check(`token-completeness(${cfg.name}): every non-skip token has a discharge path`, tc.orphans.length === 0, `orphans: ${tc.orphans.join(' ')}`);
+    check(`token-completeness(${cfg.name}): no flat token is neutered to the bare root scope`, tc.neutered.length === 0, `neutered: ${tc.neutered.join(' ')}`);
     const tm = await loadTmFromFiles(cfg.scopeName, { [cfg.scopeName]: cfg.tm, ...(cfg.tmExtra ?? {}) });
     let cov: CoverageResult = { den: 0, painted: 0, uncovered: [] };
     if (tm) cov = leafCoverage(g, tm);