Skip to content

Commit 638a23a

Browse files
johnsoncodehkjohnsoncodehk
committed
Consolidate test reports: one npm run check gate runner + TESTING.md taxonomy + grouped scripts
The suite had grown a dozen `npm run test:*` / `scope-gap:*` / `gap-ledger*` scripts with a dozen output formats and no map of what answers what. Consolidate without changing any check's behavior: - test/check.ts: ONE gate runner. Runs every CI-blocking pass/fail check (21 gates: core / conformance / highlighter / vue / yaml / the generative scope≡role check + gap-ledger selftest + --check) as subprocesses and prints one ✓/✗ table + one exit code. `node test/check.ts <substr>` runs a subset. CI's Test step is now `npm run check`. - TESTING.md: the report taxonomy. Four KINDS, each answering a different question and none interchangeable: Gate (is it correct? `npm run check`), Metric (how it compares to the official grammar/parser, the README table), Ledger (valid-input gaps the metric is structurally blind to, KNOWN-GAPS.md), Bench (perf/coverage, on demand). Spells out why the gate, the metric, and the ledger catch DIFFERENT classes and why none subsumes the others. - package.json: scripts grouped by tier with consistent prefixes — gate:* / metric:* / ledger:* / bench:*. `test` and `check` both run the gate runner. Pure rename of the internal scripts (none are referenced externally); the README/readme-bench refs (bench:issues, gate:treesitter, coverage:table) are unchanged.
1 parent 8638392 commit 638a23a

4 files changed

Lines changed: 197 additions & 73 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 6 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -34,38 +34,13 @@ jobs:
3434
npm run gen
3535
git diff --exit-code -- '*.tmLanguage.json' '*.language-configuration.json' '*.monarch.json' '*.cst-types.ts' '*.contributes.json' tree-sitter
3636
37-
# Self-contained suites (each exits non-zero on failure). The conformance /
38-
# coverage / bench tools needing the external TypeScript corpus or VS Code's
39-
# grammar are excluded; tsx/jsx conformance use only the bundled tsc.
37+
# Every correctness GATE through ONE runner — sanity / agnostic / conformance /
38+
# highlighter / vue / yaml / the generative scope≡role check / the gap-ledger selftest
39+
# + --check (stale KNOWN-GAPS.md fails). One ✓/✗ summary, one exit code. The comparative
40+
# METRICS (scope-gap / src-coverage) and BENCH tools need the external TS corpus / VS Code
41+
# grammars and run in the readme-bench workflow, not here. See TESTING.md for the taxonomy.
4042
- name: Test
41-
run: |
42-
node test/sanity-check.ts
43-
node test/agnostic.ts
44-
node test/refactor-guard.ts
45-
node test/test-issues.ts
46-
node test/tm-highlight-guards.ts
47-
node test/js-conformance.ts
48-
node test/tsx-conformance.ts
49-
node test/jsx-conformance.ts
50-
node test/redcmd-tm-diagnostics.ts
51-
node test/html-lexer-spike.ts
52-
node test/html-conformance.ts
53-
node test/html-monarch.ts
54-
node test/html-embed-js.ts
55-
node test/vue-directives.ts
56-
node test/vue-embed-boundary.ts
57-
node test/vue-interp-expr.ts
58-
node test/yaml-issue12-regressions.ts
59-
node test/yaml-depth-witnesses.ts
60-
node test/generative.ts
61-
# The gap ledger is the deterministic, oracle-classified record of the divergences the
62-
# generative check DISCOVERS. Its self-test asserts the ddmin keep-path + the oracle
63-
# drop-path + determinism; --check fails if the committed KNOWN-GAPS.md is stale (the
64-
# ledger is a pure function of the grammar, so it must be regenerated when a grammar
65-
# changes — `npm run gap-ledger`). This is the deterministic source of truth; a later
66-
# layer can turn rows into issues, but the committed artifact is gated here first.
67-
node test/gap-ledger-selftest.ts
68-
node test/gap-ledger.ts --check
43+
run: npm run check
6944

7045
# The derived tree-sitter highlighter is the strongest thesis proof (a real GLR
7146
# parser from the same grammar, beating the official hand-written one). Build its

‎TESTING.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Testing — the report taxonomy
2+
3+
The suite produces **four kinds of report**, each answering a different question. They are
4+
*not* interchangeable, and no one number is "the" test result. Read them in this order.
5+
6+
| Kind | Question it answers | One command | Output |
7+
|---|---|---|---|
8+
| **1. Gate** | Is the repo CORRECT? (pass/fail) | `npm run check` | one ✓/✗ summary + exit code |
9+
| **2. Metric** | How does Monogram compare to the OFFICIAL grammar/parser? | `npm run coverage:table` | the README coverage table |
10+
| **3. Ledger** | What valid-input highlighter GAPS exist that the metric is blind to? | `npm run ledger` | `KNOWN-GAPS.md` (+ auto-filed issues) |
11+
| **4. Bench** | How fast / how much is covered? (informational) | `npm run bench:*`, `npm run bench:perf` | per-tool numbers, on demand |
12+
13+
---
14+
15+
## 1. Gates — `npm run check`
16+
17+
The CI-blocking pass/fail checks (no external corpus needed). One runner, one ✓/✗ table, one
18+
exit code — instead of a dozen scripts with a dozen formats. Covers: sanity / agnostic /
19+
refactor-guard / issue-cases · conformance (js, tsx, jsx, html) · highlighter guards + RedCMD
20+
diagnostics + Monarch + HTML embed/lexer · Vue directives/embeds/interpolation · YAML
21+
issue-12 + depth-witnesses · the generative **scope≡role** check + the gap-ledger selftest +
22+
`--check`. `node test/check.ts <substr>` runs a subset (e.g. `… yaml`).
23+
24+
A gate is binary and authoritative: if `npm run check` is green, every guaranteed property holds.
25+
26+
## 2. Metrics — `npm run coverage:table`
27+
28+
The COMPARATIVE numbers (Monogram **vs the official** grammar / parser) the README publishes:
29+
- **Parser** (`test/src-coverage*`) — Monogram's accept/reject (and HTML tree shape) vs the
30+
official parser, weighted by the official parser's own branch coverage.
31+
- **Highlighter** (`test/scope-gap*`) — Monogram's TextMate grammar vs the official one, both
32+
graded against the parser's per-token roles ([vscode#203212](https://github.com/microsoft/vscode/issues/203212)).
33+
34+
These need the external corpora (the TS repo, Test262, yaml-test-suite) + the VS Code / official
35+
grammars — so they run locally / in the README-bench workflow, **not** in `check`. Both run
36+
through ONE data-driven driver each (`scope-gap-run.ts <lang>`, `src-coverage-run.ts <lang>`).
37+
38+
> A metric number is **comparative and bounded**: "100%" means "100% of the *graded* tokens (the
39+
> oracle has a role for, above the punctuation floor) on the *fixed corpus*" — never "perfect".
40+
41+
## 3. Ledger — `KNOWN-GAPS.md` (`npm run ledger`)
42+
43+
The metric above is corpus-bound, oracle-limited, and excludes the punctuation floor — so it is
44+
structurally blind to a whole class of real bugs. The **generative scope≡role check**
45+
(`test/generative.ts`, a gate) is **floor-blind** and **systematic** (it generates inputs from the
46+
grammar, deterministically), so it finds divergences where Monogram's own highlighter disagrees
47+
with its own parser. `test/gap-ledger.ts` operationalizes those into `KNOWN-GAPS.md`: each gap is
48+
**delta-debug-minimized**, **oracle-classified** (only valid-input gaps kept), and **fingerprinted**
49+
(stable across commits). `.github/workflows/gap-issues.yml` projects the ledger onto GitHub issues
50+
idempotently (open on appearance, auto-close on fix).
51+
52+
### Why a gate AND a metric, AND a ledger — they catch DIFFERENT classes
53+
54+
- A **metric** "Monogram-wrong" is a disagreement with the EXTERNAL oracle (tsc / the official
55+
grammar) — mostly *semantic identifier* roles (is `e` a type or a value?). Monogram's parser and
56+
highlighter **agree** there (self-consistent); they just make a different *semantic* call than
57+
tsc. The gate/ledger can't see it (no self-inconsistency, and a CFG can't decide it).
58+
- A **ledger/gate** gap is a disagreement between Monogram's highlighter and its own PARSER — a
59+
*structural* mis-paint (a `-` indicator scoped as a string/name) the metric floors out and the
60+
corpus may never contain. The metric can't see it.
61+
62+
Neither subsumes the other. `check` (self-consistency) does **not** replace the metric
63+
(external-semantic alignment) or the negative tests — round-trip proves only self-consistency.
64+
65+
## 4. Bench — on demand
66+
67+
`npm run bench:perf`, `bench:html-official`, `bench:vue-*`, `bench:issues` (the README bug ledger),
68+
`gate:treesitter`, `compat`. Informational — perf, official-comparison detail, repo-compat. Not gates.
69+
70+
---
71+
72+
### TL;DR
73+
74+
```
75+
npm run check # gates — is it correct? (one ✓/✗ summary)
76+
npm run coverage:table # metrics — how it compares to official (README table)
77+
npm run ledger # ledger — valid-input gaps the metric is blind to (KNOWN-GAPS.md)
78+
```

‎package.json‎

Lines changed: 44 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -4,52 +4,54 @@
44
"type": "module",
55
"scripts": {
66
"gen": "node src/cli.ts typescript.ts && node src/cli.ts javascript.ts && node src/cli.ts typescriptreact.ts && node src/cli.ts javascriptreact.ts && node src/cli.ts html.ts && node src/cli.ts vue.ts && node src/cli.ts yaml.ts",
7-
"test": "node test/sanity-check.ts",
8-
"generative": "node test/generative.ts",
9-
"gap-ledger": "node test/gap-ledger.ts --write",
10-
"gap-ledger:check": "node test/gap-ledger.ts --check",
11-
"gap-ledger:selftest": "node test/gap-ledger-selftest.ts",
12-
"conformance": "node test/run-conformance.ts",
13-
"conformance:js": "node test/js-conformance.ts",
14-
"conformance:tsx": "node test/tsx-conformance.ts",
15-
"conformance:jsx": "node test/jsx-conformance.ts",
16-
"conformance:html": "node test/html-conformance.ts",
17-
"test:tm-diagnostics": "node test/redcmd-tm-diagnostics.ts",
18-
"test:tm-guards": "node test/tm-highlight-guards.ts",
19-
"test:yaml-issues": "node test/yaml-issue12-regressions.ts",
20-
"test:left-recursion": "node test/left-recursion.ts",
21-
"spike:html-lexer": "node test/html-lexer-spike.ts",
7+
8+
"test": "node test/check.ts",
9+
"check": "node test/check.ts",
10+
11+
"gate:generative": "node test/generative.ts",
12+
"gate:conformance-js": "node test/js-conformance.ts",
13+
"gate:conformance-tsx": "node test/tsx-conformance.ts",
14+
"gate:conformance-jsx": "node test/jsx-conformance.ts",
15+
"gate:conformance-html": "node test/html-conformance.ts",
16+
"gate:tm-diagnostics": "node test/redcmd-tm-diagnostics.ts",
17+
"gate:tm-guards": "node test/tm-highlight-guards.ts",
18+
"gate:yaml-issues": "node test/yaml-issue12-regressions.ts",
19+
"gate:yaml-witnesses": "node test/yaml-depth-witnesses.ts",
20+
"gate:left-recursion": "node test/left-recursion.ts",
21+
"gate:html-lexer": "node test/html-lexer-spike.ts",
22+
"gate:treesitter": "(cd tree-sitter/typescript && npx tree-sitter generate && npx tree-sitter build --wasm .) && node test/treesitter-bench.ts",
23+
24+
"metric:scope-gap:ts": "node test/scope-gap-run.ts ts",
25+
"metric:scope-gap:js": "node test/scope-gap-run.ts js",
26+
"metric:scope-gap:jsx": "node test/scope-gap-run.ts jsx",
27+
"metric:scope-gap:tsx": "node test/scope-gap-run.ts tsx",
28+
"metric:scope-gap:html": "node test/scope-gap-run.ts html",
29+
"metric:scope-gap:yaml": "node test/scope-gap-run.ts yaml",
30+
"metric:scope-gap:vue": "node test/scope-gap-run.ts vue",
31+
"metric:src-coverage:ts": "node test/src-coverage-run.ts ts",
32+
"metric:src-coverage:js": "node test/src-coverage-run.ts js",
33+
"metric:src-coverage:jsx": "node test/src-coverage-run.ts jsx",
34+
"metric:src-coverage:tsx": "node test/src-coverage-run.ts tsx",
35+
"metric:src-coverage:html": "node test/src-coverage-run.ts html",
36+
"metric:src-coverage:yaml": "node test/src-coverage-run.ts yaml",
37+
"metric:scope-coverage": "node test/scope-coverage.ts",
38+
"metric:conformance": "node test/run-conformance.ts",
39+
"metric:compat": "node test/repo-compat.ts",
40+
"coverage:table": "node test/coverage-table.ts --write",
41+
42+
"ledger": "node test/gap-ledger.ts --write",
43+
"ledger:check": "node test/gap-ledger.ts --check",
44+
"ledger:selftest": "node test/gap-ledger-selftest.ts",
45+
"ledger:issues": "node test/gap-issues.ts",
46+
"ledger:issues:dry": "node test/gap-issues.ts --dry-run",
47+
48+
"bench:perf": "node test/perf-bench.ts",
49+
"bench:issues": "node test/issue-table.ts --write",
2250
"bench:html-official": "node test/html-bench.ts",
23-
"bench:html-embed": "node test/html-embed-js.ts",
24-
"bench:vue-directives": "node test/vue-directives.ts",
25-
"bench:vue-embed-boundary": "node test/vue-embed-boundary.ts",
26-
"bench:vue-interp-expr": "node test/vue-interp-expr.ts",
2751
"bench:vue-official": "node test/vue-bench.ts",
2852
"bench:vue-issues": "node test/vue-issues.ts",
2953
"bench:vue-dropin": "node test/vue-dropin.ts",
30-
"bench:vue-generic-dropin": "node test/vue-generic-dropin.ts",
31-
"compat:ts": "node test/repo-compat.ts typescript",
32-
"bench:issues": "node test/issue-table.ts --write",
33-
"gate:treesitter": "(cd tree-sitter/typescript && npx tree-sitter generate && npx tree-sitter build --wasm .) && node test/treesitter-bench.ts",
34-
"bench:perf": "node test/perf-bench.ts",
35-
"coverage": "node test/scope-coverage.ts",
36-
"compat": "node test/repo-compat.ts",
37-
"src-coverage:ts": "node test/src-coverage-run.ts ts",
38-
"src-coverage:js": "node test/src-coverage-run.ts js",
39-
"src-coverage:jsx": "node test/src-coverage-run.ts jsx",
40-
"src-coverage:tsx": "node test/src-coverage-run.ts tsx",
41-
"src-coverage:html": "node test/src-coverage-run.ts html",
42-
"src-coverage:yaml": "node test/src-coverage-run.ts yaml",
43-
"scope-gap:ts": "node test/scope-gap-run.ts ts",
44-
"scope-gap:js": "node test/scope-gap-run.ts js",
45-
"scope-gap:jsx": "node test/scope-gap-run.ts jsx",
46-
"scope-gap:tsx": "node test/scope-gap-run.ts tsx",
47-
"scope-gap:html": "node test/scope-gap-run.ts html",
48-
"scope-gap:yaml": "node test/scope-gap-run.ts yaml",
49-
"scope-gap:vue": "node test/scope-gap-run.ts vue",
50-
"coverage:table": "node test/coverage-table.ts --write",
51-
"gap-issues": "node test/gap-issues.ts",
52-
"gap-issues:dry": "node test/gap-issues.ts --dry-run"
54+
"bench:vue-generic-dropin": "node test/vue-generic-dropin.ts"
5355
},
5456
"devDependencies": {
5557
"@types/node": "^25.9.1",

‎test/check.ts‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
// ─────────────────────────────────────────────────────────────────────────────
2+
// check.ts — the SINGLE gate runner. Runs every correctness GATE as a subprocess and
3+
// prints ONE ✓/✗ summary, exiting non-zero if any fails. `npm run check` answers
4+
// "is the repo healthy?" in one command + one output — instead of running a dozen
5+
// scripts with a dozen output formats. See TESTING.md for the report taxonomy
6+
// (gate / metric / ledger / bench); this runs the GATE tier (the CI-blocking pass/fail
7+
// checks that need no external corpus). Metrics live in the README coverage table
8+
// (`npm run coverage:table`), findings in the gap ledger (`KNOWN-GAPS.md`).
9+
//
10+
// Assumes the generated grammars are current (`npm run gen`), as CI does before this.
11+
// Run: node test/check.ts # all gates
12+
// node test/check.ts yaml # only gates whose group/name contains "yaml"
13+
// ─────────────────────────────────────────────────────────────────────────────
14+
import { execFileSync } from 'node:child_process';
15+
16+
interface Gate { group: string; name: string; args: string[] }
17+
const GATES: Gate[] = [
18+
{ group: 'core', name: 'sanity', args: ['test/sanity-check.ts'] },
19+
{ group: 'core', name: 'agnostic', args: ['test/agnostic.ts'] },
20+
{ group: 'core', name: 'refactor-guard', args: ['test/refactor-guard.ts'] },
21+
{ group: 'core', name: 'issue-cases', args: ['test/test-issues.ts'] },
22+
{ group: 'conformance', name: 'js', args: ['test/js-conformance.ts'] },
23+
{ group: 'conformance', name: 'tsx', args: ['test/tsx-conformance.ts'] },
24+
{ group: 'conformance', name: 'jsx', args: ['test/jsx-conformance.ts'] },
25+
{ group: 'conformance', name: 'html', args: ['test/html-conformance.ts'] },
26+
{ group: 'highlighter', name: 'tm-guards', args: ['test/tm-highlight-guards.ts'] },
27+
{ group: 'highlighter', name: 'tm-diagnostics', args: ['test/redcmd-tm-diagnostics.ts'] },
28+
{ group: 'highlighter', name: 'html-monarch', args: ['test/html-monarch.ts'] },
29+
{ group: 'highlighter', name: 'html-embed-js', args: ['test/html-embed-js.ts'] },
30+
{ group: 'highlighter', name: 'html-lexer-spike', args: ['test/html-lexer-spike.ts'] },
31+
{ group: 'vue', name: 'directives', args: ['test/vue-directives.ts'] },
32+
{ group: 'vue', name: 'embed-boundary', args: ['test/vue-embed-boundary.ts'] },
33+
{ group: 'vue', name: 'interp-expr', args: ['test/vue-interp-expr.ts'] },
34+
{ group: 'yaml', name: 'issue12-regressions', args: ['test/yaml-issue12-regressions.ts'] },
35+
{ group: 'yaml', name: 'depth-witnesses', args: ['test/yaml-depth-witnesses.ts'] },
36+
{ group: 'generative', name: 'scope≡role', args: ['test/generative.ts'] },
37+
{ group: 'generative', name: 'gap-ledger-selftest', args: ['test/gap-ledger-selftest.ts'] },
38+
{ group: 'generative', name: 'gap-ledger-check', args: ['test/gap-ledger.ts', '--check'] },
39+
];
40+
41+
const filter = process.argv[2];
42+
const gates = filter ? GATES.filter((g) => (g.group + ' ' + g.name).includes(filter)) : GATES;
43+
if (!gates.length) { console.error(`no gate matches "${filter}"`); process.exit(1); }
44+
45+
const lastLine = (s: string): string => { const ls = s.trimEnd().split('\n').filter((l) => l.trim()); return ls.length ? ls[ls.length - 1].trim().slice(0, 70) : ''; };
46+
47+
interface Result { gate: Gate; ok: boolean; ms: number; summary: string; output: string }
48+
const results: Result[] = [];
49+
let curGroup = '';
50+
for (const gate of gates) {
51+
if (gate.group !== curGroup) { curGroup = gate.group; process.stdout.write(`\n ${curGroup}\n`); }
52+
const t0 = Date.now();
53+
let ok = true, output = '';
54+
try { output = execFileSync('node', gate.args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], maxBuffer: 64 * 1024 * 1024 }); }
55+
catch (e: any) { ok = false; output = (e.stdout ?? '') + (e.stderr ?? ''); }
56+
const ms = Date.now() - t0;
57+
const summary = lastLine(output);
58+
results.push({ gate, ok, ms, summary, output });
59+
process.stdout.write(` ${ok ? '✓' : '✗'} ${gate.name.padEnd(22)} ${String(ms).padStart(6)}ms ${ok ? summary : ''}\n`);
60+
}
61+
62+
const failed = results.filter((r) => !r.ok);
63+
console.log(`\n${'─'.repeat(70)}`);
64+
console.log(` ${results.length - failed.length}/${results.length} gates pass` + (failed.length ? ` — FAILED: ${failed.map((f) => f.gate.name).join(', ')}` : ' ✓'));
65+
for (const f of failed) {
66+
console.log(`\n── ✗ ${f.gate.name} (node ${f.gate.args.join(' ')}) ──`);
67+
console.log(f.output.trimEnd().split('\n').slice(-25).join('\n'));
68+
}
69+
process.exit(failed.length ? 1 : 0);

0 commit comments

Comments
 (0)