Generative testing: grammar-derived inputs + by-construction consistency (#25) #205
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: CI | |
| on: | |
| push: | |
| branches: [master] | |
| pull_request: | |
| permissions: | |
| contents: read | |
| jobs: | |
| test: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| submodules: true | |
| # Node 24+ runs the .ts sources directly (native type stripping) — no build, no tsx. | |
| - uses: actions/setup-node@v4 | |
| with: | |
| node-version: 24 | |
| - run: npm ci | |
| - name: Typecheck | |
| run: npx tsc --noEmit | |
| # Regenerate every grammar's artifacts and fail if any drifts from the committed | |
| # files (someone edited a grammar but forgot to regenerate). Covers all four | |
| # grammars (sources at the repo root) + the tree-sitter packages. | |
| - name: Generate editor artifacts (must be in sync) | |
| run: | | |
| npm run gen | |
| git diff --exit-code -- '*.tmLanguage.json' '*.language-configuration.json' '*.monarch.json' '*.cst-types.ts' '*.contributes.json' tree-sitter | |
| # Self-contained suites (each exits non-zero on failure). The conformance / | |
| # coverage / bench tools needing the external TypeScript corpus or VS Code's | |
| # grammar are excluded; tsx/jsx conformance use only the bundled tsc. | |
| - name: Test | |
| run: | | |
| node test/sanity-check.ts | |
| node test/agnostic.ts | |
| node test/refactor-guard.ts | |
| node test/test-issues.ts | |
| node test/tm-highlight-guards.ts | |
| node test/js-conformance.ts | |
| node test/tsx-conformance.ts | |
| node test/jsx-conformance.ts | |
| node test/redcmd-tm-diagnostics.ts | |
| node test/html-lexer-spike.ts | |
| node test/html-conformance.ts | |
| node test/html-monarch.ts | |
| node test/html-embed-js.ts | |
| node test/vue-directives.ts | |
| node test/vue-embed-boundary.ts | |
| node test/vue-interp-expr.ts | |
| node test/yaml-issue12-regressions.ts | |
| node test/yaml-depth-witnesses.ts | |
| node test/generative.ts | |
| # The gap ledger is the deterministic, oracle-classified record of the divergences the | |
| # generative check DISCOVERS. Its self-test asserts the ddmin keep-path + the oracle | |
| # drop-path + determinism; --check fails if the committed KNOWN-GAPS.md is stale (the | |
| # ledger is a pure function of the grammar, so it must be regenerated when a grammar | |
| # changes — `npm run gap-ledger`). This is the deterministic source of truth; a later | |
| # layer can turn rows into issues, but the committed artifact is gated here first. | |
| node test/gap-ledger-selftest.ts | |
| node test/gap-ledger.ts --check | |
| # The derived tree-sitter highlighter is the strongest thesis proof (a real GLR | |
| # parser from the same grammar, beating the official hand-written one). Build its | |
| # wasm and gate the accuracy so the 95.9% is verified, not just claimed. The | |
| # tree-sitter CLI bundles its own wasm toolchain — no emscripten/docker needed. | |
| treesitter: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: actions/setup-node@v4 | |
| with: | |
| node-version: 24 | |
| - run: npm ci | |
| # Cheap LR-conflict gate: `tree-sitter generate` (no wasm) for every derived | |
| # grammar that is a tree-sitter target, so a conflict introduced by a grammar | |
| # change is caught even for the dialects whose wasm is not built below (tsx/js/jsx) | |
| # — exactly the gap that let an unresolved `type`/`class_heritage` conflict ship. | |
| # yaml is now included (issue #3): its indent/scalar tokens are wired as tree-sitter | |
| # externals and the C indentation scanner is implemented, so its grammar generates + builds. | |
| - name: Generate every derived tree-sitter grammar (conflict gate, no wasm) | |
| run: | | |
| for g in typescript typescriptreact javascript javascriptreact html yaml; do | |
| echo "── tree-sitter generate: $g" | |
| ( cd "tree-sitter/$g" && npx tree-sitter generate ) | |
| done | |
| - name: Build the derived tree-sitter grammar to wasm | |
| run: | | |
| cd tree-sitter/typescript | |
| npx tree-sitter generate | |
| npx tree-sitter build --wasm . | |
| - name: Tree-sitter accuracy gate (≥ floor, must beat official) | |
| run: node test/treesitter-bench.ts | |
| - name: Build + gate the derived HTML tree-sitter grammar (v1, vs parse5) | |
| run: | | |
| cd tree-sitter/html | |
| npx tree-sitter generate | |
| npx tree-sitter build --wasm . | |
| cd ../.. | |
| node test/html-treesitter.ts | |
| # The derived YAML tree-sitter (issue #3) — build the wasm (its C indentation scanner must | |
| # compile + link). The accuracy bench (test/treesitter-yaml-bench.ts) needs the yaml-test-suite | |
| # checkout, so it runs in the readme-bench workflow where the suite is already cloned. | |
| - name: Build the derived YAML tree-sitter grammar to wasm | |
| run: | | |
| cd tree-sitter/yaml | |
| npx tree-sitter generate | |
| npx tree-sitter build --wasm . |