Skip to content

feat(ci): gate the runnable corpus — examples/ and conformance/ - #89

Merged
hyperpolymath merged 1 commit into
mainfrom
feat/gate-conformance-examples
Jul 29, 2026
Merged

feat(ci): gate the runnable corpus — examples/ and conformance/#89
hyperpolymath merged 1 commit into
mainfrom
feat/gate-conformance-examples

Conversation

@hyperpolymath

Copy link
Copy Markdown
Owner

examples/ and conformance/ existed but were executed by nothing — not CI, not the Justfile.

What that cost

examples/isotopy.tangle — the file whose entire purpose is to demonstrate the language's central claim, "two braids can be structurally different but topologically equivalent"was failing its own assertions.

The culprit is line 28:

assert simplify(braid[s1, s3, s1^-1]) ~ braid[s3]

That needs far commutation (|1−3| ≥ 2), which the old free-reduction ~ could not do. It began passing only when TG-7 (#87) made ~ decide real braid-group equivalence.

Verified rather than assumed — built the pre-TG-7 commit and ran it:

=== isotopy.tangle under PRE-TG-7 build ===
Runtime error: Assertion failed
EXIT=1

So the showcase example had been broken for as long as ~ used free reduction, and nothing noticed because nothing ran it. That is the case for this gate, and incidentally a nice independent validation of TG-7.

The gate is a ratchet

Five examples don't typecheck and five conformance/valid/ programs don't parse. Those are real gaps (filed as #88), and they can't be fixed in the change that introduces the gate. So each is recorded explicitly with its cause, and the gate enforces in both directions:

  • nothing currently working may break;
  • nothing may join the known-bad lists;
  • anything that starts working must be removed from them — a stale entry fails the build.

That last rule is what stops an "expected failures" list decaying into a permanent excuse.

Verified adversarially — I tried to break it three ways, and it caught all three:

Attack Result
Break a must-run example (append a false assert) EVAL FAILED → exit 1
Add an unlisted example UNLISTED → exit 1
Mark a working example as known-bad NOW WORKS → exit 1

Also: the conformance runner never worked

run_conformance.sh invoked dune exec -- tangle --parse-only. Both halves were wrong — the executable is main (compiler/bin/dune), not tangle, and there has never been a --parse-only flag. Every invocation failed, so the suite reported 3/19 while proving nothing: all three "passes" were invalid/ cases, and an invalid case passes precisely when the command fails. It failed for the wrong reason and scored a point for it.

Fixed (prefers the pre-built binary, falls back to dune exec). The real number is 14/19.

Current state locked in

examples parse 7/7
examples typecheck + evaluate 2/7 (echo_pd, isotopy)
conformance 14/19 (was a fake 3/19)
invalid/ correctly rejected 3/3

Path filters were widened to examples/**, conformance/** and the script itself — otherwise a change to the corpus would never trigger the gate that checks it. Adds just corpus and just check-all for local use.

Debt tracked in #88, with per-file causes and a suggested order.

🤖 Generated with Claude Code

Neither directory was executed by anything: not CI, not the Justfile. The
cost was concrete and had been sitting in the repo indefinitely.

`examples/isotopy.tangle` — the file whose entire purpose is to demonstrate
the language's central claim, that two braids can be structurally different
yet topologically equivalent — was FAILING ITS OWN ASSERTIONS. Specifically
`assert simplify(braid[s1, s3, s1^-1]) ~ braid[s3]`, which needs far
commutation (|1-3| >= 2) that the old free-reduction `~` could not do. It
began passing only when TG-7 (#87) made `~` decide real braid-group
equivalence. Verified by building the pre-TG-7 commit and running it:
"Runtime error: Assertion failed". A showcase example can stay broken for
as long as nothing runs it.

Adds scripts/check-corpus.sh, wired into ocaml-ci.yml (with the path filters
widened to examples/, conformance/ and the script itself, so a change to the
corpus actually triggers the gate that checks it), plus `just corpus` and
`just check-all` for local use.

The gate is a RATCHET, not a pass/fail. Five examples do not typecheck and
five conformance valid/ programs do not parse; those are real gaps that
cannot be fixed in the change that introduces the gate. So each is recorded
explicitly WITH ITS CAUSE, and the gate enforces in both directions:

  * nothing currently working may break
  * nothing may join the known-bad lists
  * anything that starts working MUST be removed from them — a stale entry
    fails the build

That last rule is what stops an "expected failures" list decaying into a
permanent excuse. Verified adversarially: breaking a must-run example, adding
an unlisted example, and marking a working example as known-bad were each
caught (exit 1).

Also fixes conformance/run_conformance.sh, which had never worked. It ran
`dune exec -- tangle --parse-only`; both halves were wrong — the executable
is `main` (compiler/bin/dune), not `tangle`, and there has never been a
`--parse-only` flag. Every invocation therefore failed, and the suite
reported 3/19 while proving nothing: all three "passes" were invalid/ cases,
which pass precisely WHEN the command fails. It failed for the wrong reason
and scored a point for it. The real number is 14/19.

Debt tracked in #88.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@hyperpolymath
hyperpolymath merged commit 12a32ce into main Jul 29, 2026
25 checks passed
@hyperpolymath
hyperpolymath deleted the feat/gate-conformance-examples branch July 29, 2026 00:04
Comment thread scripts/check-corpus.sh
@gitar-bot

gitar-bot Bot commented Jul 29, 2026

Copy link
Copy Markdown

Note

Automatic reviews are paused because your trial's included automatic processing has been used for this period. Upgrade now, or comment "Gitar review" to run a review anytime.
Learn more

Code Review ✅ Approved 2 resolved / 2 findings

Integrates the runnable examples and conformance test suites into CI with bidirectional ratchet gating to prevent silent regressions. Consider ensuring the invalid glob pattern handles empty matches and verifying the fallback dune binary path.

Auto-approved and auto-merge armed: No blocking issues found.
Please see Auto-approve Docs for details on setting custom approval criteria. — merges when pipeline and required approvals pass.

✅ 2 resolved
Edge Case: invalid/ glob with no matches silently passes conformance

📄 scripts/check-corpus.sh:32 📄 scripts/check-corpus.sh:153-161
In scripts/check-corpus.sh the loop for f in "${ROOT}"/conformance/invalid/*.tangle runs without nullglob, so if the directory is empty or renamed the loop iterates once over the literal pattern string. The binary then exits non-zero on the nonexistent path, which the invalid-case logic interprets as "rejected ok" — producing a false success that proves nothing (the same false-pass class this PR is fixing). Add shopt -s nullglob near the top of the script so empty globs expand to nothing, or assert a nonzero file count per directory.

Bug: dune exec fallback target './bin/main.exe' may not resolve

📄 conformance/run_conformance.sh:33
In conformance/run_conformance.sh the new fallback dune exec --root "${SCRIPT_DIR}/../compiler" -- ./bin/main.exe passes a relative target that dune resolves against the workspace root rather than the caller's CWD, so the invocation may fail to locate the executable when no pre-built binary exists (e.g. running the script directly without just corpus). Prefer the public-name form dune exec --root "${SCRIPT_DIR}/../compiler" -- main (or bin/main.exe without the ./) to match how dune names the target. CI is unaffected since it uses the pre-built binary branch.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

Comment thread conformance/run_conformance.sh
@gitar-bot

gitar-bot Bot commented Jul 29, 2026

Copy link
Copy Markdown

⚠️ Gitar auto-approved this PR but could not enable auto-merge: auto-merge is disabled for this repository — enable "Allow auto-merge" in the repository settings.

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR and enabled auto-merge (configure)

@gitar-bot gitar-bot Bot added the gitar-approved Added by Gitar label Jul 29, 2026
hyperpolymath added a commit that referenced this pull request Jul 29, 2026
Closes both acceptance criteria of #56 — though the second was already
satisfied, and the first was mis-scoped.

## Criterion 1 — template residue

The issue recorded *"5 `{{PLACEHOLDER}}` files + 8 `rsr-template-repo`
references"*. Re-measured: **8** files matched `{{...}}` and **21**
lines matched the template name — but most were **meta-references**:
files *describing* the debt, not carrying it.

Three would have been damaged by a naive search-and-replace:

| File | Why it must not be touched |
|---|---|
| `src/rust/src/eval.rs` | `"...add{{}} block"` is a Rust `format!`
**escape** — `{{}}` renders as a literal `{}`, naming the `add{}`
language construct. Working code. |
| `methodology.a2ml` | The token appears inside `reject-if-contains` —
the rule that *forbids* it. |
| `AFFIRMATION.adoc`, `docs/identity-fabric/` | Prose recording the
debt. |

**The genuine residue, now cleared:** `{{DEPS}}`,
`{{BUILD_OUTPUT_PATH}}`, `{{PACKAGE_NAME}}`, `{{LANG_STACK}}`,
`{{BUILD_CMD}}`, `{{TEST_CMD}}`, `{{MUST_INVARIANTS}}`,
`{{PROJECT_UNIQUE_STRENGTH}}` — plus four contractiles and the citation
guide whose **subject was still the template**. `Intentfile` asserted
*"This repository is the canonical template for Rhodium Standard
Repository compliance"*; `CITATIONS.adoc` credited *"Polymath, Hyper"*
for *"RSR-template-repo"* in all five citation formats.

## A worse class, found while fixing it

All three QUICKSTARTs documented commands that **do not exist** — `just
build`, `just test`, `just lint`, `just panic-scan`, `just setup-dev`,
`just build-release`, `just install`, `just run`, `just heal`, `just
setup`, `just uninstall`, `just stapeln-run` — asserted an
`$XDG_CONFIG_HOME/tangle/config.toml` the compiler never reads, and
carried a *"Multi-Instance Deployment"* section for what is a single
stateless binary.

Documentation that fails when followed is worse than none. Every command
now shown is verified to exist, and the worked example shows the
**real** output (8 `assertion passed` lines, exit 0) rather than the
template's invented `Tangle started successfully.`

## Criterion 2 — already satisfied by #78

`README.adoc` carries the correction and `AFFIRMATION.adoc` has
*"Erratum — 2026-07-21 / Correction 1 — the stack does not exist"*.

Worth noting: the criterion as worded — *"does the simplified diagram
supersede, or re-state, the nuance?"* — presupposed the stack was real.
**Both options were wrong.** `KRL → TangleIR → …` describes a dependency
that never existed (`TangleIR` appears in no source file in either
repo). It was withdrawn, not reconciled.

## Making the rule real

`methodology.a2ml` has always declared `reject-if-contains = [...,
"rsr-template-repo"]` — and **nothing enforced it**, which is precisely
how the contractiles stayed mis-instantiated. Adds
`scripts/check-rsr-instantiation.sh` (plus `just rsr-check` and a CI
step), using an explicit allowlist so meta-references and the Rust
escape aren't false-positived.

**Verified adversarially:**

| Attack | Caught |
|---|---|
| Reintroduce a `{{PLACEHOLDER}}` | ✅ |
| Claim to be `rsr-template-repo` | ✅ |
| Document a phantom `just` recipe | ✅ |
| Rust `format!` escape `add{{}}` | ✅ correctly **ignored** |

It earned itself immediately: on first run it flagged
`QUICKSTART-USER.adoc`, a file I had not inspected, with six phantom
recipes.

## Also

`AFFIRMATION.adoc` gains a dated **Erratum — 2026-07-29** (Corrections
3–5) recording the residue clearance, that the OCaml compiler is no
longer unbuilt (#87/#89 — the previous erratum's closing line said it
was), and that `==` is no longer list equality (#50/#87).

All gates pass locally: build, corpus, rsr-check, `must-check`,
`trust-verify`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
hyperpolymath added a commit that referenced this pull request Jul 29, 2026
Four of the five unparsed `conformance/valid/` programs — `v02_weave`,
`v08_crossing`, `v09_twist`, `v12_weave_expr` — use weave as a
**definition body**:

```tangle
def simple_crossing =
  weave strands a:Q, b:Q into
    (a > b)
  yield strands b:Q, a:Q
```

The grammar allowed weave only as a top-level statement, so all four
failed. Typed strands (`a:Q`) and `yield strands` were never the problem
— those already parsed. Only the **position** did.

## Why the expression form is the right fix, not a corpus edit

My first instinct was that the corpus was wrong, since
`spec/grammar.ebnf` also said statement. Then I checked what a weave
statement actually *does*:

```ocaml
(* eval.ml *)
| WeaveBlock _wb -> (env, None)          (* no-op *)

(* typecheck.ml *)
let _weave_ty = TTangle (input_boundary, output_boundary) in
gamma                                     (* type computed, then DISCARDED *)
```

**As a statement, weave is inert.** It binds nothing, produces nothing,
and its type is thrown away. The construct could be written but never
used. Binding it to a name is the only thing that makes it reachable —
which is precisely what the corpus assumed all along.

The statement form is kept, so this is **purely additive**.

## Scope — no proof implications

Weave is outside the mechanised core: **0 occurrences in
`proofs/Tangle.lean`, 0 in the TG-3 corpus**. So this touches no proof
obligation, and `Weave` joins the other non-core constructors in
`tg3_emit`'s explicit rejection branch rather than being translated.

Threaded through `ast` (new `Weave of weave_block`), `parser`
(`primary_expr`), `typecheck` (types as the `Tangle[A,B]` it denotes;
body checked in the strand context exactly as the statement form does),
`eval` (the body *is* the value), and `pretty` (re-parseable form).
`expr_calls` also traverses into a weave body — otherwise recursion
inside one would be invisible to the #91 fixpoint search.

## What this does NOT fix: `v11_add_block`

`add{ 1 + 2 + 3 }` is **not a missing parse rule**. It is the **Harvard
data sub-language**:

| | |
|---|---|
| own expression grammar | literals, vars, arithmetic, equality,
`&&`/`\|\|`/`!`, calls, `if/then/else` |
| own type system | Int, Float, Rational, Hex, Binary, Bool, String,
Symbolic |
| own environments | `Π`, with visibility rules |
| own typing judgement | `⊢_hd` |
| conversion layer | Embed / Unembed |
| calling | bidirectional with Tangle |

**288 lines of `FORMAL-SEMANTICS.md` across 12 sections.** That is a
feature, raised separately. It remains the single entry in the corpus
manifest's known-gap list.

## Results

| | before | after |
|---|---|---|
| conformance | 14/19 | **18/19** |
| corpus manifest known-unparsed | 5 | **1** |

(It reported a *fake* 3/19 before #89 fixed the runner.)

The ratchet in `check-corpus.sh` **demanded** the manifest update — it
failed the build with *"now parses — drop it from
CONFORMANCE_KNOWN_UNPARSED"* for each file. That is exactly what it was
built to do.

## Tests

- weave as a definition body (the conformance form)
- **TG-4 round-trip**: pretty-print then re-parse yields the same AST —
needed because I changed `pretty.ml`
- guard that the statement form still parses (the change is additive)

Full suite green, 0 failures; TG-3 still 1008/0. `spec/grammar.ebnf`
updated to match, with the rationale recorded.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
hyperpolymath added a commit that referenced this pull request Jul 29, 2026
…th (#95)

Implements the #92 ruling across both engines — plus two bugs found
while verifying it.

## The rule

`T-Eq-Word` required both operands at the same width. Four ways that was
inconsistent with the language itself (evidence in #92):

- **`tComposeWord` already widens** to `max n m` — you could compose two
braids you were then forbidden to compare.
- **`~` accepted differing widths** and, since TG-7, evaluates through
the *same* `Braid_equiv.equiv`. Identical operands, identical answer,
one rejected by the typechecker.
- **`eqIdBraid`/`eqBraidId`** decide "is this braid trivial?" against
`identity : word 0`, so under the old rule they could only fire when the
braid was empty. The step relation had rules for a question the typing
rule forbade asking.
- **`braid_equiv.equiv` has no width parameter at all.**

Lean's `tEqWord` now takes `(n m)`; `infer`'s `.eq` case drops the `if n
= m`; OCaml `Eq` mirrors it. Match arms join on width for words and
still require exact agreement otherwise.

## The metatheory needed no patching

**Progress, Preservation, Determinism, TypeSafety, `infer_sound`,
`infer_complete` all still hold** — `lean Tangle.lean` reports **0
errors**, sorry/axiom gate passes.

`infer_sound`'s proof got *simpler*: with no `if n = m` there's no inner
`split` to case on. A restriction whose removal shortens the proof was
carrying no weight.

**TG-3 regenerated and kernel-checked: 496 obligations, 0 errors.** 46
flipped from `= none` to `= some .bool` — exactly the `identity ==
braid` shape that was unreachable before.

## Two bugs found while verifying

**1. Statement order.** `parse_file_recovering` did `stmts := prog @
!stmts` then `List.rev` on the flattened result. That reversal is
correct for an accumulator built by prepending *single items* (as
`diagnostics` is) — but whole **segments** were prepended, so every
program came out backwards:

```
def x = 5     parses to    def z = 7
def y = 6      ------->    def y = 6
def z = 7                  def x = 5
```

Any program whose statements depend on order died with *"Unbound
variable"*. **The test suites never caught it because they call
`Tangle.Parser.program` directly — only the CLI goes through the
recovering path.**

**2. A false assertion in `examples/trefoil.tangle`:**

```tangle
assert reversed == braid[s1, s1, s1]     # reversed = reverse(trefoil)
```

`reverse` reverses the word *and negates exponents*, yielding the
inverse braid. Verified by invariant: `writhe(trefoil) = 3`,
`writhe(reversed) = -3`, and writhe is invariant under the braid
relations — so they cannot be equal. The assertion had been wrong since
it was written; nothing ran the examples until #89. Corrected to
`braid[s1^-1, s1^-1, s1^-1]`, which passes.

## Result

| | before | after |
|---|---|---|
| examples evaluating | 2/7 | **7/7** |
| `lib/stdlib.tangle` | did not typecheck | **typechecks** |
| conformance | 18/19 | 18/19 (`v11` is the Harvard sub-language, #94) |

All seven examples are now in the corpus gate's must-run set, so none
can regress.

## ⚠ Trusted base — registered as A-TG-92.1

The embedding **Bₙ ↪ Bₙ₊₁** that justifies cross-width comparison is
standard mathematics, but it is **asserted in prose, not mechanised** —
no Lean lemma states it.

What *is* machine-checked: that the metatheory holds under the widened
rule, and that OCaml `infer_expr` still agrees with Lean `infer` on the
corpus. That distinction is recorded in `ASSUMPTIONS.md` rather than
left implicit.

## Tests

6 new typecheck cases, including **two negatives** — mismatched *kinds*
still rejected, match arms of different kinds still rejected — so
widening cannot quietly become "anything compares to anything". Three
`test_check` cases repointed at an error that is still an error (and an
assertion helper left unused by the change was put back to work rather
than deleted).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gitar-approved Added by Gitar

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant