Skip to content

Commit fb27a7f

Browse files
committed
docs: extend test-anchoring to the capability matrix + file residual issues
Follow-up to #622. Applies the same "claims bound to executable tests, gated in CI" discipline to the *feature*-readiness matrix (not just the soundness ledger), and wires in the now-tracked residuals. - docs/CAPABILITY-MATRIX.adoc: add a "== Test anchors" section mapping each load-bearing component (lexer, parser, type checker, effects, borrow checker, QTT, interpreter, traits, stdlib, typed-wasm, codegen, JS-host backends, Solo core) to the test(s) that exercise it. - tools/check-capability-anchors.sh (wired into `just guard` + CI): fails the build if the Test-anchors section disappears or if any test path the matrix names goes missing — so a "works"/"enforced" row cannot outlive its test. Verified it bites (non-zero on a missing anchor). - docs/SOUNDNESS.adoc: wire the residual rows to their new tracking issues (#623 interp non-tail resume, #624 Lean/Why3 return-drop) and refresh the metatheory note to reflect the grown Coq formal/ track (#620/#621: K-1, K-1-let, F-1). Residuals filed as tracked issues so they aren't just prose: - #623 — interpreter non-tail single-shot resume (shallow continuation; needs CPS rewrite, owner-steer). - #624 — Lean/Why3 experimental backends drop `return` control flow. - #559 (already open) updated by comment: concrete-overlap coherence landed; residual narrowed to generic-subsumption overlap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbxKhXQwTvVgkYDgBMLJoa
1 parent e6c155f commit fb27a7f

5 files changed

Lines changed: 125 additions & 10 deletions

File tree

.github/workflows/ci.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,13 @@ jobs:
9797
# PROOF-NEEDS / NAVIGATION / CLAUDE.md) stops linking back to it.
9898
# See tools/check-soundness-ledger.sh.
9999
run: ./tools/check-soundness-ledger.sh
100+
- name: Capability-matrix test-anchor gate
101+
# docs/CAPABILITY-MATRIX.adoc anchors each feature-readiness claim to an
102+
# executable test ("== Test anchors"). This gate fails if that section
103+
# disappears or if any test it names goes missing — so a "works" status
104+
# row cannot outlive the test that backs it.
105+
# See tools/check-capability-anchors.sh.
106+
run: ./tools/check-capability-anchors.sh
100107
- name: Check formatting
101108
run: opam exec -- dune build @fmt
102109
lint:

docs/CAPABILITY-MATRIX.adoc

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -225,6 +225,38 @@ db-theory triplet — `Sqlite.affine` (#522 / #524 / #525), `Transaction.affine`
225225
group-by + aggregation primitives) — are the current authoring frontier.
226226
A few primitives remain `extern` builtins.
227227

228+
== Test anchors
229+
230+
Every enforcement claim in the tables above is anchored to an executable test,
231+
mirroring the link:SOUNDNESS.adoc[soundness ledger]. This makes the claims
232+
*falsifiable*: `tools/check-capability-anchors.sh` (wired into `just check` +
233+
CI) extracts every `test/…` path named below and fails the build if any has
234+
gone missing — so a `works` / `enforced` row cannot keep its status after its
235+
test is deleted or renamed without this section (and CI) noticing.
236+
237+
[cols="2,4",options="header"]
238+
|===
239+
|Component (row above) |Anchoring test(s)
240+
241+
|Lexer |`test/test_lexer.ml`
242+
|Parser / AST / Name resolution |`test/test_e2e.ml`, `test/test_qualified_paths.ml`
243+
|Type checker |`test/test_e2e.ml`, `test/test_slash_effect_row.ml`
244+
|Effect system |`test/test_effects.ml`, `test/test_effect_sites.ml`; fixtures `test/e2e/fixtures/handle_return_arm.affine`, `test/e2e/fixtures/handle_resume_tail.affine`
245+
|Borrow checker |`test/test_borrow_polonius.ml`; fixtures `test/e2e/fixtures/borrow_callee_returned_borrow_uam.affine`, `test/e2e/fixtures/borrow_callee_value_return_ok.affine`
246+
|Quantity / affine (QTT) |`test/test_e2e.ml`; fixtures `test/e2e/fixtures/linear_arrow_violation.affine`, `test/e2e/fixtures/affine_violation.affine`, `test/e2e/fixtures/erased_violation.affine`
247+
|Interpreter |`test/test_e2e.ml`, `test/test_solo_cesk.ml`, `test/test_module_mut.ml`
248+
|Traits (incl. #559 coherence) |`test/test_e2e.ml` (Trait-Coherence suite)
249+
|Stdlib (AOT + laws) |`test/test_stdlib_aot.ml`, `test/test_stdlib_laws.ml`
250+
|typed-wasm isolation |`test/test_tw_isolation.ml`
251+
|Codegen (golden snapshots) |`test/test_golden.ml`
252+
|Deno / JS host backends |`test/test_deno_builtins_consistency.ml`, `test/test_int_div_js.ml`
253+
|Solo core (executable metatheory) |`test/test_solo_cesk.ml`
254+
|===
255+
256+
NOTE: this anchors *feature-readiness* claims. *Soundness-hole* anchors live in
257+
their own ledger (link:SOUNDNESS.adoc[SOUNDNESS.adoc]), gated by
258+
`tools/check-soundness-ledger.sh`.
259+
228260
== What AffineScript is NOT (anti-over-claim)
229261

230262
* Not "production-ready". Alpha. CORE-01 (#177) closed 2026-05-30. The

docs/SOUNDNESS.adoc

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@ the reverse. This file is the structural fix:
120120
tree-walking continuation has already unwound the bind chain. This is the one
121121
genuinely-still-silent shape, on the *interpreter* path only. It needs a CPS
122122
rewrite of `eval` (blocked: OCaml 4.14 has no native effect handlers and the
123-
interpreter must stay `js_of_ocaml`-compatible — owner-steer item).
123+
interpreter must stay `js_of_ocaml`-compatible — owner-steer item). Tracked: #623.
124124
|*residual (pinned)*
125125
|`test/e2e/fixtures/handle_resume_nontail.affine` +
126126
`test_resume_nontail_known_shallow` (asserts the wrong-but-known value; flips
@@ -170,24 +170,26 @@ the reverse. This file is the structural fix:
170170
The implementation holes above are closed, but honesty requires naming what is
171171
*not* a guarantee:
172172

173-
* *Interpreter non-tail resume* (#555 residual, pinned) — see the ledger row.
173+
* *Interpreter non-tail resume* (#555 residual, pinned; tracked as #623) — see
174+
the ledger row.
174175
* *Stub backends drop `return`.* The Lean and Why3 *experimental* code
175176
generators drop `return` statements wholesale — a broader codegen-honesty gap
176-
than #555, flagged but not yet fenced. Treat all non-reference backends as
177-
experimental (see `docs/CAPABILITY-MATRIX.adoc`).
177+
than #555, flagged but not yet fenced (tracked as #624). Treat all
178+
non-reference backends as experimental (see `docs/CAPABILITY-MATRIX.adoc`).
178179
* *Generic-subsumption coherence* (#559 follow-up) — see the ledger row.
179180

180181
== Closed holes are not proofs
181182

182183
Closing an *implementation* hole is not the same as having *metatheory*. The
183184
soundness *arguments* for the holes above (`docs/academic/proofs/*.adoc`, the
184185
comments in `lib/borrow.ml`) remain `prose`, and the `Solo` core fragment's
185-
`progress` / `preservation` are still `?todo`. Mechanisation has *started* — a
186-
Wave-0 codegen-preservation seed (`formal/K1_CodegenPreservation.v`, axiom-free
187-
Coq/Rocq, minimal nat/bool fragment) landed via #620 — but it does not yet cover
188-
any hole in this ledger. The proof obligations, their rigour tiers, and their
189-
(mostly `prose` / `absent`) status are catalogued in `docs/PROOF-NEEDS.adoc`
190-
(umbrella issue #513). The one-line distinction:
186+
`progress` / `preservation` are still `?todo`. Mechanisation has *started* — an
187+
axiom-free Coq/Rocq `formal/` track (codegen-preservation: K-1
188+
`K1_CodegenPreservation.v`, K-1-with-`let` `K1Let_CodegenPreservation.v`, and
189+
the F-1 transformer-preservation composition; #620/#621) — but it does not yet
190+
cover any hole in this ledger. The proof obligations, their rigour tiers, and
191+
their (mostly `prose` / `absent`) status are catalogued in
192+
`docs/PROOF-NEEDS.adoc` (umbrella issue #513). The one-line distinction:
191193

192194
[quote]
193195
The compiler now fails loud on these shapes and the residuals are pinned by

justfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ guard:
9999
./tools/check-no-extension-ts.sh
100100
./tools/check-doc-truthing.sh
101101
./tools/check-soundness-ledger.sh
102+
./tools/check-capability-anchors.sh
102103

103104
# Re-baseline the doc-truthing over-claim ledger after a deliberate, legitimate
104105
# change (e.g. a new dated roadmap milestone). Commit the .allow diff.

tools/check-capability-anchors.sh

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: MPL-2.0
3+
# Copyright (c) 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
4+
#
5+
# Anti-staleness gate for the capability matrix's test anchors
6+
# (docs/CAPABILITY-MATRIX.adoc "== Test anchors").
7+
#
8+
# The matrix is the authoritative feature-readiness doc. Its "Test anchors"
9+
# section pins each enforcement claim ("works" / "enforced" / "partial") to the
10+
# executable test(s) that exercise it — the same test-anchoring discipline the
11+
# soundness ledger uses (tools/check-soundness-ledger.sh). This gate makes those
12+
# anchors falsifiable: it extracts every test path the matrix names and fails the
13+
# build if any has gone missing, so a feature cannot keep a green status row
14+
# after its test is deleted or renamed without this section (and CI) noticing.
15+
#
16+
# Checks:
17+
# 1. The matrix exists and carries a "== Test anchors" section.
18+
# 2. Every test/*.ml and test/e2e/fixtures/*.affine path named anywhere in the
19+
# matrix actually exists on disk.
20+
#
21+
# Usage: ./tools/check-capability-anchors.sh
22+
# Wired into: just check (via the `guard` recipe) and CI (.github/workflows/ci.yml).
23+
# Run from anywhere; it cd's to the repo root itself.
24+
25+
set -euo pipefail
26+
27+
cd "$(dirname "$0")/.."
28+
29+
MATRIX="docs/CAPABILITY-MATRIX.adoc"
30+
31+
fail=0
32+
note() { printf '%s\n' "$*" >&2; }
33+
34+
# --- 1. The matrix exists and carries its Test-anchors section ---------------
35+
if [ ! -f "$MATRIX" ]; then
36+
note "ERROR: the capability matrix is missing: $MATRIX"
37+
exit 1
38+
fi
39+
if ! grep -q "^== Test anchors" "$MATRIX"; then
40+
note "ERROR: $MATRIX lost its '== Test anchors' section."
41+
note " That section anchors each feature-readiness claim to an"
42+
note " executable test. Restore it (see the soundness ledger for the"
43+
note " same pattern: docs/SOUNDNESS.adoc)."
44+
fail=1
45+
fi
46+
47+
# --- 2. Every test path the matrix names actually exists ---------------------
48+
missing=0
49+
while IFS= read -r path; do
50+
[ -z "$path" ] && continue
51+
if [ ! -f "$path" ]; then
52+
if [ "$missing" -eq 0 ]; then
53+
note "ERROR: $MATRIX names test anchors that no longer exist:"
54+
missing=1
55+
fail=1
56+
fi
57+
note " - $path"
58+
fi
59+
done < <(grep -oE 'test/[A-Za-z0-9_./-]+\.(ml|affine)' "$MATRIX" | LC_ALL=C sort -u)
60+
if [ "$missing" -eq 1 ]; then
61+
note " Either restore the test or update the matrix to its new anchor"
62+
note " in the same change. A renamed/deleted test must not leave a"
63+
note " feature-readiness claim unmoored."
64+
fi
65+
66+
if [ "$fail" -ne 0 ]; then
67+
note ""
68+
note "Capability-anchor guard failed. Feature-readiness claims in"
69+
note "$MATRIX are drifting from the tests that back them."
70+
exit 1
71+
fi
72+
73+
echo "OK: capability anchors intact — Test-anchors section present + every named test exists."

0 commit comments

Comments
 (0)