|
| 1 | +// SPDX-License-Identifier: MPL-2.0 |
| 2 | +// SPDX-FileCopyrightText: 2026 hyperpolymath (Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>) |
| 3 | += AffineScript Soundness Work — Handoff for the Next Session (2026-06-17) |
| 4 | +:toc: |
| 5 | +:toclevels: 3 |
| 6 | + |
| 7 | +Read this file first, then `git log --oneline -12` and run the test suite. |
| 8 | +This is a curated handoff — enough to continue the soundness / Polonius-M3 |
| 9 | +work without re-deriving the map. It is NOT exhaustive repo documentation. |
| 10 | + |
| 11 | +== 0. Your mission, and how to work |
| 12 | + |
| 13 | +You are continuing a *soundness-first* pass on the AffineScript compiler |
| 14 | +(OCaml). The standing owner directive is **"holes before anything else"**: |
| 15 | +soundness holes (silent miscompiles, use-after-move / borrow false-negatives, |
| 16 | +unchecked invariants) are fixed HEAD-ON before any feature / VM / perf / doc |
| 17 | +work. After each fix, keep adversarially probing for more. |
| 18 | + |
| 19 | +Five load-bearing working rules (these override any "just keep going" flow): |
| 20 | + |
| 21 | +. **Ground-truth by running the tool.** Never trust a status doc (including |
| 22 | + `.claude/CLAUDE.md`). Run `dune build` / `dune runtest` / the compiler and |
| 23 | + read the actual result. Multiple "known holes" this session turned out |
| 24 | + already-fixed or closed; one doc said "0% implemented" for code that was at |
| 25 | + M3. Verify before you claim, and before you start work. |
| 26 | +. **Stop-first when revert-cost > ~2 min.** Build locally + commit (signed) |
| 27 | + freely. But before anything outward-facing (push, PR) or any large/risky |
| 28 | + change (e.g. an architecture-level rewrite), surface a short plan and get a |
| 29 | + go-ahead. Do not push; the owner pushes. |
| 30 | +. **NO automated licence / SPDX / attribution edits — ever.** File-by-file, |
| 31 | + owner-only, even when policy-correct. This bit us repeatedly; see §3. |
| 32 | +. **ALWAYS sign commits** with `-S` (the hook + owner both require it). Ensure |
| 33 | + `ssh-agent` has `id_ed25519_signing` (sign key `kVP7…`; `id_ed25519` is |
| 34 | + AUTH-ONLY, never sign with it). Commit/push only when it makes sense; branch |
| 35 | + off, never commit straight to `main`. |
| 36 | +. **The parallel-run gate is your safety net** for borrow-checker work — see |
| 37 | + §7.2. Any false-positive you introduce makes a currently-agreeing fixture |
| 38 | + diverge and the gate fails immediately. Use it; run the full suite after |
| 39 | + every extractor change. |
| 40 | + |
| 41 | +== 1. Repo identity — this is AffineScript, NOT ephapax |
| 42 | + |
| 43 | +`hyperpolymath/affinescript`, an OCaml compiler. It is **not** |
| 44 | +`hyperpolymath/ephapax` (a separate Rust language). They share only the |
| 45 | +compile target (`hyperpolymath/typed-wasm`). The word "affine" is a |
| 46 | +logic-family coincidence. Before applying any prior memory/snippet, check |
| 47 | +which project it was about. (`.claude/CLAUDE.md` §"Disambiguation" has the |
| 48 | +full table.) |
| 49 | + |
| 50 | +* Production borrow/affine checker: `lib/borrow.ml` (lexical, used in |
| 51 | + `bin/main.ml` via `Borrow.check_program`). |
| 52 | +* Type checker: `lib/typecheck.ml`. Parser: `lib/parser.mly`. Interpreter: |
| 53 | + `lib/interp.ml`. AST: `lib/ast.ml`. |
| 54 | +* Build: `dune-project` at root (OCaml `>= 4.14`). |
| 55 | + |
| 56 | +== 2. Environment & commands |
| 57 | + |
| 58 | +* OCaml **4.14.2** — *no native effect handlers* (matters for #555, §7.1). The |
| 59 | + interpreter must also stay `js_of_ocaml`-compatible (jsoo `>= 5.0` is a dep). |
| 60 | +* Build: `dune build`. Test: `dune runtest` (use `--force` to re-run when |
| 61 | + cached; 528 tests as of this handoff). |
| 62 | +* Run the compiler: `dune exec affinescript -- <args>`. Interpreter subcommand |
| 63 | + is `eval FILE` (it prints nothing for a pure `Int` return — tests inspect |
| 64 | + the value via `Interp.eval_program` / `Interp.apply_function` instead). The |
| 65 | + flag is NOT `-i`; `eval` is the subcommand. |
| 66 | +* Codegen targets are selected in `bin/main.ml` (~line 490-660): core-WASM |
| 67 | + (`Codegen`), WasmGC (`Codegen_gc`), Deno-ESM (`Codegen_deno`), JS-text |
| 68 | + (`Js_codegen`), C (`C_codegen`), plus ~20 experimental backends |
| 69 | + (`*_codegen.ml`) and source "faces" (`*_face.ml`). |
| 70 | +* Agda proofs (if you touch them): the stdlib path in `~/.agda/libraries` is |
| 71 | + dangling; use `AGDA_STDLIB=/home/hyperpolymath/developer/worktrees/agda-stdlib-tweak/src`. |
| 72 | + Blanket "Agda FAIL" is usually an env artifact, not a proof defect. |
| 73 | + |
| 74 | +== 3. The pre-commit hook — read before EVERY commit |
| 75 | + |
| 76 | +`.git/hooks/pre-commit` is a deliberately-kept attribution-drift detector. For |
| 77 | +this repo (MPL branch) it checks every *staged* file matching |
| 78 | +`\.(zig|ex|idr|eph|py|js|ts|rs|c|h|adoc|md)$` (diff-filter ACM) for BOTH: |
| 79 | + |
| 80 | +* `SPDX-License-Identifier: MPL-2.0`, AND |
| 81 | +* the literal owner string `Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>`. |
| 82 | + |
| 83 | +Consequences you must internalise: |
| 84 | + |
| 85 | +* `.ml` and `.affine` files are **NOT** checked — most compiler + test + |
| 86 | + fixture work commits with no header friction. |
| 87 | +* `.adoc` / `.md` files ARE checked. The header form that PASSES without |
| 88 | + editing the hook (proven this session): |
| 89 | ++ |
| 90 | +[source] |
| 91 | +---- |
| 92 | +// SPDX-License-Identifier: MPL-2.0 |
| 93 | +// SPDX-FileCopyrightText: <year(s)> hyperpolymath (Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>) |
| 94 | +---- |
| 95 | +* `.claude/CLAUDE.md` is tracked but has **no header**, so you cannot commit an |
| 96 | + edit to it without adding one — and adding a licence header is an *owner-only* |
| 97 | + decision (see §5). Do not edit-and-commit CLAUDE.md; record corrections in a |
| 98 | + ledger you own instead (that is exactly what was done this session). |
| 99 | +* Never `--no-verify`, never weaken the hook. If a pre-existing tracked file |
| 100 | + has attribution drift, that is the hook working — surface it, don't auto-fix. |
| 101 | + |
| 102 | +== 4. File map (the bits you'll actually touch) |
| 103 | + |
| 104 | +[cols="2,3",options="header"] |
| 105 | +|=== |
| 106 | +| Path | What it is |
| 107 | + |
| 108 | +| `lib/borrow.ml` | PRODUCTION borrow/affine checker (lexical). Soundness |
| 109 | + verdicts come from here. `expr_to_place`, `places_overlap`, `root_var`, |
| 110 | + `check_program`, the `BorrowOutlivesOwner` / return-escape machinery. |
| 111 | +| `lib/borrow_extract.ml` | The Polonius *extractor* (ADR-022 M3). TEST-ONLY, |
| 112 | + unwired from `bin/main.ml`; gates no production verdict. Emits datalog facts |
| 113 | + (`borrow_at`/`killed`/`cfg_edge`/`conflict_at`/`move_at`/`use_at`/ |
| 114 | + `reinit_at`) from a function body. `rloan` now has `rl_excl`. |
| 115 | +| `lib/borrow_polonius/{types,solve}.ml` | The naive-datalog solver: |
| 116 | + `compute_live`, `compute_moved_in`, `solve`. Loan liveness + UAM dataflow. |
| 117 | +| `lib/typecheck.ml` | Type checker. `check_program` (forward pass registers |
| 118 | + traits/impls, then the check pass). `type_error` + `show_type_error`. |
| 119 | + Trait-coherence call is wired here. |
| 120 | +| `lib/trait.ml` | Trait registry, impl satisfaction, method resolution, |
| 121 | + `check_coherence` / `check_all_coherence` (now implemented). |
| 122 | +| `lib/interp.ml` | Tree-walking interpreter. Effect handler dispatch |
| 123 | + (`ExprHandle`/`PerformEffect`/`resume`) ~line 282-385. Shallow single-shot. |
| 124 | +| `lib/codegen*.ml` | Backends. `ExprHandle`/`ExprResume` must fail loud, not |
| 125 | + drop arms (#555). `Async` CPS fence is in `lib/codegen.ml` (#556). |
| 126 | +| `test/test_e2e.ml` | Main E2E suite (parse→resolve→typecheck→codegen→interp). |
| 127 | + Handler-fence, resume-soundness, async-fence, trait-coherence tests live here. |
| 128 | +| `test/test_borrow_polonius.ml` | Polonius solver + extractor tests + the |
| 129 | + corpus parallel-run diff gate + the `known_divergences` allowlist. |
| 130 | +| `test/e2e/fixtures/*.affine` | Test programs (not hook-checked). |
| 131 | +| `docs/reports/TIDY-UP-LEDGER-2026-06-16.adoc` | Tidy-up + the ground-truthed |
| 132 | + soundness-survey correction (since CLAUDE.md is stale). |
| 133 | +|=== |
| 134 | + |
| 135 | +== 5. Current soundness ground-truth (the CLAUDE.md survey is STALE) |
| 136 | + |
| 137 | +`.claude/CLAUDE.md` §"Known soundness items (2026-06-11 survey)" and the |
| 138 | +disambiguation-table Proofs cell are OUT OF DATE. Do not quote them. The |
| 139 | +test-verified status (528 tests green) — also in the tidy-ledger §"Soundness |
| 140 | +survey correction" — is: |
| 141 | + |
| 142 | +* **#554 — FIXED.** Use-after-move through a callee-returned borrow is REJECTED |
| 143 | + (`let r = pick(a); consume(a); *r` → `MoveWhileBorrowed`), with hardening + |
| 144 | + anti-over-rejection tests. #177 closed. issue-draft 08 (conditional-origin |
| 145 | + escape) also FIXED. |
| 146 | +* **#555 — codegen FENCED everywhere reachable** (WASM/WasmGC/Deno-ESM/JS-text/ |
| 147 | + C all fail loud on `handle`/`resume`). Interp: tail-resume correct; multi-shot |
| 148 | + loud-fails; **non-tail single-shot still silently returns the resumed value** |
| 149 | + (residual — §7.1). Runtime handler tests now exist. Lean/Why3 are stub |
| 150 | + backends that drop `return` statements wholesale (broader gap, flagged). |
| 151 | +* **#556 — FIXED.** Async CPS table-miss fails loud, not silent sync fallback. |
| 152 | +* **#558 — N/A (closed).** Refinement & dependent types were removed in v1 |
| 153 | + (2026-04-10); `assume(...)` is rejected and `T where (P)` parse-errors — no |
| 154 | + refinement-type node for the checker to silently ignore. |
| 155 | +* **#559 — FIXED (concrete overlaps).** Trait coherence implemented + wired; |
| 156 | + overlapping impls whose self types unify are rejected. Generic-subsumption |
| 157 | + overlap is follow-up. |
| 158 | +* **#553 / Polonius** — M1–M3 implemented (NOT "0%"); TEST-ONLY / unwired; |
| 159 | + allowlist down to **7** divergences. |
| 160 | + |
| 161 | +== 6. What this session did (branch `feat/solo-core-metatheory-proofs`, unpushed) |
| 162 | + |
| 163 | +Newest first: |
| 164 | + |
| 165 | +* `414e664` docs(ledger): record ground-truthed soundness-survey correction. |
| 166 | +* `3a531e1` feat(ADR-022 M3): model loan-vs-loan exclusivity in the extractor |
| 167 | + (allowlist 9→7; both mutref fixtures now agree). |
| 168 | +* `2b3f63c` fix(#559): implement + wire trait coherence checking. |
| 169 | +* `bc061ad` fix(#555): C backend fails loud on handle/resume. |
| 170 | +* `aa1b73f` + the batch-B series (`2ba8141`…`5bb6149`, `4747e9d`): docs |
| 171 | + `.md`→`.adoc` conversion + tidy-up (separate workstream; see the ledger). |
| 172 | + |
| 173 | +Everything is local + signed. Nothing pushed. 528 tests green. |
| 174 | + |
| 175 | +== 7. Open work — prioritized, with exact next steps |
| 176 | + |
| 177 | +=== 7.1 #555 residual — delimited-continuation interpreter (BLOCKED, needs owner steer) |
| 178 | + |
| 179 | +The interpreter's non-tail single-shot resume (`let x = ask(); x + 100` with |
| 180 | +`ask() => resume(5)`) silently returns `5` instead of `105` — pinned by |
| 181 | +`test_resume_nontail_known_shallow` in `test_e2e.ml`. The correct fix is real |
| 182 | +delimited continuations, but OCaml 4.14 has no native effect handlers and the |
| 183 | +interpreter must stay `js_of_ocaml`-compatible, so it means a **CPS rewrite of |
| 184 | +`eval`** — pervasive and risky against 528 tests + the jsoo build. This is an |
| 185 | +architecture-level change: *surface a plan and get a go-ahead before starting* |
| 186 | +(stop-first). Do NOT silently begin a CPS rewrite. |
| 187 | + |
| 188 | +=== 7.2 Polonius M3 — remaining divergences (the most tractable head-on work) |
| 189 | + |
| 190 | +The extractor (`lib/borrow_extract.ml`) is diffed against the lexical checker |
| 191 | +over every borrow-checkable fixture by the corpus gate |
| 192 | +(`t_parallel_run_diff` in `test_borrow_polonius.ml`). The `known_divergences` |
| 193 | +allowlist currently has **7** entries in 3 classes. Closing one = make the |
| 194 | +extractor agree, then prune the entry; the gate proves no false-positive crept |
| 195 | +into the other ~100 fixtures. |
| 196 | + |
| 197 | +. **return-escape / borrow-outlives-owner (4)**: `borrow_outlives_owner`, |
| 198 | + `borrow_return_escape_local`, `borrow_return_escape_param`, |
| 199 | + `ref_to_ref_return_escape`. The extractor has no escape analysis; the lexical |
| 200 | + checker rejects these via `BorrowOutlivesOwner` / the return-borrow summary. |
| 201 | + Next step: model a borrow whose place is rooted at a function-local / by-value |
| 202 | + param and which escapes via `return` (or a returned aggregate) as an error |
| 203 | + fact. Mirror `lib/borrow.ml`'s escape logic; gate by the same liveness. |
| 204 | +. **captured-linear (2)**: `slice_d_captured_linear_let_rejected`, |
| 205 | + `slice_d_captured_linear_param_rejected`. Needs lambda-capture analysis in |
| 206 | + the extractor (a linear/affine value captured by a closure is consumed). |
| 207 | +. **call-aliasing mut-param-arg (1)**: `borrow_use_while_excl` |
| 208 | + (`mut_then_read(x, x)`). Model a `mut`-param argument as a call-scoped |
| 209 | + EXCLUSIVE loan born at the call point, then the existing use-while-excl rule |
| 210 | + (just added) flags the other `x` arg. This is probably the smallest of the |
| 211 | + three — good warm-up. |
| 212 | + |
| 213 | +Pattern that worked this session (loan-vs-loan, `3a531e1`): add the minimal |
| 214 | +fact-emission to the extractor, run the FULL suite, then prune the allowlist |
| 215 | +entry only after the gate stays green. Keep `&` (shared) vs `&mut` (exclusive) |
| 216 | +distinct — over-rejecting valid code is the cardinal sin. |
| 217 | + |
| 218 | +=== 7.3 #559 follow-up — generic-subsumption coherence |
| 219 | + |
| 220 | +`impl[T] Greet for Box[T]` overlapping `impl Greet for Box[Int]` is not yet |
| 221 | +detected. The coherence check itself would catch it (it instantiates impl type |
| 222 | +params via `fresh_impl_self_ty`), but generic *impl* handling has separate |
| 223 | +immaturities — `impl[T] … for Box[T]` currently trips a spurious "Trait not |
| 224 | +found" before coherence runs. Fix generic-impl registration first, then this |
| 225 | +follows. (Documented in the coherence test block in `test_e2e.ml`.) |
| 226 | + |
| 227 | +=== 7.4 Owner-manual items (do NOT auto-do) |
| 228 | + |
| 229 | +* Update the `.claude/CLAUDE.md` survey from §5 (needs an owner-authored header |
| 230 | + first — §3). |
| 231 | +* Uninstantiated RSR templates: `SECURITY.md` (root + `docs/governance/`), |
| 232 | + `docs/reference/ABI-FFI.md` (`{{PLACEHOLDER}}`s). |
| 233 | +* `docs/governance/LICENSING-GUIDE.md` format conversion (licence-adjacent). |
| 234 | +* `docs/PROOF-NEEDS.md` is owner-gated/parked (shows as modified in the tree — |
| 235 | + do not commit it). |
| 236 | + |
| 237 | +== 8. Methods that paid off (reuse them) |
| 238 | + |
| 239 | +* **Ground-truth first.** Each "hole" got reproduced/checked by running before |
| 240 | + touching code. Half were already fixed or closed — you'll waste effort if you |
| 241 | + trust the survey. |
| 242 | +* **Delegate broad reads.** An `Explore` agent audited ~18 codegen backends for |
| 243 | + silent `ExprHandle` handling in one shot — cheap, kept context clean. |
| 244 | +* **Fail loud, never silent.** The whole #555/#556 philosophy: a backend that |
| 245 | + can't implement a construct must error, not emit a plausible-wrong value. |
| 246 | + When you add a new declaration/expression shape, feed it to EVERY consumer |
| 247 | + (parse/resolve/typecheck/interp/each codegen) and assert each either handles |
| 248 | + or loudly rejects it. |
| 249 | +* **Liveness-gated fact emission.** For borrow work, emit the conflict fact and |
| 250 | + let the solver's `loan_live_at` decide — that gives NLL correctness for free |
| 251 | + and avoids over-rejection. |
| 252 | +* **Pin known incompleteness as an executable test** (e.g. |
| 253 | + `test_resume_nontail_known_shallow`) rather than leaving it silent. |
| 254 | + |
| 255 | +== 9. Suggested first moves |
| 256 | + |
| 257 | +. `git status` (expect clean apart from parked `docs/PROOF-NEEDS.md`), then |
| 258 | + `git log --oneline -12`, then `dune build && dune runtest` (expect 528 OK). |
| 259 | +. Read this file + the tidy-ledger §"Soundness survey correction". |
| 260 | +. Pick up §7.2 call-aliasing (`borrow_use_while_excl`) as the smallest M3 |
| 261 | + increment, OR ask the owner whether to take on the #555 CPS-interpreter |
| 262 | + rewrite (§7.1) — that one needs a go-ahead. |
| 263 | +. Keep adversarially probing: after any fix, look for the next hole. The owner |
| 264 | + values honest "here's what's still broken" over a tidy but false "done". |
0 commit comments