|
| 1 | +<!-- |
| 2 | +SPDX-License-Identifier: MPL-2.0 |
| 3 | +SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk> |
| 4 | +--> |
| 5 | + |
| 6 | +# GNN training run — 2026-06-02 |
| 7 | + |
| 8 | +First owner-authorised GNN training run after the post-wave3 wiring |
| 9 | +PR landed (#207, merged 2026-06-02T04:14:04Z). |
| 10 | + |
| 11 | +## Provenance |
| 12 | + |
| 13 | +| Field | Value | |
| 14 | +|---|---| |
| 15 | +| Branch | `gnn-training-run-2026-06-02` (local-only, no main pollution per Option 4) | |
| 16 | +| Worktree | `/home/hyperpolymath/developer/repos/echidna-training-2026-06-02` | |
| 17 | +| Launcher | `/tmp/echidna-training-launcher.sh` | |
| 18 | +| Log | `/tmp/echidna-training-2026-06-02.log` | |
| 19 | +| Status file | `/tmp/echidna-training-2026-06-02.status` | |
| 20 | +| Pre-requisite PRs merged | #198 saturation+typing, #206 wave3 closeout, #207 GNN wiring, #210 JSON dep fix | |
| 21 | +| Hardware | 4 CPU / 5GB RAM / WSL2 with CUDA passthrough (libcuda.so.535.309.01) | |
| 22 | +| Julia | 1.12.6 via juliaup | |
| 23 | + |
| 24 | +## Pipeline |
| 25 | + |
| 26 | +1. **Stage 0** — `Pkg.instantiate()` to verify Julia env |
| 27 | +2. **Stage 1** — `just provision-corpora` (shallow clones of all upstream sources to `external_corpora/`) |
| 28 | +3. **Stage 2** — loop over 17 saturation adapters via `just corpus-ingest-saturation <name> training_data/<name>/` |
| 29 | +4. **Stage 3** — `just train-from-corpus lean` |
| 30 | +5. **Stage 4** — `just train-from-corpus coq` |
| 31 | +6. **Stage 5** — list `models/neural/` artefacts |
| 32 | + |
| 33 | +## Run outcomes (filled in as stages complete) |
| 34 | + |
| 35 | +### Stage 0 — Julia env |
| 36 | + |
| 37 | +- _2026-06-02T10:46:36Z_ — `Pkg.status()` green; 14 direct deps + JSON (added in #210) |
| 38 | + |
| 39 | +### Stage 1 — provision-corpora |
| 40 | + |
| 41 | +- _2026-06-02T10:46:55Z_ — **43 upstream corpora present** (idempotent cache hits after v2 run) |
| 42 | +- Known per-source failures: several GitLab sources return `Gitlab::GitAccess::NotFoundError`; matita returns 404 (rocq-prover/matita repo unavailable). Non-fatal — script tolerates per-source failures. |
| 43 | + |
| 44 | +### Stage 2 — ingest 17 adapters |
| 45 | + |
| 46 | +- _2026-06-02T10:48:05Z_ — **4 of 17 adapters ingested**: `agda`, `coq` confirmed; lean + idris2 expected. |
| 47 | +- Known cause for the 13 failures: `corpus-ingest-saturation` recipe (Justfile:988) only knows the 4 dependent-typed provers. The other 13 (`isabelle`, `metamath`, `mizar`, `hol_light`, `hol4`, `dafny`, `why3`, `fstar`, `acl2_books`, `tptp`, `smtlib`, `proofnet`, `minif2f`) need per-adapter ingest scripts written. Sketch §7 promised 17; reality is 4. Tracked as roadmap follow-up. |
| 48 | + |
| 49 | +### Stage 3 — train-from-corpus lean |
| 50 | + |
| 51 | +- _TBD_ — first run blocked on Julia `JSON` dep (fixed in #210); awaiting v3 run completion. |
| 52 | +- Wall-clock target: ~10min CPU baseline / ~2hr GPU on small corpus |
| 53 | + |
| 54 | +### Stage 4 — train-from-corpus coq |
| 55 | + |
| 56 | +- _TBD_ |
| 57 | + |
| 58 | +### Stage 5 — artefacts |
| 59 | + |
| 60 | +- _TBD_ — should populate `models/neural/`. Existing `models/neural/` in the local checkout's wave3 worktree has prior weights; this run uses the dedicated branch worktree so artefacts don't pollute main. |
| 61 | + |
| 62 | +## Outcomes summary |
| 63 | + |
| 64 | +- **Models produced**: _TBD_ |
| 65 | +- **MRR vs cosine baseline**: _TBD_ |
| 66 | +- **Comparable to wave3 baseline**: _TBD — needs side-by-side once wave3 lands its first GNN run_ |
| 67 | + |
| 68 | +## Known issues discovered this run |
| 69 | + |
| 70 | +1. **JSON dep mismatch** — fixed via #210. `corpus_loader.jl` uses `JSON.parse` but `Project.toml` had only `JSON3`. |
| 71 | +2. **`CUDA.version()` MethodError** — cosmetic logging error; CUDA.jl 5.x removed the method. Fixed via this PR. |
| 72 | +3. **`corpus-ingest-saturation` 4-vs-17 gap** — known; tracked separately. |
| 73 | + |
| 74 | +## Follow-up tickets |
| 75 | + |
| 76 | +- (TBD if needed) Extend `corpus-ingest-saturation` recipe to all 17 adapters — large scope, requires per-adapter Rust extractor scripts. |
| 77 | +- (TBD) Compare `models/neural/` outputs vs wave3 baseline once both land. |
| 78 | + |
| 79 | +## Re-running |
| 80 | + |
| 81 | +Owner-side re-run after corpus refresh: |
| 82 | +```bash |
| 83 | +just provision-corpora extract-corpora |
| 84 | +for adapter in agda coq lean idris2; do |
| 85 | + just corpus-ingest-saturation "$adapter" "training_data/$adapter/" |
| 86 | +done |
| 87 | +just train-from-corpus lean |
| 88 | +just train-from-corpus coq |
| 89 | +``` |
| 90 | + |
| 91 | +Note `extract-corpora` is a *separate* recipe call — not a positional arg to `provision-corpora` (that was an early-run mistake; cost 1 launcher iteration). |
0 commit comments