Skip to content

Commit 194db20

Browse files
chore(training): EchidnaML CUDA logging fix + 2026-06-02 run doc (#211)
Two small training-infrastructure cleanups bundled. **EchidnaML.jl**: `CUDA.version()` removed in CUDA.jl 5.x; replace with `runtime_version()` / `driver_version()` wrapped in try/catch. Surfaced as noisy MethodError during the first owner-authorised GNN training run today. **docs/training-runs/2026-06-02.md**: per-stage scaffold for the run currently in progress (stages 0/1/2 outcomes captured; 3/4/5 to fill in as they complete). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent ea5f881 commit 194db20

2 files changed

Lines changed: 100 additions & 1 deletion

File tree

docs/training-runs/2026-06-02.md

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
<!--
2+
SPDX-License-Identifier: MPL-2.0
3+
SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
4+
-->
5+
6+
# GNN training run — 2026-06-02
7+
8+
First owner-authorised GNN training run after the post-wave3 wiring
9+
PR landed (#207, merged 2026-06-02T04:14:04Z).
10+
11+
## Provenance
12+
13+
| Field | Value |
14+
|---|---|
15+
| Branch | `gnn-training-run-2026-06-02` (local-only, no main pollution per Option 4) |
16+
| Worktree | `/home/hyperpolymath/developer/repos/echidna-training-2026-06-02` |
17+
| Launcher | `/tmp/echidna-training-launcher.sh` |
18+
| Log | `/tmp/echidna-training-2026-06-02.log` |
19+
| Status file | `/tmp/echidna-training-2026-06-02.status` |
20+
| Pre-requisite PRs merged | #198 saturation+typing, #206 wave3 closeout, #207 GNN wiring, #210 JSON dep fix |
21+
| Hardware | 4 CPU / 5GB RAM / WSL2 with CUDA passthrough (libcuda.so.535.309.01) |
22+
| Julia | 1.12.6 via juliaup |
23+
24+
## Pipeline
25+
26+
1. **Stage 0**`Pkg.instantiate()` to verify Julia env
27+
2. **Stage 1**`just provision-corpora` (shallow clones of all upstream sources to `external_corpora/`)
28+
3. **Stage 2** — loop over 17 saturation adapters via `just corpus-ingest-saturation <name> training_data/<name>/`
29+
4. **Stage 3**`just train-from-corpus lean`
30+
5. **Stage 4**`just train-from-corpus coq`
31+
6. **Stage 5** — list `models/neural/` artefacts
32+
33+
## Run outcomes (filled in as stages complete)
34+
35+
### Stage 0 — Julia env
36+
37+
- _2026-06-02T10:46:36Z_`Pkg.status()` green; 14 direct deps + JSON (added in #210)
38+
39+
### Stage 1 — provision-corpora
40+
41+
- _2026-06-02T10:46:55Z_**43 upstream corpora present** (idempotent cache hits after v2 run)
42+
- Known per-source failures: several GitLab sources return `Gitlab::GitAccess::NotFoundError`; matita returns 404 (rocq-prover/matita repo unavailable). Non-fatal — script tolerates per-source failures.
43+
44+
### Stage 2 — ingest 17 adapters
45+
46+
- _2026-06-02T10:48:05Z_**4 of 17 adapters ingested**: `agda`, `coq` confirmed; lean + idris2 expected.
47+
- Known cause for the 13 failures: `corpus-ingest-saturation` recipe (Justfile:988) only knows the 4 dependent-typed provers. The other 13 (`isabelle`, `metamath`, `mizar`, `hol_light`, `hol4`, `dafny`, `why3`, `fstar`, `acl2_books`, `tptp`, `smtlib`, `proofnet`, `minif2f`) need per-adapter ingest scripts written. Sketch §7 promised 17; reality is 4. Tracked as roadmap follow-up.
48+
49+
### Stage 3 — train-from-corpus lean
50+
51+
- _TBD_ — first run blocked on Julia `JSON` dep (fixed in #210); awaiting v3 run completion.
52+
- Wall-clock target: ~10min CPU baseline / ~2hr GPU on small corpus
53+
54+
### Stage 4 — train-from-corpus coq
55+
56+
- _TBD_
57+
58+
### Stage 5 — artefacts
59+
60+
- _TBD_ — should populate `models/neural/`. Existing `models/neural/` in the local checkout's wave3 worktree has prior weights; this run uses the dedicated branch worktree so artefacts don't pollute main.
61+
62+
## Outcomes summary
63+
64+
- **Models produced**: _TBD_
65+
- **MRR vs cosine baseline**: _TBD_
66+
- **Comparable to wave3 baseline**: _TBD — needs side-by-side once wave3 lands its first GNN run_
67+
68+
## Known issues discovered this run
69+
70+
1. **JSON dep mismatch** — fixed via #210. `corpus_loader.jl` uses `JSON.parse` but `Project.toml` had only `JSON3`.
71+
2. **`CUDA.version()` MethodError** — cosmetic logging error; CUDA.jl 5.x removed the method. Fixed via this PR.
72+
3. **`corpus-ingest-saturation` 4-vs-17 gap** — known; tracked separately.
73+
74+
## Follow-up tickets
75+
76+
- (TBD if needed) Extend `corpus-ingest-saturation` recipe to all 17 adapters — large scope, requires per-adapter Rust extractor scripts.
77+
- (TBD) Compare `models/neural/` outputs vs wave3 baseline once both land.
78+
79+
## Re-running
80+
81+
Owner-side re-run after corpus refresh:
82+
```bash
83+
just provision-corpora extract-corpora
84+
for adapter in agda coq lean idris2; do
85+
just corpus-ingest-saturation "$adapter" "training_data/$adapter/"
86+
done
87+
just train-from-corpus lean
88+
just train-from-corpus coq
89+
```
90+
91+
Note `extract-corpora` is a *separate* recipe call — not a positional arg to `provision-corpora` (that was an early-run mistake; cost 1 launcher iteration).

src/julia/EchidnaML.jl

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,15 @@ function __init__()
198198
# Check CUDA availability
199199
if CUDA.functional()
200200
@info "CUDA available - using GPU acceleration"
201-
@info "CUDA version: $(CUDA.version())"
201+
# CUDA.jl 5.x uses runtime_version() / driver_version();
202+
# legacy CUDA.version() was removed. Wrap defensively so a
203+
# logging-only probe can't kill the init path.
204+
try
205+
@info "CUDA runtime version: $(CUDA.runtime_version())"
206+
@info "CUDA driver version: $(CUDA.driver_version())"
207+
catch e
208+
@debug "CUDA version probe failed (non-fatal)" exception=e
209+
end
202210
@info "Device: $(CUDA.device())"
203211
else
204212
@info "CUDA not available - using CPU"

0 commit comments

Comments
 (0)