Skip to content

Commit 5ddb6c1

Browse files
feat(provers): close GNN-ranking arc — Tier-3/niche sweep (53) + doc close-out (#137)
## Summary Closes the GNN-routing arc started in S5 (commit c8a4f25) and continued via #135 (Tier-1 ext 5) + #136 (Tier-1 finisher 2 + Tier-2 sweep 33). Wires the remaining **53 Tier-3 / niche / placeholder backends** that have `suggest_tactics` impls and ships the matching doc close-out across PROVER_COUNT.md, ROADMAP.md, and the S5 verification runbook. After this PR (post-#136 merge / rebase), **every backend in `ProverKind` with a `suggest_tactics` impl prepends GNN-ranked apply tactics ahead of its heuristic hints** when the Julia `/gnn/rank` service is reachable — and no-ops gracefully otherwise. End-state target met for §S5. ## Coverage progression | Session step | Wired | Notes | |---|---|---| | pre-session | 5 | S5 pilot: rocq, lean, agda, isabelle, z3 | | fb3db5a (earlier) | +3 | abella, easycrypt, keymaerax | | #135 | +5 | Tier-1 extension: idris2, fstar, cvc5, vampire, dafny | | #136 | +35 | Tier-1 finisher (altergo, eprover) + Tier-2 sweep (33) | | **this PR** | **+53** | Tier-3 / niche / placeholder sweep | ## Backends in this PR (53) abc, acl2s, alloy, arend, boogie, cadical, **coq** *(separate from rocq)*, cryptoverif, cubical_agda, dedukti, elk, faial, framac, gnatprove, gpuverify, **hp_ecosystem** *(routes 40 HP disciplines)*, ileancop, isabelle_zf, kissat, konclude, lambda_prolog, lean3, liquid_haskell *(name: `liquid-haskell`)*, matita, mercury, mettel2, minisat, mizar_ar, mleancop, nanocop, naproche, nitpick, nunchaku, nusmv, opensmt, prism, prob, prover9, proverif, qepcad, redlog, seahorn, smtrat, spin_checker *(name: `spin`)*, stainless, storm, tamarin, tlc, **typed_wasm** *(routes 39 TypeChecker disciplines)*, uppaal, uppaal_stratego, viper, zipperposition ## Doc close-out - **`docs/PROVER_COUNT.md`**: TL;DR table updated to reflect full GNN coverage; version-drift log adds v2.3; last-revised bumped to 2026-05-30. - **`docs/ROADMAP.md`** §4 stage 4c: GNN-ranking surface marked COMPLETE; §S5 status flipped to "done 2026-05-30 ✓"; "what is not blocking" line updated. - **`docs/handover/S5-VERIFICATION-RUNBOOK.md`**: replaces hardcoded "11 passed" with a formula tied to test-file content, so future additions don't require a runbook edit. ## Test plan - [x] `cargo build --lib` clean - [x] `cargo build --tests` clean (1m 53s) - [x] `cargo test --test gnn_augment_integration` → **64 passed; 0 failed** in 0.37s (was 11). After rebase onto post-#136 main, becomes 99. - [ ] CI rust-ci / Live-Provers / MVP Smoke green - [ ] Will need rebase after #136 merges (clean — independent file sets, test-file insertion-order conflict resolves automatically) ## Follow-up Remaining work toward genuine GNN-derived suggestions (not just wire-format): 1. **S2 hardware training** — `just train-cpu`/`train` on the 553MB / 66,674-proof corpus to populate `models/neural/`. Owner has hardware; PC setup pending. 2. **Native search expansion** (Stage 4d) — currently 16 of 91 backends have native theorem search; the other 72 rely on Verisim cross-prover fallback at the dispatcher. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 34c09a4 commit 5ddb6c1

57 files changed

Lines changed: 470 additions & 147 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/PROVER_COUNT.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55

66
**Status**: canonical. Cite this file when documenting backend coverage in any
77
other doc. Other surfaces that quoted historical counts (12, 30, 48, 105) are
8-
being updated to point here. Last revised: 2026-05-26.
8+
being updated to point here. Last revised: 2026-05-30.
99

1010
## TL;DR
1111

@@ -15,7 +15,8 @@ being updated to point here. Last revised: 2026-05-26.
1515
| External prover bindings (separate binary or library) | 89 |
1616
| `TypeChecker` disciplines routed via TypedWasm Sigma | 39 |
1717
| Exposed by default REST API (`Tier 1` / core) | **12** |
18-
| With real `suggest_tactics` (not stub) | 91 of 91 (5 of 91 still use heuristic only; GNN-ranked is target end-state) |
18+
| With real `suggest_tactics` (not stub) | **91 of 91** |
19+
| Routing tactic suggestions through `gnn_augment_tactics` | **all backends with `suggest_tactics`** (S5 pilot 5 + Tier-1 extension 5 + Tier-1 finisher 2 + Tier-2 sweep 33 + Tier-3/niche sweep 53 — full coverage as of 2026-05-30; gracefully no-ops when `gnn_api_url` is None or `neural_enabled` is false) |
1920
| With native search command | 16 of 91 (72 return `Ok(vec![])` — cross-prover search via `VeriSimAdvisor` covers them) |
2021
| Trust pipeline integrity-hashed | All Tier 1; Tier 2 incrementally |
2122

@@ -47,6 +48,13 @@ History of the count drift:
4748
- v2.1 (May 2026): **105** after Wave-3 (Tamarin, ProVerif, Twelf, OR-Tools).
4849
- v2.2 (May 2026): **128** after 39 TypeChecker disciplines were Sigma-routed
4950
through TypedWasm (commit `c4bc272` and follow-on).
51+
- v2.3 (May 2026): same **128** declared, but `gnn_augment_tactics` now wraps
52+
every backend with `suggest_tactics` (S5 pilot c8a4f25 + #135 Tier-1 extension
53+
+ #136 Tier-1 finisher + Tier-2 sweep + Tier-3/niche sweep). Wiring is a
54+
no-op when the Julia `/gnn/rank` service is unreachable; once trained
55+
weights land (`models/neural/`), every backend automatically returns
56+
model-derived premise-apply tactics at the top of its `suggest_tactics`
57+
list.
5058

5159
Documents in this repo predating each milestone often quote the count current
5260
to their authoring date. When in doubt, **trust this file** and

docs/ROADMAP.md

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -69,9 +69,15 @@ Stage 3 Learning loop closes model improves from outcomes
6969
Stage 4 Interaction layer honest every declared ProverKind works
7070
4a typed_wasm → crates/typed_wasm [done 2026‑04‑22 ✓]
7171
4b 39 TypeChecker variant dispatch [done 2026‑04‑22 ✓, all Sigma‑routed]
72-
4c tactic synthesis template [91/91 real impl ✓; 5 still heuristic-only —
73-
GNN-ranking is the end-state target;
74-
see docs/PROVER_COUNT.md]
72+
4c tactic synthesis template [91/91 real impl ✓; GNN-ranking surface
73+
COMPLETE 2026‑05‑30 — every backend with
74+
suggest_tactics now routes through
75+
gnn_augment_tactics (S5 pilot + Tier-1
76+
ext + Tier-1 finisher + Tier-2 sweep +
77+
Tier-3/niche sweep). Helper no-ops when
78+
Julia /gnn/rank is unreachable. End-state
79+
target met; remaining S2 dependency is
80+
real trained weights at models/neural/]
7581
4d search_theorems template [done 2026‑04‑27 ✓ — cross-prover layer
7682
added at dispatcher (CLI/REST/REPL all
7783
call `vcl_ut::cross_prover_search_names`).
@@ -139,7 +145,7 @@ and start the self‑learning loop closure.
139145
| S2 | 2a / 2c | Run training on the fixed corpus; record new MRR; commit `metrics_baseline.jsonl` | **Hardware / you** | gated by S1 (S1 ✓; S2 hardware step still to run) |
140146
| S3 | 4a / 4b | Extract `typed_wasm` to `crates/typed_wasm/`; route 39 TypeChecker variants through Sigma parameters | **Opus‑design + Sonnet‑impl** | **done 2026‑04‑22 ✓** |
141147
| S4 | 3a / 3b | Wire Verisim cross‑prover read paths (`goal_hash` queries + `mv_prover_success_by_class` + hypatia loop) | **Opus‑design + Sonnet‑impl** | **wired 2026‑04‑27 ✓** — read paths (`query_prover_success_by_class` via `VeriSimAdvisor`, `cross_prover_search_names` via `vcl_ut`) + write path (`spawn_record_attempt`) + end-to-end test (`tests/s4_loop_closure.rs`, `just test-s4-loop`); CI workflow filing gated on `ghcr-publish.yml` in `verisimdb` upstream (runbook holds the YAML) |
142-
| S5 | 4c / 4d (pilot) | Tactic synthesis template for 5 high‑value provers (coq, lean, agda, isabelle, z3) using the GNN | **Opus‑design + Sonnet‑impl** | **mostly done**`suggest_tactics` is 91/91 real; `gnn_augment_tactics` wired into the 5 pilot backends (rocq/lean/agda/isabelle/z3) per c4bc272; remaining 5 heuristic-only backends are the GNN-ranked end-state target |
148+
| S5 | 4c / 4d (pilot) | Tactic synthesis template for 5 high‑value provers (coq, lean, agda, isabelle, z3) using the GNN | **Opus‑design + Sonnet‑impl** | **done 2026‑05‑30 ✓**`suggest_tactics` is 91/91 real; `gnn_augment_tactics` wired into ALL backends with `suggest_tactics` (S5 pilot 5 in c8a4f25, Tier-1 extension 5 in #135, Tier-1 finisher 2 + Tier-2 sweep 33 in #136, Tier-3/niche sweep 53 in this PR). Wiring is a no-op when Julia /gnn/rank is unreachable; once S2 trained weights land at `models/neural/`, every backend automatically returns model-derived premise-apply tactics |
143149

144150
After S1–S5 land, the roadmap's next sprint takes up Stage 5 (IPC +
145151
Chapel + Tier‑4 CI) and Stage 8 begins in parallel.
@@ -162,7 +168,9 @@ What is **not** blocking and can move now:
162168
- **Phase 1a** (Leo3, Satallax, Lash, AgsyHOL) — done 2026‑04‑26 ✓
163169
- **Phase 1b** (IProver, Princess, Twee, MetiTarski, CSI, AProVE) — done ✓
164170
- **S3** (typed_wasm + 39 TypeChecker Sigma routing) — done ✓
165-
- **`suggest_tactics`** — 86/91 real implementations; only 5 stubs left
171+
- **`suggest_tactics`** — 91/91 real implementations ✓; every backend now
172+
routes through `gnn_augment_tactics` (no-op until S2 weights land at
173+
`models/neural/`)
166174

167175
What **is** blocking and needs Opus + cross‑repo input:
168176

docs/handover/S5-VERIFICATION-RUNBOOK.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -28,17 +28,17 @@ Spawns an in-process mock HTTP server and asserts:
2828

2929
- `GnnClient::health_status()` returns the richer payload (model_path, vocab_size,
3030
training_records_received).
31-
- For each of rocq, lean, agda, isabelle, z3 (the S5 pilot), idris2, fstar,
32-
cvc5, vampire, dafny (the Tier-1 extension), altergo, eprover (the Tier-1
33-
finisher), and the 33 Tier-2 backends (acl2, agsyhol, aprove, athena,
34-
cameleer, cbmc, chuffed, csi, dreal, glpk, HOL4, hol_light, imandra, iprover,
35-
key, lash, leo3, metamath, metitarski, minizinc, minlog, mizar, nuprl,
36-
ortools, princess, PVS, satallax, scip, spass, tlaps, twee, twelf, why3):
31+
- For every backend in `ProverKind` whose `suggest_tactics` is wired through
32+
`gnn_augment_tactics` (S5 pilot 5 + Tier-1 extension 5 + Tier-1 finisher 2 +
33+
Tier-2 sweep 33 + Tier-3/niche sweep 53 — full coverage as of 2026-05-30):
3734
`suggest_tactics` returns `Tactic::Custom { command: "apply", args: ["lemma_foo"] }`
38-
as the first tactic, proving the `/gnn/rank` wire format is consumed
39-
correctly by every backend.
35+
as the first tactic, proving the `/gnn/rank` wire format is consumed correctly
36+
by every backend.
4037

41-
Expected output: `test result: ok. 46 passed; 0 failed`.
38+
Expected output: `test result: ok. N passed; 0 failed` where N is
39+
`1 (health) + |gnn_augment_tactics call-sites|`. The exact count tracks the
40+
test functions in `tests/gnn_augment_integration.rs`; do not hardcode it in
41+
new docs.
4242

4343
---
4444

src/rust/provers/abc.rs

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -740,8 +740,8 @@ impl ProverBackend for AbcBackend {
740740
self.to_abc_script(state)
741741
}
742742

743-
async fn suggest_tactics(&self, _state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
744-
let mut tactics = vec![
743+
async fn suggest_tactics(&self, state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
744+
let tactics = vec![
745745
// PDR (IC3) — most commonly used complete verification engine
746746
Tactic::Custom {
747747
prover: "abc".to_string(),
@@ -780,8 +780,7 @@ impl ProverBackend for AbcBackend {
780780
},
781781
];
782782

783-
tactics.truncate(limit);
784-
Ok(tactics)
783+
Ok(crate::provers::gnn_augment_tactics(&self.config, state, "abc", tactics, limit).await)
785784
}
786785

787786
async fn search_theorems(&self, _pattern: &str) -> Result<Vec<String>> {

src/rust/provers/acl2s.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ impl ProverBackend for Acl2sBackend {
125125
.unwrap_or_default())
126126
}
127127

128-
async fn suggest_tactics(&self, _state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
128+
async fn suggest_tactics(&self, state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
129129
// ACL2s (ACL2 Sedan) is a pedagogical IDE for ACL2. Its proof strategy
130130
// hints are ACL2 :hints keyword arguments and defthm patterns.
131131
let tactics = vec![
@@ -151,7 +151,7 @@ impl ProverBackend for Acl2sBackend {
151151
},
152152
Tactic::Simplify,
153153
];
154-
Ok(tactics.into_iter().take(limit).collect())
154+
Ok(crate::provers::gnn_augment_tactics(&self.config, state, "acl2s", tactics, limit).await)
155155
}
156156

157157
async fn search_theorems(&self, _pattern: &str) -> Result<Vec<String>> {

src/rust/provers/alloy.rs

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -362,8 +362,8 @@ impl ProverBackend for AlloyBackend {
362362
self.to_alloy(state)
363363
}
364364

365-
async fn suggest_tactics(&self, _state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
366-
let mut tactics = vec![
365+
async fn suggest_tactics(&self, state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
366+
let tactics = vec![
367367
Tactic::Custom {
368368
prover: "alloy".to_string(),
369369
command: "add_assertion".to_string(),
@@ -381,8 +381,7 @@ impl ProverBackend for AlloyBackend {
381381
},
382382
];
383383

384-
tactics.truncate(limit);
385-
Ok(tactics)
384+
Ok(crate::provers::gnn_augment_tactics(&self.config, state, "alloy", tactics, limit).await)
386385
}
387386

388387
async fn search_theorems(&self, _pattern: &str) -> Result<Vec<String>> {

src/rust/provers/arend.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,7 @@ impl ProverBackend for ArendBackend {
109109
.map(String::from)
110110
.unwrap_or_default())
111111
}
112-
async fn suggest_tactics(&self, _state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
112+
async fn suggest_tactics(&self, state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
113113
// Arend is a proof assistant based on Homotopy Type Theory (HoTT).
114114
// It uses term-mode proofs with the following common combinators.
115115
let tactics = vec![
@@ -141,7 +141,7 @@ impl ProverBackend for ArendBackend {
141141
},
142142
Tactic::Assumption,
143143
];
144-
Ok(tactics.into_iter().take(limit).collect())
144+
Ok(crate::provers::gnn_augment_tactics(&self.config, state, "arend", tactics, limit).await)
145145
}
146146

147147
async fn search_theorems(&self, _pattern: &str) -> Result<Vec<String>> {

src/rust/provers/boogie.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ impl ProverBackend for BoogieBackend {
133133
.unwrap_or_default())
134134
}
135135

136-
async fn suggest_tactics(&self, _state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
136+
async fn suggest_tactics(&self, state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
137137
// Boogie is an intermediate verification language; it delegates to
138138
// an SMT solver (typically Z3). Suggestions are annotation and
139139
// specification hints rather than interactive tactics.
@@ -165,7 +165,7 @@ impl ProverBackend for BoogieBackend {
165165
args: vec!["/proverOpt:O:smt.qi.max_multi_patterns=1000".to_string()],
166166
},
167167
];
168-
Ok(tactics.into_iter().take(limit).collect())
168+
Ok(crate::provers::gnn_augment_tactics(&self.config, state, "boogie", tactics, limit).await)
169169
}
170170

171171
async fn search_theorems(&self, _pattern: &str) -> Result<Vec<String>> {

src/rust/provers/cadical.rs

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -399,8 +399,8 @@ impl ProverBackend for CaDiCaLBackend {
399399
Ok(Self::to_dimacs(num_vars, &clauses))
400400
}
401401

402-
async fn suggest_tactics(&self, _state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
403-
let mut tactics = vec![
402+
async fn suggest_tactics(&self, state: &ProofState, limit: usize) -> Result<Vec<Tactic>> {
403+
let tactics = vec![
404404
Tactic::Custom {
405405
prover: "cadical".to_string(),
406406
command: "add-clause".to_string(),
@@ -419,8 +419,7 @@ impl ProverBackend for CaDiCaLBackend {
419419
Tactic::Simplify,
420420
];
421421

422-
tactics.truncate(limit);
423-
Ok(tactics)
422+
Ok(crate::provers::gnn_augment_tactics(&self.config, state, "cadical", tactics, limit).await)
424423
}
425424

426425
async fn search_theorems(&self, _pattern: &str) -> Result<Vec<String>> {

src/rust/provers/coq.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1057,7 +1057,7 @@ impl ProverBackend for CoqBackend {
10571057
args: vec![],
10581058
});
10591059

1060-
Ok(suggestions.into_iter().take(limit).collect())
1060+
Ok(crate::provers::gnn_augment_tactics(&self.config, state, "coq", suggestions, limit).await)
10611061
}
10621062

10631063
async fn search_theorems(&self, pattern: &str) -> Result<Vec<String>> {

0 commit comments

Comments
 (0)