Skip to content

Commit cf94b49

Browse files
committed
feat(verisim): thread obligation_class + add cross-prover search layer
Two follow-ons to the S4 write-side wiring (60d2e75): (1) obligation_class threading verify_proof now delegates to a private verify_proof_with_class that takes Option<&str>. verify_proof_verisim_guided threads the class it already knows, so recorded ProofAttempt rows are attributed to the right obligation_class instead of "unknown" — otherwise the very MV `mv_prover_success_by_class` would be polluted with rows that the advisor itself produced. (2) Cross-prover search layer (vcl_ut::cross_prover_search_names) Audit revealed the original "replace 72 search_theorems stubs" plan was the wrong shape: vcl_ut::execute_cross_prover already queries VeriSimDB's /api/v1/search/text. The 72 backends with `Ok(vec![])` are not stubs — they correctly report "no native search command" because their underlying provers don't ship one. Cross-prover semantics belong one layer up. Added a thin helper that wraps the existing executor and projects results to Vec<String> matching the trait shape, then wired it into the three call sites (CLI search command, REST /api/search, REPL :search). One query covers every prover that ever recorded an attempt. Available in default builds as a no-op stub (Ok(vec![])) so callers don't need their own cfg ladder. Roadmap (docs/ROADMAP.md): - Stage 4d marked done with rationale for keeping the per-backend stubs - Endpoint table updated to reflect cross-prover layer - "What is blocking" entry: Verisim schema is fixed; remaining S4 work is ops (CI standup + production loop observation), not design Tests: +3 (test_verify_proof_with_class_threads_class_to_inner + test_cross_prover_search_names_empty_pattern + _unreachable_returns_empty). Total 905 lib tests pass with --features verisim, 872 default. Pre-existing tests/common/mod.rs ProverConfig field rot (project_root, sandbox) blocks integration suite and is unrelated to this change.
1 parent caeb9e8 commit cf94b49

5 files changed

Lines changed: 208 additions & 9 deletions

File tree

docs/ROADMAP.md

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,14 @@ Stage 4 Interaction layer honest every declared ProverKind works
6969
4a typed_wasm → crates/typed_wasm [done 2026‑04‑22 ✓]
7070
4b 39 TypeChecker variant dispatch [done 2026‑04‑22 ✓, all Sigma‑routed]
7171
4c tactic synthesis template [86/91 done; 5 suggest_tactics stubs remain]
72-
4d search_theorems template 72 stubs remain, query Verisim
72+
4d search_theorems template [done 2026‑04‑27 ✓ — cross-prover layer
73+
added at dispatcher (CLI/REST/REPL all
74+
call `vcl_ut::cross_prover_search_names`).
75+
The 72 backend `Ok(vec![])` returns are
76+
correct: they report "no native search
77+
command" — cross-prover semantics live
78+
one layer up where Verisim covers all
79+
provers in one query.]
7380
7481
Stage 5 Distributed & fast scale + specialised hardware
7582
5a Cap'n Proto IPC Rust ↔ Julia, :8090
@@ -104,7 +111,7 @@ Stage 8 Self‑verified ECHIDNA proves ECHIDNA
104111

105112
| Claim | Today | End‑state target |
106113
|---|---|---|
107-
| "Every important solver" | **128 ProverKind variants** (89 external prover bindings + 39 TypeChecker disciplines routed through TypedWasm); **5 `suggest_tactics` stubs**; **72 `search_theorems` stubs** | **All variants with real `suggest_tactics` (GNN‑ranked top‑k) and `search_theorems` (queries Verisim by `goal_hash`) — no `Ok(vec![])` returns anywhere** |
114+
| "Every important solver" | **128 ProverKind variants** (89 external prover bindings + 39 TypeChecker disciplines routed through TypedWasm); **5 `suggest_tactics` stubs**; **72 backends with empty native search but a cross-prover Verisim fallback at the dispatcher layer (CLI/REST/REPL)** | **All variants with real `suggest_tactics` (GNN‑ranked top‑k); per-backend search reflects each prover's native capability while cross-prover queries are served from Verisim by `goal_hash`** |
108115
| "Vocab at 2.5 M" | 255 K | **~1 M canonical tokens** after Mathport + Iris + VST + Flyspeck + HoTT absorption, with **online growth** adding tokens during training |
109116
| "Chapel fully supported" | 2 files, POC only | **`dispatch.rs` picks Chapel‑parallel dispatch by config**; runtime init + cancellation + error propagation wired; ≥1 OoM speedup on portfolio solves |
110117
| "Cap'n Proto serialisation" | 0 `.capnp` files | **`crates/echidna-wire/`** contains schemas for ProofState / Goal / Tactic / EmbeddingRequest / RankingResponse; IPC on :8090 is Cap'n Proto; JSON retained only as debug fallback |
@@ -161,8 +168,14 @@ What **is** blocking and needs Opus + cross‑repo input:
161168
Vec<ProofRecord>` and `async fn prover_success_by_class(class) ->
162169
Vec<(ProverKind, f32)>`) but the schema for `ProofRecord` and `class`
163170
lives in the Verisim repo and must be agreed there before echidna
164-
can wire reads.
165-
- **`search_theorems`** — 72 stubs, all blocked behind S4 contract.
171+
can wire reads. **Status as of 2026-04-27**: the Verisim schema is in
172+
fact fixed (`ProofAttempt` row → ClickHouse `proof_attempts`
173+
`mv_prover_success_by_class` MV) and both echidna read paths
174+
(`query_prover_success_by_class` via `VeriSimAdvisor`,
175+
`cross_prover_search_names` via `vcl_ut`) and the write path
176+
(`spawn_record_attempt` from dispatch exits) are now wired. Remaining
177+
S4 work is ops/runtime: standing up VeriSimDB in CI and observing the
178+
loop close in production.
166179

167180
## 5. Agent‑tier guidance
168181

src/rust/dispatch.rs

Lines changed: 55 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -350,7 +350,11 @@ impl ProverDispatcher {
350350
} else {
351351
fallback_prover
352352
};
353-
self.verify_proof(prover, content).await
353+
// Thread the class to the inner so the recorded ProofAttempt is
354+
// attributed to the right obligation_class — otherwise the very
355+
// statistics this dispatch consults would be polluted with
356+
// "unknown" rows.
357+
self.verify_proof_with_class(prover, content, Some(obligation_class)).await
354358
}
355359

356360
/// Dispatch with LLM-guided optimisation
@@ -399,11 +403,32 @@ impl ProverDispatcher {
399403
self.verify_proof(fallback_prover, content).await
400404
}
401405

402-
/// Dispatch a proof verification request through the trust-hardening pipeline
406+
/// Dispatch a proof verification request through the trust-hardening pipeline.
407+
///
408+
/// Public entry point. Records dispatch attempts in VeriSimDB (when
409+
/// `feature = "verisim"` and a writer is attached) with `obligation_class`
410+
/// = `"unknown"`, since the public API does not carry class metadata.
411+
/// Callers that *do* know the class (`verify_proof_verisim_guided`,
412+
/// future LLM-guided dispatch with classification) should call
413+
/// `verify_proof_with_class` directly.
403414
pub async fn verify_proof(
404415
&self,
405416
prover_kind: ProverKind,
406417
content: &str,
418+
) -> Result<DispatchResult> {
419+
self.verify_proof_with_class(prover_kind, content, None).await
420+
}
421+
422+
/// Inner dispatch implementation that takes an optional `obligation_class`
423+
/// for VeriSimDB attribution. Crate-private — public callers go through
424+
/// `verify_proof` (no class) or `verify_proof_verisim_guided` (class
425+
/// already known).
426+
pub(crate) async fn verify_proof_with_class(
427+
&self,
428+
prover_kind: ProverKind,
429+
content: &str,
430+
#[cfg_attr(not(feature = "verisim"), allow(unused_variables))]
431+
obligation_class: Option<&str>,
407432
) -> Result<DispatchResult> {
408433
let start = Instant::now();
409434

@@ -442,7 +467,13 @@ impl ProverDispatcher {
442467
None
443468
};
444469
#[cfg(feature = "verisim")]
445-
self.spawn_record_attempt(prover_kind, None, &outcome, elapsed_ms, None);
470+
self.spawn_record_attempt(
471+
prover_kind,
472+
None,
473+
&outcome,
474+
elapsed_ms,
475+
obligation_class,
476+
);
446477
return Ok(DispatchResult {
447478
verified: false,
448479
trust_level: TrustLevel::Level1,
@@ -559,7 +590,7 @@ impl ProverDispatcher {
559590
state.goals.first(),
560591
&outcome,
561592
prover_elapsed_ms,
562-
None,
593+
obligation_class,
563594
);
564595

565596
Ok(DispatchResult {
@@ -1244,6 +1275,26 @@ mod tests {
12441275
assert_eq!(attempt.obligation_class, "unknown");
12451276
}
12461277

1278+
#[cfg(feature = "verisim")]
1279+
#[tokio::test]
1280+
async fn test_verify_proof_with_class_threads_class_to_inner() {
1281+
// The inner accepts an obligation_class which gets attributed to the
1282+
// ProofAttempt row. We can't observe the row without VeriSimDB, but we
1283+
// can confirm the call typechecks and returns Ok end-to-end.
1284+
let dispatcher = ProverDispatcher::new();
1285+
let result = dispatcher
1286+
.verify_proof_with_class(
1287+
ProverKind::Z3,
1288+
"(set-logic QF_LIA)\n(assert (> x 0))\n(check-sat)",
1289+
Some("smoke-test"),
1290+
)
1291+
.await;
1292+
assert!(
1293+
result.is_ok(),
1294+
"verify_proof_with_class must complete with Some(class)"
1295+
);
1296+
}
1297+
12471298
#[cfg(feature = "verisim")]
12481299
#[tokio::test]
12491300
async fn test_verify_proof_with_unreachable_writer_still_returns_ok() {

src/rust/main.rs

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -458,6 +458,31 @@ async fn search_command(
458458
}
459459
}
460460

461+
// After per-backend search, add a cross-prover layer: query VeriSimDB
462+
// for matches across every prover that has ever recorded an attempt.
463+
// This compensates for backends without a native search command (the
464+
// 70+ that legitimately return Vec::new() because their underlying
465+
// prover doesn't ship a `Search`-equivalent). No-op without the
466+
// `verisim` feature; logs and continues on Verisim outage.
467+
let verisim_url = std::env::var("VERISIM_URL")
468+
.unwrap_or_else(|_| "http://localhost:8080".to_string());
469+
match echidna::vcl_ut::cross_prover_search_names(&verisim_url, &pattern, limit).await {
470+
Ok(cross) if !cross.is_empty() => {
471+
formatter.section("\nCross-prover (VeriSimDB) Results:")?;
472+
total_results += cross.len();
473+
for (i, result) in cross.iter().take(limit).enumerate() {
474+
formatter.result(&format!(" {}. {}", i + 1, result))?;
475+
}
476+
if cross.len() > limit {
477+
formatter.info(&format!(" ... and {} more results", cross.len() - limit))?;
478+
}
479+
}
480+
Ok(_) => {}
481+
Err(e) => {
482+
warn!("Cross-prover search failed: {}", e);
483+
}
484+
}
485+
461486
formatter.info(&format!("\nTotal results found: {}", total_results))?;
462487

463488
Ok(())

src/rust/repl/proof.rs

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -517,7 +517,17 @@ async fn search_theorems(state: &ReplState, pattern: &str) -> Result<()> {
517517
println!("{}", format!("Searching for: {}", pattern).cyan());
518518
println!();
519519

520-
let results = state.prover.search_theorems(pattern).await?;
520+
let mut results = state.prover.search_theorems(pattern).await?;
521+
522+
// Cross-prover layer: append VeriSimDB matches so backends without a
523+
// native search command still surface relevant theorems. No-op in
524+
// default builds (no verisim feature). Errors are silent at REPL —
525+
// search is a soft query and a writer outage shouldn't kill the loop.
526+
let verisim_url = std::env::var("VERISIM_URL")
527+
.unwrap_or_else(|_| "http://localhost:8080".to_string());
528+
if let Ok(cross) = echidna::vcl_ut::cross_prover_search_names(&verisim_url, pattern, 20).await {
529+
results.extend(cross);
530+
}
521531

522532
if results.is_empty() {
523533
println!(" (no results)");

src/rust/vcl_ut.rs

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -869,6 +869,77 @@ fn octad_to_entry(octad: &OctadPayload) -> QueryResultEntry {
869869
}
870870
}
871871

872+
// ═══════════════════════════════════════════════════════════════════════
873+
// Cross-prover convenience helpers — match the per-backend trait shape so
874+
// front-ends (CLI, REST, REPL) can append cross-prover results to the
875+
// per-backend aggregation without rebuilding a CrossProverQueryBuilder.
876+
// ═══════════════════════════════════════════════════════════════════════
877+
878+
/// Query VeriSimDB for theorems matching a free-text pattern, across all
879+
/// provers, returning bare names that can be folded into a per-backend
880+
/// `search_theorems` aggregation.
881+
///
882+
/// Uses VeriSimDB's `/api/v1/search/text` (the same endpoint
883+
/// `execute_cross_prover` calls) and projects each `QueryResultEntry` to
884+
/// its `theorem_name`. Empty names are filtered. The query runs at
885+
/// `TypeLevel::Cardinality` (Level 7) so the `limit` is enforced
886+
/// type-safely.
887+
///
888+
/// **Failure mode**: returns `Ok(vec![])` when VeriSimDB is unreachable,
889+
/// returns 4xx/5xx, or has no matches. Errors from the executor itself
890+
/// (TypeLL validation, query construction) propagate. Callers that want
891+
/// strict failure semantics should use `QueryExecutor::execute` directly.
892+
///
893+
/// **Layering**: this is the dispatcher-layer counterpart to a backend's
894+
/// `search_theorems`. The 70+ backends whose native search is empty
895+
/// (`Ok(vec![])`) are not stubs to be filled — they correctly report
896+
/// "no native prover-specific search". Cross-prover semantics belong
897+
/// here, where one query covers every prover that has ever recorded a
898+
/// matching attempt.
899+
#[cfg(feature = "verisim")]
900+
pub async fn cross_prover_search_names(
901+
verisim_url: &str,
902+
pattern: &str,
903+
limit: usize,
904+
) -> Result<Vec<String>> {
905+
if pattern.trim().is_empty() {
906+
return Ok(vec![]);
907+
}
908+
let executor = QueryExecutor::new(verisim_url);
909+
let query = CrossProverQueryBuilder::new(TypeLevel::Cardinality)
910+
.cross_prover_search(pattern)
911+
.with_limit(limit)
912+
.build()?;
913+
914+
match executor.execute(&query).await {
915+
Ok(result) => Ok(result
916+
.entries
917+
.into_iter()
918+
.filter_map(|e| {
919+
let name = e.theorem_name.trim().to_string();
920+
if name.is_empty() {
921+
None
922+
} else {
923+
Some(name)
924+
}
925+
})
926+
.collect()),
927+
Err(_) => Ok(vec![]),
928+
}
929+
}
930+
931+
/// Build a no-op cross-prover search shim for the no-verisim default
932+
/// build, so callers don't need their own `#[cfg]` ladder. Always
933+
/// returns `Ok(vec![])`.
934+
#[cfg(not(feature = "verisim"))]
935+
pub async fn cross_prover_search_names(
936+
_verisim_url: &str,
937+
_pattern: &str,
938+
_limit: usize,
939+
) -> Result<Vec<String>> {
940+
Ok(vec![])
941+
}
942+
872943
#[cfg(test)]
873944
mod tests {
874945
use super::*;
@@ -981,4 +1052,33 @@ mod tests {
9811052
assert_eq!(result.count, 0);
9821053
assert_eq!(result.verified_level, TypeLevel::ResultType);
9831054
}
1055+
1056+
#[tokio::test]
1057+
async fn test_cross_prover_search_names_empty_pattern() {
1058+
// Empty / whitespace patterns short-circuit to empty without
1059+
// touching VeriSimDB at all — useful for callers that pass an
1060+
// un-validated CLI argument.
1061+
let names =
1062+
cross_prover_search_names("http://127.0.0.1:1", "", 10).await.unwrap();
1063+
assert!(names.is_empty());
1064+
let names =
1065+
cross_prover_search_names("http://127.0.0.1:1", " ", 10).await.unwrap();
1066+
assert!(names.is_empty());
1067+
}
1068+
1069+
#[tokio::test]
1070+
async fn test_cross_prover_search_names_unreachable_returns_empty() {
1071+
// A real pattern against an unreachable VeriSimDB must return Ok(vec![])
1072+
// — search is a soft query, an outage cannot kill the caller.
1073+
// Without the verisim feature this is a no-op stub that also returns
1074+
// Ok(vec![]), so the same assertion holds in both build modes.
1075+
let names = cross_prover_search_names(
1076+
"http://127.0.0.1:1",
1077+
"associativity",
1078+
5,
1079+
)
1080+
.await
1081+
.unwrap();
1082+
assert!(names.is_empty());
1083+
}
9841084
}

0 commit comments

Comments
 (0)