Skip to content

Commit 2269ebb

Browse files
hyperpolymathclaude
andcommitted
fix(server): guard against empty-ProofState false positives in verify/prove
Partial fix for the parse+export round-trip bug documented in TEST-NEEDS.md: when the selected prover's parse_string returns an empty ProofState for unrecognised input, the subsequent verify_proof regenerated an empty prover source file that the real backend accepted vacuously — producing valid=true for garbage input to Coq/Lean/Agda. Added `is_empty_state` helper that checks goals, theorems, definitions, axioms, and variables. If all are empty AND the request content is non-empty, both handlers now return valid=false / success=false with a clear message instead of round-tripping. Verified live: garbage @#$ to Coq → valid=false (was valid=true) empty to Coq → valid=true (legitimately vacuous) 'Theorem t : True. ...' Coq → valid=true (unchanged — real proof) unsat SMT to Z3 → valid=false (real Z3 result) The deeper fix (making each prover preserve original content) still outstanding — this guard catches the worst false-positive class and makes /api/verify usable for training-signal collection without bypassing it via the batch_driver's direct-binary approach. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 30b91c4 commit 2269ebb

2 files changed

Lines changed: 45 additions & 0 deletions

File tree

TEST-NEEDS.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,16 @@ Idris2 has the mirror-image bug (false-negative): `parse_string` parses
242242
- Add regression tests: `verify` on garbage, empty, cross-prover content
243243
must all return `valid: false`.
244244

245+
**Partial fix (2026-04-05):** server-level guard in `prove_handler` and
246+
`verify_handler` now returns `valid: false` when `parse_string` produces
247+
an empty `ProofState` (no goals, theorems, definitions, axioms, or variables)
248+
on non-empty input. Verified live:
249+
- garbage to Coq/Lean: `valid: false` (was `true`)
250+
- cross-prover (SMT to Coq): `valid: false`
251+
- real Coq/Z3 proofs: still work correctly
252+
The deeper fix — making each prover backend preserve original content
253+
instead of round-tripping through ProofState — remains outstanding.
254+
245255
## Priority
246256
- **CRG C COMPLETE** as of 2026-04-04
247257
- Next priority: Fix 5 shell issues found by validation

src/rust/server.rs

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,19 @@ async fn prove_handler(Json(req): Json<ProveRequest>) -> Result<Json<ProveRespon
238238
.await
239239
.map_err(|e| AppError::ParseError(e.to_string()))?;
240240

241+
// Fail-fast on empty parse results (fixes false-positive: unrecognised
242+
// content produced an empty ProofState that re-exported to an empty file
243+
// which the backend prover then happily accepted). Require at least one
244+
// goal, theorem, definition, or axiom before we claim "parse succeeded".
245+
if is_empty_state(&state) && !req.content.trim().is_empty() {
246+
return Ok(Json(ProveResponse {
247+
success: false,
248+
goals: 0,
249+
message: "Parse produced no goals, theorems, definitions, or axioms — \
250+
content not recognised by the selected prover backend".to_string(),
251+
}));
252+
}
253+
241254
// Verify proof
242255
let valid = prover
243256
.verify_proof(&state)
@@ -270,6 +283,15 @@ async fn verify_handler(Json(req): Json<VerifyRequest>) -> Result<Json<VerifyRes
270283
.await
271284
.map_err(|e| AppError::ParseError(e.to_string()))?;
272285

286+
// Fail-fast on empty parse results (see prove_handler comment).
287+
if is_empty_state(&state) && !req.content.trim().is_empty() {
288+
return Ok(Json(VerifyResponse {
289+
valid: false,
290+
goals_remaining: 0,
291+
tactics_used: 0,
292+
}));
293+
}
294+
273295
// Verify
274296
let valid = prover
275297
.verify_proof(&state)
@@ -283,6 +305,19 @@ async fn verify_handler(Json(req): Json<VerifyRequest>) -> Result<Json<VerifyRes
283305
}))
284306
}
285307

308+
/// Return true if the parsed ProofState contains no meaningful structure.
309+
/// Used to detect the parse+export round-trip bug: a prover backend's
310+
/// `parse_string` returns an empty state on unrecognised input, then
311+
/// `verify_proof` regenerates an empty file which the real backend binary
312+
/// then accepts vacuously (false positive).
313+
fn is_empty_state(state: &echidna::core::ProofState) -> bool {
314+
state.goals.is_empty()
315+
&& state.context.theorems.is_empty()
316+
&& state.context.axioms.is_empty()
317+
&& state.context.definitions.is_empty()
318+
&& state.context.variables.is_empty()
319+
}
320+
286321
/// Get tactic suggestions
287322
async fn suggest_handler(
288323
State(state): State<AppState>,

0 commit comments

Comments
 (0)