Skip to content

Commit 58e2e91

Browse files
ArghDA M6 follow-on: Lean #print axioms audit — honest Unknown→Proven (#42)
Second refinement follow-on. A clean Lean elaboration is only `Unknown` on its own (`sorry` is a mere warning; `native_decide`'s axiom is invisible to a bare run). `check_file` now runs a `#print axioms` audit on a clean elaboration and promotes the verdict **honestly**. ## Mechanism (std-only — no new dependency) - **`decl_names`** parses auditable declarations (`theorem`/`lemma`/`def`/`abbrev`/`instance`), skipping leading `@[..]` attributes + modifiers and anonymous decls. Conservative: a missed name just isn't audited; a mis-parsed name makes `lean` error and the audit falls back to `Unknown` — **never a wrong verdict**. - **`axiom_audit`** copies the source into a fresh temp dir, appends `#print axioms <name>` per decl, runs `lean`, and — only if that copy elaborated cleanly — classifies the output. Temp dir removed after. - **`classify_axioms`** (ground-truthed vs Lean 4.13.0): all decls clean or only `{propext, Classical.choice, Quot.sound}` → **Proven**; any `sorryAx` → **Admitted**; any other axiom (e.g. `native_decide`'s) → **Postulated**. ## Honest reach Import-free files (the temp-dir copy elaborates without `LEAN_PATH`). Imported files can't be audited in a bare temp dir → the audit returns `None` → they stay `Unknown` until lake-env resolution (the remaining M6 follow-on). ## Verified (ran, not inferred) - `cargo test` → **129 passing, 0 failed** (+2 unit: `decl_names`, `classify_axioms`; the pre-existing lean check_file test updated to be audit-aware). - clippy -D warnings, fmt, SPDX clean. - **Dogfooded real lean:** `rfl` → `proven-eligible`; `Classical.em` → `proven-eligible` (standard axioms); `native_decide` → **`postulated`** (the audit catches the non-standard axiom a bare elaboration hides); `sorry` → `admitted`. ## Status M6 now ~92% (remaining: lake-env for imported files). Next: M3 (workspace-fed verdicts + staleness in `reason`). --- _Generated by [Claude Code](https://claude.ai/code/session_012MpYSh6Wy8YMBH2E3qVyT7)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4f4a9f5 commit 58e2e91

2 files changed

Lines changed: 218 additions & 13 deletions

File tree

.machine_readable/6a2/STATE.a2ml

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ milestones = [
4646
{ name = "M3: Flying-Logic reasoning graph (src/reason: And|Or juncts, demote-only cycle-safe Verdict propagation, additive studio JSON)", completion = 100 },
4747
{ name = "M4: Idris2 adapter (core/ABI language: .ipkg/--check/totality; escape-hatches)", completion = 95 },
4848
{ name = "M5: SMT solver backends (Z3, CVC5): SMT-LIB2 -> sat/unsat/unknown -> Verdict", completion = 100 },
49-
{ name = "M6: Lean4 adapter (lake build + #print axioms audit)", completion = 85 },
49+
{ name = "M6: Lean4 adapter (lake build + #print axioms audit)", completion = 92 },
5050
{ name = "M7: Echidna dispatch seam (optional route to orchestrator :8090, same Outcome)", completion = 70 },
5151
{ name = "M8: Coq/Rocq adapter (--heavy provisioning; Section-aware postulate classifier)", completion = 0 },
5252
{ name = "M9: Isabelle adapter (--heavy; dogfood tropical-resource-typing .thy)", completion = 0 },
@@ -75,6 +75,22 @@ milestones = [
7575
# has recently been set up". Permanent — future PRs satisfy the gate
7676
# automatically. NB the template is fine; only extraction-origin repos drift.
7777
#
78+
# M6 FOLLOW-ON landed 2026-07-01: the Lean `#print axioms` audit — the honest
79+
# Unknown→Proven promotion. On a clean elaboration (base verdict Unknown),
80+
# check_file now runs an audit: parse decl names (theorem/lemma/def/abbrev/
81+
# instance, skipping attrs/modifiers/anonymous), copy the source to a fresh
82+
# TEMP DIR (std-only, no new dep; cleaned up) + append `#print axioms <name>`
83+
# per decl, run `lean`, and classify (ground-truthed vs Lean 4.13.0):
84+
# all decls "does not depend on any axioms" or only {propext, Classical.choice,
85+
# Quot.sound} → Proven; any sorryAx → Admitted; any OTHER axiom (e.g.
86+
# native_decide's) → Postulated. Audit failure (imports unresolvable in the
87+
# temp dir, no decls, lean absent) → None → stays Unknown (honest, never
88+
# wrong). REACH: import-free files (temp-dir copy elaborates without
89+
# LEAN_PATH); imported files need lake-env (still the remaining M6 follow-on).
90+
# 129 tests (+2 unit: decl_names, classify_axioms). Dogfooded REAL lean:
91+
# rfl→proven-eligible, Classical→proven-eligible, native_decide→POSTULATED
92+
# (the audit catches what bare elaboration hides), sorry→admitted. M6 now 92%.
93+
#
7894
# M4 FOLLOW-ON landed 2026-07-01: Idris2 `.ipkg`-declared roots. discover_roots
7995
# now unions (a) `.ipkg` `main = <Module>` resolved to its .idr (honouring an
8096
# optional quoted `sourcedir`; `--` comments stripped; first existing candidate
@@ -242,7 +258,7 @@ milestones = [
242258
actions = [
243259
"ENGINE-SPINE DELIVERABLE COMPLETE: M0-M7 + M11 all landed. v0.1 spec closed. Six backends (agda, agda-cubical, idris2, lean4 Assistant; z3, cvc5 Solver) + reasoning graph + doctor + groove + dispatch seam. Remaining work is refinement + the heavy tail:",
244260
"M7 follow-on: the real Echidna orchestrator HTTP client (feature-gated), once Echidna's API is confirmed — only Dispatch::run changes.",
245-
"M6 follow-on: lake-env/LEAN_PATH multi-file resolution; per-decl #print axioms audit to promote Unknown→Proven.",
261+
"M6 follow-on: #print axioms audit DONE (import-free files promote honestly); remaining = lake-env/LEAN_PATH so imported files can be checked + audited too.",
246262
"M4 follow-on: .ipkg-declared roots DONE; remaining = totality-hole (?name) + per-def `partial` lint.",
247263
"M3 follow-on: feed real verdicts into `reason` from a workspace `proven/` state + wire staleness from the content-hash closure (proven.rs).",
248264
"M8-M10 heavy tail (Coq/Rocq, Isabelle, Mizar) — gated on --heavy provisioning (Coq via opam ~1hr; Isabelle ~4GB; Mizar niche).",
@@ -254,8 +270,8 @@ last-result = "pass" # unknown | pass | warn | fail
254270
open-warnings = 0
255271
open-failures = 0
256272
# `just check` (fmt-check + clippy -D warnings + build + test + SPDX) green
257-
# post-M4-followon: 93 lib + 34 integration = 127 tests pass. codeql.yml added
258-
# (code-scanning gate fix). doctor/groove/dispatch
273+
# post-M6-followon: 95 lib + 34 integration = 129 tests pass. codeql.yml added
274+
# (code-scanning gate fix). Lean #print axioms audit dogfooded. doctor/groove/dispatch
259275
# dogfooded: 6/6 backends runnable; groove writes /.well-known/groove; --dispatch
260276
# local real, echidna honest-stub. All six backends dogfooded via real CLI: agda
261277
# + agda-cubical + idris2 + lean4 (Assistant), z3 + cvc5 (Solver). Dogfooded

src/prover/lean.rs

Lines changed: 198 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,19 @@
1010
//! * exit non-zero → [`Verdict::Error`] (elaboration failed).
1111
//! * exit 0, output mentions `sorry` → [`Verdict::Admitted`] (a hole rode
1212
//! along on a green elaboration).
13-
//! * exit 0, clean → [`Verdict::Unknown`] — the file elaborates, but without
14-
//! a `#print axioms` audit arghda will NOT claim it proven (it could still
15-
//! use `native_decide`, `sorryAx`, or other axioms). Promoting `Unknown` →
16-
//! `Proven` via a per-declaration `#print axioms` audit is the follow-on.
13+
//! * exit 0, clean → run the **`#print axioms` audit** and promote honestly:
14+
//! `Proven` if every declaration depends only on the standard axioms
15+
//! (`propext`, `Classical.choice`, `Quot.sound`); `Postulated` if a
16+
//! non-standard axiom sneaks in (e.g. the one `native_decide` introduces —
17+
//! which a bare elaboration would NOT reveal); `Admitted` on `sorryAx`. If
18+
//! the audit can't run (no declarations, `lean` absent, or the file imports
19+
//! modules the bare temp-dir copy can't resolve), it stays `Verdict::Unknown`.
1720
//! * binary absent → [`Verdict::Unavailable`].
1821
//!
22+
//! The audit currently reaches import-free files (a temp-dir copy elaborates
23+
//! them without `LEAN_PATH`); auditing imported files needs `lake env`
24+
//! resolution — a documented follow-on.
25+
//!
1926
//! Lean modules are dotted (`Mathlib.Data.Nat` ↔ `Mathlib/Data/Nat.lean`),
2027
//! so [`crate::graph::module_name_of`] is reused. Imports are top-level
2128
//! `import Mod` lines; Lean `open` is a *namespace* directive, not a
@@ -30,8 +37,14 @@ use anyhow::{Context, Result};
3037
use std::fs;
3138
use std::path::{Path, PathBuf};
3239
use std::process::Command;
40+
use std::time::{SystemTime, UNIX_EPOCH};
3341
use walkdir::WalkDir;
3442

43+
/// Lean's standard, trusted axioms — a proof depending only on these is
44+
/// sound (this is the mathlib convention). Anything else (sorryAx, or the
45+
/// axioms `native_decide` introduces) is not.
46+
const STANDARD_AXIOMS: &[&str] = &["propext", "Classical.choice", "Quot.sound"];
47+
3548
const TAIL_LINES: usize = 40;
3649

3750
/// The Lean4 theorem prover.
@@ -63,7 +76,18 @@ impl Backend for Lean {
6376
Ok(out) => {
6477
let mut combined = String::from_utf8_lossy(&out.stdout).into_owned();
6578
combined.push_str(&String::from_utf8_lossy(&out.stderr));
66-
let verdict = lean_verdict(&combined, out.status.success());
79+
let mut verdict = lean_verdict(&combined, out.status.success());
80+
// A clean elaboration is only `Unknown` on its own — run the
81+
// `#print axioms` audit to promote it honestly: Proven if every
82+
// declaration depends only on the standard axioms, Postulated
83+
// if a non-standard axiom (e.g. `native_decide`'s) sneaks in,
84+
// Admitted on sorryAx. If the audit can't run (imports need
85+
// lake, no declarations, tool absent) it stays `Unknown`.
86+
if verdict == Verdict::Unknown {
87+
if let Some(audited) = axiom_audit(file) {
88+
verdict = audited;
89+
}
90+
}
6791
Ok(Outcome {
6892
available: true,
6993
exit_code: out.status.code(),
@@ -170,6 +194,124 @@ fn tail(s: &str, n: usize) -> String {
170194
lines[start..].join("\n")
171195
}
172196

197+
/// The declaration names in a Lean source that can depend on axioms
198+
/// (`theorem`/`lemma`/`def`/`abbrev`/`instance`), skipping leading
199+
/// attributes/modifiers and anonymous decls. Deliberately conservative: a
200+
/// missed name just isn't audited; a mis-parsed name makes `lean` error and
201+
/// the audit falls back to `Unknown` — it never yields a wrong verdict.
202+
fn decl_names(src: &str) -> Vec<String> {
203+
const KEYWORDS: &[&str] = &["theorem", "lemma", "def", "abbrev", "instance"];
204+
const MODIFIERS: &[&str] = &[
205+
"private",
206+
"protected",
207+
"noncomputable",
208+
"partial",
209+
"unsafe",
210+
"scoped",
211+
"local",
212+
"mutual",
213+
];
214+
let mut names = Vec::new();
215+
for line in src.lines() {
216+
let tokens: Vec<&str> = line.split_whitespace().collect();
217+
let mut i = 0;
218+
// Skip leading `@[..]` attributes and declaration modifiers.
219+
while let Some(tk) = tokens.get(i) {
220+
if tk.starts_with("@[") || MODIFIERS.contains(tk) {
221+
i += 1;
222+
} else {
223+
break;
224+
}
225+
}
226+
if tokens.get(i).is_some_and(|tk| KEYWORDS.contains(tk)) {
227+
if let Some(name_tok) = tokens.get(i + 1) {
228+
let name: String = name_tok
229+
.chars()
230+
.take_while(|c| c.is_alphanumeric() || *c == '_' || *c == '.' || *c == '\'')
231+
.collect();
232+
if !name.is_empty() {
233+
names.push(name);
234+
}
235+
}
236+
}
237+
}
238+
names.dedup();
239+
names
240+
}
241+
242+
/// Classify the combined output of a batch of `#print axioms` commands:
243+
/// any `sorryAx` ⇒ Admitted; else any non-standard axiom ⇒ Postulated; else
244+
/// (all clean or standard-only) ⇒ Proven.
245+
fn classify_axioms(output: &str) -> Verdict {
246+
let mut saw_nonstandard = false;
247+
for line in output.lines() {
248+
let Some(open) = line.find("depends on axioms: [") else {
249+
continue; // "does not depend on any axioms" ⇒ clean, skip
250+
};
251+
let rest = &line[open + "depends on axioms: [".len()..];
252+
let list = rest.split(']').next().unwrap_or(rest);
253+
for ax in list.split(',') {
254+
let ax = ax.trim();
255+
if ax == "sorryAx" {
256+
return Verdict::Admitted;
257+
}
258+
if !ax.is_empty() && !STANDARD_AXIOMS.contains(&ax) {
259+
saw_nonstandard = true;
260+
}
261+
}
262+
}
263+
if saw_nonstandard {
264+
Verdict::Postulated
265+
} else {
266+
Verdict::Proven
267+
}
268+
}
269+
270+
/// Run a `#print axioms` audit on `file`'s declarations. Copies the source
271+
/// into a fresh temp dir, appends `#print axioms <name>` per declaration,
272+
/// and runs `lean`. Returns the classified verdict, or `None` when the audit
273+
/// can't be trusted — no declarations, `lean` absent, or the copy fails to
274+
/// elaborate (e.g. it imports modules that need `lake env`/`LEAN_PATH`, which
275+
/// a bare temp dir lacks). `None` ⇒ the caller keeps the honest `Unknown`.
276+
fn axiom_audit(file: &Path) -> Option<Verdict> {
277+
let src = fs::read_to_string(file).ok()?;
278+
let names = decl_names(&src);
279+
if names.is_empty() {
280+
return None;
281+
}
282+
283+
let nanos = SystemTime::now()
284+
.duration_since(UNIX_EPOCH)
285+
.map(|d| d.as_nanos())
286+
.unwrap_or(0);
287+
let dir = std::env::temp_dir().join(format!("arghda-audit-{}-{}", std::process::id(), nanos));
288+
fs::create_dir_all(&dir).ok()?;
289+
290+
let mut body = src;
291+
body.push('\n');
292+
for n in &names {
293+
body.push_str(&format!("#print axioms {n}\n"));
294+
}
295+
let audit_file = dir.join("Audit.lean");
296+
297+
let verdict = match fs::write(&audit_file, &body) {
298+
Ok(()) => match Command::new("lean").arg(&audit_file).output() {
299+
// Only trust the audit if the copy elaborated cleanly; otherwise
300+
// (imports unresolved in the temp dir, etc.) stay Unknown.
301+
Ok(out) if out.status.success() => {
302+
let mut combined = String::from_utf8_lossy(&out.stdout).into_owned();
303+
combined.push_str(&String::from_utf8_lossy(&out.stderr));
304+
Some(classify_axioms(&combined))
305+
}
306+
_ => None,
307+
},
308+
Err(_) => None,
309+
};
310+
311+
let _ = fs::remove_dir_all(&dir);
312+
verdict
313+
}
314+
173315
#[cfg(test)]
174316
mod tests {
175317
use super::*;
@@ -208,6 +350,52 @@ mod tests {
208350
assert!(!imports.iter().any(|i| i.contains("Ignored")));
209351
}
210352

353+
#[test]
354+
fn decl_names_parses_keywords_modifiers_attributes() {
355+
let src = "\
356+
@[simp] theorem foo : True := trivial\n\
357+
private def bar : Nat := 0\n\
358+
lemma baz.qux : 1 = 1 := rfl\n\
359+
noncomputable def qq : Nat := 0\n\
360+
example : True := trivial\n\
361+
-- theorem commented : ...\n\
362+
#eval 1\n";
363+
let names = decl_names(src);
364+
assert!(names.contains(&"foo".to_string()));
365+
assert!(names.contains(&"bar".to_string()));
366+
assert!(names.contains(&"baz.qux".to_string()), "dotted names kept");
367+
assert!(names.contains(&"qq".to_string()), "modifier skipped");
368+
// `example` is anonymous → not audited; `#eval` is not a decl.
369+
assert!(!names.iter().any(|n| n == "example" || n == "1"));
370+
}
371+
372+
#[test]
373+
fn classify_axioms_maps_the_ground_truthed_output() {
374+
// The three shapes ground-truthed against Lean 4.13.0.
375+
assert_eq!(
376+
classify_axioms("'t' does not depend on any axioms"),
377+
Verdict::Proven
378+
);
379+
assert_eq!(
380+
classify_axioms("'c' depends on axioms: [propext, Classical.choice, Quot.sound]"),
381+
Verdict::Proven,
382+
);
383+
assert_eq!(
384+
classify_axioms("'g' depends on axioms: [sorryAx]"),
385+
Verdict::Admitted
386+
);
387+
assert_eq!(
388+
classify_axioms("'n' depends on axioms: [Lean.ofReduceBool]"),
389+
Verdict::Postulated,
390+
"native_decide's axiom is non-standard ⇒ amber",
391+
);
392+
// Mixed: any sorryAx dominates.
393+
assert_eq!(
394+
classify_axioms("'a' depends on axioms: [propext]\n'b' depends on axioms: [sorryAx]"),
395+
Verdict::Admitted
396+
);
397+
}
398+
211399
#[test]
212400
fn verdict_is_honest_about_sorry_and_the_missing_audit() {
213401
// A green elaboration is only Unknown without a #print axioms audit;
@@ -227,13 +415,14 @@ mod tests {
227415
std::fs::write(&f, "theorem t : 1 = 1 := rfl\n").unwrap();
228416
let out = Lean.check_file(&f, tmp.path()).unwrap();
229417
if out.available {
230-
// Never fabricated Proven: a clean Lean file is Unknown (audit
231-
// absent), a sorry file Admitted, a broken file Error.
418+
// Honest verdict invariant: never fabricated. With the axioms
419+
// audit a clean `rfl` proof is promoted to Proven; `ok` iff
420+
// Proven; and the value is always one of the real states.
232421
assert!(matches!(
233422
out.verdict,
234-
Verdict::Unknown | Verdict::Admitted | Verdict::Error
423+
Verdict::Proven | Verdict::Unknown | Verdict::Admitted | Verdict::Error
235424
));
236-
assert!(!out.ok, "lean never reports `ok` without an axioms audit");
425+
assert_eq!(out.ok, out.verdict == Verdict::Proven);
237426
} else {
238427
assert_eq!(out.verdict, Verdict::Unavailable);
239428
}

0 commit comments

Comments
 (0)