Skip to content

Commit 6a1feb2

Browse files
aggregate: ingest proven-tests JSON run reports (new input kind) (#152)
## Context `proven-tests-and-benches` emits a JSON run report (schema_version 1) grading each test by three-tier provenance. This adds a self-contained `aggregate` input kind that parses that report and folds it in using the **honest** tier→verdict mapping fixed in that repo's `docs/INTEROP-PANIC-ATTACK.adoc`. ## Change New module `src/aggregate/proven_tests.rs` (registered from `src/aggregate/mod.rs`): - Serde structs for the schema-v1 report (`ProvenTestsReport`, `Entry`, `Summary`, `ProofStep`). - `ReportTier::to_verdict()` — the honest mapping: - **Actually-Proven → `Closed`** (carries a proof ladder; `actually_proven_ladder_files()` extracts the distinct cited `.idr` paths, which are the natural `aggregate --proof` inputs). - **Provisionally-Proven / Unproven → `None`** — no proof artifact, deliberately **not** read as `Holes`. Absence of a proof object is not the presence of a hole. - Rejects unknown `schema_version`; tolerates unknown tiers (surfaces them in `notes()` rather than panicking). The trust boundary is unchanged: verdicts are conditioned on the Idris checker's soundness; panic-attack does not re-check proofs. No CLI wiring in this change — the parser + mapper is the substantive, testable core; the ordinary `aggregate --proof <ladder-file>` path already handles the export direction (verified end-to-end from the proven-tests side). ## Verification ``` cargo test --lib aggregate::proven_tests # 5 passed ``` Tests cover: schema-v1 parse, honest tier mapping, distinct-ladder extraction, unknown-schema rejection, unknown-tier tolerance. Related: `hyperpolymath/proven-tests-and-benches#13`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01SW76f8XBgwsiGXU9x5vmQU --- _Generated by [Claude Code](https://claude.ai/code/session_01SW76f8XBgwsiGXU9x5vmQU)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4f4ed50 commit 6a1feb2

2 files changed

Lines changed: 248 additions & 0 deletions

File tree

src/aggregate/mod.rs

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,9 @@ use std::fs::{self, File};
5050
use std::io::Read;
5151
use std::path::{Path, PathBuf};
5252

53+
/// Ingest a `proven-tests` JSON run report as an aggregate input kind.
54+
pub mod proven_tests;
55+
5356
/// Upper bound on the text we scan for verdict / coverage classification.
5457
/// The *hash* always covers the whole file (see [`hash_file`]); this cap
5558
/// only bounds the in-memory text scan for markers.

src/aggregate/proven_tests.rs

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
// SPDX-License-Identifier: MPL-2.0
2+
// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
3+
4+
//! Ingest a `proven-tests` JSON run report (schema_version 1) as an aggregate
5+
//! input kind.
6+
//!
7+
//! `proven-tests-and-benches` grades each test by three-tier provenance
8+
//! (Actually-Proven / Provisionally-Proven / Unproven), where Actually-Proven
9+
//! carries a proof ladder citing machine-checked theorems in real `.idr` files.
10+
//! This module folds such a report into aggregate's vocabulary using the honest
11+
//! mapping fixed in that repo's `docs/INTEROP-PANIC-ATTACK.adoc`:
12+
//!
13+
//! * **Actually-Proven** — has a proof artifact (the ladder files). Expected
14+
//! to reconcile as [`ProofVerdict::Closed`]. Its cited `proof_file`s are the
15+
//! natural inputs to the ordinary `aggregate --proof` path.
16+
//! * **Provisionally-Proven** — type-safe framework + example/property
17+
//! evidence, but *no proof object*. Deliberately NOT treated as
18+
//! [`ProofVerdict::Holes`]: absence of a proof artifact is not the presence
19+
//! of a hole. Recorded as informational only.
20+
//! * **Unproven** — no artifact; informational only.
21+
//!
22+
//! The trust boundary is unchanged: panic-attack does not re-check the proofs;
23+
//! a `Closed` reading is conditioned on the Idris checker's soundness.
24+
25+
use super::ProofVerdict;
26+
use anyhow::{Context, Result};
27+
use serde::{Deserialize, Serialize};
28+
use std::collections::BTreeSet;
29+
use std::path::Path;
30+
31+
/// A cited proof step (only Actually-Proven entries carry these).
32+
#[derive(Debug, Clone, Serialize, Deserialize)]
33+
pub struct ProofStep {
34+
pub description: String,
35+
#[serde(default)]
36+
pub proof_file: Option<String>,
37+
#[serde(default)]
38+
pub theorem: Option<String>,
39+
}
40+
41+
/// One graded test entry in the run report.
42+
#[derive(Debug, Clone, Serialize, Deserialize)]
43+
pub struct Entry {
44+
pub test_id: String,
45+
#[serde(default)]
46+
pub category: Option<String>,
47+
#[serde(default)]
48+
pub aspect: Option<String>,
49+
/// "Actually-Proven" / "Provisionally-Proven" / "Unproven".
50+
pub provenance: String,
51+
#[serde(default)]
52+
pub proof_ladder: Vec<ProofStep>,
53+
}
54+
55+
/// The run-report summary block.
56+
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
57+
pub struct Summary {
58+
#[serde(default)]
59+
pub passed: usize,
60+
#[serde(default)]
61+
pub total: usize,
62+
#[serde(default)]
63+
pub actually_proven: usize,
64+
#[serde(default)]
65+
pub provisionally_proven: usize,
66+
#[serde(default)]
67+
pub unproven: usize,
68+
#[serde(default)]
69+
pub covered_cells: usize,
70+
#[serde(default)]
71+
pub cat_aspect_cells: usize,
72+
#[serde(default)]
73+
pub lattice_covered: usize,
74+
}
75+
76+
/// A parsed `proven-tests` run report.
77+
#[derive(Debug, Clone, Serialize, Deserialize)]
78+
pub struct ProvenTestsReport {
79+
pub schema_version: u32,
80+
#[serde(default)]
81+
pub suite: String,
82+
#[serde(default)]
83+
pub summary: Summary,
84+
#[serde(default)]
85+
pub entries: Vec<Entry>,
86+
}
87+
88+
/// The three provenance tiers, parsed from the report's string labels.
89+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
90+
pub enum ReportTier {
91+
ActuallyProven,
92+
ProvisionallyProven,
93+
Unproven,
94+
}
95+
96+
impl ReportTier {
97+
/// Parse the report's tier label (the exact `Show ProvenStatus` strings).
98+
pub fn parse(s: &str) -> Option<Self> {
99+
match s {
100+
"Actually-Proven" => Some(ReportTier::ActuallyProven),
101+
"Provisionally-Proven" => Some(ReportTier::ProvisionallyProven),
102+
"Unproven" => Some(ReportTier::Unproven),
103+
_ => None,
104+
}
105+
}
106+
107+
/// The honest aggregate reading of a tier. Only Actually-Proven maps to a
108+
/// proof verdict (`Closed`); the weaker tiers carry no artifact, so they
109+
/// map to `None` rather than being misread as `Holes`.
110+
pub fn to_verdict(self) -> Option<ProofVerdict> {
111+
match self {
112+
ReportTier::ActuallyProven => Some(ProofVerdict::Closed),
113+
ReportTier::ProvisionallyProven | ReportTier::Unproven => None,
114+
}
115+
}
116+
}
117+
118+
impl ProvenTestsReport {
119+
/// Parse a run report from JSON text.
120+
pub fn from_str(text: &str) -> Result<Self> {
121+
let report: ProvenTestsReport =
122+
serde_json::from_str(text).context("parsing proven-tests run report")?;
123+
if report.schema_version != 1 {
124+
anyhow::bail!(
125+
"unsupported proven-tests report schema_version {} (expected 1)",
126+
report.schema_version
127+
);
128+
}
129+
Ok(report)
130+
}
131+
132+
/// Parse a run report from a file path.
133+
pub fn from_path(path: &Path) -> Result<Self> {
134+
let text = std::fs::read_to_string(path)
135+
.with_context(|| format!("reading {}", path.display()))?;
136+
Self::from_str(&text)
137+
}
138+
139+
/// The distinct `.idr` proof-ladder files cited by Actually-Proven entries.
140+
/// These are the natural inputs to the ordinary `aggregate --proof` path.
141+
pub fn actually_proven_ladder_files(&self) -> Vec<String> {
142+
let mut files: BTreeSet<String> = BTreeSet::new();
143+
for entry in &self.entries {
144+
if ReportTier::parse(&entry.provenance) == Some(ReportTier::ActuallyProven) {
145+
for step in &entry.proof_ladder {
146+
if let Some(f) = &step.proof_file {
147+
files.insert(f.clone());
148+
}
149+
}
150+
}
151+
}
152+
files.into_iter().collect()
153+
}
154+
155+
/// A one-line, honest note per entry, applying the tier→verdict mapping.
156+
pub fn notes(&self) -> Vec<String> {
157+
self.entries
158+
.iter()
159+
.map(|e| match ReportTier::parse(&e.provenance) {
160+
Some(ReportTier::ActuallyProven) => format!(
161+
"{}: Actually-Proven → Closed ({} ladder step(s))",
162+
e.test_id,
163+
e.proof_ladder.len()
164+
),
165+
Some(ReportTier::ProvisionallyProven) => {
166+
format!("{}: Provisionally-Proven → informational (no proof artifact)", e.test_id)
167+
}
168+
Some(ReportTier::Unproven) => {
169+
format!("{}: Unproven → informational (no proof artifact)", e.test_id)
170+
}
171+
None => format!("{}: unrecognised provenance tier {:?}", e.test_id, e.provenance),
172+
})
173+
.collect()
174+
}
175+
}
176+
177+
#[cfg(test)]
178+
mod tests {
179+
use super::*;
180+
181+
const SAMPLE: &str = r#"{
182+
"schema_version": 1,
183+
"suite": "proven-tests",
184+
"summary": {"passed": 3, "total": 3, "actually_proven": 1,
185+
"provisionally_proven": 1, "unproven": 1,
186+
"covered_cells": 35, "cat_aspect_cells": 238, "lattice_covered": 35},
187+
"entries": [
188+
{"test_id": "tropical-laws", "category": "Proof-Regression",
189+
"aspect": "Dependability", "provenance": "Actually-Proven",
190+
"proof_ladder": [
191+
{"description": "comm", "proof_file": "src/ProvenTests/Tropical.idr", "theorem": "oplusComm"},
192+
{"description": "assoc", "proof_file": "src/ProvenTests/Tropical.idr", "theorem": "oplusAssoc"}
193+
],
194+
"result": "passed"},
195+
{"test_id": "typesafe-tropical", "provenance": "Provisionally-Proven",
196+
"proof_ladder": [], "result": "passed"},
197+
{"test_id": "smoke-show", "provenance": "Unproven",
198+
"proof_ladder": [], "result": "passed"}
199+
]
200+
}"#;
201+
202+
#[test]
203+
fn parses_schema_v1() {
204+
let r = ProvenTestsReport::from_str(SAMPLE).expect("parse");
205+
assert_eq!(r.schema_version, 1);
206+
assert_eq!(r.entries.len(), 3);
207+
assert_eq!(r.summary.actually_proven, 1);
208+
assert_eq!(r.summary.covered_cells, 35);
209+
}
210+
211+
#[test]
212+
fn tier_mapping_is_honest() {
213+
// Only Actually-Proven becomes a proof verdict; the weaker tiers do NOT
214+
// become Holes.
215+
assert_eq!(
216+
ReportTier::ActuallyProven.to_verdict(),
217+
Some(ProofVerdict::Closed)
218+
);
219+
assert_eq!(ReportTier::ProvisionallyProven.to_verdict(), None);
220+
assert_eq!(ReportTier::Unproven.to_verdict(), None);
221+
}
222+
223+
#[test]
224+
fn extracts_distinct_actually_proven_ladder_files() {
225+
let r = ProvenTestsReport::from_str(SAMPLE).unwrap();
226+
let files = r.actually_proven_ladder_files();
227+
// two ladder steps cite the same file → one distinct entry
228+
assert_eq!(files, vec!["src/ProvenTests/Tropical.idr".to_string()]);
229+
}
230+
231+
#[test]
232+
fn rejects_unknown_schema_version() {
233+
let bad = SAMPLE.replace("\"schema_version\": 1", "\"schema_version\": 2");
234+
assert!(ProvenTestsReport::from_str(&bad).is_err());
235+
}
236+
237+
#[test]
238+
fn unknown_tier_is_flagged_not_panicked() {
239+
assert_eq!(ReportTier::parse("Sorta-Proven"), None);
240+
let weird = SAMPLE.replace("\"Unproven\"", "\"Sorta-Proven\"");
241+
let r = ProvenTestsReport::from_str(&weird).unwrap();
242+
// notes() surfaces the unrecognised tier rather than crashing
243+
assert!(r.notes().iter().any(|n| n.contains("unrecognised provenance tier")));
244+
}
245+
}

0 commit comments

Comments
 (0)