Skip to content

Commit 49fab43

Browse files
feat(verification): unified ResultArbiter — wire the orphaned 4-mechanism arbitration stack into dispatch (#306)
## What First instance of **mathematical-object arbitration**: when several provers rule on the same goal, ECHIDNA now fuses their verdicts through a policy-selected arbiter instead of an inline boolean all-must-agree check. The four arbitration mechanisms (portfolio `reconcile`, Bayesian log-odds, Dempster-Shafer, Pareto) already existed in `src/rust/verification/` but were **orphaned** — each had its own bespoke input type and nothing outside its own tests called it. `verify_proof_cross_checked` collapsed every cross-checker to a bool, counted a timeout as disagreement, and flattened a genuine Proven-vs-Refuted conflict to `verified=false`. ## How - **`result_arbiter.rs`** (new): `ResultArbiter::arbitrate(&[ProverAttempt]) -> ArbitratedVerdict`. Adapts the 8-variant `ProverOutcome` taxonomy into each mechanism's input; `ArbitrationPolicy` (`portfolio` default | `bayesian` | `dempster_shafer`) selected via `DispatchConfig.arbitration_policy`. The verdict carries agreeing/disagreeing/inconclusive camps, a [0,1] conflict metric, `needs_review`, policy-specific posterior/belief, and a Pareto-recommended prover among the agreeing set. - **`dispatch.rs`**: cross-check path feeds rich outcomes to the arbiter; `DispatchResult` gains `needs_review` + `arbitration` (serde-defaulted → wire-compatible). Disagreements are named in the message ("PROVERS DISAGREE: … — flagged for review"). - Semantics: timeout/error = **no-information** (no longer vetoes agreement); `InconsistentPremises` from any prover taints the verdict (`suspect_premises` → review); conclusive disagreement is never silently flattened. - `verified` stays primary-anchored (arbitrated-Proven **and** primary proved) so existing consumers see no surprise flips. ## Owner decision embedded here (why this PR is NOT armed for auto-merge) The **default `ArbitrationPolicy::Portfolio`** keeps conservative categorical semantics, but the timeout-is-not-disagreement change and the `NoProofFound→Refuted` verdict mapping are policy calls on the trust pipeline — please ratify (or request `bayesian`/`dempster_shafer` as default). ## Verification - 11 new unit tests: Proven-vs-Refuted conflict flagged not flattened, DS high-conflict refusal, timeout tolerance, suspect premises, Pareto pick, serde round-trip - `cargo test --lib`: 1185 passed; edited integration suites (stage1, e2e pipeline/prover, aspect, security, concurrency): 154 passed - `cargo fmt --check` + clippy clean on the new module Refs echidnabot#58 (multi-prover consensus). Follow-up (out of scope): proof-object-level arbitration via the exchange/ translators — needs `ProverBackend` to return proof objects first. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 1e91510 commit 49fab43

11 files changed

Lines changed: 586 additions & 20 deletions

benches/routing_benchmarks.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,6 +307,7 @@ fn bench_dispatch_config_construction(c: &mut Criterion) {
307307
generate_certificates: true,
308308
timeout: 60,
309309
diagnostics: false,
310+
arbitration_policy: Default::default(),
310311
})
311312
})
312313
});

docs/wiki/Guides.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,10 @@ Motivating examples:
101101
- **Dempster-Shafer** — "Five solvers; three Proven, two Refuted; either commit a posterior or *refuse to arbitrate* because conflict is too high."
102102
- **Pareto** — "Lean took 30s and produced a 4kB certificate; Z3 took 0.2s and produced no certificate. Which dominates?" — neither; return both as Pareto-optimal.
103103

104+
### Unified entry point: `ResultArbiter`
105+
106+
You normally don't call the four mechanisms directly. `ResultArbiter` (`src/rust/verification/result_arbiter.rs`) takes the per-prover `ProverOutcome`s from a cross-checked dispatch, adapts them into whichever mechanism `DispatchConfig.arbitration_policy` selects (`portfolio` default | `bayesian` | `dempster_shafer`), and returns an `ArbitratedVerdict`: winning verdict, agreeing/disagreeing/inconclusive camps, a `[0,1]` conflict metric, `needs_review`, and a Pareto-recommended prover among the agreeing set. `Dispatcher::verify_proof_cross_checked` attaches it to `DispatchResult.arbitration`. Key semantics: timeouts and errors are no-information (they no longer veto agreement), and a genuine Proven-vs-Refuted split is flagged for review instead of being flattened to `verified = false`.
107+
104108
## Guide: Cross-prover semantic queries
105109

106110
The synonym layer carries an optional `semantic_class` tag per entry. Combined with the three cross-prover dictionaries (`_msc2020.toml`, `_wordnet_math.toml`, `_conceptnet_seed.toml`) this gives "every prover's name for the same concept" lookups, fully offline.

src/rust/dispatch.rs

Lines changed: 74 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@ use crate::provers::outcome::{classify_anyhow_error, ProverOutcome};
2525
use crate::provers::{ProverConfig, ProverFactory, ProverKind};
2626
use crate::verification::axiom_tracker::{AxiomTracker, AxiomUsage, DangerLevel};
2727
use crate::verification::confidence::{compute_trust_level, TrustFactors, TrustLevel};
28+
use crate::verification::result_arbiter::{
29+
ArbitratedVerdict, ArbitrationPolicy, ProverAttempt, ResultArbiter,
30+
};
2831

2932
#[cfg(feature = "chapel")]
3033
use crate::proof_search::{ChapelParallelSearch, ProofSearchStrategy};
@@ -89,6 +92,17 @@ pub struct DispatchResult {
8992
/// input and per-prover records.
9093
#[serde(default)]
9194
pub diagnostics: Option<RunDiagnostics>,
95+
/// True when arbitration found conclusive disagreement, all-timeout,
96+
/// high evidential conflict, or suspect premises. A `verified = false`
97+
/// with `needs_review = true` means "the provers could not settle it",
98+
/// not "the proof is bad".
99+
#[serde(default)]
100+
pub needs_review: bool,
101+
/// Full arbitration verdict for cross-checked dispatches: winning
102+
/// verdict, agreeing/disagreeing camps, conflict metric, and any
103+
/// policy-specific posterior/belief. `None` on single-prover paths.
104+
#[serde(default)]
105+
pub arbitration: Option<ArbitratedVerdict>,
92106
}
93107

94108
/// Configuration for the dispatch pipeline
@@ -109,6 +123,11 @@ pub struct DispatchConfig {
109123
/// default is `false` so production callers pay nothing.
110124
#[serde(default)]
111125
pub diagnostics: bool,
126+
/// Which mechanism fuses cross-checked prover outcomes into one
127+
/// verdict. Default `Portfolio` (categorical agreement) — the
128+
/// conservative pre-arbiter behaviour.
129+
#[serde(default)]
130+
pub arbitration_policy: ArbitrationPolicy,
112131
}
113132

114133
impl Default for DispatchConfig {
@@ -120,6 +139,7 @@ impl Default for DispatchConfig {
120139
generate_certificates: false,
121140
timeout: 300,
122141
diagnostics: false,
142+
arbitration_policy: ArbitrationPolicy::default(),
123143
}
124144
}
125145
}
@@ -568,6 +588,8 @@ impl ProverDispatcher {
568588
cross_checked: false,
569589
outcome,
570590
diagnostics,
591+
needs_review: false,
592+
arbitration: None,
571593
});
572594
},
573595
};
@@ -688,6 +710,8 @@ impl ProverDispatcher {
688710
cross_checked: false,
689711
outcome,
690712
diagnostics,
713+
needs_review: false,
714+
arbitration: None,
691715
})
692716
}
693717

@@ -769,27 +793,40 @@ impl ProverDispatcher {
769793
.collect()
770794
};
771795

772-
// Run additional provers for cross-checking
773-
let mut all_agree = primary_result.verified;
796+
// Run additional provers and collect their rich outcomes — the
797+
// arbiter needs to distinguish "refuted" from "timed out" from
798+
// "errored", which the boolean `verified` flattens away.
799+
let mut attempts = vec![ProverAttempt::new(
800+
primary_prover,
801+
primary_result.outcome.clone(),
802+
primary_result.proof_time_ms,
803+
)];
774804
let mut provers_used = vec![format!("{:?}", primary_prover)];
775-
let mut confirming_count: u32 = if primary_result.verified { 1 } else { 0 };
776805

777806
for &additional in available_cross_checkers.iter() {
778807
match self.verify_proof(additional, content).await {
779808
Ok(result) => {
780809
provers_used.push(format!("{:?}", additional));
781-
if result.verified {
782-
confirming_count += 1;
783-
} else {
784-
all_agree = false;
785-
}
810+
attempts.push(ProverAttempt::new(
811+
additional,
812+
result.outcome.clone(),
813+
result.proof_time_ms,
814+
));
786815
},
787816
Err(e) => {
788817
warn!("Cross-check with {:?} failed: {}", additional, e);
789818
},
790819
}
791820
}
792821

822+
// Arbitrate: fuse the per-prover outcomes under the configured
823+
// policy instead of the old inline all-must-agree boolean check.
824+
let arbiter =
825+
ResultArbiter::new(self.config.arbitration_policy, self.config.timeout * 1000);
826+
let arbitration = arbiter.arbitrate(&attempts);
827+
828+
let confirming_count = attempts.iter().filter(|a| a.outcome.is_proved()).count() as u32;
829+
793830
let elapsed = start.elapsed().as_millis() as u64;
794831

795832
// Recompute trust level with cross-checking info
@@ -822,13 +859,7 @@ impl ProverDispatcher {
822859
certificate_verified: false,
823860
worst_axiom_danger: worst_danger,
824861
solver_integrity_ok,
825-
portfolio_confidence: Some(if confirming_count >= 2 {
826-
crate::verification::portfolio::PortfolioConfidence::CrossChecked
827-
} else if confirming_count == 1 {
828-
crate::verification::portfolio::PortfolioConfidence::SingleSolver
829-
} else {
830-
crate::verification::portfolio::PortfolioConfidence::Inconclusive
831-
}),
862+
portfolio_confidence: Some(arbitration.confidence),
832863
};
833864

834865
let trust_level = compute_trust_level(&trust_factors);
@@ -841,14 +872,32 @@ impl ProverDispatcher {
841872
primary_result.provers_used = provers_used;
842873
primary_result.proof_time_ms = elapsed;
843874
primary_result.cross_checked = cross_checked;
844-
primary_result.verified = all_agree && primary_result.verified;
875+
// `verified` stays primary-anchored: the arbitrated verdict must be
876+
// Proven AND the primary itself proved the goal. A timeout among the
877+
// cross-checkers no longer vetoes agreement (it is no-information),
878+
// but a conclusive disagreement always does. Consumers wanting the
879+
// fused multi-prover verdict read `arbitration`.
880+
primary_result.verified =
881+
matches!(arbitration.verified, Some(true)) && primary_result.outcome.is_proved();
882+
primary_result.needs_review = arbitration.needs_review;
845883

846884
if cross_checked {
847-
primary_result.message = format!(
848-
"Proof cross-checked by {} prover(s) with {}",
849-
confirming_count, trust_level
850-
);
885+
primary_result.message = if !arbitration.disagreeing.is_empty() {
886+
format!(
887+
"PROVERS DISAGREE on this goal: {:?} vs {:?} (conflict {:.2}) — flagged for review",
888+
arbitration.agreeing, arbitration.disagreeing, arbitration.conflict
889+
)
890+
} else if arbitration.suspect_premises {
891+
"Premises flagged inconsistent by a cross-checker — result withheld for review"
892+
.to_string()
893+
} else {
894+
format!(
895+
"Proof cross-checked by {} prover(s) with {}",
896+
confirming_count, trust_level
897+
)
898+
};
851899
}
900+
primary_result.arbitration = Some(arbitration);
852901

853902
Ok(primary_result)
854903
}
@@ -1012,6 +1061,8 @@ impl ProverDispatcher {
10121061
cross_checked: false,
10131062
outcome,
10141063
diagnostics,
1064+
needs_review: false,
1065+
arbitration: None,
10151066
})
10161067
}
10171068

@@ -1186,6 +1237,8 @@ mod tests {
11861237
cross_checked: false,
11871238
outcome: ProverOutcome::Proved { elapsed_ms: 1500 },
11881239
diagnostics: None,
1240+
needs_review: false,
1241+
arbitration: None,
11891242
};
11901243

11911244
let json = serde_json::to_string(&result).unwrap();
@@ -1229,6 +1282,7 @@ mod tests {
12291282
generate_certificates: true,
12301283
timeout: 600,
12311284
diagnostics: false,
1285+
arbitration_policy: Default::default(),
12321286
};
12331287
assert!(config.cross_check);
12341288
assert_eq!(config.min_trust_level, TrustLevel::Level4);

src/rust/verification/mod.rs

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,11 @@
2828
//!
2929
//! Picking between them: see the "Guide: Picking an arbitration
3030
//! mechanism" entry in `docs/wiki/Guides.md`.
31+
//!
32+
//! Unified entry point: `result_arbiter` adapts per-prover
33+
//! [`crate::provers::outcome::ProverOutcome`]s into whichever mechanism
34+
//! the configured [`result_arbiter::ArbitrationPolicy`] selects; the
35+
//! dispatch cross-check path delegates to it.
3136
3237
pub mod axiom_tracker;
3338
pub mod bayesian_arbiter;
@@ -40,6 +45,7 @@ pub mod pareto_arbiter;
4045
pub mod portfolio;
4146
#[cfg(feature = "verisim")]
4247
pub mod proof;
48+
pub mod result_arbiter;
4349
pub mod statistics;
4450

4551
pub use axiom_tracker::{AxiomPolicy, AxiomTracker, AxiomUsage, DangerLevel};
@@ -52,4 +58,5 @@ pub use portfolio::{PortfolioConfig, PortfolioResult, PortfolioSolver};
5258
pub use proof::{
5359
theorem_identity, Proof, ProofStateRecord, ProofVersion, TacticApplication, TacticStatus,
5460
};
61+
pub use result_arbiter::{ArbitratedVerdict, ArbitrationPolicy, ProverAttempt, ResultArbiter};
5562
pub use statistics::{StatisticsTracker, StatsSummaryRecord};

0 commit comments

Comments
 (0)