Skip to content

Commit 10f009f

Browse files
hyperpolymathclaude
andcommitted
docs: add TEST-NEEDS.md and PROOF-NEEDS.md from audit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 087c27b commit 10f009f

2 files changed

Lines changed: 100 additions & 0 deletions

File tree

PROOF-NEEDS.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# PROOF-NEEDS.md — neural-foundations
2+
3+
## Current State
4+
5+
- **src/abi/*.idr**: YES (in satellites) — echidna has `BojForeign.idr`, `Foreign.idr`, `Layout.idr`, `Overlay.idr`
6+
- **Dangerous patterns**: 0 in own code (27 references are all in rule/detection code that identifies dangerous patterns in other repos)
7+
- **LOC**: ~153,000 (Rust + Agda + Julia + Idris2)
8+
- **ABI layer**: ECHIDNA has comprehensive Idris2 ABI with explicit "NO believe_me" invariant
9+
- **Existing proofs**: Agda proofs in `echidna/proofs/agda/` and `echidna/meta-checker/`
10+
11+
## What Needs Proving
12+
13+
| Component | What | Why |
14+
|-----------|------|-----|
15+
| ECHIDNA axiom safety checker | Meta-checker correctly classifies all axiom types | The tool that checks OTHER proofs must itself be proven correct |
16+
| ECHIDNA prover integrations | Each prover bridge (Dafny, dReal, Frama-C, F*, Imandra, Kissat, MetaMath, MiniZinc, Mizar, NuSMV, PRISM) faithfully translates | Incorrect translation makes proof results meaningless |
17+
| LLM-unify parser correctness | Claude/Copilot/Gemini parsers extract correct structured data | Wrong parsing produces incorrect model comparisons |
18+
| Elegant-state consensus | Voting/proposal mechanism produces correct outcomes | Distributed consensus bugs lose data |
19+
| Conative-gating SLM | Small language model gating decisions are sound | Incorrect gating passes unsafe content |
20+
21+
## Recommended Prover
22+
23+
**Agda** for ECHIDNA meta-checker (already in Agda). **Idris2** for ABI layer extensions. The prover bridge translations are ideal for **Lean4** relational proofs.
24+
25+
## Priority
26+
27+
**HIGH** — ECHIDNA is the proof verification meta-tool. If the meta-checker itself has bugs, it undermines all downstream formal verification. The existing Agda proofs need extension to cover all prover bridges.

TEST-NEEDS.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# TEST-NEEDS.md — neural-foundations
2+
3+
> Generated 2026-03-29 by punishing audit.
4+
5+
## Current State
6+
7+
| Category | Count | Notes |
8+
|-------------|-------|-------|
9+
| Unit tests | ~8 | echidna: test_agda_backend.rs, test_neural_integration.rs, ffi_integration_test.rs, prover tests (unified_test.ts, prover_test.ts). llm-antidote: test_suite.py |
10+
| Integration | ~5 | echidna: integration_test.sh, Zig core_native_test.zig, integration_test.zig. llm-unify: e2e_test.sh |
11+
| E2E | 1 | llm-unify e2e_test.sh |
12+
| Benchmarks | 4 | echidna: proof_benchmarks.rs, benchmark.zig, overlay_benchmark.zig. llm-antidote: benchmark.py |
13+
14+
**Source modules:** ~337 across 3 satellite groups. neurosymbolic/echidna: ~104 Rust src + Idris2 ABI + Zig FFI + Julia. foundation-models: llm-antidote, llm-unify, llm-unify-core. agentic: elegant-state, conative-gating, agentic-scm.
15+
16+
## What's Missing
17+
18+
### P2P (Property-Based) Tests
19+
- [ ] echidna proof engine: property tests for proof soundness (no invalid proof accepted)
20+
- [ ] Neural integration: property tests for embedding consistency
21+
- [ ] LLM antidote: adversarial input property tests
22+
- [ ] Prover backends: equivalence property tests across prover implementations
23+
24+
### E2E Tests
25+
- [ ] echidna: full proof request -> prover selection -> proof generation -> verification
26+
- [ ] llm-unify: request -> routing -> model inference -> response normalization
27+
- [ ] agentic: agent lifecycle (create -> plan -> execute -> evaluate)
28+
- [ ] Cross-satellite: echidna proof + llm-unify inference in combined workflow
29+
30+
### Aspect Tests
31+
- **Security:** No tests for proof forgery, model injection in llm-unify, agent privilege escalation
32+
- **Performance:** Benchmarks exist for echidna (good) and llm-antidote. Missing: llm-unify latency, agentic execution time
33+
- **Concurrency:** No tests for parallel proof generation, concurrent model inference, agent contention
34+
- **Error handling:** No tests for prover timeout, model unavailability, malformed proof input, invalid agent state
35+
36+
### Build & Execution
37+
- [ ] `cargo test` for echidna Rust
38+
- [ ] `zig build test` for FFI
39+
- [ ] Python test runner for llm-antidote
40+
- [ ] Shell test execution for integration tests
41+
42+
### Benchmarks Needed
43+
- [ ] Proof generation latency by complexity
44+
- [ ] LLM routing decision time
45+
- [ ] Agent planning throughput
46+
- [ ] Cross-prover comparison benchmarks
47+
- [ ] Neural overlay inference time
48+
49+
### Self-Tests
50+
- [ ] Prover health checks (each backend reachable)
51+
- [ ] Model availability verification
52+
- [ ] Proof chain integrity validation
53+
54+
### CRITICAL GAPS
55+
56+
| Subsystem | Source Files | Tests | Status |
57+
|-----------|-------------|-------|--------|
58+
| echidna | ~104 | ~8 | **7.7% — insufficient** |
59+
| llm-antidote | unknown | 1 suite | Minimal |
60+
| llm-unify | unknown | 1 e2e | Minimal |
61+
| agentic (elegant-state) | unknown | 1 archived | **Effectively 0** |
62+
| conative-gating | unknown | 0 | **Untested** |
63+
| agentic-scm | unknown | 0 | **Untested** |
64+
65+
## Priority
66+
67+
**CRITICAL.** 337 source files across a neurosymbolic AI framework with ~14 test files total. echidna at 104 Rust files with 8 tests is 7.7% coverage. The agentic subsystem is completely untested. Benchmarks for echidna are a bright spot, but the overall test story is woeful for a framework that generates formal proofs — the proof engine itself needs far more rigorous testing.
68+
69+
## FAKE-FUZZ ALERT
70+
71+
- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing
72+
- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file
73+
- Priority: P2 — creates false impression of fuzz coverage

0 commit comments

Comments
 (0)