|
| 1 | +# TEST-NEEDS.md — neural-foundations |
| 2 | + |
| 3 | +> Generated 2026-03-29 by punishing audit. |
| 4 | +
|
| 5 | +## Current State |
| 6 | + |
| 7 | +| Category | Count | Notes | |
| 8 | +|-------------|-------|-------| |
| 9 | +| Unit tests | ~8 | echidna: test_agda_backend.rs, test_neural_integration.rs, ffi_integration_test.rs, prover tests (unified_test.ts, prover_test.ts). llm-antidote: test_suite.py | |
| 10 | +| Integration | ~5 | echidna: integration_test.sh, Zig core_native_test.zig, integration_test.zig. llm-unify: e2e_test.sh | |
| 11 | +| E2E | 1 | llm-unify e2e_test.sh | |
| 12 | +| Benchmarks | 4 | echidna: proof_benchmarks.rs, benchmark.zig, overlay_benchmark.zig. llm-antidote: benchmark.py | |
| 13 | + |
| 14 | +**Source modules:** ~337 across 3 satellite groups. neurosymbolic/echidna: ~104 Rust src + Idris2 ABI + Zig FFI + Julia. foundation-models: llm-antidote, llm-unify, llm-unify-core. agentic: elegant-state, conative-gating, agentic-scm. |
| 15 | + |
| 16 | +## What's Missing |
| 17 | + |
| 18 | +### P2P (Property-Based) Tests |
| 19 | +- [ ] echidna proof engine: property tests for proof soundness (no invalid proof accepted) |
| 20 | +- [ ] Neural integration: property tests for embedding consistency |
| 21 | +- [ ] LLM antidote: adversarial input property tests |
| 22 | +- [ ] Prover backends: equivalence property tests across prover implementations |
| 23 | + |
| 24 | +### E2E Tests |
| 25 | +- [ ] echidna: full proof request -> prover selection -> proof generation -> verification |
| 26 | +- [ ] llm-unify: request -> routing -> model inference -> response normalization |
| 27 | +- [ ] agentic: agent lifecycle (create -> plan -> execute -> evaluate) |
| 28 | +- [ ] Cross-satellite: echidna proof + llm-unify inference in combined workflow |
| 29 | + |
| 30 | +### Aspect Tests |
| 31 | +- **Security:** No tests for proof forgery, model injection in llm-unify, agent privilege escalation |
| 32 | +- **Performance:** Benchmarks exist for echidna (good) and llm-antidote. Missing: llm-unify latency, agentic execution time |
| 33 | +- **Concurrency:** No tests for parallel proof generation, concurrent model inference, agent contention |
| 34 | +- **Error handling:** No tests for prover timeout, model unavailability, malformed proof input, invalid agent state |
| 35 | + |
| 36 | +### Build & Execution |
| 37 | +- [ ] `cargo test` for echidna Rust |
| 38 | +- [ ] `zig build test` for FFI |
| 39 | +- [ ] Python test runner for llm-antidote |
| 40 | +- [ ] Shell test execution for integration tests |
| 41 | + |
| 42 | +### Benchmarks Needed |
| 43 | +- [ ] Proof generation latency by complexity |
| 44 | +- [ ] LLM routing decision time |
| 45 | +- [ ] Agent planning throughput |
| 46 | +- [ ] Cross-prover comparison benchmarks |
| 47 | +- [ ] Neural overlay inference time |
| 48 | + |
| 49 | +### Self-Tests |
| 50 | +- [ ] Prover health checks (each backend reachable) |
| 51 | +- [ ] Model availability verification |
| 52 | +- [ ] Proof chain integrity validation |
| 53 | + |
| 54 | +### CRITICAL GAPS |
| 55 | + |
| 56 | +| Subsystem | Source Files | Tests | Status | |
| 57 | +|-----------|-------------|-------|--------| |
| 58 | +| echidna | ~104 | ~8 | **7.7% — insufficient** | |
| 59 | +| llm-antidote | unknown | 1 suite | Minimal | |
| 60 | +| llm-unify | unknown | 1 e2e | Minimal | |
| 61 | +| agentic (elegant-state) | unknown | 1 archived | **Effectively 0** | |
| 62 | +| conative-gating | unknown | 0 | **Untested** | |
| 63 | +| agentic-scm | unknown | 0 | **Untested** | |
| 64 | + |
| 65 | +## Priority |
| 66 | + |
| 67 | +**CRITICAL.** 337 source files across a neurosymbolic AI framework with ~14 test files total. echidna at 104 Rust files with 8 tests is 7.7% coverage. The agentic subsystem is completely untested. Benchmarks for echidna are a bright spot, but the overall test story is woeful for a framework that generates formal proofs — the proof engine itself needs far more rigorous testing. |
| 68 | + |
| 69 | +## FAKE-FUZZ ALERT |
| 70 | + |
| 71 | +- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing |
| 72 | +- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file |
| 73 | +- Priority: P2 — creates false impression of fuzz coverage |
0 commit comments