|
| 1 | +# TEST-NEEDS.md — patch-bridge |
| 2 | + |
| 3 | +> Generated 2026-03-29 by punishing audit. |
| 4 | +
|
| 5 | +## Current State |
| 6 | + |
| 7 | +| Category | Count | Notes | |
| 8 | +|-------------|-------|-------| |
| 9 | +| Unit tests | ~14 | Inline `#[test]`: classify(4), intelligence(2), lockfile(2), reachability(4), registry(2) | |
| 10 | +| Integration | 1 | src/interface/ffi/test/integration_test.zig | |
| 11 | +| E2E | 0 | None | |
| 12 | +| Benchmarks | 0 | None | |
| 13 | + |
| 14 | +**Source modules:** ~11 Rust source files in src/bridge/ (classify, intelligence, lockfile, reachability, registry + main) + 3 Idris2 ABI + 1 Zig FFI + 1 ReScript. |
| 15 | + |
| 16 | +## What's Missing |
| 17 | + |
| 18 | +### P2P (Property-Based) Tests |
| 19 | +- [ ] CVE classification: property tests for severity scoring consistency |
| 20 | +- [ ] Lockfile parsing: arbitrary lockfile format fuzzing (Cargo.lock, package-lock.json, mix.lock, etc.) |
| 21 | +- [ ] Reachability: property tests for call graph analysis correctness |
| 22 | +- [ ] Registry: property tests for advisory lookup consistency |
| 23 | + |
| 24 | +### E2E Tests |
| 25 | +- [ ] Full CVE lifecycle: detect -> classify -> assess reachability -> recommend mitigation -> verify fix |
| 26 | +- [ ] Multi-format: lockfile parsing across all supported package managers |
| 27 | +- [ ] Intelligence: advisory fetch -> parse -> match to dependencies |
| 28 | +- [ ] Adoption gate: recommend patch -> apply -> verify -> close |
| 29 | + |
| 30 | +### Aspect Tests |
| 31 | +- **Security:** A CVE mitigation tool needs security testing: false negative detection (missed CVEs), advisory tampering, lockfile injection — ZERO security tests |
| 32 | +- **Performance:** No benchmarks for lockfile parsing speed, CVE database lookup time, reachability analysis scaling |
| 33 | +- **Concurrency:** No tests for parallel vulnerability assessment, concurrent advisory fetches |
| 34 | +- **Error handling:** No tests for malformed lockfiles, unreachable registries, invalid CVE identifiers, network timeouts |
| 35 | + |
| 36 | +### Build & Execution |
| 37 | +- [ ] `cargo test` |
| 38 | +- [ ] Zig FFI test execution |
| 39 | +- [ ] CLI smoke tests with real lockfiles |
| 40 | + |
| 41 | +### Benchmarks Needed |
| 42 | +- [ ] Lockfile parsing time per format |
| 43 | +- [ ] Reachability analysis time vs dependency graph size |
| 44 | +- [ ] Advisory lookup latency |
| 45 | +- [ ] Full assessment pipeline throughput |
| 46 | + |
| 47 | +### Self-Tests |
| 48 | +- [ ] Assess its own Cargo.lock for vulnerabilities |
| 49 | +- [ ] Registry connectivity health check |
| 50 | +- [ ] Classification model consistency verification |
| 51 | + |
| 52 | +## Priority |
| 53 | + |
| 54 | +**CRITICAL.** A CVE mitigation lifecycle tool with 14 inline unit tests and ZERO E2E tests. The reachability and classification modules have the most tests (good). But no integration testing means the pieces are never validated together. A security tool that cannot test its own security posture is self-refuting. No benchmarks for what should be a performance-sensitive pipeline. |
| 55 | + |
| 56 | +## FAKE-FUZZ ALERT |
| 57 | + |
| 58 | +- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing |
| 59 | +- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file |
| 60 | +- Priority: P2 — creates false impression of fuzz coverage |
0 commit comments