|
| 1 | +# Test & Benchmark Requirements |
| 2 | + |
| 3 | +## Current State |
| 4 | +- Unit tests: ~22 Rust test files + shared-context tests — count unknown (no Cargo.toml at root to run) |
| 5 | +- Integration tests: partial (echidnabot/tests/integration_tests.rs, rhodibot/tests/integration_tests.rs) |
| 6 | +- E2E tests: NONE |
| 7 | +- Benchmarks: 5 files exist |
| 8 | +- panic-attack scan: NEVER RUN |
| 9 | + |
| 10 | +## What's Missing |
| 11 | +### Point-to-Point (P2P) |
| 12 | +205 Rust + 367 JavaScript + 42 ReScript + 34 Julia + 13 Haskell + 9 Idris2 + 85 Shell source files: |
| 13 | + |
| 14 | +#### Bots (major components): |
| 15 | +- **echidnabot/** — has integration tests, but coverage vs source count unclear |
| 16 | +- **rhodibot/** — has integration tests |
| 17 | +- **sustainabot/** — has test_sample.rs + .env.test, minimal coverage |
| 18 | +- **finishbot/** — has analyzers/testing.rs (test utilities, not tests OF finishbot) |
| 19 | +- **glambot/** — no tests found |
| 20 | +- **seambot/** — no tests found |
| 21 | +- **panicbot/** — no tests found |
| 22 | + |
| 23 | +#### Shared Context (Rust): |
| 24 | +- shared-context/tests/context_tests.rs — exists |
| 25 | +- shared-context/tests/fleet_coordination_test.rs — exists |
| 26 | +- Other shared modules may lack coverage |
| 27 | + |
| 28 | +#### Robot-repo-automaton: |
| 29 | +- No test files found |
| 30 | + |
| 31 | +#### Dashboard: |
| 32 | +- No test files found |
| 33 | + |
| 34 | +#### Campaigns: |
| 35 | +- No test files found |
| 36 | + |
| 37 | +#### Hooks: |
| 38 | +- No test files found |
| 39 | + |
| 40 | +#### Tasks/Scripts: |
| 41 | +- 85 shell scripts — no tests |
| 42 | +- 367 JavaScript files — no tests (these are likely bot action scripts) |
| 43 | +- 34 Julia files — no tests |
| 44 | + |
| 45 | +### End-to-End (E2E) |
| 46 | +- Bot fleet coordination: dispatch task -> bot processes -> report results -> aggregate |
| 47 | +- Individual bot lifecycle: start -> receive event -> analyze -> act -> report |
| 48 | +- echidnabot: receive webhook -> analyze repo -> generate findings -> submit |
| 49 | +- rhodibot: detect issue -> apply fix -> create PR |
| 50 | +- sustainabot: check dependencies -> evaluate health -> report |
| 51 | +- finishbot: analyze completion -> identify gaps -> notify |
| 52 | +- glambot: check style -> suggest fixes |
| 53 | +- seambot: check integration points -> verify |
| 54 | +- panicbot: run security scan -> report findings |
| 55 | +- Robot-repo-automaton: detect issue -> calculate confidence -> auto-fix |
| 56 | +- Fleet coordination: multiple bots on same repo -> deconflict |
| 57 | + |
| 58 | +### Aspect Tests |
| 59 | +- [ ] Security (webhook signature verification, bot credential handling, repo access scoping, auto-fix safety) |
| 60 | +- [ ] Performance (fleet throughput, bot startup latency, concurrent repo processing) |
| 61 | +- [ ] Concurrency (bot deconfliction, shared-context locking, parallel webhook processing) |
| 62 | +- [ ] Error handling (bot crash recovery, API rate limiting, malformed webhooks) |
| 63 | +- [ ] Accessibility (dashboard UI if applicable) |
| 64 | + |
| 65 | +### Build & Execution |
| 66 | +- [ ] cargo build per bot — not verified |
| 67 | +- [ ] Individual bot startup — not verified |
| 68 | +- [ ] Fleet orchestration — not verified |
| 69 | +- [ ] Dashboard build — not verified |
| 70 | +- [ ] Self-diagnostic — none |
| 71 | + |
| 72 | +### Benchmarks Needed |
| 73 | +- Per-bot analysis throughput (repos/hour) |
| 74 | +- Fleet coordination overhead |
| 75 | +- Webhook processing latency |
| 76 | +- Auto-fix confidence score accuracy |
| 77 | +- Memory usage per active bot |
| 78 | + |
| 79 | +### Self-Tests |
| 80 | +- [ ] panic-attack assail on own repo |
| 81 | +- [ ] Fleet health check |
| 82 | +- [ ] Per-bot self-test |
| 83 | +- [ ] Shared-context integrity verification |
| 84 | + |
| 85 | +## Priority |
| 86 | +- **HIGH** — Bot fleet (205 Rust + 367 JS + 42 ReScript + 34 Julia files) with tests only for echidnabot, rhodibot, and shared-context. 5 out of 7+ bots have ZERO tests. The 367 JavaScript files (bot action scripts) are completely untested. The robot-repo-automaton that auto-fixes repos based on confidence scores has no tests — this is a system that makes automated changes to other repos and needs extremely high correctness guarantees. |
| 87 | + |
| 88 | +## FAKE-FUZZ ALERT |
| 89 | + |
| 90 | +- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing |
| 91 | +- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file |
| 92 | +- Priority: P2 — creates false impression of fuzz coverage |
0 commit comments