|
| 1 | +# TEST-NEEDS.md — odds-and-sods-package-manager |
| 2 | + |
| 3 | +> Generated 2026-03-29 by punishing audit. |
| 4 | +
|
| 5 | +## Current State |
| 6 | + |
| 7 | +| Category | Count | Notes | |
| 8 | +|-------------|-------|-------| |
| 9 | +| Unit tests | ~25 | opsm_test, package/transaction, progress, maintenance, validation, config, lockfile, imp_property, registry_gateway, version_constraint, verified, verified/url_property, verified/json_property, verified/result_property | |
| 10 | +| Integration | 5 | pipeline, trust_pipeline, e2e, git_pipeline, manifest_roundtrip | |
| 11 | +| E2E | 1 | integration/e2e_test.exs | |
| 12 | +| Benchmarks | 0 | Bench files exist but are in deps/proven (not own benchmarks) | |
| 13 | + |
| 14 | +**Source modules:** ~238 own source files (Elixir + ReScript + Rust). Covers: package management, trust pipeline, registry gateway, version constraints, lockfiles, config, validation, manifest handling, Nickel plugins. |
| 15 | + |
| 16 | +## What's Missing |
| 17 | + |
| 18 | +### P2P (Property-Based) Tests |
| 19 | +- [ ] Version constraint: already has imp_property_test but needs more: arbitrary version range intersection/union |
| 20 | +- [ ] Lockfile: property tests for lockfile determinism (same inputs = same lockfile) |
| 21 | +- [ ] Manifest: arbitrary manifest structure validation |
| 22 | +- [ ] Registry gateway: response format property tests |
| 23 | + |
| 24 | +### E2E Tests |
| 25 | +- [ ] Full install: search -> resolve -> download -> verify -> install -> lockfile update |
| 26 | +- [ ] Full update: check -> resolve conflicts -> download -> verify -> swap -> lockfile update |
| 27 | +- [ ] Trust pipeline: untrusted package -> verification -> trust decision -> install/reject |
| 28 | +- [ ] Uninstall: remove -> cleanup -> lockfile update -> verify no orphans |
| 29 | + |
| 30 | +### Aspect Tests |
| 31 | +- **Security:** No tests for package tampering detection, registry MITM, dependency confusion, lockfile poisoning — CRITICAL for a package manager |
| 32 | +- **Performance:** ZERO own benchmarks. Resolution speed, download throughput, verification time all unmeasured |
| 33 | +- **Concurrency:** No tests for parallel package downloads, concurrent installs, lockfile contention |
| 34 | +- **Error handling:** No tests for network failure during download, corrupted package, version conflict deadlock, registry unavailability |
| 35 | + |
| 36 | +### Build & Execution |
| 37 | +- [ ] `mix test` for Elixir |
| 38 | +- [ ] ReScript build |
| 39 | +- [ ] Rust crate tests |
| 40 | + |
| 41 | +### Benchmarks Needed |
| 42 | +- [ ] Dependency resolution time vs dependency graph size |
| 43 | +- [ ] Package download + verification throughput |
| 44 | +- [ ] Lockfile generation time |
| 45 | +- [ ] Trust pipeline evaluation speed |
| 46 | +- [ ] Version constraint solving time |
| 47 | + |
| 48 | +### Self-Tests |
| 49 | +- [ ] Registry connectivity health check |
| 50 | +- [ ] Lockfile integrity self-validation |
| 51 | +- [ ] Package cache consistency check |
| 52 | +- [ ] Trust chain verification |
| 53 | + |
| 54 | +## Priority |
| 55 | + |
| 56 | +**CRITICAL.** A package manager with ZERO security tests is a supply chain risk. 238 source files with 30 test files is 12.6% file coverage. The property tests and integration tests are a decent start but security testing is completely absent. No own benchmarks despite the deps/proven benchmarks being present (those test the wrong thing). Lockfile and trust pipeline need immediate hardening. |
| 57 | + |
| 58 | +## FAKE-FUZZ ALERT |
| 59 | + |
| 60 | +- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing |
| 61 | +- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file |
| 62 | +- Priority: P2 — creates false impression of fuzz coverage |
0 commit comments