Skip to content

Commit ec04fd5

Browse files
hyperpolymathclaude
andcommitted
docs: add TEST-NEEDS.md and/or PROOF-NEEDS.md from audit
Documents testing and proof gaps identified during batch audit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 3a21b7e commit ec04fd5

2 files changed

Lines changed: 90 additions & 0 deletions

File tree

PROOF-NEEDS.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# PROOF-NEEDS.md — odds-and-sods-package-manager (OPSM)
2+
3+
## Current State
4+
5+
- **src/abi/*.idr**: YES — `Types.idr`, `Layout.idr`, `Foreign.idr`, `opsm-abi.ipkg`
6+
- **Dangerous patterns**: 0 in own code (hits in deps/proven vendored bindings)
7+
- **LOC**: ~102,000 (Elixir + Rust + Idris2 + vendored deps)
8+
- **ABI layer**: Complete Idris2 ABI with ipkg build
9+
10+
## What Needs Proving
11+
12+
| Component | What | Why |
13+
|-----------|------|-----|
14+
| Package installation integrity | Install operations don't corrupt existing packages | Package managers must never break the system |
15+
| Dependency resolution | Resolution algorithm terminates and produces consistent solution | Non-terminating or inconsistent resolution breaks builds |
16+
| PQ crypto (pq_crypto NIF) | Post-quantum cryptographic operations are correct | Package signing with broken crypto is worthless |
17+
| QUIC transport | Transport layer delivers data correctly | Corrupt downloads install wrong packages |
18+
| Verified HTTP (opsm/verified/http.ex) | HTTP operations enforce security policies | Unverified HTTP allows MITM attacks on package downloads |
19+
| Safe exec | Command execution respects sandboxing invariants | Package scripts must not escape sandbox |
20+
| Validation module | Input validation is complete (no bypass paths) | Malformed input to package manager is an attack vector |
21+
22+
## Recommended Prover
23+
24+
**Idris2** — ABI layer already complete with ipkg. Extend with dependency resolution termination proof and package integrity invariants. PQ crypto proofs may need **Coq** with Jasmin/EasyCrypt.
25+
26+
## Priority
27+
28+
**HIGH** — Package managers are high-value attack targets. OPSM handles package installation, dependency resolution, and cryptographic signing. A bug in any of these is a supply chain vulnerability. The PQ crypto NIF is especially critical.

TEST-NEEDS.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# TEST-NEEDS.md — odds-and-sods-package-manager
2+
3+
> Generated 2026-03-29 by punishing audit.
4+
5+
## Current State
6+
7+
| Category | Count | Notes |
8+
|-------------|-------|-------|
9+
| Unit tests | ~25 | opsm_test, package/transaction, progress, maintenance, validation, config, lockfile, imp_property, registry_gateway, version_constraint, verified, verified/url_property, verified/json_property, verified/result_property |
10+
| Integration | 5 | pipeline, trust_pipeline, e2e, git_pipeline, manifest_roundtrip |
11+
| E2E | 1 | integration/e2e_test.exs |
12+
| Benchmarks | 0 | Bench files exist but are in deps/proven (not own benchmarks) |
13+
14+
**Source modules:** ~238 own source files (Elixir + ReScript + Rust). Covers: package management, trust pipeline, registry gateway, version constraints, lockfiles, config, validation, manifest handling, Nickel plugins.
15+
16+
## What's Missing
17+
18+
### P2P (Property-Based) Tests
19+
- [ ] Version constraint: already has imp_property_test but needs more: arbitrary version range intersection/union
20+
- [ ] Lockfile: property tests for lockfile determinism (same inputs = same lockfile)
21+
- [ ] Manifest: arbitrary manifest structure validation
22+
- [ ] Registry gateway: response format property tests
23+
24+
### E2E Tests
25+
- [ ] Full install: search -> resolve -> download -> verify -> install -> lockfile update
26+
- [ ] Full update: check -> resolve conflicts -> download -> verify -> swap -> lockfile update
27+
- [ ] Trust pipeline: untrusted package -> verification -> trust decision -> install/reject
28+
- [ ] Uninstall: remove -> cleanup -> lockfile update -> verify no orphans
29+
30+
### Aspect Tests
31+
- **Security:** No tests for package tampering detection, registry MITM, dependency confusion, lockfile poisoning — CRITICAL for a package manager
32+
- **Performance:** ZERO own benchmarks. Resolution speed, download throughput, verification time all unmeasured
33+
- **Concurrency:** No tests for parallel package downloads, concurrent installs, lockfile contention
34+
- **Error handling:** No tests for network failure during download, corrupted package, version conflict deadlock, registry unavailability
35+
36+
### Build & Execution
37+
- [ ] `mix test` for Elixir
38+
- [ ] ReScript build
39+
- [ ] Rust crate tests
40+
41+
### Benchmarks Needed
42+
- [ ] Dependency resolution time vs dependency graph size
43+
- [ ] Package download + verification throughput
44+
- [ ] Lockfile generation time
45+
- [ ] Trust pipeline evaluation speed
46+
- [ ] Version constraint solving time
47+
48+
### Self-Tests
49+
- [ ] Registry connectivity health check
50+
- [ ] Lockfile integrity self-validation
51+
- [ ] Package cache consistency check
52+
- [ ] Trust chain verification
53+
54+
## Priority
55+
56+
**CRITICAL.** A package manager with ZERO security tests is a supply chain risk. 238 source files with 30 test files is 12.6% file coverage. The property tests and integration tests are a decent start but security testing is completely absent. No own benchmarks despite the deps/proven benchmarks being present (those test the wrong thing). Lockfile and trust pipeline need immediate hardening.
57+
58+
## FAKE-FUZZ ALERT
59+
60+
- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing
61+
- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file
62+
- Priority: P2 — creates false impression of fuzz coverage

0 commit comments

Comments
 (0)