Skip to content

Commit 6174705

Browse files
hyperpolymathclaude
andcommitted
docs: add TEST-NEEDS.md and/or PROOF-NEEDS.md from audit
Documents testing and proof gaps identified during batch audit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 3baf298 commit 6174705

1 file changed

Lines changed: 43 additions & 0 deletions

File tree

TEST-NEEDS.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# TEST-NEEDS: SMTLib.jl
2+
3+
## Current State
4+
5+
| Category | Count | Details |
6+
|----------|-------|---------|
7+
| **Source modules** | 12 | 3,586 lines |
8+
| **Test files** | 1 | 1,578 lines, 650 @test/@testset |
9+
| **Benchmarks** | 0 | None |
10+
| **E2E tests** | 0 | None |
11+
12+
## What's Missing
13+
14+
### E2E Tests
15+
- [ ] No test running SMTLib against actual SMT solvers (Z3, CVC5)
16+
- [ ] No roundtrip test (generate SMTLib2 -> parse back)
17+
18+
### Aspect Tests
19+
- [ ] **Performance**: SMT solving is performance-critical, 0 benchmarks
20+
- [ ] **Error handling**: No tests for unsatisfiable formulas, timeout, solver crashes
21+
- [ ] **Concurrency**: No parallel solving tests
22+
23+
### Benchmarks Needed (CRITICAL)
24+
- [ ] Formula generation throughput
25+
- [ ] Solver invocation latency
26+
- [ ] Scaling with formula complexity
27+
28+
### Self-Tests
29+
- [ ] No solver availability self-check
30+
31+
## FLAGGED ISSUES
32+
- **650 tests is excellent** -- second highest test count
33+
- **Single test file for 12 modules** -- should be split
34+
- **0 benchmarks for performance-critical SMT library** -- major gap
35+
- **No solver integration tests** -- generates SMTLib2 but never verifies it works
36+
37+
## Priority: P2 (MEDIUM) -- strong unit tests, needs benchmarks and solver E2E
38+
39+
## FAKE-FUZZ ALERT
40+
41+
- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing
42+
- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file
43+
- Priority: P2 — creates false impression of fuzz coverage

0 commit comments

Comments
 (0)