Skip to content

Commit 8f9fbf0

Browse files
hyperpolymathclaude
andcommitted
docs: add TEST-NEEDS.md and/or PROOF-NEEDS.md from audit
Documents testing and proof gaps identified during batch audit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent b20dc49 commit 8f9fbf0

2 files changed

Lines changed: 101 additions & 0 deletions

File tree

PROOF-NEEDS.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# PROOF-NEEDS.md
2+
<!-- SPDX-License-Identifier: PMPL-1.0-or-later -->
3+
4+
## Current State
5+
6+
- **LOC**: ~8,000
7+
- **Languages**: Rust, ReScript, Idris2, Zig
8+
- **Existing ABI proofs**: `src/interface/abi/*.idr` (template-level) + domain-specific Idris2: `src/core/Checker.idr`, `Grammar.idr`, `Levels.idr`, `Schema.idr`
9+
- **Dangerous patterns**: None detected
10+
11+
## What Needs Proving
12+
13+
### Query Type Checker (src/core/Checker.idr)
14+
- Already in Idris2 — verify it type-checks and that the checking algorithm is total
15+
- Prove: well-typed VQL-UT queries produce well-typed results against a schema
16+
17+
### Grammar Specification (src/core/Grammar.idr)
18+
- VQL-UT grammar defined in Idris2 — prove the grammar is unambiguous
19+
- Prove: parser (ReScript side) accepts exactly the Idris2-specified grammar
20+
21+
### Level System (src/core/Levels.idr)
22+
- 10-level type safety hierarchy — prove level ordering is a lattice
23+
- Prove: level promotion/demotion preserves query safety
24+
25+
### Schema Validation (src/core/Schema.idr)
26+
- Prove: schema-validated queries cannot produce runtime type errors
27+
- Prove: schema evolution preserves backward compatibility for existing queries
28+
29+
### Rust DAP/Formatter (src/interface/dap/, src/interface/fmt/)
30+
- Debug adapter and formatter — lower priority but should preserve query semantics
31+
32+
### ReScript Bridge (src/bridges/)
33+
- `VqlUtParser.res`, `VqlUtBridge.res` — bridge between ReScript frontend and Idris2/Rust core
34+
- Prove: bridge faithfully translates between ReScript and core representations
35+
36+
## Recommended Prover
37+
38+
- **Idris2** (already in use for core — complete the proofs in Checker.idr, Grammar.idr, Levels.idr, Schema.idr)
39+
40+
## Priority
41+
42+
**HIGH** — VQL-UT is the query language for VeriSimDB. Incorrect type checking could allow queries that corrupt data or return wrong results. The Idris2 core is already in place — completing the proofs is high value for low effort.

TEST-NEEDS.md

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# TEST-NEEDS: vql-ut
2+
3+
## Current State
4+
5+
| Category | Count | Details |
6+
|----------|-------|---------|
7+
| **Source modules** | 27 | Idris2 core (4: Checker, Grammar, Levels, Schema), ReScript bridges (3: Ast, Bridge, Parser), ReScript definitions/errors (2), Rust interfaces (LSP 2, DAP 2, fmt 2, lint 2, lib), 3 Idris2 interface ABI, Zig FFI |
8+
| **Unit tests** | 1 file | integration_test.rs (49 #[test]) |
9+
| **Integration tests** | 0 | Despite the filename, these are unit tests |
10+
| **E2E tests** | 0 | None |
11+
| **Benchmarks** | 0 | None |
12+
| **Fuzz tests** | 0 | None |
13+
14+
## What's Missing
15+
16+
### P2P Tests (CRITICAL)
17+
- [ ] No tests for ReScript parser -> Idris2 checker pipeline
18+
- [ ] No tests for LSP server handling real editor requests
19+
- [ ] No tests for DAP server with real debugger
20+
- [ ] No tests for fmt/lint tools on actual VQL-UT code
21+
22+
### E2E Tests (CRITICAL)
23+
- [ ] No test that parses VQL-UT, type-checks it through all 10 levels, and executes it
24+
- [ ] No test for LSP completion/hover/diagnostics
25+
- [ ] No test for DAP breakpoints/stepping
26+
27+
### Aspect Tests
28+
- [ ] **Security**: Query language with no injection tests
29+
- [ ] **Performance**: No benchmarks for type checking, parsing throughput
30+
- [ ] **Concurrency**: No concurrent query compilation tests
31+
- [ ] **Error handling**: No tests for malformed VQL-UT, type errors at each level
32+
33+
### Build & Execution
34+
- [ ] 4 Idris2 core modules with 0 Idris2-level tests -- are proofs checked?
35+
- [ ] 4 Rust tool interfaces (LSP, DAP, fmt, lint) with 0 tests each
36+
- [ ] Zig FFI integration_test.zig likely template placeholder
37+
38+
### Benchmarks Needed
39+
- [ ] VQL-UT parsing throughput
40+
- [ ] Type checking per level (L1-L10)
41+
- [ ] LSP response latency
42+
- [ ] Query compilation time
43+
44+
### Self-Tests
45+
- [ ] No VQL-UT self-consistency check
46+
47+
## FLAGGED ISSUES
48+
- **49 tests for 27 source modules** = thin coverage
49+
- **4 developer tools (LSP, DAP, fmt, lint) with 0 tests** -- tools that developers will use are untested
50+
- **10-level type system with 0 level-specific tests** -- can't verify any level works
51+
- **Idris2 formal core is unverified** -- the proofs exist but nobody tests that they check
52+
53+
## Priority: P0 (CRITICAL)
54+
55+
## FAKE-FUZZ ALERT
56+
57+
- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing
58+
- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file
59+
- Priority: P2 — creates false impression of fuzz coverage

0 commit comments

Comments
 (0)