|
| 1 | +# Audit Report — valence-shell (2026-04-04) |
| 2 | + |
| 3 | +## Summary |
| 4 | + |
| 5 | +Valence Shell (vsh) is an advanced research prototype (v0.9.0) of a formally verified shell with proven reversibility guarantees. The codebase exhibits mature RSR compliance, comprehensive CI automation, and disciplined handling of dangerous patterns. However, it is explicitly NOT production-ready—the Lean4 correspondence proofs are ~85% confidence (property-tested only, not mechanized), and several high-priority gaps remain. |
| 6 | + |
| 7 | +## Findings |
| 8 | + |
| 9 | +### Critical |
| 10 | + |
| 11 | +1. **Lean->Rust correspondence unproven** — Testing-based validation only (~85% confidence), no mechanized correspondence proof |
| 12 | +2. **4 proof gaps documented** — PROOF_HOLES_AUDIT.md lists completion gaps across 8 proof files (4 genuine gaps, 4 axioms, 2 structural) |
| 13 | +3. **NOT production-ready** — Explicit disclaimer in CLAUDE.md, README, and docs/INTEGRATION.md |
| 14 | +4. **Dead code detected** — lean_ffi.rs (library doesn't exist), daemon_client.rs (no daemon), multiple stale job control methods |
| 15 | + |
| 16 | +### High |
| 17 | + |
| 18 | +1. **47/58 commits author-spoofed as "Test <test@example.com>"** — Metadata integrity compromised; commits authored by Sonnet LLM without proper attribution |
| 19 | +2. **Version inflation in commit history** — Previous versions claimed 1.0.0 / "production-ready" / "256 proofs" (2026-03-08 correction brought to 0.9.0 / "200+ theorems") |
| 20 | +3. **No mechanized Echidna integration** — Formal property validation pipeline not automated |
| 21 | + |
| 22 | +### Medium |
| 23 | + |
| 24 | +1. **POSIX compliance incomplete** — Functions, script execution, word splitting ($IFS), tilde expansion, trap, alias still missing |
| 25 | +2. **GDPR compliance stubs** — RMO (Remove-Match-Obliterate) for secure deletion not implemented |
| 26 | +3. **Elixir NIF broken** — Reference implementation build broken (low priority, slated for removal) |
| 27 | +4. **10 proof holes** — Mix of genuine gaps, axioms, structural issues (see PROOF_HOLES_AUDIT.md) |
| 28 | + |
| 29 | +## RSR Compliance |
| 30 | + |
| 31 | +- **EXPLAINME.adoc**: present ✓ |
| 32 | +- **0-AI-MANIFEST.a2ml**: present ✓ |
| 33 | +- **.machine_readable/**: present ✓ |
| 34 | +- **SECURITY.md**: present ✓ |
| 35 | +- **CONTRIBUTING.md**: present ✓ |
| 36 | + |
| 37 | +## Test Coverage |
| 38 | + |
| 39 | +**Comprehensive test suite (602 tests passing, 0 failures):** |
| 40 | + |
| 41 | +- Unit tests (lib): 277 |
| 42 | +- Correspondence (Lean 4): 28 |
| 43 | +- Extended features: 55 |
| 44 | +- Integration tests: 35 |
| 45 | +- Integration (extra): 10 |
| 46 | +- Lean4 proptest: 16 |
| 47 | +- Parameter expansion: 67 |
| 48 | +- Property correspondence: 15 |
| 49 | +- Property tests: 28 |
| 50 | +- Security tests: 15 |
| 51 | +- Doctests: 52 |
| 52 | +- **Total**: 602 passing, 0 failures |
| 53 | +- Ignored (stress): 14 (run manually) |
| 54 | + |
| 55 | +**Test files:** |
| 56 | +- impl/rust-cli/tests/ — 9 integration test suites |
| 57 | +- impl/elixir/test/ — test_helper.exs (stale) |
| 58 | +- Comprehensive property-based testing via proptest |
| 59 | + |
| 60 | +## Proof Debt |
| 61 | + |
| 62 | +**Status: SIGNIFICANT DEBT, ACTIVELY MANAGED** |
| 63 | + |
| 64 | +From PROOF_HOLES_AUDIT.md: |
| 65 | +- 4 genuine proof gaps (need completion work) |
| 66 | +- 4 axioms (assumed truths, not proven) |
| 67 | +- 2 structural issues (architectural, not logical) |
| 68 | +- 10 holes total across 8 files |
| 69 | + |
| 70 | +Dangerous pattern status: |
| 71 | +- believe_me: 0 actual uses detected ✓ |
| 72 | +- assert_total: 0 actual uses detected ✓ |
| 73 | +- Admitted (Coq): ~10 remaining (see INTEGRATION.md, lines 163-174) |
| 74 | +- sorry: 0 actual uses detected ✓ |
| 75 | +- unsafeCoerce: 0 Haskell uses detected ✓ |
| 76 | +- Obj.magic: 24 calls REPLACED in ReScript (eliminated 2026-03-13, annotated 1 for MCP JSON cast) |
| 77 | + |
| 78 | +Proof integrity enforced via: |
| 79 | +- .bot_directives/echidnabot.scm (bans Admitted at severity "critical") |
| 80 | +- CI validation workflow (echidna-validation.yml) |
| 81 | +- CONTRIBUTING.md (line 283): "Avoid admit/Admitted in main branch" |
| 82 | +- Machine-readable policies (MUST.contractile) |
| 83 | + |
| 84 | +## Publication Safety |
| 85 | + |
| 86 | +**Current claim**: "Formally verified shell with proven reversibility guarantees" |
| 87 | + |
| 88 | +**Evidence gap**: Lean4→Rust correspondence is property-tested (~85% confidence) but NOT mechanized. Claims should be downscoped to: |
| 89 | +- "Advanced research prototype with formal reversibility theorems in Lean4" |
| 90 | +- "Property-tested Rust implementation (85% confidence)" |
| 91 | +- "NOT production-ready for security-critical systems" |
| 92 | + |
| 93 | +The shell **functionally works** (602 tests pass), but the **mathematical proof story is incomplete**. |
| 94 | + |
| 95 | +## CI Health |
| 96 | + |
| 97 | +**Status: COMPREHENSIVE, WELL-MAINTAINED** |
| 98 | + |
| 99 | +Active workflows: |
| 100 | +- echidna-validation.yml — Formal proof validation |
| 101 | +- cflite_pr.yml / cflite_batch.yml — Coq/Lean/Agda/Isabelle/Mizar/Z3 validation |
| 102 | +- ts-blocker.yml, npm-bun-blocker.yml — Language policy enforcement |
| 103 | +- workflow-linter.yml — CI integrity |
| 104 | +- instant-sync.yml — Git mirroring |
| 105 | + |
| 106 | +**No obvious failures** detected in workflow definitions. SHA pins verified. No banned languages in CI. |
| 107 | + |
| 108 | +## Verdict |
| 109 | + |
| 110 | +**AFTER REPAIR: Plan B (Conditional Publication)** |
| 111 | + |
| 112 | +This is sophisticated research-grade code that deserves publication—BUT ONLY with corrected claims: |
| 113 | + |
| 114 | +**Required repairs:** |
| 115 | +1. Downscope claims from "formally verified" to "formally specified" (Lean4) with "property-tested implementation" (Rust) |
| 116 | +2. Fix commit author metadata (re-sign 47 commits attributed to "Test <test@example.com>") |
| 117 | +3. Document 4 proof gaps explicitly in README (not hidden in PROOF_HOLES_AUDIT.md) |
| 118 | +4. Implement Echidna CI integration (currently manual/optional) |
| 119 | +5. Remove dead code (lean_ffi.rs, daemon_client.rs, unused job control methods) |
| 120 | + |
| 121 | +**If repairs completed**: PUBLISHABLE as "Advanced Research Prototype — v0.9.0" |
| 122 | + |
| 123 | +**Do not publish as-is** — the version number, feature claims, and proof gap documentation need alignment first. |
| 124 | + |
| 125 | +--- |
| 126 | + |
| 127 | +**Audited by**: M2 estate audit (2026-04-04) |
| 128 | +**Previous audit**: 2026-02-12 (Opus) |
| 129 | +**Confidence**: HIGH (repairs are well-scoped) |
| 130 | +**Recommendation**: REPAIR THEN PUBLISH |
0 commit comments