Skip to content

Commit 73a1c25

Browse files
hyperpolymathclaude
andcommitted
Remove template-only ABI files that falsely implied formal verification
Template Idris2 ABI files (Types.idr, Layout.idr, Foreign.idr) contained only RSR template scaffolding with unresolved placeholders and no domain-specific proofs. Removed to prevent false impression of formal verification coverage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent ac52ca6 commit 73a1c25

12 files changed

Lines changed: 1254 additions & 1869 deletions

File tree

PROOF-NEEDS.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Proof Requirements
2+
3+
## Current state
4+
- ABI directory exists (template-level)
5+
- No dangerous patterns found
6+
- 100K lines; bot fleet for repository quality enforcement
7+
- Claims: "formal mathematical/statistical verification"
8+
9+
## What needs proving
10+
- **Bot action safety**: Prove that automated bot actions (PRs, issues, fixes) are reversible and do not destroy repository state
11+
- **Confidence threshold correctness**: Prove that the robot-repo-automaton confidence thresholds correctly gate automated merges (no auto-merge below threshold)
12+
- **Rule conflict resolution**: Prove that when multiple bots (rhodibot, echidnabot, sustainabot, etc.) produce conflicting recommendations, the resolution is deterministic and documented
13+
- **Rate limiting**: Prove bots respect GitHub API rate limits and do not cause cascading failures across the fleet
14+
15+
## Recommended prover
16+
- **Idris2** — For bot action safety and confidence threshold properties
17+
- **Lean4** — For algebraic properties of confidence scoring if modeled as an ordered semiring
18+
19+
## Priority
20+
- **MEDIUM** — The fleet manages automated repository changes. Incorrect bot behavior at scale could damage many repos. The "formal verification" claim should be substantiated.
21+
22+
## Template ABI Cleanup (2026-03-29)
23+
24+
Template ABI removed -- was creating false impression of formal verification.
25+
The removed files (Types.idr, Layout.idr, Foreign.idr) contained only RSR template
26+
scaffolding with unresolved {{PROJECT}}/{{AUTHOR}} placeholders and no domain-specific proofs.

TEST-NEEDS.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Test & Benchmark Requirements
2+
3+
## Current State
4+
- Unit tests: ~22 Rust test files + shared-context tests — count unknown (no Cargo.toml at root to run)
5+
- Integration tests: partial (echidnabot/tests/integration_tests.rs, rhodibot/tests/integration_tests.rs)
6+
- E2E tests: NONE
7+
- Benchmarks: 5 files exist
8+
- panic-attack scan: NEVER RUN
9+
10+
## What's Missing
11+
### Point-to-Point (P2P)
12+
205 Rust + 367 JavaScript + 42 ReScript + 34 Julia + 13 Haskell + 9 Idris2 + 85 Shell source files:
13+
14+
#### Bots (major components):
15+
- **echidnabot/** — has integration tests, but coverage vs source count unclear
16+
- **rhodibot/** — has integration tests
17+
- **sustainabot/** — has test_sample.rs + .env.test, minimal coverage
18+
- **finishbot/** — has analyzers/testing.rs (test utilities, not tests OF finishbot)
19+
- **glambot/** — no tests found
20+
- **seambot/** — no tests found
21+
- **panicbot/** — no tests found
22+
23+
#### Shared Context (Rust):
24+
- shared-context/tests/context_tests.rs — exists
25+
- shared-context/tests/fleet_coordination_test.rs — exists
26+
- Other shared modules may lack coverage
27+
28+
#### Robot-repo-automaton:
29+
- No test files found
30+
31+
#### Dashboard:
32+
- No test files found
33+
34+
#### Campaigns:
35+
- No test files found
36+
37+
#### Hooks:
38+
- No test files found
39+
40+
#### Tasks/Scripts:
41+
- 85 shell scripts — no tests
42+
- 367 JavaScript files — no tests (these are likely bot action scripts)
43+
- 34 Julia files — no tests
44+
45+
### End-to-End (E2E)
46+
- Bot fleet coordination: dispatch task -> bot processes -> report results -> aggregate
47+
- Individual bot lifecycle: start -> receive event -> analyze -> act -> report
48+
- echidnabot: receive webhook -> analyze repo -> generate findings -> submit
49+
- rhodibot: detect issue -> apply fix -> create PR
50+
- sustainabot: check dependencies -> evaluate health -> report
51+
- finishbot: analyze completion -> identify gaps -> notify
52+
- glambot: check style -> suggest fixes
53+
- seambot: check integration points -> verify
54+
- panicbot: run security scan -> report findings
55+
- Robot-repo-automaton: detect issue -> calculate confidence -> auto-fix
56+
- Fleet coordination: multiple bots on same repo -> deconflict
57+
58+
### Aspect Tests
59+
- [ ] Security (webhook signature verification, bot credential handling, repo access scoping, auto-fix safety)
60+
- [ ] Performance (fleet throughput, bot startup latency, concurrent repo processing)
61+
- [ ] Concurrency (bot deconfliction, shared-context locking, parallel webhook processing)
62+
- [ ] Error handling (bot crash recovery, API rate limiting, malformed webhooks)
63+
- [ ] Accessibility (dashboard UI if applicable)
64+
65+
### Build & Execution
66+
- [ ] cargo build per bot — not verified
67+
- [ ] Individual bot startup — not verified
68+
- [ ] Fleet orchestration — not verified
69+
- [ ] Dashboard build — not verified
70+
- [ ] Self-diagnostic — none
71+
72+
### Benchmarks Needed
73+
- Per-bot analysis throughput (repos/hour)
74+
- Fleet coordination overhead
75+
- Webhook processing latency
76+
- Auto-fix confidence score accuracy
77+
- Memory usage per active bot
78+
79+
### Self-Tests
80+
- [ ] panic-attack assail on own repo
81+
- [ ] Fleet health check
82+
- [ ] Per-bot self-test
83+
- [ ] Shared-context integrity verification
84+
85+
## Priority
86+
- **HIGH** — Bot fleet (205 Rust + 367 JS + 42 ReScript + 34 Julia files) with tests only for echidnabot, rhodibot, and shared-context. 5 out of 7+ bots have ZERO tests. The 367 JavaScript files (bot action scripts) are completely untested. The robot-repo-automaton that auto-fixes repos based on confidence scores has no tests — this is a system that makes automated changes to other repos and needs extremely high correctness guarantees.
87+
88+
## FAKE-FUZZ ALERT
89+
90+
- `tests/fuzz/placeholder.txt` is a scorecard placeholder inherited from rsr-template-repo — it does NOT provide real fuzz testing
91+
- Replace with an actual fuzz harness (see rsr-template-repo/tests/fuzz/README.adoc) or remove the file
92+
- Priority: P2 — creates false impression of fuzz coverage

bots/echidnabot/src/abi/Foreign.idr

Lines changed: 0 additions & 216 deletions
This file was deleted.

0 commit comments

Comments
 (0)