Skip to content

Commit 516d3cd

Browse files
hyperpolymathclaude
andcommitted
docs: update evaluation with final v3 numbers — 70% token reduction confirmed
v3 completed: 89K tokens (vs 296K for v2), 2 bots (vs 5), 100% component visibility (vs 21%), 12 skipped MUSTs found including a P0 crash loop. Added tables for what each version found that the other wouldn't have. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 9338e85 commit 516d3cd

1 file changed

Lines changed: 30 additions & 5 deletions

File tree

evaluations/productive-meandering-v0-v3.md

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -71,21 +71,46 @@ making this a genuine A/B comparison. v0 and v1 data is from prior sessions.
7171
| **Completion guarantee** | Yes (trivially) | No (random walk) | Yes (constraint resolution = done) |
7272
| **Right work guarantee** | Only if user knows what's right | No (interesting > important) | Yes (critical chain enforces) |
7373

74-
## Headline Numbers (v2 vs v3)
74+
## Headline Numbers (v2 vs v3) — FINAL
7575

7676
| Metric | v2 | v3 | Change |
7777
|--------|----|----|--------|
7878
| Bots launched | 5 | 2 | **-60%** |
79-
| Tokens (est.) | 296K | ~170K | **-43%** |
79+
| Tokens (actual) | 296K | 89K | **-70%** |
80+
| Tool calls | 159 | 73 | **-54%** |
81+
| Files touched | 23 | 4 | v2 more output |
8082
| Critical chain unblocked | accidentally (wave 2) | by design (wave 1) | **1 wave faster** |
81-
| Components with coverage visibility | 6 | 29 | **+383%** |
83+
| Components with coverage visibility | 6/29 (21%) | 29/29 (100%) | **+383%** |
84+
| Skipped MUSTs detected | 0 | 12 (incl. 1 P0, 3 P1) | **v3 found what v2 missed** |
8285
| Perfective work (waste) | ~30% of effort | 0% | **eliminated** |
8386
| Constraint identified | never (found by accident) | Phase 0, before any bot launched | **systematic** |
87+
| Cherry-picking confirmed | yes (by v3 audit) | no (matrix enforced) | **structural fix** |
88+
| Schema violations fixed | 0 | 7 | **v3 found the real work** |
89+
90+
### What v3 found that v2 missed (skipped MUSTs)
91+
92+
| Priority | Component | Issue |
93+
|----------|-----------|-------|
94+
| P0 | session-sentinel | 56 SIGABRTs/4 days, disabled, D-Bus race |
95+
| P1 | nano-aider | Wrong author email (gmail.com) |
96+
| P1 | panoptes | MIT license, should be PMPL |
97+
| P1 | cicada | Stale "Palimpsest v0.4" naming |
98+
| P2 | total-recall | Uncustomised RSR template placeholder |
99+
| P2 | total-update | Same — {{PROJECT}} throughout |
100+
| P2 | reasonably-good-token-vault | Missing Idris2 ABI layer |
101+
102+
### What v2 found that v3 wouldn't have
103+
104+
| Finding | Value |
105+
|---------|-------|
106+
| Duplicate observatory in system-tools/ | Structural (prevents future confusion) |
107+
| Stale Python in stale/ | Cleanup (removes banned code) |
108+
| TOPOLOGY.md missing 25+ components | Documentation (onboarding) |
109+
| broad-spectrum is a real ReScript project | Discovery (unknown component found) |
84110

85111
## Honest Caveats
86112

87-
1. **v3 hasn't fully completed at time of writing** — final numbers may differ.
88-
This evaluation will be updated when the v3 bots finish.
113+
1. **v3 completed and final numbers are in.** Updated 2026-03-23.
89114

90115
2. **v3 benefits from v2's work.** HCT compiles because v2 fixed main.rs. The
91116
constraint analysis was easier because v2 built the components. v3 on a fresh

0 commit comments

Comments
 (0)