Skip to content

Commit 3638eb5

Browse files
hyperpolymathclaude
andcommitted
docs: add cross-project v3 consistency evaluation (3 project types)
Tested v3 on ambientops (infra), burble (platform), proven (research). Consistent constraint identification across all 3. Mode selection correct in all cases. Two gaps found: STATE file reliability (2/3 broken) and convergent gravity in divergent mode. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 042cc64 commit 3638eb5

1 file changed

Lines changed: 97 additions & 0 deletions

File tree

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# SPDX-License-Identifier: PMPL-1.0-or-later
2+
# Evaluation: v3 Cross-Project Consistency Test
3+
# Date: 2026-03-23
4+
# Projects: ambientops (infra), burble (platform), proven (research)
5+
6+
## Question
7+
8+
Does v3 methodology produce consistent results across different project
9+
types, and does it correctly adapt its mode (convergent/divergent/hybrid)?
10+
11+
## Results
12+
13+
| Dimension | ambientops | burble | proven |
14+
|-----------|-----------|--------|--------|
15+
| **Type** | Infrastructure | Platform | Research |
16+
| **Mode selected** | Convergent | Convergent | Divergent |
17+
| **Mode correct?** | Yes | Yes | Yes |
18+
| **Constraint found?** | Yes (envelope flow) | Yes (missing deps) | Yes (build unverified) |
19+
| **Constraint type** | Wiring gap | Dependency gap | Pipeline gap |
20+
| **Phase 0 worked?** | Yes (STATE valid) | Partial (STATE = template) | Partial (STATE empty) |
21+
| **v3 quality score** | 8/10 | 7/10 | 8/10 |
22+
23+
## Key Findings
24+
25+
### 1. Constraint identification is consistent (3/3)
26+
27+
All three projects had a critical chain with a clear first-unresolved-link.
28+
The constraint types varied (wiring, deps, pipeline) but the identification
29+
method (build dependency graph → find longest chain → find first break)
30+
worked identically across all three.
31+
32+
### 2. Mode selection is correct (3/3)
33+
34+
v3 correctly chose convergent for infrastructure/platform and divergent
35+
for research. The signals were clear in all cases.
36+
37+
### 3. Phase 0 has a STATE file reliability problem (2/3 failures)
38+
39+
Only ambientops had a valid STATE.a2ml. Burble's was an uncustomised
40+
template. Proven's was empty after SCM→A2ML migration. This means Phase 0
41+
falls back to TODO.md/ROADMAP.adoc in most cases.
42+
43+
**New refinement needed:** Validate STATE before trusting it:
44+
- Check for {{PLACEHOLDER}} tokens
45+
- Check project name matches repo name
46+
- Check last-updated within 90 days
47+
- Fall back to TODO/ROADMAP if validation fails
48+
49+
### 4. Convergent gravity in divergent mode (proven-specific)
50+
51+
The coverage audit step naturally pulls attention toward empty components
52+
(102 scaffolded bindings, 12 stub apps). In convergent mode this is
53+
correct (fill gaps). In divergent mode this is wrong (deepening > breadth).
54+
The 70/30 budget split mitigates but doesn't eliminate the pull.
55+
56+
**New refinement needed:** In divergent mode, the coverage audit should
57+
ask "which components carry the unique strength?" not "which components
58+
are empty?" Empty components are acceptable in divergent projects.
59+
60+
### 5. "Silent fallback" is a new constraint pattern (burble-specific)
61+
62+
Burble had fully-written QUIC and LMDB code that silently fell back to
63+
WebSocket/ETS because the deps weren't in mix.exs. This is invisible to
64+
code review — the code looks complete. Only the constraint analysis
65+
(tracing the dependency graph to find what's actually wired) revealed it.
66+
67+
This pattern may exist in other projects: code that looks done but is
68+
secretly using a fallback path because a dependency/config/flag is missing.
69+
70+
## Consistency Verdict
71+
72+
**v3 is consistent across project types.** The core mechanism (read state →
73+
build dependency graph → find constraint → work it) produces the right
74+
priorities in infrastructure, platform, and research contexts. The mode
75+
selection (convergent vs divergent) correctly adapts.
76+
77+
The two weak points (STATE reliability, convergent gravity in divergent mode)
78+
are addressable with refinements #25 and #26 to the methodology.
79+
80+
## What v3 Works Better For
81+
82+
| Situation | v3 Advantage |
83+
|-----------|-------------|
84+
| Infrastructure with hidden wiring gaps | Strong — constraint analysis finds silent failures |
85+
| Platform with dependency issues | Strong — traces dep graph to find missing links |
86+
| Research with deep proof needs | Good — divergent mode focuses on unique strength |
87+
| Multi-component monorepos | Strong — coverage audit catches skipped MUSTs |
88+
| Projects with valid STATE files | Ideal — Phase 0 works perfectly |
89+
90+
## What v3 Works Less Well For
91+
92+
| Situation | Issue |
93+
|-----------|-------|
94+
| Projects with broken/template STATE | Phase 0 starts with bad data |
95+
| Divergent projects with many stubs | Coverage audit tempts convergent filling |
96+
| Very small projects (1-3 files) | Overhead not justified |
97+
| Exploratory/greenfield work | No state to read, no constraint to find |

0 commit comments

Comments
 (0)