|
| 1 | +# SPDX-License-Identifier: PMPL-1.0-or-later |
| 2 | +# Evaluation: v3 Cross-Project Consistency Test |
| 3 | +# Date: 2026-03-23 |
| 4 | +# Projects: ambientops (infra), burble (platform), proven (research) |
| 5 | + |
| 6 | +## Question |
| 7 | + |
| 8 | +Does v3 methodology produce consistent results across different project |
| 9 | +types, and does it correctly adapt its mode (convergent/divergent/hybrid)? |
| 10 | + |
| 11 | +## Results |
| 12 | + |
| 13 | +| Dimension | ambientops | burble | proven | |
| 14 | +|-----------|-----------|--------|--------| |
| 15 | +| **Type** | Infrastructure | Platform | Research | |
| 16 | +| **Mode selected** | Convergent | Convergent | Divergent | |
| 17 | +| **Mode correct?** | Yes | Yes | Yes | |
| 18 | +| **Constraint found?** | Yes (envelope flow) | Yes (missing deps) | Yes (build unverified) | |
| 19 | +| **Constraint type** | Wiring gap | Dependency gap | Pipeline gap | |
| 20 | +| **Phase 0 worked?** | Yes (STATE valid) | Partial (STATE = template) | Partial (STATE empty) | |
| 21 | +| **v3 quality score** | 8/10 | 7/10 | 8/10 | |
| 22 | + |
| 23 | +## Key Findings |
| 24 | + |
| 25 | +### 1. Constraint identification is consistent (3/3) |
| 26 | + |
| 27 | +All three projects had a critical chain with a clear first-unresolved-link. |
| 28 | +The constraint types varied (wiring, deps, pipeline) but the identification |
| 29 | +method (build dependency graph → find longest chain → find first break) |
| 30 | +worked identically across all three. |
| 31 | + |
| 32 | +### 2. Mode selection is correct (3/3) |
| 33 | + |
| 34 | +v3 correctly chose convergent for infrastructure/platform and divergent |
| 35 | +for research. The signals were clear in all cases. |
| 36 | + |
| 37 | +### 3. Phase 0 has a STATE file reliability problem (2/3 failures) |
| 38 | + |
| 39 | +Only ambientops had a valid STATE.a2ml. Burble's was an uncustomised |
| 40 | +template. Proven's was empty after SCM→A2ML migration. This means Phase 0 |
| 41 | +falls back to TODO.md/ROADMAP.adoc in most cases. |
| 42 | + |
| 43 | +**New refinement needed:** Validate STATE before trusting it: |
| 44 | +- Check for {{PLACEHOLDER}} tokens |
| 45 | +- Check project name matches repo name |
| 46 | +- Check last-updated within 90 days |
| 47 | +- Fall back to TODO/ROADMAP if validation fails |
| 48 | + |
| 49 | +### 4. Convergent gravity in divergent mode (proven-specific) |
| 50 | + |
| 51 | +The coverage audit step naturally pulls attention toward empty components |
| 52 | +(102 scaffolded bindings, 12 stub apps). In convergent mode this is |
| 53 | +correct (fill gaps). In divergent mode this is wrong (deepening > breadth). |
| 54 | +The 70/30 budget split mitigates but doesn't eliminate the pull. |
| 55 | + |
| 56 | +**New refinement needed:** In divergent mode, the coverage audit should |
| 57 | +ask "which components carry the unique strength?" not "which components |
| 58 | +are empty?" Empty components are acceptable in divergent projects. |
| 59 | + |
| 60 | +### 5. "Silent fallback" is a new constraint pattern (burble-specific) |
| 61 | + |
| 62 | +Burble had fully-written QUIC and LMDB code that silently fell back to |
| 63 | +WebSocket/ETS because the deps weren't in mix.exs. This is invisible to |
| 64 | +code review — the code looks complete. Only the constraint analysis |
| 65 | +(tracing the dependency graph to find what's actually wired) revealed it. |
| 66 | + |
| 67 | +This pattern may exist in other projects: code that looks done but is |
| 68 | +secretly using a fallback path because a dependency/config/flag is missing. |
| 69 | + |
| 70 | +## Consistency Verdict |
| 71 | + |
| 72 | +**v3 is consistent across project types.** The core mechanism (read state → |
| 73 | +build dependency graph → find constraint → work it) produces the right |
| 74 | +priorities in infrastructure, platform, and research contexts. The mode |
| 75 | +selection (convergent vs divergent) correctly adapts. |
| 76 | + |
| 77 | +The two weak points (STATE reliability, convergent gravity in divergent mode) |
| 78 | +are addressable with refinements #25 and #26 to the methodology. |
| 79 | + |
| 80 | +## What v3 Works Better For |
| 81 | + |
| 82 | +| Situation | v3 Advantage | |
| 83 | +|-----------|-------------| |
| 84 | +| Infrastructure with hidden wiring gaps | Strong — constraint analysis finds silent failures | |
| 85 | +| Platform with dependency issues | Strong — traces dep graph to find missing links | |
| 86 | +| Research with deep proof needs | Good — divergent mode focuses on unique strength | |
| 87 | +| Multi-component monorepos | Strong — coverage audit catches skipped MUSTs | |
| 88 | +| Projects with valid STATE files | Ideal — Phase 0 works perfectly | |
| 89 | + |
| 90 | +## What v3 Works Less Well For |
| 91 | + |
| 92 | +| Situation | Issue | |
| 93 | +|-----------|-------| |
| 94 | +| Projects with broken/template STATE | Phase 0 starts with bad data | |
| 95 | +| Divergent projects with many stubs | Coverage audit tempts convergent filling | |
| 96 | +| Very small projects (1-3 files) | Overhead not justified | |
| 97 | +| Exploratory/greenfield work | No state to read, no constraint to find | |
0 commit comments