Skip to content

Commit 6db1302

Browse files
apstenku123apstenku123
committed
docs(v4): real-data flow + remaining UI→train coverage (epic cppmega-mlx-br1)
V3 honest closure audit found 20 remaining un-tested UI→train flows across 3 categories: A. Backend ready, UI not plumbed (parquet/tokenizer selection decorative for training — biggest honesty gap) B. Untested mutations: 9 of 10 ActivationName, 5 of 6 schedules, 3 of 4 LossKind, all 4 rewriters, hybrid optimizer split, side_channels — UI surface exists, math effect never asserted C. Real-world honesty: convergence test uses synthetic targets; inference-after-train never asserted; cross-arch deep skipped activation/norm; gotcha trigger faked via page.route v4 spec: 14 stages, focus on real parquet+tokenizer reaching train, all activation/schedule/loss/rewriter axes through UI→train, and post-train inference output divergence as proof model actually changed. Explicitly out of scope: sharding apply / WS reconnect / memory peak / checkpoint resume (deferred to v5+).
1 parent 82ab1be commit 6db1302

2 files changed

Lines changed: 450 additions & 0 deletions

File tree

‎VisualBuilderPlan-v4.md‎

Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
# VisualBuilderPlan-v4 — Real Data Flow + Remaining UI→Train Coverage
2+
3+
**Status**: planned 2026-05-21 (epic `cppmega-mlx-br1` — id to fill after `bd create`)
4+
**Driver**: honest V3 closure audit found 20 remaining un-tested UI→train flows.
5+
Most damning: backend `stage_train` accepts `parquet_path` (V3-2) but the
6+
DataInspector selection does not forward it, so user-chosen parquet shards
7+
still have zero effect on training. UI Tokenizer Playground is fully decorative
8+
for training. 6 of 10 activations, 5 of 6 schedules, 3 of 4 loss kinds untested
9+
through full UI→train chain.
10+
11+
v4 supersedes v3 §7 "out of scope" entries 1, 2, 5, and adds 11 more.
12+
13+
## 1. Why v4 exists
14+
15+
Three classes of remaining gaps:
16+
17+
### A. Backend ready, UI not plumbed (data-flow honesty)
18+
19+
| ID | Gap | Why bad |
20+
|-----|-----------------------------------------------------------|-------------------------------------------------------------------------------------------|
21+
| G1 | DataInspector parquet → train opts.parquet_path | User selects parquet in Data tab; train silently uses synthetic random targets |
22+
| G2 | Tokenizer Playground tokenizer → train | Tokenizer selection is decorative; train uses raw int IDs from parquet, no tokenize step |
23+
| G3 | UI exposes parquet_path testid for e2e drive | No way for Playwright to set parquet_path through UI walks |
24+
25+
### B. Untested mutations (UI changes but math effect never asserted)
26+
27+
| ID | Gap | Coverage |
28+
|-----|-----------------------------------------------------------|------------|
29+
| G4 | 9 of 10 ActivationName values untested through UI→train | 1/10 |
30+
| G5 | 5 of 6 schedule kinds untested through UI→train | 1/6 |
31+
| G6 | 3 of 4 LossKind values untested through UI→train | 1/4 |
32+
| G7 | RewritersTab apply → train math change | 0/4 |
33+
| G8 | muon_adamw_hybrid split (which params → Muon vs AdamW) | wire only |
34+
| G9 | side_channels (doc_ids, token_ids) reach forward pass | 0 |
35+
36+
### C. Real-world honesty gaps
37+
38+
| ID | Gap |
39+
|-----|-----------------------------------------------------------|
40+
| G10 | Convergence test uses synthetic targets; real-data conv untested |
41+
| G11 | Inference after train: model output differs from initial — never asserted |
42+
| G12 | activation/norm propagation across all 57 presets (V3-11 did 4 × 2) |
43+
| G13 | Real backend gotcha trigger (not page.route fake) — does incompatible loss config actually produce severity=error gotcha? |
44+
| G14 | Tokenizer roundtrip OK actually means byte-exact decode (V3-10 only tested FAIL non-blocking) |
45+
46+
### Explicitly out of scope (defer to v5+)
47+
48+
- WS reconnect mid-train (lifecycle)
49+
- Sharding apply → distributed train (needs multi-device)
50+
- Memory peak vs estimate (needs runtime instrumentation)
51+
- Concurrent Train clicks / Cancel button (lifecycle)
52+
- Spec save/load (V3-12 deferred, keep deferred)
53+
- fp8 / mixed_precision (subset of sharding)
54+
- 100+ layer realistic depth on stage_train (synthetic-test architecture limit)
55+
- Checkpoint save/resume (no stage_train checkpoint code)
56+
57+
## 2. Goal
58+
59+
After v4: for every UI surface that claims to mutate training, an e2e
60+
test reads back the effect via extras or inference output. No silent
61+
decoration left.
62+
63+
## 3. Stages
64+
65+
14 stages in epic `cppmega-mlx-br1`. P0/P1 first; defer P3 if context-limited.
66+
67+
| Stage | Title | Type | Pri | Depends |
68+
|-------|----------------------------------------------------------------------|---------|-----|---------|
69+
| V4-1 | UI: DataInspector forwards parquet_path via stage_options.train | feature | P0 | — |
70+
| V4-2 | Backend: stage_train accepts tokenizer_path; tokenize parquet text | feature | P0 | — |
71+
| V4-3 | UI: Tokenizer Playground selection forwards tokenizer_path | feature | P0 | V4-2 |
72+
| V4-4 | E2E: real-data convergence (parquet+tokenizer → losses fall) | test | P1 | V4-1,3 |
73+
| V4-5 | E2E: all 10 activations × UI→train propagation | test | P1 | — |
74+
| V4-6 | E2E: all 6 schedule kinds × UI→train propagation | test | P1 | — |
75+
| V4-7 | E2E: all 4 LossKind × UI→train + extras | test | P1 | — |
76+
| V4-8 | Backend+E2E: rewriters apply → spec graph change → train extras | test | P1 | — |
77+
| V4-9 | E2E: muon_adamw_hybrid split — extras reports Muon/AdamW buckets | test | P2 | — |
78+
| V4-10 | Backend+E2E: side_channels reach forward → forward output differs | feature | P2 | — |
79+
| V4-11 | E2E: inference after train — output diverges from initial | feature | P2 | V4-1 |
80+
| V4-12 | E2E: cross-arch activation/norm propagation × 12 presets | test | P2 | V4-5 |
81+
| V4-13 | E2E: real config triggers backend gotcha (no page.route fake) | test | P3 | — |
82+
| V4-14 | Closure report `tests/fixtures/e2e_matrix_v4_report.md` | doc | P3 | all |
83+
84+
## 4. Per-stage acceptance criteria
85+
86+
### V4-1 DataInspector parquet path threading
87+
88+
- DataInspector has a "Use for train" button (`data-testid="data-use-for-train"`)
89+
that records the loaded parquet path in App state (`trainParquetPath`).
90+
- App.handleRunPipeline forwards `trainParquetPath` via
91+
`stage_options.train.parquet_path` when set.
92+
- Visible indicator near Train button: `data-testid="train-data-source"`
93+
reads "parquet: <basename>" or "synthetic".
94+
- pytest: 4 new tests (UI state, wire payload, backend receives it,
95+
fallback when null). vitest: 3 new (DataInspector button + App
96+
state + TopBar indicator).
97+
98+
### V4-2 Backend tokenize parquet text
99+
100+
- stage_train opts.tokenizer_path → loads tokenizer via existing
101+
`tokenizer.load_preset` or path-based loader. If parquet has a `text`
102+
column, encode rows; else fall through to V3-2's raw-int path.
103+
- extras gains `tokenizer_used: str | null` (path basename when used).
104+
- 5 new pytest in `test_stage_train_tokenize.py`: with text column,
105+
without, missing tokenizer (fallback), corrupted parquet, vocab clip.
106+
107+
### V4-3 UI tokenizer threading
108+
109+
- Tokenizer Playground "Use for train" button records the loaded
110+
preset name in App state. App forwards via stage_options.train.tokenizer_path.
111+
- TopBar `train-data-source` upgrades to `parquet+tokenizer: <names>`.
112+
- vitest: 3 new (Playground button, App state, TopBar text).
113+
114+
### V4-4 Real-data convergence
115+
116+
- New spec `18_real_data_convergence.spec.ts`. Picks a known-good
117+
parquet fixture from `tests/fixtures/build_e2e_matrix.py`, picks
118+
its matching tokenizer, sets N=8 steps, asserts:
119+
- `extras.data_source === "parquet"`
120+
- `extras.token_count > 0`
121+
- `extras.tokenizer_used` matches selection
122+
- For deep presets (llama3_8b, mistral_small_3_1):
123+
`losses[7] < losses[0] * 0.95` (real data convergence floor — stricter
124+
than synthetic V3-6's tailAvg<headAvg because real tokens give signal).
125+
126+
### V4-5 All 10 activations through UI→train
127+
128+
- New spec `19_activation_propagation.spec.ts`. For each
129+
ActivationName in `["glu", "gelu", "relu", "relu2", "sqrelu", "silu",
130+
"mish", "swiglu", "geglu", "reglu", "xielu"]` (11 entries — the IS_GATED
131+
set varies, BrickContextPanel filters compatible):
132+
- Drop mlp brick to canvas, open BrickContextPanel, set activation,
133+
Apply, Train. Assert `extras.model_summary.mlp_activation === selected`.
134+
- Loss is finite.
135+
136+
### V4-6 All 6 schedules through UI→train
137+
138+
- New spec `20_schedule_propagation.spec.ts`. For each ScheduleKind in
139+
`["constant", "linear_warmup", "cosine", "wsd", "inv_sqrt", "polynomial"]`:
140+
- Open OptimTab, toggle schedule, select kind, fill required fields
141+
(warmup_steps for warmup; total_steps for cosine/polynomial; etc),
142+
Apply, Train. Assert `extras.schedule_kind === selected`.
143+
- Assert `lr_trajectory` shape matches kind's analytical formula
144+
(e.g. cosine → monotone-non-increasing after warmup;
145+
polynomial → power-law decay).
146+
147+
### V4-7 All 4 LossKind through UI→train
148+
149+
- New spec `21_loss_kind_propagation.spec.ts`. For each LossKind in
150+
`["cross_entropy", "mtp_weighted", "ifim_weighted", "mhc_weighted"]`:
151+
- Open LossTab, set kind, set required params (k, betas for MTP),
152+
Apply, Train. Assert losses finite, model_summary contains
153+
`loss_kind` (NEW extras field — V4-7 backend work).
154+
- Skip kinds that have hard preconditions not present in default preset
155+
(e.g. mtp_weighted requires MTPRewriter — included or test selects
156+
a preset that has it).
157+
158+
### V4-8 Rewriters apply → train math change
159+
160+
- New backend: extras.model_summary gains `rewriters_applied: list[str]`.
161+
- New spec `22_rewriter_propagation.spec.ts`. Enable MTPRewriter (or
162+
IFIMRewriter / MHCRewriter), Apply, Train. Assert:
163+
- `extras.model_summary.rewriters_applied` contains the rewriter name.
164+
- For MTP: extras.losses count multi-head loss contribution
165+
(or just verify shape change in some downstream extras key).
166+
167+
### V4-9 muon_adamw_hybrid split
168+
169+
- Backend: when `optimizer_kind === "muon_adamw_hybrid"`, extras gains
170+
`muon_group_size` and `adamw_group_size` (param counts in each bucket).
171+
- New spec `23_hybrid_optimizer_split.spec.ts`. UI selects
172+
muon_adamw_hybrid, Train. Assert both counters > 0, sum equals total.
173+
174+
### V4-10 side_channels reach forward
175+
176+
- Backend: stage_train accepts `opts.side_channels: {doc_ids?, token_ids?}`.
177+
When present, asserts forward pass observes the channels (via a probe
178+
hook). extras gains `side_channels_observed: list[str]`.
179+
- New spec `24_side_channels.spec.ts`. UI enables doc_ids side-channel,
180+
Train. Assert `extras.side_channels_observed` includes `"doc_ids"`.
181+
182+
### V4-11 Inference after train — output diverges
183+
184+
- Backend: extras gains `inference_probe`: result of a single
185+
forward pass over a fixed seed both before training (initial weights)
186+
and after. Returned as `{l2_diff: float, cos_sim: float}`.
187+
- New spec `25_inference_after_train.spec.ts`. Run Train. Assert
188+
`extras.inference_probe.l2_diff > 0.01` (model actually changed
189+
observable output, not just optimizer state).
190+
191+
### V4-12 Cross-arch activation/norm × 12 presets
192+
193+
- New spec `26_cross_arch_brick_mutations.spec.ts`. 12 presets × 2
194+
brick-context mutations (activation + pre_norm) = 24 cells. Each
195+
scenario clicks the preset-suffixed brick node, opens the
196+
BrickContextPanel for its mlp/attention, mutates, runs Train, asserts
197+
propagation via model_summary.
198+
- Solves the V3-11 testid gap by introspecting actual `data-testid`
199+
values (`brick-context-*`) at runtime instead of hardcoding.
200+
201+
### V4-13 Real backend gotcha trigger
202+
203+
- New spec `27_real_gotcha.spec.ts`. Set up an actually-incompatible
204+
config (e.g. MTPRewriter without enough head outputs, OR pre_norm=none
205+
+ post_norm=none on attention). Wait for verify. Assert
206+
`top-bar-train-disabled-reason` appears WITHOUT page.route() shim.
207+
- Proves V3-8/V3-9 gating logic catches real-world gotchas, not just
208+
injected ones.
209+
210+
### V4-14 Closure report
211+
212+
- ≤150-line markdown mirroring v3 layout. Stage table, gaps closed,
213+
remaining v5+ items, regression totals.
214+
215+
## 5. Workflow per stage (Goal directive)
216+
217+
Same as v3: claim → implement → review → perf → regression → commit
218+
specific files → push → close bd ticket.
219+
220+
## 6. Acceptance counters
221+
222+
| Surface | Before v4 | Target v4 |
223+
|----------------------------------------|-----------|-----------|
224+
| pytest | ~2330 | ≥2350 |
225+
| vitest | 166 | ≥176 |
226+
| Playwright deep e2e (scenarios 11-17) | 26 | ≥60 |
227+
| Activations through UI→train | 1 | 10 |
228+
| Schedules through UI→train | 1 | 6 |
229+
| LossKinds through UI→train | 1 | 4 |
230+
| stage_train extras keys | 9 | 13 |
231+
| Real-data train scenarios | 0 | 1+ |
232+
| Inference-after-train scenarios | 0 | 1+ |
233+
234+
## 7. Done definition
235+
236+
All P0+P1+P2 tickets closed (P3 may defer). Real parquet+tokenizer
237+
selection in UI demonstrably reaches train. 10 activations + 6 schedules
238+
+ 4 loss kinds all proven through UI→train. Inference output asserted
239+
post-train. Closure doc at `tests/fixtures/e2e_matrix_v4_report.md`.

0 commit comments

Comments
 (0)