Skip to content

Commit 921c3bb

Browse files
apstenku123apstenku123
committed
docs(matrix): goal #1 fully characterized with measured numbers (seq-scale ramp)
Real gb10 bf16-adamw tok/s across the feasible envelope: - path_b: bs1/seq512=156, bs2/seq512=259, bs1/seq1024=191 (scales w/ batch AND seq); OOM at seq4096. - path_c: 117 tok/s ONLY at bs1/seq512; BLOCKED at bs>1 OR seq>512 (direct_fusion_chain_logical_buffers_missing — runtime built for one shape). - MLX OOMs at seq4096 (121G) — root cause moe.py:214 dense 16-expert loop + O(n^2) reference sparse-MLA; C++ does bs4/seq4096 in ~26G (sparse alltoall MoE). The literal bs4/seq4096 like-for-like is unreachable for two pinpointed code walls; the achievable envelope is measured. Receipt: cppmega_1b_seqscale_gb10_b1s1024_results.json.
1 parent fc0f75a commit 921c3bb

2 files changed

Lines changed: 75 additions & 8 deletions

File tree

‎reports/MATRIX-REVIEW-INDEX.md‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -43,14 +43,19 @@ gb10 1B, flag-OFF (serial `path_c`) → flag-ON (`path_c_chunked`, `CPPMEGA_PATH
4343

4444
## Fair comparison (goal #1) — RESOLVED with measured evidence
4545

46-
The like-for-like **batch=4×seq=4096** comparison the goal asked for is **provably not achievable in MLX-eager on this model** — it OOMs. Measured on gb10 (121 GB unified), memory-guarded ramp:
47-
48-
| config | result (gb10, bf16) |
49-
| --- | --- |
50-
| bs=1 × seq=512 (baseline) | path_b/path_c/path_c_chunked all run (full 54-cell matrix) |
51-
| **bs=2 × seq=512** | path_b **scales** (muon 92→159, adamw 156→259 tok/s); **path_c BLOCKED** — `direct_fusion_chain_training_runtime_missing` (Path-C direct-chain runtime is **bs=1-only** on CUDA) |
52-
| **bs=1 × seq=4096** (seq-matched to C++) | **OOM** — memory spiked 22→**121 GB** instantly, guard SIGTERM'd. MLX-eager cannot reach seq=4096 **even at batch=1** |
53-
| bs=4 × seq=4096 (the literal goal config) | **OOM** (≥100 GB), never completes |
46+
The like-for-like **batch=4×seq=4096** comparison the goal asked for is **not achievable in MLX-eager on this model** — measured, memory-guarded ramp on gb10 (121 GB unified), bf16 adamw, real tok/s:
47+
48+
| config | path_b | path_c | peak mem | note |
49+
| --- | --- | --- | --- | --- |
50+
| bs=1 × seq=512 | 156 tok/s | **117 tok/s** ✅ | ~22 GB | the one shape path_c runs |
51+
| bs=2 × seq=512 | **259** ✅ | blocked | — | path_b scales w/ batch |
52+
| bs=1 × seq=1024 | **191** ✅ | blocked | 33 GB (path_b) | path_b scales w/ seq |
53+
| bs=1 × seq=4096 | **OOM (121 GB)** | OOM | — | dense MoE blows up |
54+
| bs=4 × seq=4096 (literal goal) | **OOM** | n/a | — | unreachable |
55+
56+
(muon mirrors adamw: bs1→bs2 path_b 92→159.) Two independent, code-pinpointed walls:
57+
- **path_c (CUDA direct-chain) is pinned to exactly bs=1×seq=512** — it blocks at bs>1 *or* seq>512 with `direct_fusion_chain_logical_buffers_missing` (the fused direct-chain runtime was built for that one shape). So *steady-state path_c-vs-path_b at scale is unmeasurable* — path_c doesn't run off its build shape. **path_b runs everywhere and scales** with both batch (156→259) and seq (156→191).
58+
- **MLX-eager OOMs at seq=4096 even at bs=1** (121 GB) — root cause pinpointed in code: `cppmega_mlx/nn/moe.py:214` `ReferenceMoE` runs a **dense loop over all 16 experts on all tokens** (docstring: *"Dense ... reference suitable for smoke tests"*) + gather O(n²) reference sparse-MLA. C++/Megatron does the same bs=4×seq=4096 in **~26 GB** via sparse all-to-all MoE. **That ~5× memory ratio IS the concrete MLX-vs-C++ finding.**
5459

5560
**Why MLX OOMs where C++ doesn't:** C++/Megatron does bs=4×seq=4096 in **~26 GB** (3700 tok/s) using **sparse all-to-all MoE**; the MLX path uses a **dense `ReferenceMoE`** that materializes all 16 experts (O(seq×experts)) plus gather-based O(n²)-memory reference sparse-MLA. So MLX-eager's memory blows up with seq, capping reachable seq well below 4096. **This is the concrete MLX-vs-C++ finding** — not a tuning gap but a different (reference vs production) MoE/attention implementation.
5661

‎reports/cppmega_1b_seqscale_gb10_b1s1024_results.json‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
[
2+
{
3+
"status": "ok",
4+
"tok_sec": 191.3761949718963,
5+
"step_sec": 0.1983322736306009,
6+
"compile_s": 11.78363276598975,
7+
"peak_gb": 32.57734424620867,
8+
"loss_check": true,
9+
"initial_loss": 11.289926528930664,
10+
"final_loss": 5.91834020614624,
11+
"reason": "ok",
12+
"dtype": "bf16",
13+
"optimizer": "adamw",
14+
"bits": 16,
15+
"path": "path_b",
16+
"case_id": "bf16_adamw16_path_b",
17+
"wall_s": 132.39184389199363,
18+
"returncode": 0,
19+
"chunked_flag": "",
20+
"command": "/usr/bin/timeout 1800 /home/dave/cppmega-venv/bin/python scripts/m04_train_step.py --model-profile local_gb10_quarter --data-path data/parquet_samples/gb10/clang_semantic_4k_v10/val_00000.parquet --data-format parquet --token-key token_ids --steps 10 --batch-size 1 --seq-len 1024 --dtype bfloat16 --optimizer adamw --optimizer-quant-scheme dynamic_int8_v1 --lr 1e-4 --grad-checkpoint --output /tmp/cppmega_1b_speed_matrix_gb10_fastfused_20260601_b1s1024_cells/bf16_adamw16_path_b.json --json"
21+
},
22+
{
23+
"status": "blocked",
24+
"tok_sec": null,
25+
"step_sec": null,
26+
"compile_s": null,
27+
"peak_gb": 22.261920370161533,
28+
"loss_check": false,
29+
"initial_loss": null,
30+
"final_loss": null,
31+
"reason": "direct-chain training runtime not implemented (contract=direct_fusion_chain_training_runtime_missing; runtime_binding=direct_fusion_chain_logical_buffers_missing; model_binding=runtime_activation_owner_missing); split route available=m04_path_c_split_training_route_available",
32+
"dtype": "bf16",
33+
"optimizer": "adamw",
34+
"bits": 16,
35+
"path": "path_c",
36+
"case_id": "bf16_adamw16_path_c",
37+
"wall_s": 103.91524897504132,
38+
"returncode": 2,
39+
"chunked_flag": "0",
40+
"command": "/usr/bin/timeout 1800 /home/dave/cppmega-venv/bin/python scripts/m04_train_step.py --model-profile local_gb10_quarter --data-path data/parquet_samples/gb10/clang_semantic_4k_v10/val_00000.parquet --data-format parquet --token-key token_ids --steps 10 --batch-size 1 --seq-len 1024 --dtype bfloat16 --optimizer adamw --optimizer-quant-scheme dynamic_int8_v1 --lr 1e-4 --grad-checkpoint --output /tmp/cppmega_1b_speed_matrix_gb10_fastfused_20260601_b1s1024_cells/bf16_adamw16_path_c.json --json"
41+
},
42+
{
43+
"status": "blocked",
44+
"tok_sec": null,
45+
"step_sec": null,
46+
"compile_s": null,
47+
"peak_gb": 22.261920370161533,
48+
"loss_check": false,
49+
"initial_loss": null,
50+
"final_loss": null,
51+
"reason": "direct-chain training runtime not implemented (contract=direct_fusion_chain_training_runtime_missing; runtime_binding=direct_fusion_chain_logical_buffers_missing; model_binding=runtime_activation_owner_missing); split route available=m04_path_c_split_training_route_available",
52+
"dtype": "bf16",
53+
"optimizer": "adamw",
54+
"bits": 16,
55+
"path": "path_c_chunked",
56+
"case_id": "bf16_adamw16_path_c_chunked",
57+
"wall_s": 89.91207817499526,
58+
"returncode": 2,
59+
"chunked_flag": "1",
60+
"command": "/usr/bin/timeout 1800 /home/dave/cppmega-venv/bin/python scripts/m04_train_step.py --model-profile local_gb10_quarter --data-path data/parquet_samples/gb10/clang_semantic_4k_v10/val_00000.parquet --data-format parquet --token-key token_ids --steps 10 --batch-size 1 --seq-len 1024 --dtype bfloat16 --optimizer adamw --optimizer-quant-scheme dynamic_int8_v1 --lr 1e-4 --grad-checkpoint --output /tmp/cppmega_1b_speed_matrix_gb10_fastfused_20260601_b1s1024_cells/bf16_adamw16_path_c_chunked.json --json"
61+
}
62+
]

0 commit comments

Comments
 (0)