Skip to content

Commit 52dd692

Browse files
committed
docs: ParallelismSpec.md — planning layer for FSDP/TP/EP/PP sizing
Visual builder planning/sizing layer для распределённой памяти. НЕ runtime — MLX/Apple Silicon target. Цель: GUI предсказывает OOM на конкретной топологии (H100:8 / TPU v6e-8 / GB10 quarter) и подсвечивает известные грабли ДО запуска тренировки. Импортирует уроки из: - ../cppmega: Megatron EP=4/8 production (PP=1, TP=1, no FSDP, no ZeRO) — кernel fusion + dtype ladder (BF16 grads, int8 momentum) - ../nanochat: FSDP2 + Megatron TP + SPMD multi-stack, в т.ч. /memory_estimator.py (1265 строк с формулами) и накопленные lesson documents (CLAUDE.md, Memory.md, CHANGELOG.md, базаа граблей) 5 этапов: A — topology.py + sharding_spec.py (data layer: DeviceKind, ParallelismKind, builtin device topologies, validated meshes) B — distributed_memory.py (per-rank accounting с FP8/bf16/fp32 mix; FSDP all-gather peak; Megatron RowParallel boundary materialisation; SP allreduce buffers; master-fp32 weights duplication) C — gotcha_checker.py (table-driven известные footguns ≥10: fsdp2_whole_compile, megatron_nan_step1, fp8_grad_duplication, master_fp32_duplication, ep_more_than_16_experts_xla, pp_comm_stream_broken, megatron_row_parallel_boundary, fsdp_allgather_peak_unsharded — каждый со ссылкой на конкретный nanochat/cppmega lesson) D — auto_shard.py (3-5 ranked proposals heuristics from nanochat practice) E — verify_distributed_plan + GUI workflow + perf gate <100ms per preset bd epic: cppmega-mlx-rwt (5 stage children rwt.1-rwt.5).
1 parent 8c724a8 commit 52dd692

1 file changed

Lines changed: 360 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)