Commit 655c636
committed
fix(nnx): optimize scan memory by excluding read-only parameters from scan outputs
When using `jax.lax.scan` in Flax NNX decoder layers (e.g. DeepSeek-V3 / Gemma 2), returning `nnx.state(layer)` inside `layer_fn` caused `jax.lax.scan` to stack read-only model parameters (`nnx.Param`) across all 58 MoE layers into Python scan outputs. This allocated ~12.7 GiB of extra HBM memory and triggered `RESOURCE_EXHAUSTED` OOM errors during large model execution.
This change uses `_, _, updated_state = nnx.split(layer, nnx.Param, ...)` inside `layer_fn` to exclude read-only parameters from scan carry/output states, returning only mutable layer states to `jax.lax.scan`.
Verification:
- Local NNX decoder unit tests pass 100% (44/44 in `tests/unit/nnx_decoders_test.py`).
- Pre-commit code quality checks pass 100% (`codespell`, `pylint`, `pyink`, `yamllint`).
- End-to-end logit check on `mlperf-v5p` cluster (`v5p-128`) for DeepSeek-V3 671B ran forward pass to completion without OOM.
TAG=agy
CONV=0c977f36-644a-472a-981c-1856f8cac8a61 parent d557e88 commit 655c636
1 file changed
Lines changed: 10 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
356 | 356 | | |
357 | 357 | | |
358 | 358 | | |
359 | | - | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
360 | 363 | | |
361 | 364 | | |
362 | 365 | | |
363 | 366 | | |
364 | 367 | | |
365 | | - | |
| 368 | + | |
| 369 | + | |
366 | 370 | | |
367 | 371 | | |
368 | 372 | | |
| |||
957 | 961 | | |
958 | 962 | | |
959 | 963 | | |
960 | | - | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
961 | 968 | | |
962 | 969 | | |
963 | 970 | | |
| |||
0 commit comments