[fmoe][gfx950] Add GLM-5.2 MXFP8/MXFP4 moe tune config#4074
Closed
zejunchen-zejun wants to merge 7 commits into
Closed
[fmoe][gfx950] Add GLM-5.2 MXFP8/MXFP4 moe tune config#4074zejunchen-zejun wants to merge 7 commits into
zejunchen-zejun wants to merge 7 commits into
Conversation
GLM-5.2 runs its MoE as MXFP8 (per_1x32, e8m0 block scale) after online requant. Without a tuned fmoe config the runtime fell back to the CK-tile mfma path (mfma_moe1/2_afp8_wfp8) instead of the faster FlyDSL MXFP8 a8w8 kernels. Add tuned + matching untuned configs so the FlyDSL path is picked up by default via model_configs/ auto-merge. Coverage: gfx950 (cu_num=256), model_dim=6144, expert=257, topk=9, q_type=per_1x32, a/w=fp8_e4m3fn, token 1..32768 for both inter_dim=512 (tp4) and inter_dim=256 (tp8). All rows err1/err2=0.0%. Kept as a new file (not the existing a8w8_blockscale_tuned_fmoe_glm5.csv, which is the per_1x128 CK path) since the quant scheme differs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds GLM-5.2 MoE model-specific tuned/untuned config CSVs for gfx950 MXFP8 (per_1x32) so runtime selection prefers the FlyDSL MXFP8 a8w8 kernels instead of falling back to the slower CK-tile MFMA path.
Changes:
- Add a new untuned MoE shape list for GLM-5.2 MXFP8 per_1x32 (inter_dim 512 and 256; tokens 1..32768).
- Add a new tuned MoE config for gfx950 (cu_num=256) mapping those shapes to specific FlyDSL stage1/stage2 kernel names and measured timings.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| aiter/configs/model_configs/mxfp8_untuned_fmoe_glm5_2.csv | Adds GLM-5.2 MXFP8 per_1x32 untuned shape keys to support config lookup/tuning workflows. |
| aiter/configs/model_configs/mxfp8_tuned_fmoe_glm5_2.csv | Adds gfx950 tuned kernel selections for those shapes to drive FlyDSL MXFP8 path selection via auto-merge. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| gfx950,256,512,6144,512,257,9,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float8_e4m3fn,QuantType.per_1x32,1,0,32,0,265.9479,flydsl_moe1_afp8_wfp8_bf16_t32x64x256_w2_gui_kw2_fp8,0.0%,161.7576,flydsl_moe2_afp8_wfp8_bf16_t32x128x128_atomic_persist,0.0%,427.7055,0,0,0,203.35,5692.69, | ||
| gfx950,256,1024,6144,512,257,9,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float8_e4m3fn,QuantType.per_1x32,1,0,64,0,297.0182,flydsl_moe1_afp8_wfp8_bf16_t64x128x256_w4_gui_fp8,0.0%,177.7574,flydsl_moe2_afp8_wfp8_bf16_t64x128x128_atomic_bnt2,0.0%,474.7756,0,0,0,366.38,5148.18, | ||
| gfx950,256,2048,6144,512,257,9,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float8_e4m3fn,QuantType.per_1x32,1,0,128,0,348.9894,flydsl_moe1_afp8_wfp8_bf16_t128x128x256_w4_gui_fp8,0.0%,246.1611,flydsl_moe2_afp8_wfp8_bf16_t64x128x128_atomic_sbm128,0.0%,595.1505,0,0,0,584.55,4138.63, | ||
| gfx950,256,4096,6144,512,257,9,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float8_e4m3fn,QuantType.per_1x32,1,0,64,0,529.8860000000001,flydsl_moe1_afp8_wfp8_bf16_t64x128x256_w4_bnt0_gui,0.0%,388.0813,flydsl_moe2_afp8_wfp8_bf16_t64x128x256_atomic_bnt2_persist,0.0%,917.9673,0,0,0,757.96,2724.34, |
This was referenced Jul 3, 2026
Companion to the MXFP8 configs already on this branch: adds the MXFP4 (per_1x32, a4w4, fp4x2 act+weight) fused-MoE tuning for GLM-5.2 on gfx950, covering TP4 (inter_dim=512) and TP8 (inter_dim=256), tokens 1..32768 (32 rows). Auto-merged via model_configs/ glob; no aiter code change. Dedup keeps one row per key = fastest measured (min us): FlyDSL where it is fastest, CK where CK is fastest; redundant slower duplicates dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zejunchen-zejun
changed the title
…glm5_fp4) The GLM-5.2 MXFP4 (per_1x32) configs collided 32/32 on all shape-key columns with the existing glm5_fp4_tuned_fmoe.csv (GLM-5/5.2 share MoE dims: model_dim=6144, expert=257, topk=9). After #3946 that file already carries FlyDSL-tuned primary rows plus CK flydsl_fallback rows for these shapes, and its tuning is on par with (or better than) the re-tune here on 17/32 shapes. Keeping both would trip the fmoe config auto-merge duplicate check (RuntimeError + source-file rewrite). Remove mxfp4_{tuned,untuned}_fmoe_glm5_2.csv and rely on glm5_fp4. The MXFP8 (per_1x32) configs are genuinely new (no existing GLM MXFP8 fmoe config) and are kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…into zejun/glm5.2_fp8_tune_moe_0703
The GLM-5.2 MXFP4 (per_1x32) shapes collide 1:1 with glm5_fp4_tuned_fmoe.csv (GLM-5/5.2 share MoE dims). Rather than duplicate (which trips the fmoe config auto-merge) or blindly overwrite, take the faster row per shape between this tune (2026-07-03) and main's FlyDSL tune (#3946, 2026-07-06): 15/32 shapes switch to the faster rows from here, 17/32 keep main's, and the CK flydsl_fallback rows are preserved. Total us over the 32 shapes drops to 14604.8 (vs main 14748.1 / this-tune-only 14668.8) — strictly better than either alone. The standalone mxfp4_{tuned,untuned}_fmoe_glm5_2.csv are removed; MXFP8 (per_1x32) configs are new and kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gbyu-amd
added a commit
that referenced
this pull request
Jul 7, 2026
…4074) Add GLM-5.2 MXFP8 (per_1x32) tuned/untuned MoE configs and merge best-of-both GLM-5.2 MXFP4 tuning into glm5_fp4_tuned_fmoe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
this config merged into #4095 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-5.2 runs its MoE as MXFP8 (per_1x32, e8m0 block scale) after online requant. Without a tuned fmoe config the runtime fell back to the CK-tile mfma path (mfma_moe1/2_afp8_wfp8) instead of the faster FlyDSL MXFP8 a8w8 kernels. Add tuned + matching untuned configs so the FlyDSL path is picked up by default via model_configs/ auto-merge.
Coverage: gfx950 (cu_num=256), model_dim=6144, expert=257, topk=9, q_type=per_1x32, a/w=fp8_e4m3fn, token 1..32768 for both inter_dim=512 (tp4) and inter_dim=256 (tp8). All rows err1/err2=0.0%.
Kept as a new file (not the existing a8w8_blockscale_tuned_fmoe_glm5.csv, which is the per_1x128 CK path) since the quant scheme differs.
Motivation
Technical Details
Test Plan
Test Result
Submission Checklist