Skip to content

[config] Add tuned configs for GLM-5.2-FP8 block-scale fp8 GEMM and MoE#4038

Closed
gbyu-amd wants to merge 2 commits into
mainfrom
add-glm5.2-fp8-gemm-moe-tuned-configs
Closed

[config] Add tuned configs for GLM-5.2-FP8 block-scale fp8 GEMM and MoE#4038
gbyu-amd wants to merge 2 commits into
mainfrom
add-glm5.2-fp8-gemm-moe-tuned-configs

Conversation

@gbyu-amd

@gbyu-amd gbyu-amd commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Adds tuned (and source untuned) configs for GLM-5.2-FP8 running TP4 on gfx950:

  • a8w8_blockscale_bpreshuffle GEMM: 4 shapes derived from GLM-5.2 linear layers (N6144/K3072, N6144/K4096, N6144/K6144, N7168/K512), swept over M in {1..32768}.
  • fused MoE (a8w8_blockscale, per_1x128): hidden_dim=6144, inter_dim=512, expert=257, topk=9, swept over tokens {1..32768}.

Configs are auto-discovered by the model_configs glob merge; no code change required. Verified end-to-end: server startup + gsm8k 3-shot show zero fp8 GEMM / MoE default-config fallbacks, accuracy 0.9356 strict / 0.9348 flexible exact_match.

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Adds tuned (and source untuned) configs for GLM-5.2-FP8 running TP4 on
gfx950:

- a8w8_blockscale_bpreshuffle GEMM: 4 shapes derived from GLM-5.2 linear
  layers (N6144/K3072, N6144/K4096, N6144/K6144, N7168/K512), swept over
  M in {1..32768}.
- fused MoE (a8w8_blockscale, per_1x128): hidden_dim=6144, inter_dim=512,
  expert=257, topk=9, swept over tokens {1..32768}.

Configs are auto-discovered by the model_configs glob merge; no code
change required. Verified end-to-end: server startup + gsm8k 3-shot show
zero fp8 GEMM / MoE default-config fallbacks, accuracy 0.9356 strict /
0.9348 flexible exact_match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gbyu-amd gbyu-amd requested a review from a team July 1, 2026 10:25
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4038 --add-label <label>

@zejunchen-zejun

Copy link
Copy Markdown
Contributor

#4074
Hi @gbyu-amd
We use online quant to make GLM5.2 FP8 MOE 1x32+E8M0 scale, so I tunned this moe kernel, could you help review

@yifehuan yifehuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one thing is the gfx column is not needed in fmoe tuned csv (not found in other csv), but it will auto-derive, it is ok

@zejunchen-zejun

Copy link
Copy Markdown
Contributor

LGTM, one thing is the gfx column is not needed in fmoe tuned csv (not found in other csv), but it will auto-derive, it is ok

Hi @yifehuan
Thank you for review. This PR will be closed as the new PR is here: #4095. It adds all tuned config. Could you help review this new PR?
Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants