Skip to content

feat(overlap): global gemm_num_sms SM cap for frontier GEMMs#57

Open
A-nnonymous wants to merge 1 commit into
PFCCLab:paddlefrom
A-nnonymous:feat/gemm-sm-limit
Open

feat(overlap): global gemm_num_sms SM cap for frontier GEMMs#57
A-nnonymous wants to merge 1 commit into
PFCCLab:paddlefrom
A-nnonymous:feat/gemm-sm-limit

Conversation

@A-nnonymous

Copy link
Copy Markdown
Collaborator

背景 / Motivation

要在训练/推理中做计算-通信多流 overlap(把 MoE 的 GEMM 与 DeepEP/HybridEP 通信放到不同 CUDA stream 并发),必须限制 GEMM 占用的 SM 数,给通信 kernel 留出资源。参考 DeepGEMM 的 set_num_sms 单旋钮语义。

sonic-moe 的 FP8/BF16 frontier GEMM 全走 CUTLASS DSL persistent tile scheduler,grid.z (num_persistent_clusters) 由 max_active_clusters 决定,即 SM 占用上界。本 PR 加一个统一的全局旋钮把它按预算收窄。

改动 / What

  • SonicMoEConfig.gemm_num_sms(+ resolve_gemm_num_sms()):优先级 config 字段 > 环境变量 SONIC_MOE_GEMM_NUM_SMS > None(用满全部 SM)。<=0 视为未设。
  • 新增 quack_utils/sm_limit.pycapped_max_active_clusters(cluster_size, persistent=) 返回 min(硬件满占用, num_sms // cluster_size),下限 1。只依赖 ..config + quack.cute_dsl_utils,无循环依赖。
  • 将所有 frontier max_active_clusters 走此收口:gemm_gatedgemm_dgatedgemm_sm100_fp8_zeromat(3 处)、blockscaled_fp8_gemm(7 处)、以及 BF16 wgrad varlen_k GEMM。不改任何公开签名

正确性保证 / Correctness

收窄只降低 grid.z,不改变归约顺序,因此输出与不限流 bit 完全一致。max_active_clusters 以运行时动态 Int32TileSchedulerOptions)传入,不进 compile_key,改值无需重编译

验证 / Verification(B300 / SM103,148 SMs)

  • tests/ops/test_gemm_sm_cap.py:resolution 优先级、数值收窄、bf16/fp8 gated GEMM 在 cap=64/112 下 bit-exact + 无重编译——8 passed。
  • 现有回归(gated/dgated/varlen/blockscaled/zeromat/quant 变体、bf16-mode wgrad)在 SONIC_MOE_GEMM_NUM_SMS=64 下全部通过。
  • grid.z 随 cap 变化:max_active_clusters 148→112→64。
  • 单算子延迟(up-proj gated,ERNIE shape T=8192,H=3072,I=1536,E=8,K=8,fp8):全 SM 699µs → cap 64–112 约 720–730µs(~1.03–1.05x),换取释放 30–50% SM 给通信。tests/ops/bench_gemm_sm_cap.py 可复现。

用法 / Usage

from sonicmoe import SonicMoEConfig
cfg = SonicMoEConfig(use_fp8=True, gemm_num_sms=112)   # 留 36 SM 给通信
with cfg.activate():
    out = moe(x)
# 或:SONIC_MOE_GEMM_NUM_SMS=112

🤖 Generated with Claude Code

Add a single knob to cap how many SMs the frontier CuTe-DSL persistent
GEMMs may occupy, so callers can leave SMs free for communication kernels
when overlapping compute and comm on separate CUDA streams. Modeled after
DeepGEMM's set_num_sms single-knob semantics.

- SonicMoEConfig.gemm_num_sms (+ resolve_gemm_num_sms): config field > env
  SONIC_MOE_GEMM_NUM_SMS > None. <=0 treated as unset.
- New quack_utils/sm_limit.py: capped_max_active_clusters(cluster_size,
  persistent=) returns min(hw_full_occupancy, num_sms // cluster_size),
  clamped to >=1. Depends only on ..config + quack.cute_dsl_utils (no cycle).
- Route every frontier max_active_clusters through the cap: gemm_gated,
  gemm_dgated, gemm_sm100_fp8_zeromat (3), blockscaled_fp8_gemm (7), and the
  BF16 wgrad varlen_k GEMM. No public signatures changed.

The cap only lowers grid.z (num_persistent_clusters); it never changes the
reduction order, so output is bit-identical to the uncapped run. Because
max_active_clusters is a runtime dynamic Int32 (TileSchedulerOptions), it is
not part of the compile key — changing the cap does not force a recompile.

Tests (tests/ops/test_gemm_sm_cap.py): resolution priority, numeric capping,
and bit-exactness + no-recompile for bf16/fp8 gated GEMM at cap 64/112.
bench_gemm_sm_cap.py reports per-call latency across caps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant