feat(overlap): global gemm_num_sms SM cap for frontier GEMMs#57
Open
A-nnonymous wants to merge 1 commit into
Open
feat(overlap): global gemm_num_sms SM cap for frontier GEMMs#57A-nnonymous wants to merge 1 commit into
A-nnonymous wants to merge 1 commit into
Conversation
Add a single knob to cap how many SMs the frontier CuTe-DSL persistent GEMMs may occupy, so callers can leave SMs free for communication kernels when overlapping compute and comm on separate CUDA streams. Modeled after DeepGEMM's set_num_sms single-knob semantics. - SonicMoEConfig.gemm_num_sms (+ resolve_gemm_num_sms): config field > env SONIC_MOE_GEMM_NUM_SMS > None. <=0 treated as unset. - New quack_utils/sm_limit.py: capped_max_active_clusters(cluster_size, persistent=) returns min(hw_full_occupancy, num_sms // cluster_size), clamped to >=1. Depends only on ..config + quack.cute_dsl_utils (no cycle). - Route every frontier max_active_clusters through the cap: gemm_gated, gemm_dgated, gemm_sm100_fp8_zeromat (3), blockscaled_fp8_gemm (7), and the BF16 wgrad varlen_k GEMM. No public signatures changed. The cap only lowers grid.z (num_persistent_clusters); it never changes the reduction order, so output is bit-identical to the uncapped run. Because max_active_clusters is a runtime dynamic Int32 (TileSchedulerOptions), it is not part of the compile key — changing the cap does not force a recompile. Tests (tests/ops/test_gemm_sm_cap.py): resolution priority, numeric capping, and bit-exactness + no-recompile for bf16/fp8 gated GEMM at cap 64/112. bench_gemm_sm_cap.py reports per-call latency across caps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景 / Motivation
要在训练/推理中做计算-通信多流 overlap(把 MoE 的 GEMM 与 DeepEP/HybridEP 通信放到不同 CUDA stream 并发),必须限制 GEMM 占用的 SM 数,给通信 kernel 留出资源。参考 DeepGEMM 的
set_num_sms单旋钮语义。sonic-moe 的 FP8/BF16 frontier GEMM 全走 CUTLASS DSL persistent tile scheduler,grid.z (
num_persistent_clusters) 由max_active_clusters决定,即 SM 占用上界。本 PR 加一个统一的全局旋钮把它按预算收窄。改动 / What
SonicMoEConfig.gemm_num_sms(+resolve_gemm_num_sms()):优先级 config 字段 > 环境变量SONIC_MOE_GEMM_NUM_SMS> None(用满全部 SM)。<=0视为未设。quack_utils/sm_limit.py:capped_max_active_clusters(cluster_size, persistent=)返回min(硬件满占用, num_sms // cluster_size),下限 1。只依赖..config+quack.cute_dsl_utils,无循环依赖。max_active_clusters走此收口:gemm_gated、gemm_dgated、gemm_sm100_fp8_zeromat(3 处)、blockscaled_fp8_gemm(7 处)、以及 BF16 wgrad varlen_k GEMM。不改任何公开签名。正确性保证 / Correctness
收窄只降低 grid.z,不改变归约顺序,因此输出与不限流 bit 完全一致。
max_active_clusters以运行时动态Int32(TileSchedulerOptions)传入,不进 compile_key,改值无需重编译。验证 / Verification(B300 / SM103,148 SMs)
tests/ops/test_gemm_sm_cap.py:resolution 优先级、数值收窄、bf16/fp8 gated GEMM 在 cap=64/112 下 bit-exact + 无重编译——8 passed。SONIC_MOE_GEMM_NUM_SMS=64下全部通过。max_active_clusters148→112→64。tests/ops/bench_gemm_sm_cap.py可复现。用法 / Usage
🤖 Generated with Claude Code