feat(gemm): unified gemm_num_sms SM-count cap for all frontier GEMMs#58
Draft
A-nnonymous wants to merge 2 commits into
Draft
feat(gemm): unified gemm_num_sms SM-count cap for all frontier GEMMs#58A-nnonymous wants to merge 2 commits into
A-nnonymous wants to merge 2 commits into
Conversation
Add a single knob `SonicMoEConfig.gemm_num_sms` (env
`SONIC_MOE_GEMM_NUM_SMS`) that caps the number of SMs every frontier
CUTLASS-DSL persistent GEMM may occupy, mirroring DeepGEMM's
`set_num_sms`. Intended for compute/communication multi-stream overlap:
capping the GEMM leaves SMs free for a comm kernel on another stream.
Mechanism (new helper `quack_utils/sm_limit.py`), applied identically at
every frontier call site:
1. `capped_max_active_clusters` narrows the runtime-dynamic
max_active_clusters (SM budget -> cluster budget, floor div, min 1).
2. `clc_persistence_default` forces use_clc_persistence=False (STATIC
scheduler) whenever the cap is set. On Blackwell (SM100+) the default
CLC scheduler ignores max_active_clusters entirely (grid is always
the full problem grid), so the cap only bites under STATIC. This is a
compile-time constant, so `sm_cap_enabled()` is folded into every
compile key (only the STATIC-vs-CLC bool, not the SM count -- the
count only drives the runtime-dynamic grid, no recompile).
When gemm_num_sms is None, behavior is unchanged bit-for-bit (CLC path,
full grid). Covers: gemm_gated, gemm_dgated, the 3 gather_A+blockscaled
zeromat variants, the 7 blockscaled_fp8_gemm call sites, and
bf16_wgrad_gemm. SM90/Hopper is out of scope (no CLC; STATIC already
honors max_active_clusters there).
Verified on B300 (SM103, 148 SM, CUDA 13.2):
- All GEMM unit tests pass with cap off AND cap forced to 64 (grid-only
change, no reduction-order change -> numerics bit-identical).
- ncu (real knob): grid.z tracks the cap (none=CLC full / 112->56 /
64->32 / 32->16 clusters), SM_active% ~= residentCTA/148, per-SM
efficiency stays ~1.0 (near-linear -- this is a compute-bound GEMM, so
capping trades throughput ~linearly; the knob is for freeing SMs, not
a free lunch).
- Compile cache grows by exactly one STATIC entry when toggling cap on;
different non-None caps reuse the same STATIC compilation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b77bf27 to
4b175dd
Compare
The deferred `from ..config import get_active_config` inside resolve_gemm_num_sms re-resolved the relative name at call time and, under the paddlefleet_ops.sonicmoe install alias, raised ModuleNotFoundError: No module named 'sonicmoe'. Hoisting the import to module load time (as functional/utils.py and functional/fp8_config.py already do) binds it once against the correct package. config imports nothing from quack_utils, so there is no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a single config knob
SonicMoEConfig.gemm_num_sms(envSONIC_MOE_GEMM_NUM_SMS) that caps the number of SMs every frontier CUTLASS-DSL persistent GEMM may occupy — one knob for all GEMMs (BF16 and FP8), à la DeepGEMM'sset_num_sms.Purpose: compute/communication multi-stream overlap. Capping the GEMM leaves SMs free for a DeepEP/HybridEP comm kernel on another stream.
None(default) = use all SMs, behavior unchanged bit-for-bit.How
New helper
sonicmoe/quack_utils/sm_limit.py, applied identically at every frontier call site:capped_max_active_clustersnarrows the runtime-dynamicmax_active_clusters(SM budget → cluster budget, floor division, min 1).clc_persistence_defaultforcesuse_clc_persistence=False(STATIC scheduler) whenever the cap is set. On Blackwell (SM100+) the default CLC scheduler path ignoresmax_active_clustersentirely — the grid is always the full problem grid — so the cap only takes effect under STATIC.use_clc_persistenceis a compile-time constant, sosm_cap_enabled()is folded into every compile key.Only the STATIC-vs-CLC bool enters the compile key, not the SM count: the count drives only the runtime-dynamic grid, so different non-None caps reuse the same STATIC compilation (no recompile per cap value).
Coverage
gemm_gated,gemm_dgated, the 3 gather_A+blockscaled zeromat variants (gemm_sm100_fp8_zeromat.py), the 7blockscaled_fp8_gemm.pycall sites (grouped / varlen_k ×3 / plain / wgrad ×2), andbf16_wgrad_gemm.py. All changes confined tosonicmoe/.Out of scope: SM90/Hopper — no CLC there, so STATIC already honors
max_active_clusters; the SM90 branches are left untouched (the helper returns the base value when no cap is set).Verification (B300 / SM103, 148 SM, CUDA 13.2)
Numerics — bit-identical. All GEMM unit tests pass both with the cap off and with
SONIC_MOE_GEMM_NUM_SMS=64forced on. Capping changes onlygrid.z/ scheduling, never the reduction order, so results are bit-for-bit equal:ncu (real knob, up-proj gated FP8, cluster=1→2 SM/cluster):
grid.ztracks the cap exactly; SM_active% ≈ resident CTAs / 148.Notes