Skip to content

feat(gemm): unified gemm_num_sms SM-count cap for all frontier GEMMs#58

Draft
A-nnonymous wants to merge 2 commits into
paddlefrom
feat/gemm-sm-limit
Draft

feat(gemm): unified gemm_num_sms SM-count cap for all frontier GEMMs#58
A-nnonymous wants to merge 2 commits into
paddlefrom
feat/gemm-sm-limit

Conversation

@A-nnonymous

Copy link
Copy Markdown
Collaborator

What

Adds a single config knob SonicMoEConfig.gemm_num_sms (env SONIC_MOE_GEMM_NUM_SMS) that caps the number of SMs every frontier CUTLASS-DSL persistent GEMM may occupy — one knob for all GEMMs (BF16 and FP8), à la DeepGEMM's set_num_sms.

Purpose: compute/communication multi-stream overlap. Capping the GEMM leaves SMs free for a DeepEP/HybridEP comm kernel on another stream. None (default) = use all SMs, behavior unchanged bit-for-bit.

How

New helper sonicmoe/quack_utils/sm_limit.py, applied identically at every frontier call site:

  1. capped_max_active_clusters narrows the runtime-dynamic max_active_clusters (SM budget → cluster budget, floor division, min 1).
  2. clc_persistence_default forces use_clc_persistence=False (STATIC scheduler) whenever the cap is set. On Blackwell (SM100+) the default CLC scheduler path ignores max_active_clusters entirely — the grid is always the full problem grid — so the cap only takes effect under STATIC. use_clc_persistence is a compile-time constant, so sm_cap_enabled() is folded into every compile key.

Only the STATIC-vs-CLC bool enters the compile key, not the SM count: the count drives only the runtime-dynamic grid, so different non-None caps reuse the same STATIC compilation (no recompile per cap value).

Coverage

gemm_gated, gemm_dgated, the 3 gather_A+blockscaled zeromat variants (gemm_sm100_fp8_zeromat.py), the 7 blockscaled_fp8_gemm.py call sites (grouped / varlen_k ×3 / plain / wgrad ×2), and bf16_wgrad_gemm.py. All changes confined to sonicmoe/.

Out of scope: SM90/Hopper — no CLC there, so STATIC already honors max_active_clusters; the SM90 branches are left untouched (the helper returns the base value when no cap is set).

Verification (B300 / SM103, 148 SM, CUDA 13.2)

Numerics — bit-identical. All GEMM unit tests pass both with the cap off and with SONIC_MOE_GEMM_NUM_SMS=64 forced on. Capping changes only grid.z / scheduling, never the reduction order, so results are bit-for-bit equal:

test file cap=off cap=64
test_gemm_gated.py 45 passed 45 passed
test_gemm_dgated.py 45 passed 45 passed
test_gemm_gated_postact_quant.py 5 passed 5 passed
test_gemm_gated_combined_quant.py 3 passed 3 passed
test_gemm_dgated_savez0_zeromat.py 3 passed 3 passed
test_varlen_gemm.py 45 passed
test_fused_zy1_quant.py 30 passed

ncu (real knob, up-proj gated FP8, cluster=1→2 SM/cluster):

cap grid.z active SM SM_active% time (µs) per-SM eff
none (CLC) 1 (full) 148 98.3 651 1.00
112 56 112 73.6 849 1.01
64 32 64 42.1 1430 1.05
32 16 32 21.0 2830 1.06
  • grid.z tracks the cap exactly; SM_active% ≈ resident CTAs / 148.
  • Per-SM efficiency stays ≈1.0 — this GEMM is compute-bound, so capping trades wall-clock roughly linearly for freed SMs. The knob is for freeing SMs to overlap, not a free speedup. There is no "sweet spot" that beats full occupancy for this kernel in isolation; the sweet spot only exists once a concurrent comm kernel is filling the freed SMs (out of scope for this PR, which lands the mechanism only).
  • Compile cache grows by exactly one STATIC entry when the cap is first enabled; different non-None caps reuse it.

Notes

  • No public function signatures changed; the knob flows through config/env only.

Add a single knob `SonicMoEConfig.gemm_num_sms` (env
`SONIC_MOE_GEMM_NUM_SMS`) that caps the number of SMs every frontier
CUTLASS-DSL persistent GEMM may occupy, mirroring DeepGEMM's
`set_num_sms`. Intended for compute/communication multi-stream overlap:
capping the GEMM leaves SMs free for a comm kernel on another stream.

Mechanism (new helper `quack_utils/sm_limit.py`), applied identically at
every frontier call site:
  1. `capped_max_active_clusters` narrows the runtime-dynamic
     max_active_clusters (SM budget -> cluster budget, floor div, min 1).
  2. `clc_persistence_default` forces use_clc_persistence=False (STATIC
     scheduler) whenever the cap is set. On Blackwell (SM100+) the default
     CLC scheduler ignores max_active_clusters entirely (grid is always
     the full problem grid), so the cap only bites under STATIC. This is a
     compile-time constant, so `sm_cap_enabled()` is folded into every
     compile key (only the STATIC-vs-CLC bool, not the SM count -- the
     count only drives the runtime-dynamic grid, no recompile).

When gemm_num_sms is None, behavior is unchanged bit-for-bit (CLC path,
full grid). Covers: gemm_gated, gemm_dgated, the 3 gather_A+blockscaled
zeromat variants, the 7 blockscaled_fp8_gemm call sites, and
bf16_wgrad_gemm. SM90/Hopper is out of scope (no CLC; STATIC already
honors max_active_clusters there).

Verified on B300 (SM103, 148 SM, CUDA 13.2):
  - All GEMM unit tests pass with cap off AND cap forced to 64 (grid-only
    change, no reduction-order change -> numerics bit-identical).
  - ncu (real knob): grid.z tracks the cap (none=CLC full / 112->56 /
    64->32 / 32->16 clusters), SM_active% ~= residentCTA/148, per-SM
    efficiency stays ~1.0 (near-linear -- this is a compute-bound GEMM, so
    capping trades throughput ~linearly; the knob is for freeing SMs, not
    a free lunch).
  - Compile cache grows by exactly one STATIC entry when toggling cap on;
    different non-None caps reuse the same STATIC compilation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@A-nnonymous A-nnonymous force-pushed the feat/gemm-sm-limit branch from b77bf27 to 4b175dd Compare July 2, 2026 06:51
The deferred `from ..config import get_active_config` inside
resolve_gemm_num_sms re-resolved the relative name at call time and, under
the paddlefleet_ops.sonicmoe install alias, raised ModuleNotFoundError:
No module named 'sonicmoe'. Hoisting the import to module load time (as
functional/utils.py and functional/fp8_config.py already do) binds it once
against the correct package. config imports nothing from quack_utils, so
there is no cycle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@A-nnonymous A-nnonymous marked this pull request as draft July 2, 2026 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant