Skip to content

perf: fuse SiLU and FP8 group quant before DeepGEMM#109

Closed
yeahdongcn wants to merge 9 commits into
v0.24.0-devfrom
xd/musa-0759-silu-deepgemm
Closed

perf: fuse SiLU and FP8 group quant before DeepGEMM#109
yeahdongcn wants to merge 9 commits into
v0.24.0-devfrom
xd/musa-0759-silu-deepgemm

Conversation

@yeahdongcn

@yeahdongcn yeahdongcn commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

This is the first PR in a four-PR MUSA kernel stack:

  1. MUSA-0759 (this PR): fuse SiLU and row-major group-128 FP8 activation quantization before MATE DeepGEMM.
  2. MUSA-0760: fuse residual-add RMSNorm and row-major group-128 FP8 activation quantization.
  3. MUSA-0763: add guarded chunked min-p sampling.
  4. MUSA-0765: add guarded uniform top-k=50 renormalization.

Review/merge order: #109 -> #110 -> #111 -> #112.

The compiler pass is selected automatically for the validated Qwen3-8B-FP8,
single-S5000 contract. Unsupported model, parallelism, shape, dtype, device,
or quantization contracts keep the existing path. No campaign- or
fusion-specific environment control is introduced or read to select this
path; guarded automatic dispatch is the selection mechanism.

Exact source:

  • base: v0.24.0-dev at 68b5ef7a7eac9959dd71f600f7df2ebd756de1f8
  • head: 7d6dd02c4b6a45e00d0131fd42b885030d7891c2
  • head ref: xd/musa-0759-silu-deepgemm
  • pinned vLLM: ee0da84ab9e04ac7610e28580af62c365e898389

Validation

Historical per-ticket performance evidence

The figures below are retained from the earlier MUSA-0759-only serving
campaign. They predate the current environment-independent and logical-device
follow-ups, so they are directional historical evidence, not a result from the
current exact head and not the final combined four-PR A/B. The run used
Qwen3-8B-FP8, TP1, S5000, normal Inductor,
FULL_AND_PIECEWISE graph capture, greedy decoding, no MTP/speculative
decoding, and separate candidate/native-control source trees. openai-chat
adds eight Qwen template tokens, so --random-input-len 4096 measures 4104
input tokens; every request generated exactly 1024 output tokens.

concurrency mean TTFT change mean TPOT reduction
1 +1.89% +0.38%
4 -4.49% +0.80%
16 +2.39% +0.77%
64 +0.84% +0.32%

TPOT improved in all four historical cells. TTFT was noisy, especially at
concurrency 4, and is not presented as a general win.

Superseded descendant-stack evidence

The former descendant head f53cf6cbbd7f4d85e82270a86db1938c713d948a
passed dense, MoE, and MLA functional serving checks. That SHA is no longer the
final stack, and the same host-81 campaign had a foreign ExecBench workload,
so its combined latency rows are rejected and none of this evidence is used as
an exact-head readiness claim.

Exact-final MUSA-0767 gate

Status: EXACT-HEAD FUNCTIONAL REGRESSION COMPLETE; PERFORMANCE GATES
PENDING -- keep this PR in draft.

  • final stacked head: 95c3661c8658a861a1891f981c81dcadabea0654
  • final source tree: a766855ed712d61f81acb6b11d456ea5a128cac5
  • source archive SHA-256:
    03dbe0526f72d4dcba6b743c500d4d5eaaf9676e263f9bd91cbfd265db140feb
  • source manifest SHA-256:
    3ab55a222ea7fc8324c611bda9d3d842bc94850bbc7ccf4d869d0692132768a7
  • exact environment and build: 8x MTT S5000, driver 3.3.5-server,
    MUSA runtime 50200, torch==2.9.1.post1,
    torch_musa==2.9.1.post1+4c646b6, torchada==0.1.75,
    MATE/FlashAttention3 0.2.4, candidate extension SHA-256
    d0baaa8ba11981438a549115d50e89c10ec566dfac1ba27c0ccdc8364c5c2bd7
  • focused native correctness and graph-capture tests: 60 passed in 44.72s
  • non-identity logical-device sampling smoke: PASS with
    MUSA_VISIBLE_DEVICES=4,5; physical devices 4 and 5 were exposed as
    logical devices 0 and 1, and the smoke selected both optimized samplers on
    logical device 0
  • compiled/captured Qwen dense, Qwen MoE, and DeepSeek MLA regression:
    PASS, 3/3 rows and 10/10 semantic requests (Qwen3-8B-FP8 TP1
    FLASH_ATTN, Qwen3.5-35B-A3B-FP8 TP4 FLASH_ATTN, and
    DeepSeek-V2-Lite-Chat-FP8 TP1 FLASHMLA; Inductor,
    FULL_AND_PIECEWISE, MTP off)
  • clean-node combined Qwen3-8B-FP8 candidate/control result at exact
    4104/1024 and concurrency 1/4/16/64, without MTP:
    PENDING -- not present in this regression-only bundle
    • the first control-only attempt is rejected because default prefix caching
      plus repeated deterministic prompts invalidated cold-prefill TTFT; every
      A/B leg will restart with prefix caching disabled
  • Qwen3.5-2B and Qwen3.6-27B exact-token widening A/B evidence:
    PENDING -- not present in this regression-only bundle
  • regression evidence bundle SHA-256:
    ea1b38559be0458343b85f3cd6bba275d320d2b1c729071f455e2dccb04c5272;
    its focused monitor verdict was FAIL because foreign ExecBench GPU work
    was present, so the clean performance-monitor gate remains PENDING

Remaining gate

The exact-head functional regression is complete. This PR remains draft until
the combined and widening A/B plus clean performance-monitor gates are
complete and accepted. The stacked PRs must be reviewed and merged in order.

@yeahdongcn

Copy link
Copy Markdown
Collaborator Author

Superseded by #115, which consolidates the four logical kernel changes on the current v0.24.0-dev head (4416dc24) while preserving per-change commits. This draft and its historical evidence remain available for traceability; it is not a current-head readiness claim.

@yeahdongcn yeahdongcn closed this Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant