MUSA: add calibrated FP8 MoE backend selection on S5000

[yeahdongcn]

Summary

• add an exact-shape S5000 FP8 MoE dispatcher;
• use native GEMV only in calibrated small-M decode windows;
• preserve the default contiguous/fused-glue DeepGEMM path for eligible large
  eager prefill calls, and use the established upstream path in between;
• remove the DeepSeek-V4-specific GEMV enablement environment variable from the
  normal serving path while retaining generic diagnostic overrides;
• add cold-cache crossover, FP32-oracle, folded-shared-expert, production
  DeepGEMM identity, and graph-replay evidence tooling.

M is hidden_states.shape[0] for one MoE invocation, not HTTP batch size or
per-expert route count.

Rebase status

• base: v0.24.0-dev@4416dc24b00afcb46ebe96b34308fda96d30ed36
  (Qwen3.5 MoE optimizations, Qwen3.5 moe optimizations #113)
• measured head: 599cb804080e886344098a68c4e2a01fbbd90210
• review head: 943e5e0eaefc8d25cb24c249da0842b6205add64
• merge state: clean

The measured head has tree b1e8cc7bb4b0b4b38985b64220bc60e1c6c8238b.
The review head has tree dfee058a43cc3755318ea8fdc86aed01479b08ff and differs
only by a fail-closed DeepGEMM scale-expansion guard plus its regression test
(two files, 37 added and two replaced lines). Calibrated block-scale layouts
do not enter the new exception branch, so the measured production paths and
hardware attribution are unchanged; the latest head was rechecked with the
focused Python/meta suites rather than rerunning the full serving matrix.

The review commits are:

1. MUSA: add calibrated FP8 MoE backend dispatch
2. bench: add S5000 FP8 MoE calibration tools
3. build: package the MUSA JIT include marker
4. docs: document S5000 FP8 MoE backend selection

Qwen3.5 compiled serving on the current base also needs the GDN compile fix
from #115 (511cde18fe6a80cec15d8083a9e70a28c07ee057, currently open). The
Qwen A/B below applied that identical validation-only fix to both arms, so its
performance delta remains attributable to this dispatcher. DSV2/DSV4 do not
depend on #115, and this PR intentionally does not duplicate the GDN change.

The offline folded-Qwen TP4 shape is E=257, N=256, K=2048, topk=9. Its
multi-route sweep and graph replay qualify M <= 48 for the synthetic folded
shape, but the live Qwen3_5MoeForConditionalGeneration checkpoint follows
the qwen3_5.py path and does not activate the post-#113 shared-expert fold.
The production auto policy therefore intentionally retains the existing
E=256/topk=8 behavior and has no E=257 threshold.

Final auto policy

Per-rank FP8 MoE shape	Small M	Large eligible eager M	Otherwise
Qwen3.5 live TP4, E=256/topk=8	GEMV at M <= 13	contiguous DeepGEMM	upstream
Qwen3.5 folded TP4, E=257/topk=9	unobserved in live path (upstream)	contiguous DeepGEMM	upstream
DeepSeek-V4 TP8, block 32x8	GEMV at M <= 5	contiguous DeepGEMM	upstream
DeepSeek-V4 TP8, block 16x8	GEMV at M <= 5	contiguous DeepGEMM	upstream
DeepSeek-V2-Lite TP1 eager	GEMV at M <= 3	not eligible (E=64)	upstream
DeepSeek-V2-Lite TP1 capture	GEMV at M = 1	not eligible (E=64)	upstream

Unknown devices/shapes, unsupported layouts, and unsafe capture states fail
closed. Explicit upstream bypasses both GEMV and DeepGEMM for diagnostic
rollback. Diagnostic grouped GEMM remains disabled in auto until it has
separate correctness and serving evidence.

S5000 serving A/B

Environment: isolated 8x S5000, driver 3.3.5-server, torch/torch_musa
2.9.1.post1+musa5.2.0s5000, torchada 0.1.72, MATE/flash_attn_3 0.2.4.
Every row used exact 4096 input and 1024 output tokens at BS 1/4/16/64, prefix
caching off, independent cache roots, normal compiled/captured serving, zero
failed requests, and exact per-request output lengths. DSV2 rows are
three-repetition medians; DSV4 and Qwen3-VL rows are one paired repetition.

Model	BS	output tok/s base -> candidate	TTFT ms base -> candidate	TPOT ms base -> candidate
DSV4 Flash TP8	1	4.573 -> 4.578 (+0.11%)	1315.0 -> 1305.8 (-0.70%)	217.618 -> 217.384 (-0.11%)
DSV4 Flash TP8	4	17.576 -> 17.575 (-0.01%)	4413.6 -> 4413.9 (+0.01%)	223.220 -> 223.237 (+0.01%)
DSV4 Flash TP8	16	34.143 -> 34.319 (+0.51%)	30342.5 -> 30167.4 (-0.58%)	235.937 -> 233.784 (-0.91%)
DSV4 Flash TP8	64	52.210 -> 52.609 (+0.76%)	438236.6 -> 434228.1 (-0.91%)	241.255 -> 239.092 (-0.90%)
DSV2 Lite FP8	1	76.584 -> 83.007 (+8.39%)	302.1 -> 303.1 (+0.33%)	12.775 -> 11.762 (-7.92%)
DSV2 Lite FP8	4	195.820 -> 199.334 (+1.79%)	935.6 -> 932.7 (-0.31%)	19.507 -> 19.145 (-1.85%)
DSV2 Lite FP8	16	530.996 -> 533.729 (+0.51%)	2827.2 -> 2832.3 (+0.18%)	27.250 -> 27.090 (-0.59%)
DSV2 Lite FP8	64	1150.712 -> 1140.382 (-0.90%)	9987.4 -> 9980.3 (-0.07%)	45.059 -> 45.578 (+1.15%)
Qwen3-VL-30B-A3B	1	75.684 -> 74.920 (-1.01%)	138.5 -> 139.6 (+0.82%)	13.090 -> 13.224 (+1.02%)
Qwen3-VL-30B-A3B	4	242.315 -> 241.789 (-0.22%)	395.5 -> 396.6 (+0.27%)	16.107 -> 16.142 (+0.21%)
Qwen3-VL-30B-A3B	16	710.945 -> 712.854 (+0.27%)	1280.3 -> 1290.2 (+0.78%)	21.143 -> 21.074 (-0.33%)
Qwen3-VL-30B-A3B	64	928.995 -> 937.646 (+0.93%)	13219.0 -> 13169.6 (-0.37%)	55.337 -> 54.764 (-1.04%)

No measured regression crosses the predeclared 3% remediation gate. The DSV2
single-token decode gain survives the stricter capture boundary without a
material regression at the larger batch sizes. Qwen3-VL uses an uncalibrated
E=128,N=192 MoE shape and verifies that auto remains on upstream.

The missing Qwen3.5-35B-A3B base-vs-PR comparison was rerun on an isolated TP4
S5000 lease on 10.20.35.71. The base arm was exact post-#113 plus the common
PR115-equivalent GDN compile fix; the candidate was the measured PR head plus
the identical fix. The arms used their separately compiled native extensions, torchada
0.1.76, normal compilation/CUDAGraph, independent measurement caches, exact
4096/1024 token lengths, and three repetitions except BS64. Logs prove that
base had no auto dispatcher, while candidate selected the E=256/topk=8 M13
GEMV policy in both eager and capture and retained DeepGEMM for large prefill.

BS	output tok/s base -> PR	TPOT ms base -> PR	TTFT ms base -> PR
1	61.752 -> 67.831 (+9.84%)	16.067 -> 14.617 (-9.03%)	144.8 -> 142.6 (-1.54%)
4	227.352 -> 247.811 (+9.00%)	17.090 -> 15.685 (-8.22%)	465.0 -> 459.3 (-1.22%)
16	726.556 -> 723.877 (-0.37%)	20.738 -> 20.835 (+0.47%)	1201.4 -> 1198.3 (-0.26%)
64	1028.423 -> 1025.842 (-0.25%)	46.907 -> 47.072 (+0.35%)	15068.9 -> 15068.3 (-0.00%)

This directly validates the shipped E256/M13 policy: it improves BS1/4 decode
by about 9% and leaves BS16/64 within 0.5% of the post-#113 base.

A separate folded-shape experiment was completed on 10.20.35.88 with the
common GDN fix applied identically to both validation arms. It passed normal
compiled startup, CUDAGraph capture, the DeepGEMM prefill gate, and semantic
output. Both arms already contained the existing E=256/topk=8 policy; the
candidate-only E=257/M48 policy was not observed, so this table is a no-op
policy check rather than a base-vs-PR comparison.

BS	output tok/s control -> M48	TPOT ms control -> M48	TTFT delta
1	79.012 -> 78.924 (-0.11%)	12.532 -> 12.508 (+0.19%)	-2.64%
4	283.929 -> 288.062 (+1.46%)	13.646 -> 13.452 (+1.42%)	+0.66%
16	807.171 -> 803.010 (-0.52%)	18.589 -> 18.735 (-0.79%)	+2.08%
64	1076.800 -> 1068.413 (-0.78%)	44.431 -> 44.889 (-1.03%)	-0.12%

Because the new policy was not exercised and the conservative no-regression
screen failed at BS64 TPOT by 1.03%, the M48 candidate was removed from the
PR worktree. The measured result therefore remains an upstream-path
no-op check, not evidence for an E=257 threshold.

Correctness and tests

• rebased onto the exact post-Qwen3.5 moe optimizations #113 base
• dispatch-policy tests: 12/12
• final-head fused-MoE chunking/dispatch tests: 35/35 in a no-GPU
  torch/MUSA container using meta tensors; device capability was stubbed only
  to avoid driver initialization and no MUSA kernel was executed
• unaligned per-tensor FP8 scales now fail closed to upstream instead of
  raising before the DeepGEMM eligibility gate
• Ruff check/format, py_compile, and git diff --check
• exact-head DSV2 graph replay at capture M=1/2/3: eight
  balanced/random/hot replays per point, bitwise-equal outputs and zero max
  absolute difference
• exact capture call counts: M=1 GEMV=1/upstream=0; M=2/3
  GEMV=0/upstream=1
• post-Qwen3.5 moe optimizations #113 4096/1024 serving A/B at BS 1/4/16/64 across three MoE models
• exact compiled Qwen3.5 control/candidate A/B at BS 1/4/16/64 on
  10.20.35.88; candidate E=257/M48 policy not observed, so upstream behavior
  retained
• exact post-Qwen3.5 moe optimizations #113-vs-PR Qwen3.5 E256/M13 A/B at BS 1/4/16/64 on
  10.20.35.71; BS1/4 improve by 9.84%/9.00% and BS16/64 stay within 0.5%

The earlier headline serving table remains withdrawn: its Qwen baseline
compiled TileLang kernels inside the measured BS1 run while the candidate
inherited a warm global cache, and it predates #113.

Scope

This optimizes S5000 block-128 E4M3 FP8 MoE execution. It does not make a raw
MXFP8 checkpoint directly executable; MXFP8 weights still require transcoding
to the supported representation.