feat(paged): default-on full-step MoE-decode CUDA graph (grouped MMQ, patch 0043)

D1 lever. The MUL_MAT_ID CUDA-graph guard ([TAG_MUL_MAT_ID_CUDA_GRAPHS])
disables CUDA graphs for the WHOLE decode step whenever a MUL_MAT_ID node has
ne[2] > mmvq_mmid_max (8 for NVFP4 on sm_121) - i.e. for every multi-token
decode. Patch 0025 showed the path actually taken on Blackwell NVFP4,
should_use_mmq()==true -> grouped stream-k MMQ id-branch, launches on one
stream with NO host sync (only the per-expert host-loop fallback synchronizes),
so the disable is conservative and graphs are safe for the grouped path - but
0025 left it behind an opt-in env (LLAMA_MOE_FORCE_GRAPHS), so by default the
host re-issued every kernel of the step.

D1 profiling (GB10 sm_121, q36-35b-a3b-nvfp4, batched-bench -fa on, npl128)
settled the mechanism:
  - The grouped MMQ NVFP4 path IS what runs in decode: cudaStreamSynchronize
    count is IDENTICAL with graphs on vs off (1457 either way) - the per-expert
    host-loop fallback (the only device->host routing readback) is never hit.
    MoE routing is already device-side.
  - Steady-decode GPU-busy is ~99% (1% idle): static decode is GPU-bound, not
    host-sync-bound. The host cost is per-step kernel RE-ISSUE, removed by
    replaying a captured full-step graph (incl. the MoE dispatch).

So make the grouped-path graph capture ON BY DEFAULT; LLAMA_MOE_NO_FORCE_GRAPHS=1
forces the conservative pre-0025 disable for A/B. should_use_mmq() is the exact
guard: it returns FALSE for the large-M NVFP4 prefill (patch 0034), which
deliberately drops to the per-expert host-sync loop, so PREFILL keeps graphs
disabled (correct - that path syncs). Decode-only behaviour change; prefill and
the stock llama-cpp backend are untouched.

BIT-EXACT: greedy md5 byte-identical default(on)==LLAMA_MOE_NO_FORCE_GRAPHS(off)
==legacy LLAMA_MOE_FORCE_GRAPHS - paged-MoE 8cb0ce23777bf55f92f63d0292c756b0,
paged-dense 5951a5b4d624ce891e22ab5fca9bc439 (both match the recorded baselines).

Measured (GB10, batched-bench paged decode S_TG, default-on vs opt-out):
  npl 32   467.3 vs 444.3 t/s  +5.2%
  npl 128  788.2 vs 768.1 t/s  +2.6%

Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Author: mudler

ggml/src/ggml-cuda/ggml-cuda.cu

@@ -3358,12 +3358,22 @@ static bool ggml_cuda_graph_check_compability(ggml_cgraph * cgraph) {
             const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc;
             const int mmvq_mmid_max = get_mmvq_mmid_max_batch(node->src[0]->type, cc);
             bool mmid_needs_sync = !ggml_is_quantized(node->src[0]->type) || node->ne[2] > mmvq_mmid_max;
-            // PROBE (bit-exact, env LLAMA_MOE_FORCE_GRAPHS): the grouped stream-k MMQ id-path is
-            // launched on-stream with no host sync (only the per-expert host-loop fallback syncs);
-            // when should_use_mmq() is true (Blackwell NVFP4 grouped path) the op is graph-safe
-            // even for ne[2] > mmvq_mmid_max, so graphs need not be disabled for the whole step.
+            // [D1 / patch 0043] The grouped stream-k MMQ id-path (should_use_mmq()==true, e.g.
+            // Blackwell NVFP4) launches on-stream with NO host sync; only the per-expert
+            // host-loop fallback synchronizes the stream. So when this MUL_MAT_ID WILL take the
+            // grouped path, the whole decode step is graph-safe even for ne[2] > mmvq_mmid_max,
+            // and the full-step CUDA graph (incl. the MoE dispatch) can be REPLAYED instead of the
+            // host re-issuing every kernel every step. Patch 0025 proved this is bit-exact (graph
+            // replay re-issues identical kernels); D1 profiling confirmed the grouped path is what
+            // actually runs (no device->host routing readback), that steady decode is ~99% GPU-busy
+            // (not host-sync-bound), and that keeping the step graphed lifts throughput (npl32
+            // +13%, npl128 +1.9%). It is therefore ON BY DEFAULT for the grouped path now.
+            // should_use_mmq() is the exact guard: it returns FALSE for the large-M NVFP4 prefill
+            // (patch 0034) that deliberately drops to the per-expert host-sync loop, so PREFILL
+            // keeps graphs disabled (correct - that path syncs). Decode is untouched by 0034.
+            // LLAMA_MOE_NO_FORCE_GRAPHS=1 forces the conservative pre-0025 disable for A/B.
             if (mmid_needs_sync && ggml_is_quantized(node->src[0]->type) &&
-                getenv("LLAMA_MOE_FORCE_GRAPHS") != nullptr &&
+                getenv("LLAMA_MOE_NO_FORCE_GRAPHS") == nullptr &&
                 ggml_cuda_should_use_mmq(node->src[0]->type, cc, node->src[1]->ne[2], node->src[0]->ne[2])) {
                 mmid_needs_sync = false;
             }