fix: recover PR 2380 ingest via sweep reuse (#2418)

* fix(kimik3): keep only TRTLLM_RAGGED in the DSpark recipe, revert the rest

Reduce the flag alignment to its one surviving piece and rebuild on main.

The full upstream Blackwell/NVIDIA block killed the engine core on every
completed concurrency -- c1/c2/c4/c8 across both KV arms in run 30378755933 --
roughly 13-16 min in, during agentic warmup:

  vllm/model_executor/layers/fused_moe/runner/shared_experts.py:165
      assert self._output[self._output_idx] is None
  AssertionError            (all 8 TP ranks simultaneously)

K3 declares num_shared_experts 2 and this image resolves its MoE backend to
FLASHINFER_TRTLLM_MXFP4_MXFP8 / TrtLlmMxfp4ExpertsMonolithic, so that reads as an
image-side invariant violation rather than a bad configuration value. The same
crash hit the non-DSpark recipe, so it is not spec-decoding specific.

Retained:  --attention-config mla_prefill_backend=TRTLLM_RAGGED
Reverted:  gpu-memory-utilization 0.95 -> 0.90
           max-num-seqs 32 -> 2*CONC
           max-model-len 1000000 -> 1048576
           max-num-batched-tokens 32768 -> dropped (8192 default)
           --no-enable-flashinfer-autotune -> dropped
           VLLM_USE_V2_MODEL_RUNNER=1 -> dropped (V2 is on by default anyway)

cudagraph capture consequently enumerates 2*CONC token-multiples of (1+7)=8
again, instead of a fixed 32-entry ladder to 256.

Rebased onto main rather than amending, since main had advanced four commits and
the changelog diff had started reading main's tail as deletions.

* fix(kimik3): enable VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION on the DSpark recipe

The upstream recipe sets this on BOTH its blackwell and nvidia paths
(recipes.vllm.ai/moonshotai/Kimi-K3), and this repo has never set it. It
defaults to 0.

It matters because of where it lands: vllm/models/kimi_k3/nvidia/model.py:549
threads it into LatentMoERunner as
runner_args={"enable_k3_latent_moe_tail_fusion": ...}, and that is the same
runner whose shared-experts output buffer asserted when the wider flag alignment
was tried:

  vllm/model_executor/layers/fused_moe/runner/shared_experts.py:165
      assert self._output[self._output_idx] is None   (all 8 TP ranks)

So with it unset we have been running a MoE tail path upstream never exercises,
on every K3 run here including the green ones. That is a better explanation for
the crash than any of the six flags reverted earlier.

Enabled on its own so its effect is isolated. If this holds, the remaining
upstream deltas can be re-added in order: gpu-memory-utilization 0.95,
max-num-seqs 32 (spec arm), --no-enable-flashinfer-autotune,
VLLM_USE_V2_MODEL_RUNNER=1.

Not re-adding --max-num-batched-tokens 32768: re-reading the recipe YAML shows
that value appears nowhere for blackwell (AMD is 4096, PD prefill 16384, PD
decode 32), so it was introduced in error. --max-model-len is likewise 1048576
upstream, not 1000000, and already correct here.

* feat(kimik3): DSpark at level 2 with synthetic acceptance pinned to golden AL

Adopt the AgentX-prescribed measurement methodology for the DSpark arm, per
golden_al_distribution/README.md: "a submission may choose any supported draft
length, but it may not substitute a different acceptance target", because
"InferenceX is evaluating inference-system performance, not the ability to
fine-tune a benchmark-specific speculative head". The README also states up front
that AL is workload-dependent, so pinning the committed acceptance is what makes
submissions comparable -- running the acceptance this corpus happens to produce
would be the non-conforming choice.

  throughput: rejection_sample_method synthetic, synthetic_acceptance_length 2.51
              (committed golden AL at K=2, probabilistic curve)
  EVAL_ONLY:  rejection_sample_method block (real verification)

The eval split matters: synthetic acceptance commits drafts regardless of target
logits, so text is wrong and SWE-bench scores 0.0000. This follows
dsv4_fp4_b300_vllm_mtp.sh; kimik2.5_fp4_b300_mtp.sh omits the split.

Level 2, not 7. Draft length IS ours to choose, and verify width is a real system
cost synthetic acceptance does not remove -- at K=7 the target verifies 8
positions per step, which loses to no-speculation once the GPU saturates even at
the forced golden AL of 3.84. With cost ~ 1 + m(K-1) + f: K=2 clears break-even
across the m range (~1.7x low conc, ~1.2x saturated), K=3 is a wash at saturation
(~0.95x), K=7 is negative (~0.54x). cudagraph capture therefore enumerates
multiples of (1+2)=3.

Recorded in the script and config headers, because the throughput numbers will
otherwise be misread: the draft's REAL acceptance on this corpus is 1.16-2.01
with 13-41% position-1 acceptance. Its native window is 32k (rope yarn factor 32
over original_max_position_embeddings 32768, stretched to the declared 1M), and
the identical image, draft and spec config reach AL 4.18 / 0.826 position-1 on
short-context work. The synthetic numbers measure the system at a prescribed
acceptance; they are not evidence the draft predicts well at 100k+ context.

* chore: prepare PR 2380 ingest recovery

* fix: recover PR 2380 ingest

Author: Oseltamivir

perf-changelog.yaml

@@ -5296,3 +5296,18 @@
     - "Enable multiple frontends and MessagePack request-plane encoding, and reduce 1P4D decode GPU memory utilization to 0.85"
     - "Drop the recipe setup_script entirely (removes the DeepGEMM NVLink barrier timeout patch); required container deps now ship in vLLM 0.25.1, on NVIDIA/srt-slurm v1.0.31"
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2139
+
+- config-keys:
+    - kimik3-fp4-b300-vllm-agentic-dspark
+  description:
+    - "Add the DSpark speculative-decoding variant of kimik3-fp4-b300-vllm-agentic: Kimi-K3 (MXFP4, 2.8T MoE, KDA/MLA hybrid, 1M native context) on B300 with vLLM, agentic-coding scenario only -- no fixed-seq-len (1k1k / 8k1k) arms."
+    - "Speculative config: method dspark on the Inferact/Kimi-K3-DSpark draft head, num_speculative_tokens 2, attention_backend FLASHINFER_MLA, draft_sample_method probabilistic, with rejection_sample_method synthetic and synthetic_acceptance_length 2.51 -- the committed golden AL at K=2 on the probabilistic curve (golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml). This follows the AgentX policy in golden_al_distribution/README.md: a submission may choose any supported draft length but may not substitute a different acceptance target, because InferenceX measures inference-system performance rather than the fitness of a benchmark-specific speculative head. The greedy/standard curve gives 2.45 at K=2; the curves are not mixed."
+    - "EVAL_ONLY runs switch to real block verification. Synthetic acceptance commits drafted tokens regardless of the target logits, so generated text is wrong and SWE-bench scores 0.0000 -- the same split dsv4_fp4_b300_vllm_mtp.sh makes, and which kimik2.5_fp4_b300_mtp.sh omits; dsv4 is the pattern followed here."
+    - "Level 2 rather than 7. Draft length is the submission's choice and verify width is a real system cost synthetic acceptance does not remove: at K=7 the target verifies 8 positions per step, which loses to no-speculation once the GPU saturates even at the forced golden AL of 3.84 (K=2 clears break-even at both ends of the cost range; K=3 is a wash at saturation; K=7 is negative). cudagraph capture consequently enumerates multiples of (1+2)=3, since vLLM rounds configured sizes to multiples of (1+N) and would otherwise cover only MAX_NUM_SEQS/3 sequences. The drafter's KV footprint costs ~25% of GPU KV regardless of K and remains a genuine cost of enabling spec decoding on 100k+ prompts."
+    - "The Inferact/Kimi-K3-DSpark draft is not pre-staged on the B300 cluster, so it is downloaded once to the launcher's writable models dir (/data/models, same placement as DeepSeek-V4-Pro-DSpark) and cached across the sweep. It must not go next to the target: Kimi-K3 is in STAGED_MODELS, so dirname(MODEL_PATH) is the read-only /scratch/models mount."
+    - "Config key uses the -dspark suffix rather than the repo-wide -mtp convention, to name the actual speculative method. Note the schema field stays spec-decoding: mtp (its Literal has no dspark member), so generated exp-names, artifacts and job labels still read _spec-mtp, and the benchmark script remains kimik3_fp4_b300_vllm_mtp.sh because the launcher derives its _mtp suffix from that field rather than from the key."
+    - "Serve flags stay on the values that ran 12/12 green in run 30326393603, with one exception adopted from recipes.vllm.ai/moonshotai/Kimi-K3: --attention-config mla_prefill_backend=TRTLLM_RAGGED (the non-DSpark sibling gets the same swap in #2385). Adopting the rest of the upstream Blackwell/NVIDIA block at the same time -- gpu-memory-utilization 0.95, max-num-seqs 32, max-model-len 1000000, max-num-batched-tokens 32768, --no-enable-flashinfer-autotune, VLLM_USE_V2_MODEL_RUNNER=1 -- killed the engine core on every completed concurrency (c1/c2/c4/c8, both KV arms, run 30378755933) about 13-16 min in during agentic warmup, with an assertion in the FlashInfer MoE runner's shared-experts output buffer (vllm/model_executor/layers/fused_moe/runner/shared_experts.py:165, assert self._output[self._output_idx] is None) firing on all 8 TP ranks at once. So max-num-seqs stays at 2*CONC and cudagraph capture enumerates 2*CONC token-multiples of (1+7)=8 rather than a fixed 32/256 ladder."
+    - "Measured acceptance does NOT reproduce the golden curve on this workload. A partial run (30337911387, 8 of 24 jobs before an unrelated enroot/pyxis flake cancelled the rest) recorded mean acceptance length 1.18-1.68 against the golden 3.84, per-position acceptance decaying 0.26 / 0.12 / 0.087 / 0.056 / 0.037 / 0.019 / 0.007 and an 8-10% average draft acceptance rate -- drafting 489-515 tok/s to accept 41-48. Throughput regressed 20-63% vs the non-MTP arms at conc 4 and 8. The spec config itself matches upstream exactly, and the draft supports the full 1M context (max_position_embeddings 1048576), so the gap is workload: the golden curve was collected on SPEED-Bench coding at max_model_len 16384 with short single-turn prompts, whereas agentic replays 118-230k-token multi-turn tool-calling traces. Treat the SPEED-Bench curve as non-transferable here and re-measure AL on the agentic corpus before trusting a level-7 configuration. Note rejection_sample_method=block is what exposed this: a synthetic_acceptance_length pinned to 3.84 would have reported a phantom speedup."
+    - "Search space mirrors the non-MTP entry's KV arms -- TP8 GPU-resident and TP8 host-DRAM offload -- so the spec-decoding delta is readable at equal concurrency, but stops at conc 16 rather than 24. The non-MTP bring-up sweep (run 30326393603) showed the GPU-resident arm already thrashing at conc >= 16 (prefix cache hit rate 2.7%, TTFT p50 86-191s) because GPU KV holds only ~3.1 max-length requests, so conc 24 would spend a full job per arm re-measuring that regime; conc 16 still exercises the DRAM tier meaningfully (62% external prefix cache hit rate). TP8-only for the same memory reason: a ~1.5 TB MXFP4 checkpoint needs ~188 GB/GPU across 8 B300s and does not fit below 8 GPUs."
+    - "Sets VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, which the upstream recipe requires on both its blackwell and nvidia paths and which this repo had never set. It defaults to 0 and is threaded into LatentMoERunner as runner_args={\"enable_k3_latent_moe_tail_fusion\": ...} at vllm/models/kimi_k3/nvidia/model.py:549 -- the same runner whose shared-experts output buffer asserted (fused_moe/runner/shared_experts.py:165, all 8 TP ranks at once) when the wider flag alignment was attempted, so every K3 run here so far has been exercising a MoE tail path upstream does not use. Enabled on its own, ahead of re-attempting gpu-memory-utilization 0.95, max-num-seqs 32, --no-enable-flashinfer-autotune or VLLM_USE_V2_MODEL_RUNNER=1, to isolate its effect."
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2418