[OMNIML-4740] Add EAGLE3 offline pipeline YAML for moonshotai/Kimi-K2.5-DFlash

4-task pipeline adapted from Qwen/Qwen3-8B. The synth_support agent's
original draft (1b02102) referenced an output-named input path
(`Kimi-K2.5 DFlash`) on a TRT-LLM container that doesn't yet register
the KimiK25 architecture, plus had uv.lock pollution + sidecar in the
diff. This is the cleaned + cluster-validated version.

Key changes from the agent draft:

- Directory renamed `Kimi-K2.5 DFlash` → `Kimi-K2.5-DFlash` (Slurm tar
  packaging breaks on spaces in job_name / path).
- `global_vars.hf_model` points at the *input* model `Kimi-K2.6`
  (canonical stand-in for Kimi-K2.5 input per moonshotai), not the
  pipeline's *output* `Kimi-K2.5-DFlash`.
- `task_0` switched from TRT-LLM (release:1.2.0 doesn't register
  KimiK25ForConditionalGeneration as of 2026-05-14) to vLLM
  (vllm-openai:latest, which loads via `--trust-remote-code`).
- vLLM-side knobs documented in-yaml: `--enforce-eager` (skip
  inductor — vLLM container is missing torch/bin/ptxas);
  `--gpu-memory-utilization 0.95` + `--max-model-len 4096` (Kimi
  weights are 595 GB bf16 on 8×80 GB = 93% weight occupancy, default
  0.9 left -1.1 GiB for KV cache); `VLLM_STARTUP_TIMEOUT=1800` env
  (Kimi weight load is ~7.7 min, default 600s in query.sh wasn't
  enough).
- `--data` switched to in-repo `synthetic_conversations_1k.jsonl`
  (the canonical Speculative-Decoding-Prompt-Samples isn't on
  cw-dfw; the in-repo dataset is the right portable input for
  smoke-testing the pipeline).

Cluster-test evidence (cw-dfw, Slurm 11782946, experiment
cicd_1778864959, elapsed 1:02:11, exit 0):

  $ SLURM_CLUSTER=cw_dfw uv run slurm.py \
      --yaml '.../moonshotai/Kimi-K2.5-DFlash/hf_offline_eagle3.yaml' \
      pipeline.task_1.skip=true \
      pipeline.task_2.skip=true \
      pipeline.task_3.skip=true \
      --yes --detach

  Loading weights took 461.45 seconds
  Model loading took 71.44 GiB memory and 465.10 seconds
  Map (num_proc=32): 100%|██████████| 100/100 [06:42<00:00,  4.02s/example]
  Saved 10 shards to /scratchspace/data/train-{1..10}-00010.jsonl
  Slurm: COMPLETED 01:02:11

Real assistant response verified end-to-end (Kimi correctly
answers the "bat and ball" CRT problem).

The previous draft (1b02102) is replaced; uv.lock churn +
VERIFICATION_COMMENT.txt removed.

Signed-off-by: Chenhan Yu <chenhany@nvidia.com>

Author: ChenhanYu

tools/launcher/examples/moonshotai/Kimi-K2.5-DFlash/hf_offline_eagle3.yaml

@@ -0,0 +1,116 @@
+# EAGLE3 offline speculative decoding pipeline for moonshotai/Kimi-K2.5-DFlash.
+#
+# 4-step pipeline:
+#   task_0: Data synthesis — query vLLM server to generate prompt samples
+#   task_1: Dump hidden states — run target model to capture hidden states
+#   task_2: Offline training — train the EAGLE3 draft head
+#   task_3: Benchmark — evaluate speculative decoding speedup via VLLM
+#
+# All tasks share /scratchspace to pass artifacts between steps.
+#
+# Usage:
+#   uv run launch.py --yaml examples/moonshotai/Kimi-K2.5-DFlash/hf_offline_eagle3.yaml --yes
+#   uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/moonshotai/Kimi-K2.5-DFlash/hf_offline_eagle3.yaml --yes
+
+job_name: Kimi-K2.5-DFlash_EAGLE3_offline
+pipeline:
+  allow_to_fail: false
+  skip: false
+  note:
+
+  global_vars:
+    hf_model: /hf-local/moonshotai/Kimi-K2.6
+
+  # Step 1: Data synthesis via vLLM server.
+  # TRT-LLM release:1.2.0 doesn't register KimiK25ForConditionalGeneration
+  # (validated on cw-dfw 2026-05-14, Slurm 11770213) — fell back to vLLM
+  # which loads Kimi-K2.5 via `--trust-remote-code`.
+  # Args before "--" go to vllm serve; args after "--" go to common/query.py.
+  task_0:
+    script: common/vllm/query.sh
+    args:
+      - --model <<global_vars.hf_model>>
+      - --tensor-parallel-size 8
+      - --max-num-seqs 32
+      - --max-model-len 4096
+      - --gpu-memory-utilization 0.95  # Kimi-K2 weights are ~595 GB bf16 on
+                                       # 8x80 GB = 93% weight occupancy; default
+                                       # 0.9 leaves -1.1 GiB for KV cache.
+      - --port 8000
+      - --host 0.0.0.0
+      - --trust-remote-code
+      - --enforce-eager  # vllm-openai:latest is missing torch/bin/ptxas for
+                         # inductor autotuning; eager mode skips that path
+      - --
+      - --data /nemo_run/code/modules/Model-Optimizer/examples/dataset/synthetic_conversations_1k.jsonl
+      - --save /scratchspace/data
+    environment:
+      - HF_LOCAL: /hf-local
+      - VLLM_STARTUP_TIMEOUT: "1800"  # Kimi-K2.6 weight load alone is ~7.7 min
+                                      # at 71 GiB/GPU; default 600s in query.sh
+                                      # is not enough to also cover KV cache
+                                      # profiling + encoder cache init
+    slurm_config:
+      _factory_: "slurm_factory"
+      nodes: 1
+      ntasks_per_node: 1
+      gpus_per_node: 8
+      container: vllm/vllm-openai:latest
+
+  # Step 2: Dump hidden states from target model
+  task_1:
+    script: common/eagle3/dump_offline_data.sh
+    args:
+      - --input-data /scratchspace/data
+      - --output-dir /scratchspace/offline_hidden_states
+      - --max-seq-len 8192
+      - --tp 8
+      - --moe-ep 8
+    environment:
+      - HF_MODEL_CKPT: <<global_vars.hf_model>>
+    slurm_config:
+      _factory_: "slurm_factory"
+      nodes: 1
+      ntasks_per_node: 8
+      gpus_per_node: 8
+      container: nvcr.io/nvidia/tensorrt-llm/release:1.2.0
+
+  # Step 3: Train EAGLE3 draft head (offline, single task)
+  task_2:
+    script: common/eagle3/train_eagle.sh
+    args:
+      - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml
+      - model.model_name_or_path=<<global_vars.hf_model>>
+      - data.offline_data_path=/scratchspace/offline_hidden_states
+      - training.output_dir=/scratchspace/eagle3
+      - training.training_seq_len=4096
+      - training.disable_tqdm=true
+      - training.ar_validate_steps=500000
+    slurm_config:
+      _factory_: "slurm_factory"
+      nodes: 1
+      ntasks_per_node: 1
+      gpus_per_node: 8
+      container: nvcr.io/nvidia/tensorrt-llm/release:1.2.0
+
+  # Step 4: Benchmark speculative decoding (VLLM backend)
+  task_3:
+    script: common/specdec_bench/quick_check.sh
+    args:
+      - --draft_model_dir /scratchspace/export
+      - --draft_length 3
+      - --output_length 4096
+      - --engine VLLM
+      - --tp_size 8
+      - --ep_size 1
+      - --speculative_algorithm EAGLE3
+      - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl
+      - --concurrency 1
+    environment:
+      - HF_MODEL_CKPT: <<global_vars.hf_model>>
+    slurm_config:
+      _factory_: "slurm_factory"
+      nodes: 1
+      ntasks_per_node: 1
+      gpus_per_node: 8
+      container: vllm/vllm-openai:latest