[AMD][AgentX] Kimi-K2.7-Code FP4 MI355X agentX vLLM

configs/amd-master.yaml

@@ -548,6 +548,76 @@ kimik2.5-fp4-mi355x-vllm-agentic:
       - { tp: 4, kv-offloading: none, conc-list: [16, 24, 32, 40] }
       - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-native }, conc-list: [16, 24, 32, 40] }
 
+kimik2.7-fp4-mi355x-vllm-agentic:
+  image: vllm/vllm-openai-rocm:v0.24.0
+  model: amd/Kimi-K2.7-Code-MXFP4
+  model-prefix: kimik2.7
+  runner: cluster:mi355x-amds
+  precision: fp4
+  framework: vllm
+  multinode: false
+  scenarios:
+    agentic-coding:
+    # TP8, 3600s. Best-config sweep on the agentic trace, one KV tier per regime:
+    # GPU-only (none) is only viable at low conc [4,8] (peaks ~26.7 tok/s/GPU at
+    # c8, then collapses at the KV-capacity wall); LMCache DRAM offload carries the
+    # high-conc side and holds throughput where none thrashes, so it is swept at
+    # c16 (the crossover where it overtakes none). Mooncake and c32/48 dropped:
+    # they added no throughput headroom over this best-config envelope.
+    - dram-utilization: 0.80
+      duration: 3600
+      search-space:
+      - { tp: 8, kv-offloading: none, conc-list: [4, 8] }
+      - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "aaf7c0d3" }, conc-list: [16] }
+
+# EAGLE3 speculative-decoding ("MTP") variant of kimik2.7-fp4-mi355x-vllm-agentic.
+# spec-decoding: mtp routes to kimik2.7_fp4_mi355x_mtp.sh, which adds the
+# K2.7-Code-matched EAGLE3.1 draft (lightseekorg/kimi-k2.7-coder-eagle3.1-mla,
+# num_speculative_tokens 3). Mirrors the base best-config envelope (none [4,8] +
+# LMCache [16]) so the MTP throughput delta is apples-to-apples; spec decoding
+# lifts decode most at low conc, where the draft's acceptance is not
+# capacity-starved. Accuracy (SWE-bench agentic) runs via the shared EVAL_ONLY
+# path in the recipe.
+kimik2.7-fp4-mi355x-vllm-agentic-mtp:
+  image: vllm/vllm-openai-rocm:v0.24.0
+  model: amd/Kimi-K2.7-Code-MXFP4
+  model-prefix: kimik2.7
+  runner: cluster:mi355x-amds
+  precision: fp4
+  framework: vllm
+  multinode: false
+  scenarios:
+    agentic-coding:
+    - dram-utilization: 0.80
+      duration: 3600
+      search-space:
+      - { tp: 8, kv-offloading: none, conc-list: [4, 8], spec-decoding: mtp }
+      - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "aaf7c0d3" }, conc-list: [16], spec-decoding: mtp }
+
+# Smoke-test key for the MTP recipe: TP4 + TP8 x {none c4, LMCache c16}, 900s.
+# Same model-prefix (kimik2.7) + spec-decoding: mtp -> routes to
+# kimik2.7_fp4_mi355x_mtp.sh. Validates the EAGLE3 draft + flydsl/compile +
+# LMCache-alignment changes launch and produce aiperf numbers at both TP
+# widths. Temporary; drop once the recipe is verified.
+kimik2.7-fp4-mi355x-vllm-agentic-mtp-smoke:
+  image: vllm/vllm-openai-rocm:v0.24.0
+  model: amd/Kimi-K2.7-Code-MXFP4
+  model-prefix: kimik2.7
+  runner: cluster:mi355x-amds
+  precision: fp4
+  framework: vllm
+  multinode: false
+  scenarios:
+    agentic-coding:
+    - dram-utilization: 0.80
+      duration: 900
+      search-space:
+      - { tp: 4, kv-offloading: none, conc-list: [4], spec-decoding: mtp }
+      - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "aaf7c0d3" }, conc-list: [16], spec-decoding: mtp }
+      - { tp: 8, kv-offloading: none, conc-list: [4], spec-decoding: mtp }
+      - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "aaf7c0d3" }, conc-list: [16], spec-decoding: mtp }
+
+
 kimik2.5-fp4-mi355x-atom:
   image: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_202607091539
   model: amd/Kimi-K2.5-MXFP4

perf-changelog.yaml

@@ -4974,3 +4974,31 @@
     - "Run 29651235293 showed the 1M-context corpus working set outgrowing the HBM KV pool past conc 8 (TP8) / conc 64 (DP8): gpu_kv_cache_usage pinned at 1.0 and the radix hit rate collapsed from a ~0.97 theoretical ceiling to 0.04-0.06, so every post-knee turn re-prefilled its full history and throughput fell together with interactivity"
     - "HiCache spills evicted prefixes to host DRAM and restores them at C2C bandwidth instead of recomputing; sizing follows the qwen3.5-fp8-b300-sglang-agentic-hicache recipe (GLM-5.2 is plain GQA: one host pool per rank, GB-based --hicache-size)"
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2280
+
+- config-keys:
+    - kimik2.7-fp4-mi355x-vllm-agentic
+  description:
+    - "Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X (runner: cluster:mi355x-amds) via vLLM v0.24.0, TP4"
+    - "Sweep KV-offloading none (conc [1,4,8]) vs dram/LMCache (conc [16,32]); LMCache MP server + LMCacheMPConnector, dram-utilization 0.80"
+    - "Recipe benchmarks/single_node/agentic/kimik2.7_fp4_mi355x.sh clones LMCache to container-local /opt/lmcache-src (pinned) so CI checkout never trips over root-owned build artifacts"
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2126
+
+- config-keys:
+    - kimik2.7-fp4-mi355x-vllm-agentic
+  description:
+    - "Move the agentic-coding sweep from TP4 to TP8 and compare three KV tiers on the agentic trace: GPU-only (none) at conc [4,8], and dram/LMCache + dram/Mooncake at conc [16,32,48] (3600s trace replay)"
+    - "TP4 was GPU-KV-capacity bound: the ~121 GiB pool held only ~12 resident ~90k-token contexts, so throughput plateaued (~160 tok/s) and requests preempted/thrashed from conc16 up. GPU-only still collapses under load at TP8, so it is swept only at low conc; the dram offload backends carry the high-conc range"
+    - "Add a Mooncake KV-offload backend to the recipe (embedded MooncakeStoreConnector store). Built from source for ROCm via Mooncake's build_wheel.sh and pip-installed so the mooncake package lands in site-packages and is importable from the vLLM worker subprocesses (a bare make install left workers unable to import it)"
+    - "Adds optional per-scenario 'duration' override to the agentic-coding config schema (defaults to 3600s when unset)"
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2126
+
+- config-keys:
+    - kimik2.7-fp4-mi355x-vllm-agentic-mtp
+    - kimik2.7-fp4-mi355x-vllm-agentic
+  description:
+    - "Add EAGLE3 speculative-decoding ('MTP') variant kimik2.7-fp4-mi355x-vllm-agentic-mtp: spec-decoding: mtp routes to benchmarks/single_node/agentic/kimik2.7_fp4_mi355x_mtp.sh, which serves the K2.7-Code-matched draft lightseekorg/kimi-k2.7-coder-eagle3.1-mla (EAGLE3.1 MLA, num_speculative_tokens 3, env-overridable via DRAFT_MODEL/NUM_SPEC_TOKENS/VLLM_SPEC_CONFIG). Kimi-K2 has no native MTP, so the draft is external; it is pre-staged with hf download before serving so a cold pull cannot blow READY_TIMEOUT at engine init"
+    - "Mirrors the base best-config envelope (GPU-only none [4,8] + LMCache dram [16], TP8, 3600s) so the MTP throughput delta is apples-to-apples"
+    - "Add the SWE-bench agentic accuracy path (EVAL_ONLY -> run_eval) to both the base and MTP recipes, plus the Kimi eval parser flags on the vLLM server (--tool-call-parser kimi_k2 --enable-auto-tool-choice --reasoning-parser kimi_k2) so message.tool_calls parse for the agent harness; harmless for the throughput replay path"
+    - "Align both recipes' server flags with the official recipes.vllm.ai/moonshotai/Kimi-K2.7-Code MI355X recipe: add --moe-backend flydsl (FlyDSL W4A16 MoE kernel for gfx950) and --compilation-config {pass_config:{fuse_allreduce_rms:false}} (required on MI350X/MI355X). Retains the sweep-tuned --block-size=1 + VLLM_ROCM_USE_AITER_MLA=1 (~3x TRITON_MLA on the agentic trace), which the recipe page does not specify"
+    - "Harden the LMCache MP server config (all flags verified present at the pinned aaf7c0d3 build): split the worker pools --max-gpu-workers 1 / --max-cpu-workers 8 (was --max-workers TP*2, which set both pools equal) to avoid concurrent-GPU-transfer stalls under async-load (the conc64 stall class), and add --chunk-size 1024, --l1-align-bytes 16384, --eviction-trigger-watermark 0.85, --eviction-ratio 0.10, --supported-transfer-mode lmcache_driven. Keeps the dynamic SHM-aware --l1-size-gb sizing rather than a hardcoded value"
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2126

utils/matrix_logic/generate_sweep_configs.py

@@ -700,7 +700,7 @@ def generate_full_sweep(args, all_config_data, runner_data):
 
         for agentic_config in agentic_configs:
             bmk_space = agentic_config[Fields.SEARCH_SPACE.value]
-            duration = DEFAULT_AGENTIC_DURATION_SECONDS
+            duration = agentic_config.get(Fields.DURATION.value) or DEFAULT_AGENTIC_DURATION_SECONDS
 
             for bmk in bmk_space:
                 if is_multinode:
@@ -1003,7 +1003,7 @@ def generate_test_config_sweep(args, all_config_data, runner_data=None):
         # ---- Agentic-coding scenarios ----
         agentic_configs = val[Fields.SCENARIOS.value].get(Fields.AGENTIC_CODING.value, []) if (scenario_filter is None or 'agentic-coding' in scenario_filter) else []
         for agentic_config in agentic_configs:
-            duration = DEFAULT_AGENTIC_DURATION_SECONDS
+            duration = agentic_config.get(Fields.DURATION.value) or DEFAULT_AGENTIC_DURATION_SECONDS
             bmk_space = agentic_config[Fields.SEARCH_SPACE.value]
 
             for bmk in bmk_space:

utils/matrix_logic/test_validation.py

@@ -575,10 +575,33 @@ def test_available_cpu_dram_is_not_a_master_config_field(self):
                 }],
             })
 
-    def test_duration_is_not_a_master_config_field(self):
+    def test_duration_is_an_optional_master_config_field(self):
+        # Per-scenario duration override (falls back to the default 3600s in the
+        # sweep generator when unset). Used for shorter agentic smoke sweeps.
+        cfg = AgenticCodingConfig(**{
+            "duration": 1800,
+            "search-space": [{
+                "tp": 8,
+                "kv-offloading": "none",
+                "conc-list": [16],
+            }],
+        })
+        assert cfg.duration == 1800
+
+    def test_duration_defaults_to_none_when_unset(self):
+        cfg = AgenticCodingConfig(**{
+            "search-space": [{
+                "tp": 8,
+                "kv-offloading": "none",
+                "conc-list": [16],
+            }],
+        })
+        assert cfg.duration is None
+
+    def test_duration_must_be_positive(self):
         with pytest.raises(Exception, match="duration"):
             AgenticCodingConfig(**{
-                "duration": 1800,
+                "duration": 0,
                 "search-space": [{
                     "tp": 8,
                     "kv-offloading": "none",

utils/matrix_logic/validation.py

@@ -638,6 +638,11 @@ class AgenticCodingConfig(BaseModel):
     dram_utilization: Optional[float] = Field(
         default=None, alias=Fields.DRAM_UTILIZATION.value, gt=0, le=1
     )
+    # Per-scenario trace-replay duration override (seconds). Falls back to
+    # DEFAULT_AGENTIC_DURATION_SECONDS in the sweep generator when unset.
+    duration: Optional[int] = Field(
+        default=None, alias=Fields.DURATION.value, gt=0
+    )
 
     @model_validator(mode='after')
     def validate_dram_offload_capacity(self):