[AMD][AgentX] kimik3: try lazy offload for fp8 KV; --block-size is inert on K3

The block-size attempt (run 30505656455) is void. K3 is hybrid, so vLLM raises
the attention block size until the attention page is at least the mamba/KDA
page (interface.py:911) and silently overrode --block-size 64 to 1536. Measured
pages are bf16 768 / fp8 1536, so that cell was byte-identical to the earlier
eager failures and died the same way. Keep the KV_BLOCK_SIZE knob but document
it as inert here. This also explains the connector's identical 14122 CPU blocks
across dtypes: fp8 halves bytes/token but doubles page tokens, so page bytes --
and manager.py:199's pool -- are unchanged.

New lever: lazy offload. Eager stores every completed block to CPU at once;
every fp8+connector death has come while the GPU pool was nearly empty (11.6%
usage, 0.0% external hit rate in 30505656455), i.e. mid store-storm with
nothing yet to gain from the host tier, and the last two were a 600s RCCL
_ALLGATHER_BASE hang rather than an OOM. Lazy only stores under GPU-block
pressure, removing nearly all connector traffic in that regime. It is also the
mode kimik3_fp4_mi355x_mtp.sh already pins for gfx950 (PR #2367).

Add a vllm-simple-fp8-lazy backend and swap the fp8b64 key for fp8lazy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: hyukjlee

benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh

@@ -436,17 +436,22 @@ except Exception:
         )
     fi
     ;;
-  vllm-simple|vllm-simple-fp8|vllm-simple-fp8-b64)
+  vllm-simple|vllm-simple-fp8|vllm-simple-fp8-lazy)
     require_agentic_kv_offload_backend "$KV_OFFLOAD_BACKEND"
-    # vllm-simple-fp8-b64 is vllm-simple-fp8 with the KV page pinned to 64
-    # tokens. Under fp8 vLLM otherwise picks 128, which routes MLA into aiter's
-    # Gluon b128 kernel; fp8 survives that alone but every fp8 cell that also
-    # ran this connector died with a GPU fault (3/3, g17/g19/g12, at <10% pool
-    # usage) while bf16 + the same connector ran to 99.9%. 64 is the page bf16
-    # used, so this holds the one geometry that is known to work with the
-    # connector while keeping fp8.
-    if [ "$KV_OFFLOAD_BACKEND" = "vllm-simple-fp8-b64" ]; then
-        KV_BLOCK_SIZE="${KV_BLOCK_SIZE:-64}"
+    # vllm-simple-fp8-lazy is vllm-simple-fp8 with lazy offload. In eager mode
+    # the connector stores every completed block to CPU immediately; in lazy
+    # mode it only stores under GPU-block pressure (manager.py _lazy_mode /
+    # _estimate_lazy_target_blocks walk the GPU free queue). Every fp8+connector
+    # cell so far has died while the GPU pool was nearly EMPTY -- 11.6% usage
+    # with a 0.0% external hit rate in run 30505656455, i.e. mid store-storm
+    # with nothing yet to gain from the host tier -- so the store path is where
+    # to look, and lazy removes almost all of it in that regime.
+    #
+    # Also the mode Wenyao's kimik3_fp4_mi355x_mtp.sh pins (SIMPLE_LAZY_OFFLOAD
+    # =true) as part of the gfx950 bundle validated in PR #2367 against an
+    # eight-rank GPU memory access fault.
+    if [ "$KV_OFFLOAD_BACKEND" = "vllm-simple-fp8-lazy" ]; then
+        SIMPLE_LAZY_OFFLOAD="${SIMPLE_LAZY_OFFLOAD:-true}"
     fi
     # vllm-simple-fp8 is the same connector with an fp8 KV cache. fp8 halves
     # bytes/token in the GPU pool, which on this KV-bound corpus moves the
@@ -456,7 +461,7 @@ except Exception:
     # state and gated-MLA latent, and fp8 support across both spec types is
     # unconfirmed on this build.
     case "$KV_OFFLOAD_BACKEND" in
-      vllm-simple-fp8|vllm-simple-fp8-b64)
+      vllm-simple-fp8|vllm-simple-fp8-lazy)
         KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}"
         ;;
     esac
@@ -558,17 +563,17 @@ if [ -n "${KV_CACHE_DTYPE:-}" ] && [ "${KV_CACHE_DTYPE}" != "auto" ]; then
     KV_CACHE_DTYPE_ARGS=(--kv-cache-dtype "$KV_CACHE_DTYPE")
 fi
 
-# Unset by default: vLLM sizes the KV page itself. Under fp8 it picks a 128-token
-# page (bf16 gets 64) -- measured as an identical 14122 CPU blocks in the offload
-# connector across both dtypes while GPU token capacity went 1,957,290 ->
-# 3,859,490, which per manager.py:199 (num_cpu_blocks scales with num_gpu_blocks)
-# can only mean the block count held and tokens/block doubled.
+# INERT ON K3 -- kept only so the override stays discoverable. K3 is hybrid, and
+# vLLM raises the attention block size until the attention page is at least the
+# mamba/KDA page (interface.py:911, "Setting attention block size to N tokens to
+# ensure that attention page size is >= mamba page size"). Measured: bf16 -> 768
+# tokens, fp8 -> 1536. --block-size 64 was silently overridden to 1536 in run
+# 30505656455, so this knob cannot move the page on this model; only
+# kv_cache_dtype can.
 #
-# A 128-token page routes MLA into aiter's Gluon b128 kernel. That path is fine
-# on its own (the gist patch above; fp8 kvnone cells pass), but every fp8 cell
-# that ALSO ran SimpleCPUOffloadConnector has died with a GPU fault -- 3/3 on
-# g17/g19/g12, at <10% pool usage, while the same connector on bf16 ran to 99.9%.
-# Set KV_BLOCK_SIZE=64 to hold fp8 on the geometry bf16 used.
+# That also explains the connector's identical 14122 CPU blocks across dtypes:
+# fp8 halves bytes/token but the page doubles in tokens, so page BYTES are
+# unchanged and manager.py:199 derives the same pool.
 KV_BLOCK_SIZE_ARGS=()
 if [ -n "${KV_BLOCK_SIZE:-}" ] && [ "${KV_BLOCK_SIZE}" != "0" ]; then
     KV_BLOCK_SIZE_ARGS=(--block-size "$KV_BLOCK_SIZE")

configs/amd-master.yaml

@@ -798,28 +798,27 @@ kimik3-fp4-mi355x-vllm-agentic-fp8eval:
       search-space:
       - { tp: 8, kv-offloading: none, conc-list: [8] }
 
-kimik3-fp4-mi355x-vllm-agentic-fp8b64:
-  # One variable off the cell that has now failed 3/3: fp8 KV + the
-  # SimpleCPUOffloadConnector, with the KV page pinned to 64 tokens.
+kimik3-fp4-mi355x-vllm-agentic-fp8lazy:
+  # fp8 KV + SimpleCPUOffloadConnector in LAZY mode. One variable off the cell
+  # that has now failed 4/4 (g17/g19/g12/g12), and the block-size attempt in run
+  # 30505656455 is void: vLLM overrode --block-size 64 to 1536 because K3's
+  # attention page must be >= its mamba/KDA page (interface.py:911), so that
+  # cell was byte-identical to the eager failures.
   #
-  # Runs 30453589555 and 30457683686 killed every fp8+connector cell with a GPU
-  # fault (g17/g19/g12) at under 10% pool usage, while the SAME connector on
-  # bf16 (run 30431115226) ran to 99.9% with zero faults, and fp8 WITHOUT the
-  # connector passes (kvnone c4/c8/c16). The two runs are config-identical
-  # except kv_cache_dtype -- same vLLM g5f76ae224, GMU 0.88, max-num-seqs 128,
-  # same cpu_bytes_to_use_per_rank -- and diffing vllm_command.txt leaves
-  # --kv-transfer-config as the only delta.
+  # Why lazy. Eager stores every completed block to CPU immediately. Every
+  # fp8+connector death has happened while the GPU pool was nearly empty --
+  # 11.6% usage, 0.0% external hit rate in 30505656455 -- i.e. deep in a store
+  # storm with nothing yet to gain from the host tier, and the last two deaths
+  # were a 600 s RCCL _ALLGATHER_BASE hang rather than an OOM. Lazy only stores
+  # under GPU-block pressure, so it removes nearly all connector traffic in
+  # exactly the regime that is killing the cell.
   #
-  # Both runs allocate an identical 14122 CPU blocks while GPU token capacity
-  # goes 1,957,290 -> 3,859,490. manager.py:199 scales num_cpu_blocks with
-  # num_gpu_blocks, so the block count held and tokens/block doubled: fp8 picks
-  # a 128-token page, bf16 gets 64. A 128 page routes MLA into aiter's Gluon
-  # b128 kernel -- survivable alone (the gist patch; run 30442578333 got 696
-  # tok/s/GPU) but not, so far, alongside block-granular connector copies.
+  # It is also the mode kimik3_fp4_mi355x_mtp.sh already pins for gfx950
+  # (PR #2367) against an eight-rank GPU memory access fault.
   #
-  # If this passes, the b128 page is the culprit and fp8 keeps the host tier.
-  # If it dies the same way, the page is not the variable and the next step is
-  # reading the worker-side copy path rather than guessing at geometry.
+  # Control: bf16 + the same connector in EAGER mode ran to 99.9% pool usage
+  # with zero faults (run 30431115226), and fp8 WITHOUT the connector peaks at
+  # 754 tok/s/GPU at c8 (run 30453589555).
   image: vllm/vllm-openai-rocm:kimi-k3
   model: moonshotai/Kimi-K3
   model-prefix: kimik3
@@ -831,7 +830,7 @@ kimik3-fp4-mi355x-vllm-agentic-fp8b64:
     agentic-coding:
     - dram-utilization: 0.80
       search-space:
-      - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8-b64 }, conc-list: [8] }
+      - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple-fp8-lazy }, conc-list: [8] }
 
 kimik3-fp4-mi355x-vllm-agentic-fp8kv:
   # fp8 KV on the OFFLOAD arm. This is the fp8kvtest arm (run 30442578333: