fix(agentx): adopt additive AIPerf main warmup

.github/workflows/README.md

@@ -159,11 +159,12 @@ are also reusable.
 
 ## AgentX Fast Mode
 
-Add `agentx-fast` alongside one primary sweep label to run the 5-minute cache
-warmup and 20-minute profiling preset for single- and multi-node AgentX
-throughput jobs. Fixed-sequence throughput and eval jobs retain their canonical
-settings. Adding or removing the modifier restarts the active sweep. Fast-mode
-runs are not eligible for artifact reuse after merge.
+Add `agentx-fast` alongside one primary sweep label to run one additional
+warmup request per AgentX lane after mandatory primers and a 20-minute profile
+for single- and multi-node AgentX throughput jobs. Fixed-sequence throughput
+and eval jobs retain their canonical settings. Adding or removing the modifier
+restarts the active sweep. Fast-mode runs are not eligible for artifact reuse
+after merge.
 
 ## Reusing an Approved PR Full Sweep
 

.github/workflows/benchmark-multinode-tmpl.yml

@@ -155,7 +155,7 @@ on:
         required: false
         default: "3600"
       agentx-fast:
-        description: "Use the AgentX 5-minute warmup and 20-minute profiling preset"
+        description: "Use one warmup request per AgentX lane and a 20-minute profile"
         type: boolean
         required: false
         default: false

.github/workflows/benchmark-tmpl.yml

@@ -130,7 +130,7 @@ on:
         type: string
         default: '3600'
       agentx-fast:
-        description: "Use the AgentX 5-minute warmup and 20-minute profiling preset"
+        description: "Use one warmup request per AgentX lane and a 20-minute profile"
         required: false
         type: boolean
         default: false

.github/workflows/e2e-tests.yml

@@ -22,7 +22,7 @@ on:
                 type: string
                 default: ""
             agentx-fast:
-                description: "AgentX fast feedback: 5-minute cache warmup and 20-minute profile."
+                description: "AgentX fast feedback: one warmup request per lane and a 20-minute profile."
                 required: false
                 type: boolean
                 default: false
@@ -56,7 +56,7 @@ on:
                 type: string
                 default: ""
             agentx-fast:
-                description: "AgentX fast feedback: 5-minute cache warmup and 20-minute profile."
+                description: "AgentX fast feedback: one warmup request per lane and a 20-minute profile."
                 required: false
                 type: boolean
                 default: false

.gitmodules

@@ -1,4 +1,3 @@
 [submodule "utils/aiperf"]
 	path = utils/aiperf
 	url = https://github.com/SemiAnalysisAI/aiperf.git
-	branch = cquil11/aiperf-agentx-v1.0

AGENTS.md

@@ -95,7 +95,7 @@ PRs do not run the sweep automatically - `run-sweep.yml` is gated on a primary s
 - `full-sweep-fail-fast` - runs the full intermediate concurrency sweep behind the same sequential single-node canary gate as `full-sweep-enabled` (so a globally broken change burns one job, not the whole fan-out), and with `strategy.fail-fast` enabled on every matrix: the first failure in a matrix cancels that matrix's remaining jobs. Fail-fast is matrix-scoped, so the other matrices (1k1k vs 8k1k vs agentic vs evals) keep running and self-terminate on their own first failure; their completed results remain valid. The failing job keeps its red *failure* conclusion and the run concludes failed. **This is the strongly recommended default for full sweeps** (image bumps, recipe changes, bring-up) - a failure means the rest of that matrix is wasted GPU time. Caveat: one flaky job kills its matrix's in-flight results; if that repeatedly bites a specific config, fall back to `full-sweep-enabled` for that PR.
 - `full-sweep-fail-fast-no-canary` - same as `full-sweep-fail-fast` but without the canary gate: all matrices fan out immediately. Use when the canary is flaky or not representative of the affected configuration but you still want per-matrix fail-fast.
 
-`all-evals`, `evals-only`, and `agentx-fast` are optional modifier labels. Combine them with one primary sweep label. `all-evals` expands eval selection to every generated fixed-sequence configuration without changing throughput. `evals-only` suppresses throughput while keeping the default eval subset; combining both eval modifiers runs every eval and no throughput. `agentx-fast` applies the 5-minute warmup and 20-minute profiling preset to single- and multi-node AgentX throughput jobs only; fixed-sequence and eval jobs are unchanged. `all-evals` remains eligible for artifact reuse when paired with an eligible full-sweep label. Runs with `evals-only` or `agentx-fast` are not eligible for artifact reuse.
+`all-evals`, `evals-only`, and `agentx-fast` are optional modifier labels. Combine them with one primary sweep label. `all-evals` expands eval selection to every generated fixed-sequence configuration without changing throughput. `evals-only` suppresses throughput while keeping the default eval subset; combining both eval modifiers runs every eval and no throughput. `agentx-fast` reduces deterministic warmup to one request per lane and profiling to 20 minutes for single- and multi-node AgentX throughput jobs; fixed-sequence and eval jobs are unchanged. `all-evals` remains eligible for artifact reuse when paired with an eligible full-sweep label. Runs with `evals-only` or `agentx-fast` are not eligible for artifact reuse.
 
 **The sweep does not trigger while the PR has merge conflicts.** Even with a sweep label applied, the `run-sweep.yml` workflow will not start until the PR cleanly merges into main — a stale claude/* or update-* branch with a `perf-changelog.yaml` conflict (the common case) will sit in NO_SWEEP / NO_SUCCESS until rebased. Resolution recipe is documented in `KLAUD_DEBUG.md §1.1`: `git merge origin/main`, then `git checkout origin/main -- perf-changelog.yaml`, then re-append the PR's own changelog entry at the tail. Don't 3-way merge `perf-changelog.yaml`; whitespace edits silently re-trigger the deletion check.
 
@@ -119,7 +119,7 @@ gh api -X POST \
 
 Inputs: top-level `ref` (required) is the workflow ref to dispatch from, almost always `main`. `inputs[ref]` is the repo ref under test (defaults to the dispatch ref's `github.sha`). `inputs[generate-cli-command]` (required) is passed verbatim to `generate_sweep_configs.py` - test locally first. `inputs[test-name]` is the display name in the Actions UI. `inputs[duration-override]` overrides per-config duration (seconds); empty = use matrix value.
 
-For an AgentX preflight, add `-F 'inputs[agentx-fast]=true'` to the dispatch command. This selects a 5-minute cache warmup and 20-minute profile for every AgentX job in that e2e dispatch and takes precedence over `duration-override`. Use it to get faster signal while debugging, then run the official sweep without `agentx-fast`; canonical AgentX timing remains a 10-minute cache warmup and 1-hour profile.
+For an AgentX preflight, add `-F 'inputs[agentx-fast]=true'` to the dispatch command. This reduces deterministic warmup to one request per lane and profiling to 20 minutes for every AgentX job. Use it to get faster feedback while debugging, then run the official sweep without `agentx-fast`; canonical AgentX warmup remains 10 requests per lane with a 1-hour profile.
 
 The POST returns no body and no run ID - find the run with `gh run list` below.
 

benchmarks/benchmark_lib.sh

@@ -1736,22 +1736,19 @@ build_replay_cmd() {
     # the worker threads the server's live assistant response back into the
     # session.
     #
-    # The scenario plugin locks: --cache-bust first_turn_prefix and
-    # --trace-idle-gap-cap-seconds 10 (per-trace idle-gap compression
-    # against parent + subagent request-start timestamps; supersedes the
-    # legacy --use-think-time-only / --inter-turn-delay-cap-seconds path),
-    # and auto-injects them — so we do not pass them. See
-    # utils/aiperf/docs/tutorials/agentx-mvp.md.
+    # The scenario plugin locks --cache-bust first_turn_prefix and a 10-second
+    # whole-system idle cap. Source end-to-start delays remain intact; the cap
+    # shifts all pending timers uniformly only when no request is active or
+    # ready. See utils/aiperf/docs/tutorials/agentx-mvp.md.
     local result_dir="$1"
     local duration="$DURATION"
-    local cache_warmup_duration="${AIPERF_AGENTIC_CACHE_WARMUP_DURATION:-600}"
+    local warmup_requests_per_lane="${AIPERF_WARMUP_REQUESTS_PER_LANE:-10}"
 
-    # Fast mode is an e2e-only feedback preset used before canonical one-hour
-    # sweeps. AIPerf already exposes both controls, so no AIPerf patch is
-    # required.
+    # Fast mode minimizes setup by advancing each trajectory lane only once
+    # and shortens profiling to 20 minutes.
     if [[ "${AIPERF_EXPERIMENTAL_FAST:-0}" == "1" ]]; then
         duration=1200
-        cache_warmup_duration=300
+        warmup_requests_per_lane=1
     fi
 
     export AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES="${AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES:-0}"
@@ -1765,6 +1762,9 @@ build_replay_cmd() {
     # aiperf validates that SERVICE_PROFILE_CONFIGURE_TIMEOUT >=
     # DATASET_CONFIGURATION_TIMEOUT at startup. Bump it in lockstep.
     export AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT=1800
+    # Headless realtime metrics are opt-in on current AIPerf main. Enable the
+    # rolling TTFT/ITL/throughput block and emit it every 30 seconds.
+    export AIPERF_UI_REALTIME_METRICS_ENABLED=true
     REPLAY_CMD="$AIPERF_CLI profile --scenario inferencex-agentx-mvp"
     REPLAY_CMD+=" --url http://localhost:$PORT"
     REPLAY_CMD+=" --endpoint /v1/chat/completions"
@@ -1773,6 +1773,7 @@ build_replay_cmd() {
     REPLAY_CMD+=" --model $MODEL"
     REPLAY_CMD+=" --concurrency $CONC"
     REPLAY_CMD+=" --benchmark-duration $duration"
+    REPLAY_CMD+=" --stats-interval 30"
     REPLAY_CMD+=" --random-seed 42"
     # Fail runs once more than 10% of requests error. This keeps known
     # transient low-rate failures from killing long sweeps while still
@@ -1784,10 +1785,12 @@ build_replay_cmd() {
     # least one profile turn after warmup.
     REPLAY_CMD+=" --trajectory-start-min-ratio 0.25"
     REPLAY_CMD+=" --trajectory-start-max-ratio 0.75"
-    # After the normal t* snapshot warmup, continue those exact trajectories
-    # with one-token outputs and no idle delays. Profiling begins only after
-    # those requests drain and resumes from the resulting live state.
-    REPLAY_CMD+=" --agentic-cache-warmup-duration $cache_warmup_duration"
+    # After the normal t* snapshot primers, advance every trajectory lane by
+    # this many additional one-token requests with no idle delay. Profiling
+    # begins after those requests drain and resumes from the resulting live
+    # state. Do not pass --burst-phase-starts: AIPerf main's spread default
+    # preserves each lane's recorded phase-start offset.
+    REPLAY_CMD+=" --warmup-requests-per-lane $warmup_requests_per_lane"
     # Give long-context warmup requests up to 30 minutes to drain before
     # declaring warmup failed. Recipes whose saturation arms carry a larger
     # in-flight working set may override via AGENTIC_WARMUP_GRACE_PERIOD

benchmarks/multi_node/amd_utils/server_sglang.sh

@@ -986,7 +986,7 @@ if [ "$NODE_RANK" -eq 0 ]; then
                       DECODE_NUM_WORKERS DECODE_TP DECODE_EP DECODE_DP_ATTN DECODE_HARDWARE \
                       KV_OFFLOADING KV_OFFLOAD_BACKEND KV_OFFLOAD_BACKEND_METADATA TOTAL_CPU_DRAM_GB KV_P2P_TRANSFER \
                       WEKA_LOADER_OVERRIDE AIPERF_FAILED_REQUEST_THRESHOLD \
-                      AIPERF_AGENTIC_CACHE_WARMUP_DURATION AIPERF_UNSAFE_OVERRIDE \
+                      AIPERF_WARMUP_REQUESTS_PER_LANE AIPERF_EXPERIMENTAL_FAST AIPERF_UNSAFE_OVERRIDE \
                       AIPERF_TRAJECTORY_START_MIN_RATIO AIPERF_TRAJECTORY_START_MAX_RATIO \
                       AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES ROUTER_PORT TQDM_MININTERVAL; do
                 if [[ -n "${!_v+x}" ]]; then

benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/agg-gb300-tp4-mtp-agentic.yaml

@@ -141,7 +141,6 @@ benchmark:
     PORT: "8000"
     IS_MULTINODE: "true"
     AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
-    AIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600"
     AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
     HF_HUB_CACHE: "/hf_hub_cache"
     WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"

benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/agg-gb300-tp8-mtp-agentic.yaml

@@ -144,7 +144,6 @@ benchmark:
     # the zero decode-worker count instead of duplicating TP into P and D.
     IS_MULTINODE: "true"
     AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
-    AIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600"
     AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
     HF_HUB_CACHE: "/hf_hub_cache"
     WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"

benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb300-1p1d-dep4-dep8-c128-mtp-agentic.yaml

@@ -198,7 +198,6 @@ benchmark:
     IS_MULTINODE: "true"
     AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
     AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
-    AIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600"
     AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
     HF_HUB_CACHE: "/hf_hub_cache"
     WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"

benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb300-1p1d-dep8-dep8-c384-mtp-agentic.yaml

@@ -199,7 +199,6 @@ benchmark:
     IS_MULTINODE: "true"
     AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
     AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
-    AIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600"
     AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
     HF_HUB_CACHE: "/hf_hub_cache"
     WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"

utils/process_changelog.py

@@ -16,6 +16,17 @@
 SCENARIO_TYPES = ("fixed-seq-len", "agentic-coding")
 
 
+def _freeze_config_value(value):
+    """Convert JSON-shaped config values into deterministic hashable values."""
+    if isinstance(value, dict):
+        return tuple(
+            sorted((key, _freeze_config_value(item)) for key, item in value.items())
+        )
+    if isinstance(value, list):
+        return tuple(_freeze_config_value(item) for item in value)
+    return value
+
+
 def get_added_lines(base_ref: str, head_ref: str, filepath: str) -> str:
     result = subprocess.run(
         ["git", "diff", base_ref, head_ref, "--", filepath],
@@ -54,8 +65,8 @@ def trim_conc(entries: list[dict]) -> list[dict]:
     the source ordering of ``conc-list`` / ``conc-start``.
 
     Input comes from ``json.loads(subprocess.stdout)`` so ``conc`` is always
-    ``int`` (single-node) or ``list`` (multi-node); other single-node fields
-    are hashable scalars.
+    ``int`` (single-node) or ``list`` (multi-node). Other fields may contain
+    nested dictionaries or lists, such as KV-offload backend metadata.
 
     - Single-node entries: group by every other field and keep only the entry
       with the lowest ``conc`` per group.
@@ -72,7 +83,13 @@ def trim_conc(entries: list[dict]) -> list[dict]:
             out.append(entry)
             continue
 
-        key = tuple(sorted((k, v) for k, v in entry.items() if k != "conc"))
+        key = tuple(
+            sorted(
+                (k, _freeze_config_value(v))
+                for k, v in entry.items()
+                if k != "conc"
+            )
+        )
         groups.setdefault(key, []).append(len(out))
         out.append(entry)
 

utils/test_process_changelog.py

@@ -15,6 +15,34 @@ def _scenario_values(command):
     return command[index:]
 
 
+def test_trim_conc_supports_nested_backend_metadata():
+    common = {
+        "model": "moonshotai/Kimi-K3",
+        "kv-offloading": "dram",
+        "kv-offload-backend": {
+            "name": "vllm-simple",
+            "settings": {"tiers": ["cpu", "gpu"]},
+        },
+    }
+    entries = [
+        {**common, "conc": 8},
+        {**common, "conc": 2},
+        {
+            **common,
+            "kv-offload-backend": {"name": "lmcache"},
+            "conc": 4,
+        },
+    ]
+
+    trimmed = process_changelog.trim_conc(entries)
+
+    assert [entry["conc"] for entry in trimmed] == [2, 4]
+    assert [entry["kv-offload-backend"]["name"] for entry in trimmed] == [
+        "vllm-simple",
+        "lmcache",
+    ]
+
+
 def test_config_key_expansion_is_deterministic_and_deduplicated():
     master_config = {
         "config-b": {},