Skip to content

Commit 3f788b0

Browse files
committed
fix(agentx): pin additive AIPerf main warmup
Pin the shared AIPerf submodule to the synchronized main revision with additive per-lane warmup after mandatory primers while retaining timing-faithful spread phase starts. Align the workflow documentation with the request-count fast preset. 中文:将共享 AIPerf 子模块固定到已同步的 main 版本,在必需预热请求之后执行额外的逐 lane 预热,并保留符合轨迹时序的分散阶段启动。同步更新工作流文档,使其准确描述基于请求数量的快速预设。
1 parent 5ba0b0b commit 3f788b0

15 files changed

Lines changed: 80 additions & 36 deletions

.github/workflows/README.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -159,11 +159,12 @@ are also reusable.
159159

160160
## AgentX Fast Mode
161161

162-
Add `agentx-fast` alongside one primary sweep label to run the 5-minute cache
163-
warmup and 20-minute profiling preset for single- and multi-node AgentX
164-
throughput jobs. Fixed-sequence throughput and eval jobs retain their canonical
165-
settings. Adding or removing the modifier restarts the active sweep. Fast-mode
166-
runs are not eligible for artifact reuse after merge.
162+
Add `agentx-fast` alongside one primary sweep label to run one additional
163+
warmup request per AgentX lane after mandatory primers and a 20-minute profile
164+
for single- and multi-node AgentX throughput jobs. Fixed-sequence throughput
165+
and eval jobs retain their canonical settings. Adding or removing the modifier
166+
restarts the active sweep. Fast-mode runs are not eligible for artifact reuse
167+
after merge.
167168

168169
## Reusing an Approved PR Full Sweep
169170

.github/workflows/benchmark-multinode-tmpl.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ on:
155155
required: false
156156
default: "3600"
157157
agentx-fast:
158-
description: "Use the AgentX 5-minute warmup and 20-minute profiling preset"
158+
description: "Use one warmup request per AgentX lane and a 20-minute profile"
159159
type: boolean
160160
required: false
161161
default: false

.github/workflows/benchmark-tmpl.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -130,7 +130,7 @@ on:
130130
type: string
131131
default: '3600'
132132
agentx-fast:
133-
description: "Use the AgentX 5-minute warmup and 20-minute profiling preset"
133+
description: "Use one warmup request per AgentX lane and a 20-minute profile"
134134
required: false
135135
type: boolean
136136
default: false

.github/workflows/e2e-tests.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ on:
2222
type: string
2323
default: ""
2424
agentx-fast:
25-
description: "AgentX fast feedback: 5-minute cache warmup and 20-minute profile."
25+
description: "AgentX fast feedback: one warmup request per lane and a 20-minute profile."
2626
required: false
2727
type: boolean
2828
default: false
@@ -56,7 +56,7 @@ on:
5656
type: string
5757
default: ""
5858
agentx-fast:
59-
description: "AgentX fast feedback: 5-minute cache warmup and 20-minute profile."
59+
description: "AgentX fast feedback: one warmup request per lane and a 20-minute profile."
6060
required: false
6161
type: boolean
6262
default: false

.gitmodules

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
11
[submodule "utils/aiperf"]
22
path = utils/aiperf
33
url = https://github.com/SemiAnalysisAI/aiperf.git
4-
branch = cquil11/aiperf-agentx-v1.0

AGENTS.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ PRs do not run the sweep automatically - `run-sweep.yml` is gated on a primary s
9595
- `full-sweep-fail-fast` - runs the full intermediate concurrency sweep behind the same sequential single-node canary gate as `full-sweep-enabled` (so a globally broken change burns one job, not the whole fan-out), and with `strategy.fail-fast` enabled on every matrix: the first failure in a matrix cancels that matrix's remaining jobs. Fail-fast is matrix-scoped, so the other matrices (1k1k vs 8k1k vs agentic vs evals) keep running and self-terminate on their own first failure; their completed results remain valid. The failing job keeps its red *failure* conclusion and the run concludes failed. **This is the strongly recommended default for full sweeps** (image bumps, recipe changes, bring-up) - a failure means the rest of that matrix is wasted GPU time. Caveat: one flaky job kills its matrix's in-flight results; if that repeatedly bites a specific config, fall back to `full-sweep-enabled` for that PR.
9696
- `full-sweep-fail-fast-no-canary` - same as `full-sweep-fail-fast` but without the canary gate: all matrices fan out immediately. Use when the canary is flaky or not representative of the affected configuration but you still want per-matrix fail-fast.
9797

98-
`all-evals`, `evals-only`, and `agentx-fast` are optional modifier labels. Combine them with one primary sweep label. `all-evals` expands eval selection to every generated fixed-sequence configuration without changing throughput. `evals-only` suppresses throughput while keeping the default eval subset; combining both eval modifiers runs every eval and no throughput. `agentx-fast` applies the 5-minute warmup and 20-minute profiling preset to single- and multi-node AgentX throughput jobs only; fixed-sequence and eval jobs are unchanged. `all-evals` remains eligible for artifact reuse when paired with an eligible full-sweep label. Runs with `evals-only` or `agentx-fast` are not eligible for artifact reuse.
98+
`all-evals`, `evals-only`, and `agentx-fast` are optional modifier labels. Combine them with one primary sweep label. `all-evals` expands eval selection to every generated fixed-sequence configuration without changing throughput. `evals-only` suppresses throughput while keeping the default eval subset; combining both eval modifiers runs every eval and no throughput. `agentx-fast` reduces deterministic warmup to one request per lane and profiling to 20 minutes for single- and multi-node AgentX throughput jobs; fixed-sequence and eval jobs are unchanged. `all-evals` remains eligible for artifact reuse when paired with an eligible full-sweep label. Runs with `evals-only` or `agentx-fast` are not eligible for artifact reuse.
9999

100100
**The sweep does not trigger while the PR has merge conflicts.** Even with a sweep label applied, the `run-sweep.yml` workflow will not start until the PR cleanly merges into main — a stale claude/* or update-* branch with a `perf-changelog.yaml` conflict (the common case) will sit in NO_SWEEP / NO_SUCCESS until rebased. Resolution recipe is documented in `KLAUD_DEBUG.md §1.1`: `git merge origin/main`, then `git checkout origin/main -- perf-changelog.yaml`, then re-append the PR's own changelog entry at the tail. Don't 3-way merge `perf-changelog.yaml`; whitespace edits silently re-trigger the deletion check.
101101

@@ -119,7 +119,7 @@ gh api -X POST \
119119

120120
Inputs: top-level `ref` (required) is the workflow ref to dispatch from, almost always `main`. `inputs[ref]` is the repo ref under test (defaults to the dispatch ref's `github.sha`). `inputs[generate-cli-command]` (required) is passed verbatim to `generate_sweep_configs.py` - test locally first. `inputs[test-name]` is the display name in the Actions UI. `inputs[duration-override]` overrides per-config duration (seconds); empty = use matrix value.
121121

122-
For an AgentX preflight, add `-F 'inputs[agentx-fast]=true'` to the dispatch command. This selects a 5-minute cache warmup and 20-minute profile for every AgentX job in that e2e dispatch and takes precedence over `duration-override`. Use it to get faster signal while debugging, then run the official sweep without `agentx-fast`; canonical AgentX timing remains a 10-minute cache warmup and 1-hour profile.
122+
For an AgentX preflight, add `-F 'inputs[agentx-fast]=true'` to the dispatch command. This reduces deterministic warmup to one request per lane and profiling to 20 minutes for every AgentX job. Use it to get faster feedback while debugging, then run the official sweep without `agentx-fast`; canonical AgentX warmup remains 10 requests per lane with a 1-hour profile.
123123

124124
The POST returns no body and no run ID - find the run with `gh run list` below.
125125

benchmarks/benchmark_lib.sh

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1736,22 +1736,19 @@ build_replay_cmd() {
17361736
# the worker threads the server's live assistant response back into the
17371737
# session.
17381738
#
1739-
# The scenario plugin locks: --cache-bust first_turn_prefix and
1740-
# --trace-idle-gap-cap-seconds 10 (per-trace idle-gap compression
1741-
# against parent + subagent request-start timestamps; supersedes the
1742-
# legacy --use-think-time-only / --inter-turn-delay-cap-seconds path),
1743-
# and auto-injects them — so we do not pass them. See
1744-
# utils/aiperf/docs/tutorials/agentx-mvp.md.
1739+
# The scenario plugin locks --cache-bust first_turn_prefix and a 10-second
1740+
# whole-system idle cap. Source end-to-start delays remain intact; the cap
1741+
# shifts all pending timers uniformly only when no request is active or
1742+
# ready. See utils/aiperf/docs/tutorials/agentx-mvp.md.
17451743
local result_dir="$1"
17461744
local duration="$DURATION"
1747-
local cache_warmup_duration="${AIPERF_AGENTIC_CACHE_WARMUP_DURATION:-600}"
1745+
local warmup_requests_per_lane="${AIPERF_WARMUP_REQUESTS_PER_LANE:-10}"
17481746

1749-
# Fast mode is an e2e-only feedback preset used before canonical one-hour
1750-
# sweeps. AIPerf already exposes both controls, so no AIPerf patch is
1751-
# required.
1747+
# Fast mode minimizes setup by advancing each trajectory lane only once
1748+
# and shortens profiling to 20 minutes.
17521749
if [[ "${AIPERF_EXPERIMENTAL_FAST:-0}" == "1" ]]; then
17531750
duration=1200
1754-
cache_warmup_duration=300
1751+
warmup_requests_per_lane=1
17551752
fi
17561753

17571754
export AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES="${AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES:-0}"
@@ -1765,6 +1762,9 @@ build_replay_cmd() {
17651762
# aiperf validates that SERVICE_PROFILE_CONFIGURE_TIMEOUT >=
17661763
# DATASET_CONFIGURATION_TIMEOUT at startup. Bump it in lockstep.
17671764
export AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT=1800
1765+
# Headless realtime metrics are opt-in on current AIPerf main. Enable the
1766+
# rolling TTFT/ITL/throughput block and emit it every 30 seconds.
1767+
export AIPERF_UI_REALTIME_METRICS_ENABLED=true
17681768
REPLAY_CMD="$AIPERF_CLI profile --scenario inferencex-agentx-mvp"
17691769
REPLAY_CMD+=" --url http://localhost:$PORT"
17701770
REPLAY_CMD+=" --endpoint /v1/chat/completions"
@@ -1773,6 +1773,7 @@ build_replay_cmd() {
17731773
REPLAY_CMD+=" --model $MODEL"
17741774
REPLAY_CMD+=" --concurrency $CONC"
17751775
REPLAY_CMD+=" --benchmark-duration $duration"
1776+
REPLAY_CMD+=" --stats-interval 30"
17761777
REPLAY_CMD+=" --random-seed 42"
17771778
# Fail runs once more than 10% of requests error. This keeps known
17781779
# transient low-rate failures from killing long sweeps while still
@@ -1784,10 +1785,12 @@ build_replay_cmd() {
17841785
# least one profile turn after warmup.
17851786
REPLAY_CMD+=" --trajectory-start-min-ratio 0.25"
17861787
REPLAY_CMD+=" --trajectory-start-max-ratio 0.75"
1787-
# After the normal t* snapshot warmup, continue those exact trajectories
1788-
# with one-token outputs and no idle delays. Profiling begins only after
1789-
# those requests drain and resumes from the resulting live state.
1790-
REPLAY_CMD+=" --agentic-cache-warmup-duration $cache_warmup_duration"
1788+
# After the normal t* snapshot primers, advance every trajectory lane by
1789+
# this many additional one-token requests with no idle delay. Profiling
1790+
# begins after those requests drain and resumes from the resulting live
1791+
# state. Do not pass --burst-phase-starts: AIPerf main's spread default
1792+
# preserves each lane's recorded phase-start offset.
1793+
REPLAY_CMD+=" --warmup-requests-per-lane $warmup_requests_per_lane"
17911794
# Give long-context warmup requests up to 30 minutes to drain before
17921795
# declaring warmup failed. Recipes whose saturation arms carry a larger
17931796
# in-flight working set may override via AGENTIC_WARMUP_GRACE_PERIOD

benchmarks/multi_node/amd_utils/server_sglang.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -986,7 +986,7 @@ if [ "$NODE_RANK" -eq 0 ]; then
986986
DECODE_NUM_WORKERS DECODE_TP DECODE_EP DECODE_DP_ATTN DECODE_HARDWARE \
987987
KV_OFFLOADING KV_OFFLOAD_BACKEND KV_OFFLOAD_BACKEND_METADATA TOTAL_CPU_DRAM_GB KV_P2P_TRANSFER \
988988
WEKA_LOADER_OVERRIDE AIPERF_FAILED_REQUEST_THRESHOLD \
989-
AIPERF_AGENTIC_CACHE_WARMUP_DURATION AIPERF_UNSAFE_OVERRIDE \
989+
AIPERF_WARMUP_REQUESTS_PER_LANE AIPERF_EXPERIMENTAL_FAST AIPERF_UNSAFE_OVERRIDE \
990990
AIPERF_TRAJECTORY_START_MIN_RATIO AIPERF_TRAJECTORY_START_MAX_RATIO \
991991
AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES ROUTER_PORT TQDM_MININTERVAL; do
992992
if [[ -n "${!_v+x}" ]]; then

benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/agg-gb300-tp4-mtp-agentic.yaml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,6 @@ benchmark:
141141
PORT: "8000"
142142
IS_MULTINODE: "true"
143143
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
144-
AIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600"
145144
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
146145
HF_HUB_CACHE: "/hf_hub_cache"
147146
WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"

benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/agg-gb300-tp8-mtp-agentic.yaml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,6 @@ benchmark:
144144
# the zero decode-worker count instead of duplicating TP into P and D.
145145
IS_MULTINODE: "true"
146146
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
147-
AIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600"
148147
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
149148
HF_HUB_CACHE: "/hf_hub_cache"
150149
WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"

0 commit comments

Comments
 (0)