Skip to content

Commit 3355132

Browse files
csahithixinli-swcquil11weireweire
authored
Add GB300 DeepSeek-V4 Dynamo-SGLang AgentX aggregated and disaggregated recipes / 新增 GB300 DeepSeek-V4 Dynamo-SGLang AgentX 聚合式与分离式配方 (#2157)
* Add GB300 DeepSeek-V4 dynamo-sglang agentic recipes Add aggregated (TP4) and disaggregated (DEP8 pareto) GB300 DeepSeek-V4 agentic-coding recipes with prefill-side MTP to match decode, along with the master-config sweep entries, GB300 launcher support, and the sgl-deep-gemm downgrade helper. * fix * Bump agentic image to 20260711-7de33ce8; add conc 4 to agg; drop deep-gemm patch - Bump GB300 DeepSeek-V4 dynamo-sglang agentic image (agg + disagg) to lmsysorg/sglang:nightly-dev-cu13-20260711-7de33ce8. - Add concurrency 4 to the agg search-space (conc-list [1] -> [1, 4]). - Drop the sgl-deep-gemm downgrade patch (downgrade-sgl-deep-gemm.sh setup script + its recipe setup_script: refs + launcher wiring); the regression (sglang #30399, cross-node DEP8 prefill grid-sync timeout) is fixed in the 20260711 image. * agg: hicache-ratio 6, extend conc to [1, 4, 8, 16] - Set the aggregated recipe's hierarchical-cache ratio to 6. - Extend the agg search-space concurrencies to [1, 4, 8, 16]. * Fix agg GPU accounting: decode num-worker 0 + zero decode contribution The GB300 DeepSeek-V4 agentic-agg is the first aggregated (disagg:false) config on the multi-node agentic path, so process_agentic_result.py double-counted its GPUs: num_gpus = prefill(1x4) + decode(1x4) = 8 and tp = 4+4 = 8, i.e. the dashboard showed TP8 (and halved per-GPU throughput) for a single TP4 worker. - master config: set the agg's decode num-worker to 0. The aggregated worker's GPUs also serve decode, so there is no separate decode allocation. Deployment is recipe-driven (srtctl apply -f CONFIG_FILE, agg_workers: 1), so the actual benchmark is unchanged; only num_gpus now counts 4. - process_agentic_result.py: mirror process_result.py -- when there are no decode GPUs, zero the decode-side parallelism so tp/ep and the per-GPU throughput denominator reflect the single aggregated worker. Disagg configs (decode num-worker >= 1) are unaffected. * Install dynamo from wheel 1.3.0.dev1; drop disagg prefill MTP; pin srt-slurm v1.0.10 - dynamo: install all 6 GB300 DeepSeek-V4 agentic recipes from the staged wheel 1.3.0.dev1 (dynamo.wheel) instead of a source-build commit hash. - Remove prefill-side MTP (speculative-* in the prefill sglang_config) from the 5 disagg recipes; decode-side MTP is retained. Update the changelog description to reflect decode-only MTP. - launcher: pin srt-slurm to the v1.0.10 release tag (identical commit to the previously pinned 03c0e8c) instead of the raw hash. * chore(config): add serving component metadata * fix(changelog): omit null optional metadata * fix(agentx): align DSV4 MTP golden AL settings * fix(agentx): restore prefill MTP for DSV4 disagg Root cause: the disaggregated recipes dropped the prefill speculative-decoding flags even though the pinned SGLang image already includes the DSV4 draft SWA transfer fix. Fix: restore the EAGLE three-step, top-k 1, four-draft-token settings on every DSV4 disaggregated prefill worker while keeping the existing SGLang image unchanged. Validation: generated the six-entry DSV4 agentic matrix, verified matching prefill/decode MTP settings in all five disaggregated recipes, and passed all 221 matrix-logic tests. 中文：恢复所有 DSV4 分离式配置的预填充端 MTP 参数。当前固定的 SGLang 镜像已经包含 DSV4 draft SWA 传输修复，因此无需升级镜像。已验证 6 个 AgentX 矩阵条目，且 221 项矩阵逻辑测试全部通过。 * Update perf-changelog.yaml --------- Co-authored-by: Xin Li <xinli@nvidia.com> Co-authored-by: Cameron Quilici <cjquilici@gmail.com> Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
1 parent 13f55eb commit 3355132

13 files changed

Lines changed: 1390 additions & 5 deletions

‎.github/workflows/benchmark-multinode-tmpl.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -240,7 +240,7 @@ jobs:
240240
${{ inputs.spec-decoding != 'none' && inputs.spec-decoding || '' }}
241241
${{ inputs.kv-offloading != '' && inputs.kv-offloading != 'none' && format('{0} KV offload', inputs.kv-offloading) || '' }}
242242
${{ inputs.kv-offload-backend != '' && inputs.kv-offload-backend != 'none' && inputs.kv-offload-backend != 'default' && inputs.kv-offload-backend || '' }}
243-
c${{ inputs.conc != '' && inputs.conc || join(fromJson(inputs.conc-list), 'x') }}${{ inputs.eval-only && ' | eval-only' || (inputs.run-eval && ' | eval' || '') }}
243+
c${{ join(fromJson(inputs.conc-list), 'x') }}${{ inputs.eval-only && ' | eval-only' || (inputs.run-eval && ' | eval' || '') }}
244244
245245
steps:
246246
- name: Slurm cleanup (pre-run)

‎.github/workflows/run-sweep.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -556,7 +556,7 @@ jobs:
556556
precision: ${{ matrix.config.precision }}
557557
router: ${{ matrix.config.router && toJson(matrix.config.router) || '' }}
558558
kv-p2p-transfer: ${{ matrix.config['kv-p2p-transfer'] || '' }}
559-
conc-list: '[${{ matrix.config.conc }}]'
559+
conc-list: ${{ toJson(matrix.config.conc) }}
560560
spec-decoding: ${{ matrix.config.spec-decoding }}
561561
disagg: ${{ matrix.config.disagg }}
562562
prefill-hardware: ${{ matrix.config.prefill.hardware }}
@@ -577,7 +577,7 @@ jobs:
577577
decode-ep: ${{ matrix.config.decode.ep }}
578578
decode-dp-attn: ${{ matrix.config.decode.dp-attn }}
579579
decode-additional-settings: ${{ toJson(matrix.config.decode.additional-settings) }}
580-
conc: ${{ matrix.config.conc }}
580+
conc: ${{ matrix.config.conc[0] }}
581581
kv-offloading: ${{ matrix.config.kv-offloading }}
582582
kv-offload-backend: ${{ matrix.config['kv-offload-backend'].name }}
583583
kv-offload-backend-metadata: ${{ matrix.config['kv-offload-backend'] && toJson(matrix.config['kv-offload-backend']) || '' }}

‎benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/agentic/agg-gb300-tp4-mtp-kvoffload.yaml‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
name: "agg-gb300-tp4-mtp-kvoffload"
2+
3+
# Agentic-coding SGLang aggregated recipe for DeepSeek-V4-Pro on GB300
4+
# (single aggregated worker, TP4, MTP + hierarchical-cache KV offload).
5+
#
6+
# Uses the flat single-variant schema the agentic CI flow expects. Concurrency is NOT
7+
# a recipe field here: the GHA matrix fans out one job per concurrency from
8+
# the master-config conc-list, exporting CONC into agentic_srt.sh. Modeled on
9+
# the flat vllm/deepseek-v4/agentic recipes (agentic infra + benchmark wiring)
10+
# and the flat sglang/deepseek-v4/8k1k recipes (sglang backend schema).
11+
12+
model:
13+
path: "deepseek-v4-pro"
14+
container: "dynamo-sglang"
15+
precision: "fp4"
16+
17+
dynamo:
18+
install: true
19+
wheel: "1.3.0.dev1"
20+
21+
slurm:
22+
time_limit: "8:00:00"
23+
24+
health_check:
25+
max_attempts: 1440
26+
interval_seconds: 10
27+
28+
resources:
29+
gpu_type: "gb300"
30+
gpus_per_node: 4
31+
agg_nodes: 1
32+
agg_workers: 1
33+
gpus_per_agg: 4
34+
35+
infra:
36+
# Dedicated etcd/nats node — matches the vllm agentic recipes; under the
37+
# dynamo router + multiple frontends the infra services need their own node.
38+
etcd_nats_dedicated_node: true
39+
# cc-traces prompts (~125K tokens) serialize to ~2.8MB; the 1 MiB NATS
40+
# default drops them. 32 MiB matches the agentic vllm recipes (~10x headroom
41+
# over the largest observed payload); schema recommends 24+ for long ISL.
42+
nats_max_payload_mb: 32
43+
44+
frontend:
45+
type: dynamo
46+
nginx_session_affinity: true
47+
nginx_session_affinity_header: X-Correlation-ID
48+
enable_multiple_frontends: true
49+
num_additional_frontends: 4
50+
env:
51+
DYN_ROUTER_TEMPERATURE: "10000000"
52+
# The sglang image is PEP 668 externally-managed; the runtime dynamo
53+
# install (dynamo_wheels.py / source build) runs pip and fails with
54+
# "externally-managed-environment" without this. Lets pip install into
55+
# the system env. Applies to both dynamo.hash (source) and dynamo.wheel.
56+
PIP_BREAK_SYSTEM_PACKAGES: "1"
57+
args:
58+
router-mode: "kv"
59+
router-reset-states: true
60+
active-decode-blocks-threshold: "None"
61+
active-prefill-tokens-threshold: "None"
62+
active-prefill-tokens-threshold-frac: "None"
63+
64+
backend:
65+
type: sglang
66+
67+
aggregated_environment:
68+
SGLANG_DEFAULT_THINKING: '1'
69+
SGLANG_DSV4_REASONING_EFFORT: high
70+
SGLANG_SIMULATE_ACC_LEN: '2.49'
71+
SGLANG_SIMULATE_ACC_METHOD: match-expected
72+
SGLANG_SIMULATE_ACC_TOKEN_MODE: "real-draft-token"
73+
# Lets the runtime dynamo install's pip bypass PEP 668 on the sglang image.
74+
PIP_BREAK_SYSTEM_PACKAGES: "1"
75+
# from submission for B300
76+
SGLANG_JIT_DEEPGEMM_PRECOMPILE: "1"
77+
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: "1"
78+
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: "1"
79+
SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2: "1"
80+
SGLANG_OPT_USE_ONLINE_COMPRESS: "0"
81+
SGLANG_OPT_USE_JIT_INDEXER_METADATA: "1"
82+
SGLANG_OPT_USE_JIT_NORM: "1"
83+
SGLANG_OPT_USE_TOPK_V2: "True"
84+
# for kvcache offload
85+
SGLANG_ENABLE_UNIFIED_RADIX_TREE: "1"
86+
87+
sglang_config:
88+
aggregated:
89+
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
90+
trust-remote-code: true
91+
stream-interval: 50
92+
watchdog-timeout: 1000000
93+
mem-fraction-static: 0.90
94+
chunked-prefill-size: 8192 # can only support 11000 for TP
95+
moe-runner-backend: "flashinfer_mxfp4"
96+
disable-flashinfer-autotune: true
97+
swa-full-tokens-ratio: 0.1
98+
max-running-requests: 128
99+
cuda-graph-max-bs: 128
100+
101+
scheduler-recv-interval: 30
102+
dp-size: 1
103+
tp-size: 4
104+
ep-size: 1
105+
# kv-cache offload
106+
enable-hierarchical-cache: true
107+
hicache-ratio: 6
108+
hicache-write-policy: write_through
109+
hicache-io-backend: kernel
110+
# mtp section
111+
speculative-algorithm: EAGLE
112+
speculative-num-steps: 3
113+
speculative-eagle-topk: 1
114+
speculative-num-draft-tokens: 4
115+
116+
sbatch_directives:
117+
mem: "0"
118+
cpus-per-task: "144"
119+
120+
srun_options:
121+
mem: "0"
122+
# gb300-nv: pyxis maps the calling user into the container as a non-root
123+
# uid; agentic_srt.sh's apt-get install git step needs EUID 0. Remap to
124+
# uid 0 inside the container. srt-slurm renders empty-string values as
125+
# flag-only srun args.
126+
container-remap-root: ""
127+
128+
benchmark:
129+
type: custom
130+
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
131+
env:
132+
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
133+
RESULT_DIR: /logs/agentic
134+
PORT: "8000"
135+
# Aggregated: one worker serves both prefill and decode, so GPU accounting
136+
# is the single-worker form num_gpus = TP (not the disagg prefill+decode
137+
# sum). The multinode post-processing path assumes disagg and would divide
138+
# throughput by prefill_gpus + decode_gpus; force the single-node code path
139+
# in process_agentic_result.py instead. TP must match sglang_config tp-size.
140+
IS_MULTINODE: "false"
141+
TP: "4"
142+
# Enable dynamo conv-aware routing (per-session affinity so multi-turn
143+
# KV cache is reused across turns). Previously opted out because the
144+
# frontend 400'd on aiperf's nvext.session_control actions; re-enabled
145+
# to test with the current build.
146+
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "1"
147+
# Container-side path of the aiperf mmap dataset cache; host-side mount
148+
# wired via launch_gb300-*.sh srtslurm.yaml default_mounts. Without it,
149+
# aiperf re-tokenizes + re-writes the dataset mmap on every run.
150+
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
151+
# Persistent HF hub cache (also via default_mounts) so the trace dataset
152+
# isn't re-downloaded each run; overrides the workflow-level HF_HUB_CACHE.
153+
HF_HUB_CACHE: "/hf_hub_cache"
154+

‎benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/agentic/disagg-gb300-12p4d-dep8-dep16-c1024-mtp-kvoffload.yaml‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
name: "disagg-gb300-12p4d-dep8-dep16-c1024-mtp-kvoffload"
2+
3+
# Agentic-coding SGLang disaggregated Pareto recipe for DeepSeek-V4-Pro on GB300
4+
# (12P x DEP8 / 4D x DEP16, MTP + hierarchical-cache KV offload), tuned for concurrency 1024.
5+
#
6+
# Uses the flat single-variant srtctl schema the agentic CI flow expects;
7+
# resources + backend (prefill/decode env + sglang_config) are normalized
8+
# from the Pareto run.
9+
# Concurrency is exported into agentic_srt.sh from the master-config conc-list.
10+
11+
model:
12+
path: "deepseek-v4-pro"
13+
container: "dynamo-sglang"
14+
precision: "fp4"
15+
16+
dynamo:
17+
install: true
18+
wheel: "1.3.0.dev1"
19+
20+
slurm:
21+
time_limit: "8:00:00"
22+
23+
health_check:
24+
max_attempts: 1440
25+
interval_seconds: 10
26+
27+
resources:
28+
gpu_type: gb300
29+
gpus_per_node: 4
30+
prefill_nodes: 12
31+
decode_nodes: 4
32+
prefill_workers: 6
33+
decode_workers: 1
34+
gpus_per_prefill: 8
35+
gpus_per_decode: 16
36+
37+
infra:
38+
etcd_nats_dedicated_node: true
39+
nats_max_payload_mb: 32
40+
41+
frontend:
42+
type: dynamo
43+
nginx_session_affinity: true
44+
nginx_session_affinity_header: X-Correlation-ID
45+
enable_multiple_frontends: true
46+
num_additional_frontends: 4
47+
env:
48+
DYN_ROUTER_TEMPERATURE: "10000000"
49+
PIP_BREAK_SYSTEM_PACKAGES: "1"
50+
args:
51+
router-mode: "kv"
52+
router-reset-states: true
53+
active-decode-blocks-threshold: "None"
54+
active-prefill-tokens-threshold: "None"
55+
active-prefill-tokens-threshold-frac: "None"
56+
57+
backend:
58+
type: sglang
59+
60+
prefill_environment:
61+
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "900"
62+
SGLANG_DSV4_MHC_PREWARM: '1'
63+
PIP_BREAK_SYSTEM_PACKAGES: '1'
64+
PYTHONUNBUFFERED: '1'
65+
SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK: '1'
66+
SGLANG_DG_CACHE_DIR: /configs/deepgemm_cache
67+
SGLANG_JIT_DEEPGEMM_PRECOMPILE: '1'
68+
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: '1'
69+
SGLANG_DEFAULT_THINKING: '1'
70+
SGLANG_DSV4_REASONING_EFFORT: high
71+
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: '1'
72+
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: "1"
73+
SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2: '1'
74+
SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE: '1'
75+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS: '1'
76+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND: '1'
77+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '9216'
78+
SGLANG_OPT_USE_ONLINE_COMPRESS: '0'
79+
NCCL_MNNVL_ENABLE: '1'
80+
NCCL_CUMEM_ENABLE: '1'
81+
SGLANG_MOONCAKE_CUSTOM_MEM_POOL: 'True'
82+
MC_FORCE_MNNVL: '1'
83+
NCCL_TIMEOUT: '100000'
84+
NCCL_MNNVL_UUID_QUERY_TIMEOUT: '100000'
85+
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW: '1'
86+
DYN_SKIP_SGLANG_LOG_FORMATTING: '1'
87+
DYN_LOG: info,dynamo_runtime::pipeline::network::ingress::push_handler=warn
88+
SGLANG_LOG_FORWARD_ITERS: '1'
89+
SGLANG_LOG_MS: '1'
90+
SGLANG_REQUEST_STATE_WAIT_TIMEOUT: '60'
91+
SGLANG_ENABLE_UNIFIED_RADIX_TREE: '1'
92+
93+
decode_environment:
94+
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "900"
95+
SGLANG_SIMULATE_ACC_LEN: '2.49'
96+
SGLANG_SIMULATE_ACC_METHOD: match-expected
97+
SGLANG_SIMULATE_ACC_TOKEN_MODE: "real-draft-token"
98+
PIP_BREAK_SYSTEM_PACKAGES: '1'
99+
PYTHONUNBUFFERED: '1'
100+
SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK: '1'
101+
SGLANG_DG_CACHE_DIR: /configs/deepgemm_cache
102+
SGLANG_JIT_DEEPGEMM_PRECOMPILE: '1'
103+
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: '1'
104+
SGLANG_DEFAULT_THINKING: '1'
105+
SGLANG_DSV4_REASONING_EFFORT: high
106+
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: '1'
107+
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: "1"
108+
SGLANG_OPT_DEEPGEMM_MEGA_MOE: '1'
109+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS: '1'
110+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND: '1'
111+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '4096'
112+
SGLANG_OPT_USE_ONLINE_COMPRESS: '0'
113+
NCCL_MNNVL_ENABLE: '1'
114+
NCCL_CUMEM_ENABLE: '1'
115+
SGLANG_MOONCAKE_CUSTOM_MEM_POOL: 'True'
116+
MC_FORCE_MNNVL: '1'
117+
NCCL_TIMEOUT: '100000'
118+
NCCL_MNNVL_UUID_QUERY_TIMEOUT: '100000'
119+
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW: '1'
120+
SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION: '8'
121+
DYN_SKIP_SGLANG_LOG_FORMATTING: '1'
122+
DYN_LOG: info,dynamo_runtime::pipeline::network::ingress::push_handler=warn
123+
SGLANG_LOG_FORWARD_ITERS: '1'
124+
SGLANG_LOG_MS: '1'
125+
SGLANG_REQUEST_STATE_WAIT_TIMEOUT: '60'
126+
SGLANG_ENABLE_UNIFIED_RADIX_TREE: '1'
127+
128+
sglang_config:
129+
prefill:
130+
host: 0.0.0.0
131+
served-model-name: deepseek-ai/DeepSeek-V4-Pro
132+
model-path: /model/
133+
trust-remote-code: true
134+
watchdog-timeout: 86400
135+
stream-interval: 60
136+
tp-size: 8
137+
dp-size: 8
138+
ep-size: 8
139+
enable-dp-attention: true
140+
enable-dp-lm-head: true
141+
moe-dense-tp-size: 1
142+
moe-a2a-backend: megamoe
143+
disaggregation-transfer-backend: mooncake
144+
disaggregation-mode: prefill
145+
load-balance-method: total_tokens
146+
mem-fraction-static: 0.9
147+
swa-full-tokens-ratio: 0.02
148+
max-running-requests: 1024
149+
cuda-graph-max-bs: 1024
150+
chunked-prefill-size: 32768
151+
disable-flashinfer-autotune: true
152+
model-loader-extra-config: '{"enable_multithread_load":true,"num_threads":8}'
153+
kv-events-config: '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}'
154+
speculative-algorithm: EAGLE
155+
speculative-num-steps: 3
156+
speculative-eagle-topk: 1
157+
speculative-num-draft-tokens: 4
158+
enable-hierarchical-cache: true
159+
hicache-write-policy: write_back
160+
hicache-ratio: 1
161+
hicache-io-backend: direct
162+
163+
decode:
164+
host: 0.0.0.0
165+
served-model-name: deepseek-ai/DeepSeek-V4-Pro
166+
model-path: /model/
167+
trust-remote-code: true
168+
watchdog-timeout: 86400
169+
stream-interval: 60
170+
tp-size: 16
171+
dp-size: 16
172+
ep-size: 16
173+
enable-dp-attention: true
174+
enable-dp-lm-head: true
175+
moe-dense-tp-size: 1
176+
moe-a2a-backend: megamoe
177+
disaggregation-transfer-backend: mooncake
178+
disaggregation-mode: decode
179+
load-balance-method: total_tokens
180+
mem-fraction-static: 0.9
181+
swa-full-tokens-ratio: 0.02
182+
max-running-requests: 3072
183+
cuda-graph-max-bs: 1024
184+
disable-flashinfer-autotune: true
185+
model-loader-extra-config: '{"enable_multithread_load":true,"num_threads":8}'
186+
kv-events-config: '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5567"}'
187+
speculative-algorithm: EAGLE
188+
speculative-num-steps: 3
189+
speculative-eagle-topk: 1
190+
speculative-num-draft-tokens: 4
191+
192+
193+
sbatch_directives:
194+
mem: "0"
195+
cpus-per-task: "144"
196+
197+
srun_options:
198+
mem: "0"
199+
container-remap-root: ""
200+
201+
benchmark:
202+
type: custom
203+
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
204+
env:
205+
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
206+
RESULT_DIR: /logs/agentic
207+
PORT: "8000"
208+
IS_MULTINODE: "true"
209+
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "1"
210+
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
211+
HF_HUB_CACHE: "/hf_hub_cache"
212+

0 commit comments

Comments
 (0)