Skip to content

Commit 5291955

Browse files
cquil11claude
andcommitted
benchmarks(agentic): default DSv4 recipes to v6 (060226) corpus
Removes the DSv4 carveout in resolve_trace_source that pinned DSv4 to the older 052726 v5 corpus. With v6 (060226) validated on both b300 and gb300 DSv4 runs, there's no reason to keep DSv4 on the older trace set. - benchmark_lib.sh: the model-prefix branch on MODEL_PREFIX==dsv4 is gone; default_loader is now 060226 unconditionally. WEKA_LOADER_OVERRIDE still works for any recipe that wants to pin an older or different variant (e.g. the 256k-capped flavor used by max_model_len-constrained setups). - dsv4_fp4_b300_vllm.sh: drops its now-redundant explicit WEKA_LOADER_OVERRIDE=...060226 export since that's the default now. Effect: every DSv4 agentic recipe (b200_vllm, b300_vllm, fp8_h200, mi355x_sglang, gb300_dynamo_vllm) automatically rides v6 with no per-recipe edit needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 923186d commit 5291955

2 files changed

Lines changed: 4 additions & 11 deletions

File tree

benchmarks/benchmark_lib.sh

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -926,17 +926,12 @@ resolve_trace_source() {
926926
# scenario. Used by recipes whose servers have non-default context
927927
# caps (e.g. minimaxm2.5 at max_model_len ~256k can't replay the
928928
# unfiltered corpus and switches to the 256k-capped variant), or
929-
# by recipes that want to pin a specific corpus generation rather
930-
# than ride the model-prefix-aware default below.
929+
# by recipes that want to pin an older corpus generation.
931930
#
932-
# Default (no override) is model-prefix-aware:
933-
# DSv4 recipes -> 052726 (v5 corpus, the original baseline)
934-
# everything else -> 060226 (v6 corpus, newer CC versions)
935-
# DSv4 stays on 052726 for continuity with prior published baselines.
931+
# Default (no override): semianalysis_cc_traces_weka_with_subagents_060226
932+
# (v6 corpus, newer CC versions). All recipes — including DSv4 — ride
933+
# this default unless they opt out via WEKA_LOADER_OVERRIDE.
936934
local default_loader="semianalysis_cc_traces_weka_with_subagents_060226"
937-
if [[ "${MODEL_PREFIX:-}" == "dsv4" ]]; then
938-
default_loader="semianalysis_cc_traces_weka_with_subagents"
939-
fi
940935
local loader="${WEKA_LOADER_OVERRIDE:-$default_loader}"
941936
local dataset
942937
case "$loader" in

benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,8 +52,6 @@ fi
5252
nvidia-smi
5353

5454
# ---- Resolve traces and install deps ----------------------------------------
55-
# Opt this recipe out of the DSv4 052726 default; use the v6 corpus.
56-
export WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_with_subagents_060226
5755
resolve_trace_source
5856
install_agentic_deps
5957

0 commit comments

Comments
 (0)