Skip to content

Commit f632aa4

Browse files
agentic(trace-source): default non-DSv4 to v6 (060226) corpus
resolve_trace_source() now picks a model-prefix-aware default: MODEL_PREFIX == dsv4 -> semianalysis_cc_traces_weka_with_subagents (052726, the v5 baseline, unchanged for continuity with prior DSv4 published runs) everything else -> semianalysis_cc_traces_weka_with_subagents_060226 (060226, newer v6 corpus with fresher CC recording windows) WEKA_LOADER_OVERRIDE still wins. Allowed values widened from the two 052726 loaders to all four: semianalysis_cc_traces_weka_with_subagents (052726) semianalysis_cc_traces_weka_with_subagents_256k (052726-256k) semianalysis_cc_traces_weka_with_subagents_060226 (060226) semianalysis_cc_traces_weka_with_subagents_060226_256k (060226-256k) Bumps utils/aiperf submodule to de3ad1c1, which registers the two 060226 plugin entries those new loader names resolve through. The pre-cache log line now also includes MODEL_PREFIX so it's obvious in CI which default fired. Signed-off-by: Cam Quilici <cameron@semianalysis.com>
1 parent 1b23499 commit f632aa4

2 files changed

Lines changed: 22 additions & 5 deletions

File tree

benchmarks/benchmark_lib.sh

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -924,8 +924,19 @@ resolve_trace_source() {
924924
# public-dataset loader names allowed by the inferencex-agentx-mvp
925925
# scenario. Used by recipes whose servers have non-default context
926926
# caps (e.g. minimaxm2.5 at max_model_len ~256k can't replay the
927-
# unfiltered 052726 corpus and switches to the 256k-capped variant).
928-
local loader="${WEKA_LOADER_OVERRIDE:-semianalysis_cc_traces_weka_with_subagents}"
927+
# unfiltered corpus and switches to the 256k-capped variant), or
928+
# by recipes that want to pin a specific corpus generation rather
929+
# than ride the model-prefix-aware default below.
930+
#
931+
# Default (no override) is model-prefix-aware:
932+
# DSv4 recipes -> 052726 (v5 corpus, the original baseline)
933+
# everything else -> 060226 (v6 corpus, newer CC versions)
934+
# DSv4 stays on 052726 for continuity with prior published baselines.
935+
local default_loader="semianalysis_cc_traces_weka_with_subagents_060226"
936+
if [[ "${MODEL_PREFIX:-}" == "dsv4" ]]; then
937+
default_loader="semianalysis_cc_traces_weka_with_subagents"
938+
fi
939+
local loader="${WEKA_LOADER_OVERRIDE:-$default_loader}"
929940
local dataset
930941
case "$loader" in
931942
semianalysis_cc_traces_weka_with_subagents)
@@ -934,13 +945,19 @@ resolve_trace_source() {
934945
semianalysis_cc_traces_weka_with_subagents_256k)
935946
dataset="semianalysisai/cc-traces-weka-with-subagents-052726-256k"
936947
;;
948+
semianalysis_cc_traces_weka_with_subagents_060226)
949+
dataset="semianalysisai/cc-traces-weka-with-subagents-060226"
950+
;;
951+
semianalysis_cc_traces_weka_with_subagents_060226_256k)
952+
dataset="semianalysisai/cc-traces-weka-with-subagents-060226-256k"
953+
;;
937954
*)
938-
echo "Error: unknown WEKA_LOADER_OVERRIDE='$loader'. Allowed: semianalysis_cc_traces_weka_with_subagents, semianalysis_cc_traces_weka_with_subagents_256k" >&2
955+
echo "Error: unknown WEKA_LOADER_OVERRIDE='$loader'. Allowed: semianalysis_cc_traces_weka_with_subagents, semianalysis_cc_traces_weka_with_subagents_256k, semianalysis_cc_traces_weka_with_subagents_060226, semianalysis_cc_traces_weka_with_subagents_060226_256k" >&2
939956
exit 1
940957
;;
941958
esac
942959
TRACE_SOURCE_FLAG="--public-dataset $loader"
943-
echo "Loading traces via aiperf public-dataset: $loader ($dataset)"
960+
echo "Loading traces via aiperf public-dataset: $loader ($dataset) [MODEL_PREFIX=${MODEL_PREFIX:-unset}]"
944961
# Pre-download the dataset into the shared HF_HUB_CACHE (same mount used
945962
# for model weights) so subsequent runs read from cache instead of
946963
# re-downloading every job.

0 commit comments

Comments
 (0)