Skip to content

Commit c8fcc4b

Browse files
fix(inference): default scenario to 8K/1K instead of Agentic Traces (#552)
The bare /inference landing (and any page without an explicit ?i_seq=) now defaults the Scenario selector to the 8K/1K fixed-sequence scenario instead of Agentic Traces. Explicit ?i_seq= links, per-route initialSequence seeds (/compare pages), and the availability fallback ordering (selection → 8k/1k → first available) are unchanged. Agentic e2e specs now pin ?i_seq=agentic-traces explicitly, and a new e2e asserts the 8K/1K default holds even when the model has agentic data. 中文:将默认场景从 Agentic Traces 改为 8K/1K。未携带 ?i_seq= 参数的 /inference 页面现在默认选中 8K/1K 固定序列场景;显式 ?i_seq= 链接、 /compare 页面的 initialSequence 预设以及可用性回退顺序保持不变。 agentic 相关的 E2E 测试改为显式指定 ?i_seq=agentic-traces,并新增 E2E 用例验证即使模型有 agentic 数据,默认场景仍为 8K/1K。 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 098f611 commit c8fcc4b

9 files changed

Lines changed: 83 additions & 62 deletions

File tree

packages/app/cypress/e2e/gpu-compare-agentic-detail.cy.ts

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,10 @@ import { unlockAgenticGate } from '../support/e2e';
44
// Spec-scoped fixture helpers
55
//
66
// The shared cypress/fixtures/api/*.json files contain ZERO agentic_traces rows
7-
// (by design — adding them flips the bare /inference default to the agentic
8-
// scenario and regresses other specs). This spec therefore injects minimal
9-
// agentic data via spec-scoped cy.intercept overrides that shadow the fixture
10-
// server, following the same pattern used in ttft-x-axis-toggle.cy.ts.
7+
// (by design — agentic coverage is injected per-spec so fixed-seq specs stay
8+
// lean). This spec therefore injects minimal agentic data via spec-scoped
9+
// cy.intercept overrides that shadow the fixture server, following the same
10+
// pattern used in ttft-x-axis-toggle.cy.ts.
1111
// ---------------------------------------------------------------------------
1212

1313
const DEFAULT_MODEL_DB_KEY = 'dsv4'; // DeepSeek-V4-Pro

packages/app/cypress/e2e/ttft-x-axis-toggle.cy.ts

Lines changed: 26 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,12 @@ const interceptDerivedMetrics = () => {
2222

2323
// This spec exercises the agentic x-axis modes, which only exist when the
2424
// selected model resolves to the Agentic Traces scenario. The default e2e
25-
// fixtures (cypress/fixtures/api/*.json) have NO agentic rows for any model, so
26-
// after the availability-gated effectiveSequence fix the bare-/inference default
27-
// correctly resolves to a fixed-seq scenario. We therefore inject agentic
28-
// availability + benchmark rows for the default model VIA SPEC-SCOPED INTERCEPTS
29-
// (not the shared fixtures) so this test — and only this test — sees the agentic
30-
// view. Scoping to intercepts keeps every other spec's default fixed-seq.
25+
// fixtures (cypress/fixtures/api/*.json) have NO agentic rows for any model, and
26+
// the app default scenario is 8K/1K, so bare /inference always resolves to a
27+
// fixed-seq scenario. We therefore inject agentic availability + benchmark rows
28+
// for the default model VIA SPEC-SCOPED INTERCEPTS (not the shared fixtures) and
29+
// visit with an explicit ?i_seq=agentic-traces so this test — and only this
30+
// test — sees the agentic view.
3131
const DEFAULT_MODEL_DB_KEY = 'dsv4'; // DeepSeek-V4-Pro is the default model
3232
const AGENTIC_DATE = '2026-06-12';
3333

@@ -66,8 +66,8 @@ const agenticGpus = [
6666
{ hardware: 'b300', framework: 'vllm', disagg: false },
6767
];
6868

69-
// Availability: default model has BOTH agentic and fixed-seq, so the default
70-
// resolves to agentic (the product-intended, agentic-preferred behavior).
69+
// Availability: default model has BOTH agentic and fixed-seq. The agentic view
70+
// is selected explicitly via ?i_seq=agentic-traces (the app default is 8K/1K).
7171
const agenticAvailability = [
7272
...agenticGpus.map((g) => ({
7373
model: DEFAULT_MODEL_DB_KEY,
@@ -144,7 +144,7 @@ const interceptFixedSequenceData = () => {
144144
describe('X-Axis Mode Toggle (inference chart)', () => {
145145
before(() => {
146146
interceptAgenticData();
147-
cy.visit('/inference', {
147+
cy.visit('/inference?i_seq=agentic-traces', {
148148
onBeforeLoad(win) {
149149
win.localStorage.setItem('inferencex-star-modal-dismissed', String(Date.now()));
150150
unlockAgenticGate(win);
@@ -173,7 +173,7 @@ describe('X-Axis Mode Toggle (inference chart)', () => {
173173

174174
it('honors explicit label URL overrides for the agentic view', () => {
175175
interceptAgenticData();
176-
cy.visit('/inference?i_label=0&i_advlabel=0&i_linelabel=1', {
176+
cy.visit('/inference?i_seq=agentic-traces&i_label=0&i_advlabel=0&i_linelabel=1', {
177177
onBeforeLoad(win) {
178178
win.localStorage.setItem('inferencex-star-modal-dismissed', String(Date.now()));
179179
unlockAgenticGate(win);
@@ -234,6 +234,21 @@ describe('X-Axis Mode Toggle (inference chart)', () => {
234234
});
235235
});
236236

237+
describe('Default scenario', () => {
238+
it('bare /inference defaults to 8K / 1K even when the model has agentic data', () => {
239+
// Availability contains BOTH agentic and fixed-seq rows for the default
240+
// model; the default selection must still resolve to 8K/1K, not agentic.
241+
interceptFixedSequenceData();
242+
cy.visit('/inference', {
243+
onBeforeLoad(win) {
244+
win.localStorage.setItem('inferencex-star-modal-dismissed', String(Date.now()));
245+
},
246+
});
247+
cy.get('[data-testid="scenario-selector"]').should('contain.text', '8K / 1K');
248+
cy.get('[data-testid="chart-figure"]').should('have.length.at.least', 1);
249+
});
250+
});
251+
237252
describe('Label defaults for fixed-sequence scenarios', () => {
238253
it('keeps parallelism labels off and line labels on by default', () => {
239254
interceptFixedSequenceData();
@@ -325,7 +340,7 @@ const interceptAgenticDataWithOverlay = () => {
325340
describe('X-Axis Mode Toggle — overlay path (finding #8 regression guard)', () => {
326341
before(() => {
327342
interceptAgenticDataWithOverlay();
328-
cy.visit(`/inference?unofficialrun=${OVERLAY_RUN_ID}`, {
343+
cy.visit(`/inference?unofficialrun=${OVERLAY_RUN_ID}&i_seq=agentic-traces`, {
329344
onBeforeLoad(win) {
330345
win.localStorage.setItem('inferencex-star-modal-dismissed', String(Date.now()));
331346
unlockAgenticGate(win);

packages/app/src/components/GlobalFilterContext.tsx

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,8 @@ const RUNID_RE = /^[A-Za-z0-9_-]{1,64}$/u;
4545
// before availability has loaded. It must be a fixed-seq scenario — never
4646
// AgenticTraces — so the scenario selector doesn't flash "Agentic Traces" for a
4747
// fixed-seq-only model while the chart shows its loading skeleton. `8k/1k` is
48-
// the pre-agentic default for non-agentic models. Consumers that must not act on
49-
// an unresolved sequence gate on `sequenceResolved` instead.
48+
// the app default scenario. Consumers that must not act on an unresolved
49+
// sequence gate on `sequenceResolved` instead.
5050
// (Declared after the import block so it never references `Sequence` above its import.)
5151
const PRE_AVAILABILITY_SEQUENCE = Sequence.EightK_OneK;
5252

@@ -170,9 +170,9 @@ export function GlobalFilterProvider({
170170
if (initialSequence) return initialSequence;
171171
const urlSeq = getUrlParam('i_seq');
172172
if (urlSeq && Object.values(Sequence).includes(urlSeq as Sequence)) return urlSeq as Sequence;
173-
// Prefer Agentic Traces by default when the selected model has it; the
174-
// effectiveSequence fallback below handles models without agentic data.
175-
return Sequence.AgenticTraces;
173+
// Default to the 8K/1K fixed-seq scenario; the effectiveSequence fallback
174+
// below handles models that lack it (e.g. agentic-only models).
175+
return Sequence.EightK_OneK;
176176
});
177177

178178
const initialValidPrecisions = useMemo(
@@ -311,9 +311,10 @@ export function GlobalFilterProvider({
311311
// may arrive from the DB (`availabilityRows`) OR from a loaded unofficial run
312312
// (`unofficialAvailable` for this model) — either source lets us resolve a
313313
// trustworthy effectiveSequence. Until then `availableSequences` is the static
314-
// SEQUENCE_OPTIONS fallback (which contains AgenticTraces), so resolving
315-
// eagerly would fetch + label an agentic scenario for fixed-seq-only models,
316-
// then snap once availability lands (flash + wasted request).
314+
// SEQUENCE_OPTIONS fallback (which contains every scenario), so resolving
315+
// eagerly would honor a selection the model may not have (e.g. agentic-traces
316+
// from a shared link on a fixed-seq-only model), fetch it, then snap once
317+
// availability lands (flash + wasted request).
317318
const availabilityLoaded = useMemo(
318319
() =>
319320
availabilityRows !== undefined || unofficialAvailable.some((a) => a.model === selectedModel),

packages/app/src/components/inference/InferenceContext.tsx

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -469,10 +469,10 @@ export function InferenceProvider({
469469
userPowers,
470470
effectiveRunDate,
471471
// Gate benchmark fetching on sequenceResolved: before availability loads we
472-
// don't yet know the model's real sequence, and the selectedSequence default
473-
// is AgenticTraces. Fetching now would fire the agentic data path for a
474-
// fixed-seq-only model, then refetch once availability snaps the sequence.
475-
// The chart's normal loading state covers this brief window.
472+
// don't yet know the model's real sequence, and the selection (e.g. an
473+
// agentic `?i_seq=` link) may be a scenario the model doesn't have. Fetching
474+
// now would fire the wrong data path, then refetch once availability snaps
475+
// the sequence. The chart's normal loading state covers this brief window.
476476
isActive && sequenceResolved,
477477
latestDate,
478478
selectedPercentile,

packages/app/src/components/ui/chart-selectors.tsx

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -315,7 +315,8 @@ export function ScenarioSelector({
315315
<SelectValue />
316316
</SelectTrigger>
317317
<SelectContent>
318-
{/* Agentic first — preferred default scenario when available. */}
318+
{/* Agentic group listed first when available (display order only —
319+
the app default scenario is 8K/1K). */}
319320
{agentic.length > 0 && (
320321
<SelectGroup>
321322
<SelectLabel>Agentic</SelectLabel>

packages/app/src/lib/default-sequence.test.ts

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,8 @@ describe('resolveEffectiveSequence', () => {
3636

3737
describe('honors a valid selection (rule 2a)', () => {
3838
it('keeps AgenticTraces when the model actually has agentic data (dsr1 case)', () => {
39-
// DeepSeek-R1 in the seeded DB has both agentic and 8k/1k — the agentic
40-
// default must survive so the PR intent (agentic-preferred) holds.
39+
// DeepSeek-R1 in the seeded DB has both agentic and 8k/1k — an explicit
40+
// agentic selection (e.g. a shared ?i_seq= link) must survive.
4141
expect(
4242
resolveEffectiveSequence({
4343
selectedSequence: Sequence.AgenticTraces,
@@ -59,8 +59,8 @@ describe('resolveEffectiveSequence', () => {
5959
});
6060

6161
describe('fallback ordering when the selection is unavailable (rule 2b/2c)', () => {
62-
it('for a fixed-seq-only model, agentic default falls back to 8k/1k, not the raw first entry (llama70b case)', () => {
63-
// Llama-3.3-70B has only 8k/1k in the seeded DB. The agentic default is
62+
it('for a fixed-seq-only model, an agentic selection falls back to 8k/1k, not the raw first entry (llama70b case)', () => {
63+
// Llama-3.3-70B has only 8k/1k in the seeded DB. An agentic selection is
6464
// unavailable, so it must resolve to a fixed-seq scenario — here the sole
6565
// available one.
6666
expect(
@@ -73,8 +73,8 @@ describe('resolveEffectiveSequence', () => {
7373
});
7474

7575
it('prefers 8k/1k over availableSequences[0] when both 1k/1k and 8k/1k exist', () => {
76-
// DB row order can surface 1k/1k first. Master defaulted non-agentic
77-
// models to 8k/1k, so prefer it rather than snapping to 1k/1k.
76+
// DB row order can surface 1k/1k first. 8k/1k is the app default
77+
// scenario, so prefer it rather than snapping to 1k/1k.
7878
expect(
7979
resolveEffectiveSequence({
8080
selectedSequence: Sequence.AgenticTraces,

packages/app/src/lib/default-sequence.ts

Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -3,29 +3,30 @@ import { Sequence } from './data-mappings';
33
/**
44
* Effective-sequence resolution.
55
*
6-
* `selectedSequence` defaults to {@link Sequence.AgenticTraces} (a deliberate
7-
* product choice — agentic-preferred), but not every model has agentic data.
8-
* This helper turns the raw user/default selection into the sequence the chart
9-
* should actually render, given what the selected model offers.
6+
* `selectedSequence` defaults to {@link Sequence.EightK_OneK}, but the user may
7+
* have selected a scenario the model doesn't offer (e.g. Agentic Traces via a
8+
* shared `?i_seq=` link on a fixed-seq-only model). This helper turns the raw
9+
* user/default selection into the sequence the chart should actually render,
10+
* given what the selected model offers.
1011
*
1112
* Two rules, in order:
1213
*
1314
* 1. **Availability gate.** Until availability rows have loaded we do NOT know
1415
* which sequences the model has. Resolving eagerly here would pick the static
15-
* fallback list (which contains AgenticTraces) and make the page fetch + label
16-
* an agentic scenario for fixed-seq-only models (e.g. Llama-3.3-70B), then
17-
* snap to a fixed-seq scenario once availability arrives — a visible flash of
18-
* "Agentic Traces" plus a wasted request. When `availabilityLoaded` is false
19-
* we return `null`; callers gate data fetching and selector display on a
20-
* non-null result (a loading skeleton covers this window, which is short).
16+
* fallback list (which contains every scenario) and make the page fetch +
17+
* label a scenario the model may not have — e.g. an agentic `?i_seq=` link on
18+
* Llama-3.3-70B — then snap to an available scenario once availability
19+
* arrives: a visible flash plus a wasted request. When `availabilityLoaded`
20+
* is false we return `null`; callers gate data fetching and selector display
21+
* on a non-null result (a loading skeleton covers this window, which is
22+
* short).
2123
*
2224
* 2. **Fallback ordering.** Once availability is known: keep the user's
2325
* `selectedSequence` if the model has it. Otherwise fall back to a sensible
2426
* fixed-seq scenario. `availableSequences[0]` follows DB row order, which can
25-
* surface `1k/1k` even when `8k/1k` exists — but `8k/1k` was the pre-agentic
26-
* default for non-agentic models, so prefer it when present to match that
27-
* long-standing behavior. Only if neither the selection nor `8k/1k` is
28-
* available do we fall to `availableSequences[0]`.
27+
* surface `1k/1k` even when `8k/1k` exists — but `8k/1k` is the app default
28+
* scenario, so prefer it when present. Only if neither the selection nor
29+
* `8k/1k` is available do we fall to `availableSequences[0]`.
2930
*/
3031
export function resolveEffectiveSequence({
3132
selectedSequence,
@@ -42,8 +43,8 @@ export function resolveEffectiveSequence({
4243
// Rule 2a: honor the user's / default selection when the model supports it.
4344
if (availableSequences.includes(selectedSequence)) return selectedSequence;
4445

45-
// Rule 2b: prefer 8k/1k (the pre-agentic default for non-agentic models) over
46-
// whatever availableSequences[0] happens to be (DB row order can yield 1k/1k).
46+
// Rule 2b: prefer 8k/1k (the app default scenario) over whatever
47+
// availableSequences[0] happens to be (DB row order can yield 1k/1k).
4748
if (availableSequences.includes(Sequence.EightK_OneK)) return Sequence.EightK_OneK;
4849

4950
// Rule 2c: last resort — first available, or the selection itself if the model

packages/app/src/lib/url-state.test.ts

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -31,10 +31,11 @@ describe('PARAM_DEFAULTS', () => {
3131
});
3232

3333
it('has an EMPTY default for i_seq so the selected scenario is always written', async () => {
34-
// The UI default scenario (gate-unlocked) is AgenticTraces, not 8k/1k. An
35-
// '8k/1k' default would strip an explicit 8K/1K selection from the URL, which
36-
// then resolves back to the agentic default on reload/share. Empty means no
37-
// scenario value ever matches the default, so it's always persisted.
34+
// Per-route `initialSequence` seeds (e.g. /compare pages) make the no-param
35+
// resolution route-dependent. An '8k/1k' default would strip an explicit
36+
// 8K/1K selection from the URL, which then resolves back to the route's
37+
// seeded scenario on reload/share. Empty means no scenario value ever
38+
// matches the default, so it's always persisted.
3839
const { PARAM_DEFAULTS } = await import('@/lib/url-state');
3940
expect(PARAM_DEFAULTS.i_seq).toBe('');
4041
});
@@ -190,8 +191,9 @@ describe('writeUrlParams + buildShareUrl', () => {
190191
setupWindow('', '/inference');
191192
const { writeUrlParams, buildShareUrl } = await import('@/lib/url-state');
192193

193-
// Picking the fixed-seq scenario must survive into the share URL; before the
194-
// fix this matched the '8k/1k' default and was dropped, reverting to agentic.
194+
// Picking the fixed-seq scenario must survive into the share URL; on routes
195+
// seeded with a different initialSequence (e.g. /compare pages), stripping
196+
// it would revert the pick back to the seeded scenario on reload.
195197
writeUrlParams({ i_seq: '8k/1k' });
196198
await vi.advanceTimersByTimeAsync(200);
197199

packages/app/src/lib/url-state.ts

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -74,12 +74,13 @@ export const PARAM_DEFAULTS: Record<UrlStateKey, string> = {
7474
g_model: 'DeepSeek-V4-Pro',
7575
g_rundate: '',
7676
g_runid: '',
77-
// No strippable default: the UI default scenario (gate-unlocked) is
78-
// AgenticTraces, not 8k/1k, so an '8k/1k' default here would strip an explicit
79-
// 8K/1K selection from the URL — on reload the empty i_seq resolves back to the
80-
// agentic default. Empty means the resolved scenario is ALWAYS written
81-
// explicitly (effectiveSequence is never ''), so a shared/reloaded link keeps
82-
// whatever the user picked. The no-param case still resolves via availability.
77+
// No strippable default: per-route `initialSequence` seeds (e.g. the /compare
78+
// pages) make the no-param resolution route-dependent, so stripping '8k/1k'
79+
// (the global default) would revert an explicit 8K/1K pick back to the route's
80+
// seeded scenario on reload. Empty means the resolved scenario is ALWAYS
81+
// written explicitly (effectiveSequence is never ''), so a shared/reloaded
82+
// link keeps whatever the user picked. The no-param case still resolves via
83+
// availability.
8384
i_seq: '',
8485
// No strippable default: precision is only written to the URL once chosen
8586
// explicitly, so an explicit FP4 selection must survive (not be stripped as a

0 commit comments

Comments
 (0)