fix(inference): scope run picker to selected scenario

On the /inference dashboard, the "Run N/M" picker listed every workflow
run for the date regardless of the selected scenario (Agentic Traces vs
Single Turn): it filtered only by changelog config_keys (model +
precision, no benchmark_type). A same-day single_turn sweep therefore
leaked into the Agentic Traces picker, and selecting it poisoned the
asOfRunId cutoff, blanking the chart.

Fix, DB-side and non-circular: the runConfigs coverage in the
workflow-info response (which enumerates all runs on a date
independently of the asOf cutoff) now carries the raw benchmark_type /
isl / osl columns. New pure helpers scenarioRunIdsForDate()
(runEnumeration.ts) and restrictRunsToScenario() (utils.ts) derive the
run ids with coverage for the selected model + scenario and restrict
the picker list to them, with fallbacks so the selector still renders
when coverage data is unavailable. GlobalFilterContext exposes
runConfigs; InferenceContext filters the picker's runs by the selected
scenario before deriving asOf. Filtering from loaded chart rows was
rejected as circular (those rows depend on asOfRunId). Unofficial-run
overlays are unaffected: dataRunsForDate() dedupes by run id, so the
finer-grained DISTINCT rows collapse.

中文：修复 /inference 面板中 "Run N/M" 选择器不区分场景的问题。此前
选择器仅按 changelog 的 config_keys（模型 + 精度）过滤，不含
benchmark_type，导致同一天的 single_turn 运行泄漏进 Agentic Traces
的选择器，选中后会污染 asOfRunId 截断点、使图表变空。修复方案在数据
库侧且无循环依赖：workflow-info 响应中的 runConfigs 覆盖数据（独立于
asOf 截断点、枚举当天全部运行）新增 benchmark_type / isl / osl 原始
列；新增纯函数 scenarioRunIdsForDate()（runEnumeration.ts）和
restrictRunsToScenario()（utils.ts），按所选模型 + 场景推导有数据覆盖
的运行 id 并据此收窄选择器列表，覆盖数据缺失时回退到原列表以保证选择
器正常渲染。GlobalFilterContext 暴露 runConfigs；InferenceContext 在
推导 asOf 之前按所选场景过滤选择器的运行列表。基于已加载图表行的客户
端过滤方案因循环依赖（这些行本身依赖 asOfRunId）被否决。非官方运行
叠加层不受影响：dataRunsForDate() 按运行 id 去重，更细粒度的
DISTINCT 行会被合并。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: cquil11

packages/app/cypress/support/mock-data.ts

@@ -447,6 +447,7 @@ export function createMockGlobalFilterContext(
     availabilityRows: undefined,
     workflowInfo: null,
     availableRuns: {},
+    runConfigs: [],
     workflowLoading: false,
     workflowError: null,
     ...overrides,

packages/app/src/components/GlobalFilterContext.tsx

@@ -36,7 +36,7 @@ import {
 import { computeAutoSwitchDecision } from '@/lib/unofficial-run-auto-switch';
 import { countCurvesByPrecision, resolveEffectivePrecisions } from '@/lib/default-precisions';
 import { resolveEffectiveSequence } from '@/lib/default-sequence';
-import type { AvailabilityRow, WorkflowInfoResponse } from '@/lib/api';
+import type { AvailabilityRow, RunConfigRow, WorkflowInfoResponse } from '@/lib/api';
 
 const RUNDATE_RE = /^\d{4}-\d{2}-\d{2}$/u;
 const RUNID_RE = /^[A-Za-z0-9_-]{1,64}$/u;
@@ -107,6 +107,14 @@ export interface GlobalFilterContextType {
   // Workflow info
   workflowInfo: { runInfoBySequence: Record<string, RunInfo> }[] | null;
   availableRuns: Record<string, RunInfo>;
+  /**
+   * Per-(run, config) coverage for the current date — which workflow runs
+   * produced benchmark data for which model / scenario / precision. Data-driven
+   * (from the benchmark rows), so it enumerates every run on the date including
+   * ones without a changelog entry. The run picker uses this to scope its list
+   * to the selected model + scenario (see `scenarioRunIdsForDate`).
+   */
+  runConfigs: RunConfigRow[];
   workflowLoading: boolean;
   workflowError: string | null;
 }
@@ -442,6 +450,8 @@ export function GlobalFilterProvider({
     [workflowData],
   );
 
+  const runConfigs = useMemo(() => workflowData?.runConfigs ?? [], [workflowData]);
+
   const workflowInfo = useMemo(
     () => (Object.keys(availableRuns).length > 0 ? [{ runInfoBySequence: availableRuns }] : null),
     [availableRuns],
@@ -530,6 +540,7 @@ export function GlobalFilterProvider({
       availabilityRows,
       workflowInfo,
       availableRuns,
+      runConfigs,
       workflowLoading,
       workflowError,
     }),
@@ -551,6 +562,7 @@ export function GlobalFilterProvider({
       availabilityRows,
       workflowInfo,
       availableRuns,
+      runConfigs,
       workflowLoading,
       workflowError,
     ],

packages/app/src/components/inference/InferenceContext.tsx

@@ -63,7 +63,7 @@ import {
   resolveExclusionToggle,
   type ExclusionConflictPolicy,
 } from '@/lib/exclusion';
-import { filterRunsByModel, getDisplayLabel } from '@/lib/utils';
+import { filterRunsByModel, getDisplayLabel, restrictRunsToScenario } from '@/lib/utils';
 
 import {
   isAgenticOnlyXAxisMode,
@@ -72,6 +72,7 @@ import {
   type XAxisMode,
 } from './hooks/useChartData';
 import { resolveComparisonEntries } from './utils/comparisonEntry';
+import { scenarioRunIdsForDate } from './utils/runEnumeration';
 import { resolveLabelState, serializeLabelState } from './utils/label-defaults';
 import {
   EMPTY_QUICK_FILTERS,
@@ -135,6 +136,7 @@ export function InferenceProvider({
     availabilityRows,
     workflowInfo,
     availableRuns,
+    runConfigs,
     workflowError,
   } = useGlobalFilters();
 
@@ -368,11 +370,39 @@ export function InferenceProvider({
     [selectedModel],
   );
 
-  const filteredAvailableRuns = useMemo(
+  // DB model keys for the selected display model (e.g. ['dsv4']). Used both by
+  // the scenario run scoping below and the GPU comparison date picker further down.
+  const dbModelKeys = useMemo<string[]>(
+    () => DISPLAY_MODEL_TO_DB[selectedModel] ?? [selectedModel],
+    [selectedModel],
+  );
+
+  const changelogFilteredRuns = useMemo(
     () => filterRunsByModel(availableRuns, modelPrefixes, [...effectivePrecisions]),
     [availableRuns, modelPrefixes, effectivePrecisions],
   );
 
+  // Run ids that actually produced data for the selected model + scenario on the
+  // date, derived from per-(run, config) coverage (independent of the chart's
+  // "as of run" query, so no circular dependency). `/api/v1/workflow-info`
+  // returns every run on a date across all models/scenarios, so without this a
+  // same-day run for a different scenario (e.g. a single_turn sweep while viewing
+  // Agentic Traces) would leak into the picker and poison the "as of run" cutoff.
+  const scenarioRunIds = useMemo(
+    () =>
+      scenarioRunIdsForDate(runConfigs, dbModelKeys, effectiveSequence, [...effectivePrecisions]),
+    [runConfigs, dbModelKeys, effectiveSequence, effectivePrecisions],
+  );
+
+  // The picker list: the changelog/precision-scoped runs, further restricted to
+  // the ones that shipped data for the selected scenario. Falls back to the
+  // changelog-scoped list when coverage data is unavailable (old dates / JSON
+  // mode) so the selector always renders.
+  const filteredAvailableRuns = useMemo(
+    () => restrictRunsToScenario(changelogFilteredRuns, availableRuns, scenarioRunIds),
+    [changelogFilteredRuns, availableRuns, scenarioRunIds],
+  );
+
   const effectiveSelectedRunId = useMemo(() => {
     if (!filteredAvailableRuns) return selectedRunId;
     const filteredRunIds = Object.keys(filteredAvailableRuns);
@@ -484,11 +514,6 @@ export function InferenceProvider({
   );
 
   // For GPU comparison date picker — use shared availability data from global filters
-  const dbModelKeys = useMemo<string[]>(
-    () => DISPLAY_MODEL_TO_DB[selectedModel] ?? [selectedModel],
-    [selectedModel],
-  );
-
   const dateRangeAvailableDates = useMemo(() => {
     if (selectedGPUs.length === 0) return availableDates;
     if (!availabilityRows) return availableDates;

packages/app/src/components/inference/utils/runEnumeration.test.ts

@@ -2,7 +2,7 @@ import { describe, expect, it } from 'vitest';
 
 import type { RunConfigRow } from '@/lib/api';
 
-import { dataRunsForDate } from './runEnumeration';
+import { dataRunsForDate, scenarioRunIdsForDate } from './runEnumeration';
 
 function rc(over: Partial<RunConfigRow>): RunConfigRow {
   return {
@@ -16,10 +16,24 @@ function rc(over: Partial<RunConfigRow>): RunConfigRow {
     framework: 'vllm',
     spec_method: 'none',
     disagg: false,
+    // Defaults to a single_turn 8k/1k row; agentic tests override these.
+    benchmark_type: 'single_turn',
+    isl: 8192,
+    osl: 1024,
     ...over,
   };
 }
 
+/** A single_turn (fixed-seq) run config for the given isl/osl. */
+function single(over: Partial<RunConfigRow>): RunConfigRow {
+  return rc({ benchmark_type: 'single_turn', isl: 8192, osl: 1024, ...over });
+}
+
+/** An agentic_traces run config (null isl/osl, as ingested for agentic rows). */
+function agentic(over: Partial<RunConfigRow>): RunConfigRow {
+  return rc({ benchmark_type: 'agentic_traces', isl: null, osl: null, ...over });
+}
+
 const SCOPE = {
   modelDbKeys: ['minimaxm3'],
   selectedGPUs: ['mi300x_vllm'],
@@ -107,3 +121,66 @@ describe('dataRunsForDate', () => {
     expect(dataRunsForDate([rc({ model: 'dsr1' })], SCOPE)).toEqual([]);
   });
 });
+
+// Mirrors the repro: on one date the same model has a single_turn run and an
+// agentic run; each scenario must list ONLY its own run.
+const ids = (rows: RunConfigRow[], model: string[], seq: string, prec: string[] = []) =>
+  [...scenarioRunIdsForDate(rows, model, seq, prec)].toSorted();
+
+describe('scenarioRunIdsForDate', () => {
+  it('agentic scenario excludes runs that only produced single_turn data', () => {
+    const rows = [
+      agentic({ github_run_id: 28955639528, model: 'dsv4' }), // dsv4 agentic
+      single({ github_run_id: 28900000001, model: 'dsv4' }), // dsv4 single_turn (leaks today)
+      single({ github_run_id: 28900000002, model: 'glm5' }), // other model
+    ];
+    expect(ids(rows, ['dsv4'], 'agentic-traces')).toEqual(['28955639528']);
+  });
+
+  it('single_turn scenario excludes agentic-only runs', () => {
+    const rows = [
+      agentic({ github_run_id: 1, model: 'dsv4' }),
+      single({ github_run_id: 2, model: 'dsv4' }),
+    ];
+    expect(ids(rows, ['dsv4'], '8k/1k')).toEqual(['2']);
+  });
+
+  it('lists a run that produced data for both scenarios under each scenario', () => {
+    const rows = [
+      agentic({ github_run_id: 42, model: 'dsv4' }),
+      single({ github_run_id: 42, model: 'dsv4' }), // same run, both types
+    ];
+    expect(ids(rows, ['dsv4'], 'agentic-traces')).toEqual(['42']);
+    expect(ids(rows, ['dsv4'], '8k/1k')).toEqual(['42']);
+  });
+
+  it('scopes to the selected model DB keys', () => {
+    const rows = [
+      agentic({ github_run_id: 1, model: 'dsv4' }),
+      agentic({ github_run_id: 2, model: 'glm5' }),
+    ];
+    expect(ids(rows, ['dsv4'], 'agentic-traces')).toEqual(['1']);
+  });
+
+  it('respects precision scoping when precisions are provided', () => {
+    const rows = [
+      agentic({ github_run_id: 1, model: 'dsv4', precision: 'fp4' }),
+      agentic({ github_run_id: 2, model: 'dsv4', precision: 'fp8' }),
+    ];
+    expect(ids(rows, ['dsv4'], 'agentic-traces', ['fp4'])).toEqual(['1']);
+    // Empty precisions = no precision constraint.
+    expect(ids(rows, ['dsv4'], 'agentic-traces')).toEqual(['1', '2']);
+  });
+
+  it('dedupes a run appearing across multiple matching configs', () => {
+    const rows = [
+      agentic({ github_run_id: 7, model: 'dsv4', hardware: 'b200' }),
+      agentic({ github_run_id: 7, model: 'dsv4', hardware: 'gb200' }),
+    ];
+    expect(ids(rows, ['dsv4'], 'agentic-traces')).toEqual(['7']);
+  });
+
+  it('returns an empty set when there is no coverage data', () => {
+    expect(scenarioRunIdsForDate([], ['dsv4'], 'agentic-traces').size).toBe(0);
+  });
+});

packages/app/src/components/inference/utils/runEnumeration.ts

@@ -15,6 +15,8 @@
  * chart keys them.
  */
 
+import { rowToSequence } from '@semianalysisai/inferencex-constants';
+
 import type { AggDataEntry } from '@/components/inference/types';
 import type { RunConfigRow } from '@/lib/api';
 import { getHardwareKey } from '@/lib/chart-utils';
@@ -78,3 +80,38 @@ export function dataRunsForDate(runConfigs: RunConfigRow[], scope: RunScope): Da
 
   return [...byRun.values()].toSorted((a, b) => a.runStartedAt.localeCompare(b.runStartedAt));
 }
+
+/**
+ * GitHub run ids (as strings) that produced benchmark data for the given model
+ * DB keys + scenario (sequence) — and, when `precisions` is non-empty, only for
+ * those precisions — on a date. Derived from per-(run, config) coverage
+ * ({@link RunConfigRow}, i.e. the benchmark rows themselves), so a run that
+ * shipped data without a changelog entry still counts.
+ *
+ * The run picker uses this to list ONLY the runs that actually produced data for
+ * the currently selected model + scenario. `/api/v1/workflow-info` returns every
+ * run on a date across all models and scenarios, so without this scoping a
+ * same-day run for a different scenario (e.g. a `single_turn` sweep while the
+ * user is viewing Agentic Traces) leaks into the picker — and selecting it
+ * poisons the "as of run" cutoff, blanking the chart.
+ *
+ * Scenario matching mirrors the chart's own row filter (`rowToSequence(row) ===
+ * selectedSequence` in useChartData), so the picker and the plotted data always
+ * agree on which runs are relevant.
+ */
+export function scenarioRunIdsForDate(
+  runConfigs: RunConfigRow[],
+  modelDbKeys: string[],
+  sequence: string,
+  precisions: string[] = [],
+): Set<string> {
+  const precSet = precisions.length > 0 ? new Set(precisions) : null;
+  const ids = new Set<string>();
+  for (const rc of runConfigs) {
+    if (!modelDbKeys.includes(rc.model)) continue;
+    if (precSet && !precSet.has(rc.precision)) continue;
+    if (rowToSequence(rc) !== sequence) continue;
+    ids.add(String(rc.github_run_id));
+  }
+  return ids;
+}

packages/app/src/lib/api.ts

@@ -96,6 +96,12 @@ export interface RunConfigRow {
   framework: string;
   spec_method: string;
   disagg: boolean;
+  /** Benchmark scenario: `single_turn` (fixed-seq isl/osl) or `agentic_traces`. */
+  benchmark_type: string;
+  /** ISL in tokens — null for agentic_traces. With osl + benchmark_type this maps to a sequence. */
+  isl: number | null;
+  /** OSL in tokens — null for agentic_traces. */
+  osl: number | null;
 }
 
 export interface WorkflowInfoResponse {

packages/app/src/lib/utils.test.ts

@@ -10,6 +10,7 @@ import {
   computeOutputCostFields,
   computeInputCostFields,
   filterRunsByModel,
+  restrictRunsToScenario,
   getFrameworkLabel,
   getHardwareLabel,
   getDisplayLabel,
@@ -539,6 +540,51 @@ describe('filterRunsByModel', () => {
   });
 });
 
+// ---------------------------------------------------------------------------
+// restrictRunsToScenario
+// ---------------------------------------------------------------------------
+describe('restrictRunsToScenario', () => {
+  const runs = {
+    '10': makeRun([['dsv4-fp4-b200-sglang']], { runId: '10' }), // agentic run
+    '20': makeRun([['dsv4-fp4-b200-trt']], { runId: '20' }), // single_turn run
+  };
+
+  it('keeps only runs present in the scenario coverage set', () => {
+    const result = restrictRunsToScenario(runs, runs, new Set(['10']));
+    expect(Object.keys(result!)).toEqual(['10']);
+  });
+
+  it('falls back to the changelog-scoped list when coverage is empty', () => {
+    const result = restrictRunsToScenario(runs, runs, new Set());
+    expect(result).toBe(runs);
+  });
+
+  it('pulls a scenario run from the raw map when the changelog list omits it', () => {
+    // A run that shipped data without a changelog entry: filterRunsByModel drops
+    // it, but it is still a valid scenario run the picker must show.
+    const changelogFiltered = { '10': runs['10'] };
+    const allRuns = runs;
+    const result = restrictRunsToScenario(changelogFiltered, allRuns, new Set(['10', '20']));
+    expect(Object.keys(result!).toSorted()).toEqual(['10', '20']);
+    expect(result!['20']).toBe(runs['20']);
+  });
+
+  it('prefers the changelog-scoped entry over the raw one', () => {
+    const scoped = makeRun([['dsv4-fp4-b200-sglang']], { runId: '10', conclusion: 'scoped' });
+    const result = restrictRunsToScenario({ '10': scoped }, runs, new Set(['10']));
+    expect(result!['10']).toBe(scoped);
+  });
+
+  it('falls back to the changelog list when no scenario run survives', () => {
+    const result = restrictRunsToScenario(runs, runs, new Set(['999']));
+    expect(result).toBe(runs);
+  });
+
+  it('returns null changelog list unchanged', () => {
+    expect(restrictRunsToScenario(null, null, new Set(['10']))).toBeNull();
+  });
+});
+
 // ===========================================================================
 // getFrameworkLabel
 // ===========================================================================

packages/app/src/lib/utils.ts

@@ -288,6 +288,39 @@ export function filterRunsByModel(
   return Object.keys(fallback).length > 0 ? fallback : null;
 }
 
+/**
+ * Restrict the run-picker list to the runs that actually produced data for the
+ * currently selected scenario, given `scenarioRunIds` (the run ids with coverage
+ * for the selected model + scenario, from `scenarioRunIdsForDate`).
+ *
+ * `changelogFiltered` is the model/precision-scoped list (`filterRunsByModel`);
+ * `allRuns` is the raw, date-wide map. For each scenario run we prefer the
+ * changelog-scoped entry (its changelog is already precision-filtered) and fall
+ * back to the raw entry — a run can ship benchmark data without a changelog
+ * entry, so `filterRunsByModel` may omit it while it is still a legitimate
+ * scenario run the user must be able to select.
+ *
+ * Fallbacks that keep the selector rendering:
+ *  - `scenarioRunIds` empty → no coverage data (dates predating per-run
+ *    coverage, or JSON/fixtures mode): return the changelog-scoped list
+ *    unchanged rather than blanking the picker.
+ *  - Non-empty coverage but zero surviving runs (shouldn't happen in practice)
+ *    → also fall back to the changelog-scoped list.
+ */
+export function restrictRunsToScenario(
+  changelogFiltered: Record<string, RunInfo> | null,
+  allRuns: Record<string, RunInfo> | null,
+  scenarioRunIds: Set<string>,
+): Record<string, RunInfo> | null {
+  if (scenarioRunIds.size === 0) return changelogFiltered;
+  const restricted: Record<string, RunInfo> = {};
+  for (const runId of scenarioRunIds) {
+    const info = changelogFiltered?.[runId] ?? allRuns?.[runId];
+    if (info) restricted[runId] = info;
+  }
+  return Object.keys(restricted).length > 0 ? restricted : changelogFiltered;
+}
+
 /**
  * Computes missing energy fields (jTotal, jOutput, jInput) at runtime.
  * This handles backwards compatibility with historical data that doesn't have these fields.