Skip to content

Commit b81e456

Browse files
christsoCopilot
andauthored
feat(studio): add Run Eval flow with suite filter, test-id filter, and target override (#947)
- Add eval-runner.ts: Hono API endpoints for discovery, launch, and status polling - GET /api/eval/discover: discovers eval files in project - GET /api/eval/targets: lists available target names - POST /api/eval/run: spawns CLI eval process with validated args - GET /api/eval/status/:id: polls running eval status - POST /api/eval/preview: generates CLI command preview - All endpoints also available project-scoped under /api/projects/:projectId/eval/* - Add RunEvalModal component: two-step wizard modal - Step 1: suite filter (text input with discovered file suggestions), test-id pills (repeatable with glob support), target override (searchable dropdown) - Step 2: advanced options (threshold, workers, dry-run) collapsed by default - Live CLI preview before launch - Run status view with stdout/stderr streaming after launch - Add entry points on every relevant page: - Home page (both single-project and multi-project): 'Run Eval' button - Run detail page: 'Re-run with Filters' (prefilled with current target) - Eval detail page: 'Run this Test' (prefilled with test ID and target) - All project-scoped variants included Closes #945 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 312cee3 commit b81e456

11 files changed

Lines changed: 1249 additions & 25 deletions

File tree

apps/cli/src/commands/results/eval-runner.ts

Lines changed: 493 additions & 0 deletions
Large diffs are not rendered by default.

apps/cli/src/commands/results/serve.ts

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ import { parseJsonlResults } from '../eval/artifact-writer.js';
4646
import { resolveRunManifestPath } from '../eval/result-layout.js';
4747
import { loadRunCache, resolveRunCacheFile } from '../eval/run-cache.js';
4848
import { listResultFiles } from '../trace/utils.js';
49+
import { registerEvalRoutes } from './eval-runner.js';
4950
import {
5051
loadLightweightResults,
5152
loadManifestResults,
@@ -892,6 +893,18 @@ export function createApp(
892893
app.get('/api/projects/:projectId/targets', (c) => withProject(c, handleTargets));
893894
app.get('/api/projects/:projectId/feedback', (c) => withProject(c, handleFeedbackRead));
894895

896+
// ── Eval runner routes (discovery, launch, status) ────────────────────
897+
898+
registerEvalRoutes(app, (c) => {
899+
// For project-scoped routes, resolve to project path; otherwise use searchDir
900+
const projectId = c.req.param('projectId');
901+
if (projectId) {
902+
const project = getProject(projectId);
903+
if (project) return project.path;
904+
}
905+
return searchDir;
906+
});
907+
895908
// ── Static file serving for Studio SPA ────────────────────────────────
896909

897910
const studioDistPath = options?.studioDir ?? resolveStudioDistDir();

0 commit comments

Comments
 (0)