feat(eval): load explicit TypeScript eval configs

[christso]

Summary

AgentV can now run and validate explicit TypeScript eval configs without treating every .ts file as runnable eval authoring. The supported CLI/programmatic contract is a default-exported EvalConfig from *.eval.ts or *.eval.mts; Promptfoo promptfooconfig.ts, agentvconfig.ts, generic config files, and .agentv/assertions/*.ts remain outside eval config loading.

This also tightens the published SDK authoring surface around one public EvalConfig type. defineEval(config) remains only as a thin optional helper, while EvalDefinition and evalSuite() are removed from the SDK TypeScript eval authoring exports and docs.

Design Notes

• Runtime TypeScript config loading and lowering lives in packages/core, and CLI eval/validate paths route through that shared loader.
• Directory/default discovery includes *.eval.ts and *.eval.mts, while broad globs and directory scans filter arbitrary TypeScript helpers.
• agentv validate now materializes TypeScript eval configs through the core loader so parse/export/contract errors match eval execution behavior.
• Docs, examples, and the bundled eval-writer skill now teach plain typed default exports as the recommended TypeScript authoring path.

Validation

• bun --filter @agentv/core build
• bun --filter @agentv/sdk build
• bun test apps/cli/test/commands/validate/validate-files.test.ts apps/cli/test/commands/eval/shared.test.ts apps/cli/test/commands/eval/discover.test.ts packages/core/test/evaluation/loaders/ts-eval-loader.test.ts packages/sdk/test/eval-authoring.test.ts packages/sdk/test/grader-helpers.test.ts packages/sdk/test/package-graph.test.ts
• bun apps/cli/src/cli.ts validate examples/features/sdk-eval-authoring/evals/greeting.eval.ts
• bun apps/cli/src/cli.ts eval examples/features/sdk-eval-authoring/evals/greeting.eval.ts

Red/green UAT: on the pre-branch main checkout, agentv validate /tmp/.../red.eval.ts reported Total files: 0; on this branch, the same direct *.eval.ts shape validates as one valid file.

Live dogfood: bun apps/cli/src/cli.ts eval /tmp/agentv-ts-eval-dogfood-*/live.eval.ts --targets /tmp/agentv-ts-eval-dogfood-*/targets-gemini.yaml --target live-gemini --grader-target live-gemini-grader --workers 1 --threshold 0.8 passed 1/1 with a live Gemini target and real Gemini llm-rubric grader.

Evidence

Private evidence branch: EntityProcess/agentv-private:evidence/av-kfik-47-eval-config-ts at commit 525dc45.

Related: av-kfik.47

---

[cloudflare-workers-and-pages]

Deploying agentv with    Cloudflare Pages

Latest commit:	0c0e375
Status:	✅  Deploy successful!
Preview URL:	https://be998c49.agentv.pages.dev
Branch Preview URL:	https://feat-av-kfik-47-eval-config.agentv.pages.dev

View logs