Skip to content

feat(config): lower scenarios into tests#1692

Closed
christso wants to merge 1 commit into
feat/av-kfik-49-eval-cases-alias-removalfrom
feat/av-kfik-50-scenarios-lowering
Closed

feat(config): lower scenarios into tests#1692
christso wants to merge 1 commit into
feat/av-kfik-49-eval-cases-alias-removalfrom
feat/av-kfik-50-scenarios-lowering

Conversation

@christso

@christso christso commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

AgentV can now load Promptfoo-style scenarios as a pre-expansion matrix helper: each scenario config row is lowered with each scenario test row into ordinary AgentV tests before the normal prompts x tests x targets expansion.

This keeps scenarios as authoring sugar rather than a new run artifact or grouping model. providerPromptMap and provider_prompt_map remain rejected because prompt selection should stay explicit through AgentV prompts, targets, files, tags, and CLI selection rather than Promptfoo provider identity baggage.

Validation

After rebasing this stack onto merged PR #1690 and the alias-removal branch:

  • bun test packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts packages/core/test/evaluation/loaders/jsonl-parser.test.ts packages/core/test/evaluation/validation/eval-schema-sync.test.ts
  • bun --filter @agentv/core typecheck
  • bun --filter @agentv/core lint
  • bun --filter @agentv/core build

The worker also regenerated skills-data/agentv-eval-writer/references/eval.schema.json and verified the pinned Promptfoo clone at 6bfc5a0c7f16f9c4717ac731d276b578e63d0769.

Live provider dogfood is deferred to the final docs/examples pass (av-kfik.16) because this PR changes deterministic schema/validation/loading and does not alter provider/grader adapter behavior or run artifact contracts.

Related

Related: av-kfik.50
Depends on: #1691


Compound Engineering
GPT--5.5_high

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 1f5df35
Status: ✅  Deploy successful!
Preview URL: https://be92da53.agentv.pages.dev
Branch Preview URL: https://feat-av-kfik-50-scenarios-lo.agentv.pages.dev

View logs

@christso christso deleted the branch feat/av-kfik-49-eval-cases-alias-removal July 6, 2026 06:17
@christso christso closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant