feat(config): lower scenarios into tests#1693
Merged
Merged
Conversation
Deploying agentv with
|
| Latest commit: |
c2d5e12
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://275b7db3.agentv.pages.dev |
| Branch Preview URL: | https://feat-av-kfik-50-scenarios-lo.agentv.pages.dev |
7265c58 to
c2d5e12
Compare
Collaborator
Author
|
Coordinator review update: after rebasing onto merged main, I found and fixed one validation bug before merge. Scenario tests without their own prompts/provider_output now validate when every scenario config row supplies prompts/provider_output, matching runtime lowering. Added positive and mixed-config negative regression tests. Local focused tests now pass with 212 cases, plus core typecheck, lint, build, and git diff --check. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AgentV can now load Promptfoo-style
scenariosas a pre-expansion matrix helper: each scenario config row is lowered with each scenario test row into ordinary AgentV tests before the normal prompts x tests x targets expansion.This keeps scenarios as authoring sugar rather than a new run artifact or grouping model.
providerPromptMapandprovider_prompt_mapremain rejected because prompt selection should stay explicit through AgentV prompts, targets, files, tags, and CLI selection rather than Promptfoo provider identity baggage.Validation
After PR #1691 merged to main, this branch was rebased onto
origin/mainand validated with:bun test packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts packages/core/test/evaluation/loaders/jsonl-parser.test.ts packages/core/test/evaluation/validation/eval-schema-sync.test.tsbun --filter @agentv/core typecheckbun --filter @agentv/core lintbun --filter @agentv/core buildThe worker also regenerated
skills-data/agentv-eval-writer/references/eval.schema.jsonand verified the pinned Promptfoo clone at6bfc5a0c7f16f9c4717ac731d276b578e63d0769.Live provider dogfood is deferred to the final docs/examples pass (
av-kfik.16) because this PR changes deterministic schema/validation/loading and does not alter provider/grader adapter behavior or run artifact contracts.Related
Related: av-kfik.50
Supersedes closed stacked PR #1692.