feat(config): lower scenarios into tests#1692
Closed
christso wants to merge 1 commit into
Closed
Conversation
Deploying agentv with
|
| Latest commit: |
1f5df35
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://be92da53.agentv.pages.dev |
| Branch Preview URL: | https://feat-av-kfik-50-scenarios-lo.agentv.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AgentV can now load Promptfoo-style
scenariosas a pre-expansion matrix helper: each scenario config row is lowered with each scenario test row into ordinary AgentV tests before the normal prompts x tests x targets expansion.This keeps scenarios as authoring sugar rather than a new run artifact or grouping model.
providerPromptMapandprovider_prompt_mapremain rejected because prompt selection should stay explicit through AgentV prompts, targets, files, tags, and CLI selection rather than Promptfoo provider identity baggage.Validation
After rebasing this stack onto merged PR #1690 and the alias-removal branch:
bun test packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts packages/core/test/evaluation/loaders/jsonl-parser.test.ts packages/core/test/evaluation/validation/eval-schema-sync.test.tsbun --filter @agentv/core typecheckbun --filter @agentv/core lintbun --filter @agentv/core buildThe worker also regenerated
skills-data/agentv-eval-writer/references/eval.schema.jsonand verified the pinned Promptfoo clone at6bfc5a0c7f16f9c4717ac731d276b578e63d0769.Live provider dogfood is deferred to the final docs/examples pass (
av-kfik.16) because this PR changes deterministic schema/validation/loading and does not alter provider/grader adapter behavior or run artifact contracts.Related
Related: av-kfik.50
Depends on: #1691