Focused examples for specific AgentV capabilities. Find your use case below, then open the linked directory for a runnable example.
| Example | Description |
|---|---|
| basic | Core schema: input, expected output, file references, multi-turn |
| basic-jsonl | Load test cases from an external JSONL file |
| default-graders | Apply the same assertions to every test without repeating them |
| readme-quickstart | Root README quickstart with default_test.options.rubric_prompt |
| Example | Description |
|---|---|
| rubric | Boolean rubric criteria — pass/fail each with a script grader or LLM check |
| weighted-graders | Multiple named llm-rubric assertions with per-grader weights |
| assert-set | Safety gate and weighted assertion groups |
| threshold-grader | Pass a test if a configurable percentage of sub-graders pass |
| multi-turn-conversation | Grade a multi-turn conversation with per-turn score breakdowns |
| file-transforms | Convert ContentFile outputs with default_test.options.transform before llm-rubric runs |
| Example | Description |
|---|---|
| assert | Core built-ins: contains, regex, is-json, equals, starts-with, ends-with |
| assert-extended | Extended variants: contains-any, icontains, icontains-all, regex flags |
| deterministic-graders | Full showcase of all deterministic assertion types |
| nlp-metrics | ROUGE, BLEU, cosine/Jaccard similarity, Levenshtein as script graders |
| Example | Description |
|---|---|
| script-grader-sdk | TypeScript script graders using defineScriptGrader() from @agentv/sdk |
| script-grader-with-llm-calls | script graders that make LLM calls via a target proxy |
| eval-assert-demo | script graders runnable both in a suite and individually via agentv eval assert |
| functional-grading | Install dependencies, compile, and run tests against agent-generated code |
| Example | Description |
|---|---|
| tool-trajectory-simple | Validate expected tool call sequences |
| tool-trajectory-advanced | Tool trajectory checks with expected_output and per-call assertions |
| latency-assertions | Assert max_duration_ms per tool call to catch performance regressions |
| tool-evaluation-plugins | F1 precision/recall scoring for tool-call accuracy |
| trace-evaluation | Inspect agent internals: LLM call counts, tool executions, step durations |
| Example | Description |
|---|---|
| copilot-log-eval | Replay Copilot CLI session transcripts from disk — no LLM API key needed |
| trace-analysis | Inspect eval results with agentv trace — summaries, trees, latency percentiles |
| agent-skills-evals | Evaluate Claude Code skills with EVAL.yaml or Agent Skills evals.json |
| Example | Description |
|---|---|
| external-datasets | Load test cases from YAML/JSONL files using file:// references and globs |
| input-files-shorthand | Attach files to every test using a compact shorthand |
| suite-level-input | Prepend a shared system prompt to every test in the suite |
| suite-level-input-files | Share file attachments across every test in the suite |
| env-interpolation | Inject environment variables into eval config with {{ env.VAR }} |
| test-vars-templating | Render prompt templates and chat prompt files from default_test.vars and per-test vars |
| Example | Description |
|---|---|
| benchmark-tooling | N-way benchmarking with agentv results compare over completed runs |
| trials | Configure repeated attempts with evaluate_options.repeat |
| trial-output-consistency | Measure output consistency across trials using pairwise cosine similarity |
| compare | Compare a run against a stored baseline |
| Example | Description |
|---|---|
| execution-metrics | Assert on token count, cost, and latency per test |
| latency-assertions | Per-tool-call latency constraints (also listed under tool evaluation) |
| Example | Description |
|---|---|
| workspace-setup-script | Multi-step setup with a beforeAll lifecycle extension |
| workspace-multi-repo | Multi-repo workspace using a VS Code .code-workspace file |
| workspace-shared-config | Define a workspace.yaml once and reference it across eval files |
| repo-lifecycle | Clone a git repo into the workspace and target the agent at it |
| file-changes | Capture workspace file changes made by the agent across test runs |
| file-changes-graders | Grade file diffs with rubrics and LLM graders |
| local-cli | Define and invoke local CLI targets |
| batch-cli | Run bulk evaluations from the CLI |
| Example | Description |
|---|---|
| document-extraction | Evaluate structured data extracted from documents |
| Example | Description |
|---|---|
| sdk-custom-assertion | Custom assertion types using defineAssertion() |
| sdk-eval-authoring | YAML-aligned .eval.ts authoring using defineEval() |
| sdk-programmatic-api | Programmatic evaluation using evaluate() |
| sdk-config-file | Typed configuration with defineConfig() |
| prompt-template-sdk | Custom LLM grader prompts using definePromptTemplate() |
| Example | Use Case |
|---|---|
| agent-skills-evals | Offline evaluation of Claude Code skills |
| assert | Deterministic assertions |
| assert-extended | Deterministic assertions |
| basic | Getting started |
| basic-jsonl | Getting started |
| batch-cli | Workspace & targets |
| benchmark-tooling | Benchmarking |
| script-grader-sdk | Custom graders |
| script-grader-with-llm-calls | Custom graders |
| compare | Benchmarking |
| assert-set | LLM grading |
| copilot-log-eval | Offline evaluation |
| default-graders | Getting started |
| deterministic-graders | Deterministic assertions |
| document-extraction | Document extraction |
| env-interpolation | Dataset & input |
| eval-assert-demo | Custom graders |
| execution-metrics | Cost, latency & tokens |
| external-datasets | Dataset & input |
| file-changes | Workspace & targets |
| file-changes-graders | Workspace & targets |
| functional-grading | Custom graders |
| input-files-shorthand | Dataset & input |
| latency-assertions | Tool & agent evaluation |
| local-cli | Workspace & targets |
| multi-turn-conversation | LLM grading |
| nlp-metrics | Deterministic assertions |
| file-transforms | LLM grading |
| prompt-template-sdk | TypeScript SDK |
| repo-lifecycle | Workspace & targets |
| rubric | LLM grading |
| sdk-config-file | TypeScript SDK |
| sdk-custom-assertion | TypeScript SDK |
| sdk-eval-authoring | TypeScript SDK |
| sdk-programmatic-api | TypeScript SDK |
| suite-level-input | Dataset & input |
| suite-level-input-files | Dataset & input |
| test-vars-templating | Dataset & prompt templates |
| threshold-grader | LLM grading |
| tool-evaluation-plugins | Tool & agent evaluation |
| tool-trajectory-advanced | Tool & agent evaluation |
| tool-trajectory-simple | Tool & agent evaluation |
| trace-analysis | Offline evaluation |
| trace-evaluation | Tool & agent evaluation |
| trial-output-consistency | Benchmarking |
| trials | Benchmarking |
| weighted-graders | LLM grading |
| workspace-multi-repo | Workspace & targets |
| workspace-setup-script | Workspace & targets |
| workspace-shared-config | Workspace & targets |