document v5 result sample terminology

christso · christso · commit 74dc6b0d42ac · 2026-07-06T17:27:52.000+10:00
diff --git a/.agents/verification.md b/.agents/verification.md
@@ -139,7 +139,7 @@ bun apps/cli/src/cli.ts eval examples/features/rubric/evals/dataset.eval.yaml --
 
 - the correct grader type ran by checking `scores[].type`
 - scores are calculated as expected
-- `grading.json.assertion_results` reflects the evaluation logic, score, verdict, and evidence
+- `grading.json.assertion_results` reflects the evaluation logic, pass boolean, score, and evidence
 
 4. Update baseline files if output format changes. Baselines live next to eval YAML files as `*.baseline.jsonl`.
 5. `agentv validate` is the cheap schema/config check. For no-live-provider quality validation, run graders against a real reference/oracle target or frozen transcript/replay fixture.
diff --git a/README.md b/README.md
@@ -193,13 +193,13 @@ Run bundle layout:
 ├── 2026-06-30T08-30-00-000Z/     # <run_id> — one committed run bundle
 │   ├── summary.json              # run rollup: metadata, pass rate, counts, cost
 │   ├── fizzbuzz--a1b2c3d4/       # <result_dir> for one test/target row
-│   │   ├── summary.json          # optional per-case rollup across attempts
+│   │   ├── summary.json          # optional per-case rollup across samples
 │   │   ├── test/                 # generated test bundle: frozen inputs for reproducibility
 │   │   │   ├── EVAL.yaml         #   resolved eval spec
 │   │   │   ├── targets.yaml      #   resolved target config
 │   │   │   └── graders/          #   grader files used
 │   │   └── sample-1/             # one materialized sample
-│   │       ├── result.json       # compact attempt manifest
+│   │       ├── result.json       # compact sample manifest
 │   │       ├── grading.json      # pass, score, reason, component_results
 │   │       ├── metrics.json      # tool calls, transcript stats, behavior metrics
 │   │       ├── transcript.json        # normalized agent transcript
diff --git a/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx b/apps/web/src/content/docs/docs/next/evaluation/eval-cases.mdx
@@ -391,7 +391,7 @@ tests:
 
 ### Required Gates
 
-Any grader in `assert` can be marked as `required`. When a required grader fails, the overall test verdict is `fail` regardless of the aggregate score.
+Any grader in `assert` can be marked as `required`. When a required grader fails, the overall case status is `fail` regardless of the aggregate score.
 
 | Value | Behavior |
 |-------|----------|
@@ -412,7 +412,7 @@ assert:
         weight: 1.0
 ```
 
-Required gates are evaluated after all graders run. If any required grader falls below its threshold, the verdict is forced to `fail`.
+Required gates are evaluated after all graders run. If any required grader falls below its threshold, the case status is forced to `fail`.
 
 ### Assert Merge Behavior
 
diff --git a/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx b/apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx
@@ -11,7 +11,7 @@ controls for an evaluation run. The reserved `tags.experiment` key is the
 run/result grouping label, top-level `target` identifies the system under test,
 and fields such as `evaluate_options.repeat`, `threshold`, `timeout_seconds`,
 `evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated
-attempts and gates. Coding-agent testbeds, workdirs, Docker config, repository
+samples and gates. Coding-agent testbeds, workdirs, Docker config, repository
 materialization, setup, and reset policy belong in `environment`. Provider
 environment-variable overrides belong in top-level `env`. Lifecycle hooks belong
 in `extensions`; runner-specific setup belongs in the `target` object, in
diff --git a/apps/web/src/content/docs/docs/next/evaluation/rubrics.mdx b/apps/web/src/content/docs/docs/next/evaluation/rubrics.mdx
@@ -78,7 +78,7 @@ assert:
 Use `min_score` for analytic rubric gating. The only 0–10 values in authored llm-rubric are `score_ranges` bands and grader outputs.
 :::
 
-`weight` controls score contribution. `required` controls the verdict: a required checklist criterion can fail the eval even when the weighted score would otherwise pass.
+`weight` controls score contribution. `required` controls the pass gate: a required checklist criterion can fail the eval even when the weighted score would otherwise pass.
 
 `llm-rubric.value` can also be free-form text or an arbitrary JSON/YAML object. AgentV uses structured scoring when `value` is an array of rubric strings or rubric objects with fields such as `outcome` and `score_ranges`; other objects are passed to the rubric prompt as rubric data.
 
diff --git a/apps/web/src/content/docs/docs/next/graders/assert-set.mdx b/apps/web/src/content/docs/docs/next/graders/assert-set.mdx
@@ -26,7 +26,7 @@ assert:
 
 Child assertions run independently. The parent score is the weighted average of child scores.
 
-When `threshold` is omitted, the assert set passes only when every nonzero-weight child assertion passes. When `threshold` is present, the aggregate score determines the parent pass/fail verdict, so a partially failing set can pass if its weighted score meets the threshold.
+When `threshold` is omitted, the assert set passes only when every nonzero-weight child assertion passes. When `threshold` is present, the aggregate score determines the parent pass/fail result, so a partially failing set can pass if its weighted score meets the threshold.
 
 ## Patterns
 
@@ -112,7 +112,7 @@ An assert set returns nested child scores:
   "name": "release_gate",
   "type": "assert-set",
   "score": 0.85,
-  "verdict": "pass",
+  "pass": true,
   "scores": [
     { "name": "safety", "type": "llm-rubric", "score": 1 },
     { "name": "correctness", "type": "contains", "score": 0.75 }
diff --git a/apps/web/src/content/docs/docs/next/graders/script-graders.mdx b/apps/web/src/content/docs/docs/next/graders/script-graders.mdx
@@ -77,7 +77,7 @@ and `reason`.
 
 For simple pass/fail checks, skip the JSON protocol entirely. The exit code determines the score and stdout becomes the check text:
 
-| Exit code | Score | Verdict |
+| Exit code | Score | Pass |
 |-----------|-------|---------|
 | 0 | 1.0 | pass |
 | non-zero (no stderr) | 0.0 | fail |
diff --git a/apps/web/src/content/docs/docs/next/tools/prepare.mdx b/apps/web/src/content/docs/docs/next/tools/prepare.mdx
@@ -6,9 +6,9 @@ sidebar:
 slug: docs/tools/prepare
 ---
 
-`agentv prepare` materializes one eval case without launching the target provider. Use it when a human, a separate agent process, or another harness should attempt the task in the same workspace state AgentV would have provided immediately before target execution.
+`agentv prepare` materializes one eval case without launching the target provider. Use it when a human, a separate agent process, or another harness should perform the task in the same workspace state AgentV would have provided immediately before target execution.
 
-This is the manual-attempt workflow:
+This is the manual sample workflow:
 
 ```bash
 agentv prepare evals/foo.eval.yaml --test-id case-1 --target codex-cli --out /tmp/agentv-case-1
@@ -49,7 +49,7 @@ agentv grade evals/foo.eval.yaml \
 
 ## Add Trace or Session Evidence
 
-Trace-aware graders can use a local trace/session artifact from the manual attempt:
+Trace-aware graders can use a local trace/session artifact from the manual sample:
 
 ```bash
 agentv grade evals/foo.eval.yaml \
diff --git a/examples/features/trials/README.md b/examples/features/trials/README.md
@@ -2,7 +2,7 @@
 
 This example keeps the runnable contract in one eval file. Top-level `target`
 selects the system under test and `evaluate_options.repeat` configures repeated
-attempts.
+samples.
 
 ## Files
 
@@ -14,7 +14,7 @@ attempts.
 bun agentv eval examples/features/trials/evals/suite.yaml
 ```
 
-Edit `evaluate_options.repeat.count` to change how many attempts AgentV makes
+Edit `evaluate_options.repeat.count` to change how many samples AgentV records
 for each case:
 
 ```yaml
diff --git a/examples/features/trials/evals/suite.yaml b/examples/features/trials/evals/suite.yaml
@@ -1,5 +1,5 @@
 name: trials
-description: Repeat runs example with 2 attempts configured inline
+description: Repeat runs example with 2 samples configured inline
 target: llm
 evaluate_options:
   repeat:
diff --git a/plugins/agentv-self/skills/image-compress-and-docs/SKILL.md b/plugins/agentv-self/skills/image-compress-and-docs/SKILL.md
@@ -50,7 +50,7 @@ kill $(lsof -ti:14800) 2>/dev/null
 
 Synthetic JSONL record format:
 ```json
-{"test_id": "my-test", "score": 0.95, "target": "claude-sonnet", "experiment": "default", "timestamp": "2026-04-08T09:15:44.003Z", "execution_status": "success", "suite": "my-suite", "duration_ms": 3500, "token_usage": {"input_tokens": 1200, "output_tokens": 400}, "scores": [{"type": "llm-rubric", "score": 0.95, "verdict": "pass"}], "grading_path": "my-test--demo/sample-1/grading.json", "metrics_path": "my-test--demo/sample-1/metrics.json", "error": null}
+{"test_id": "my-test", "score": 0.95, "target": "claude-sonnet", "experiment": "default", "timestamp": "2026-04-08T09:15:44.003Z", "status": "pass", "suite": "my-suite", "duration_ms": 3500, "token_usage": {"input_tokens": 1200, "output_tokens": 400}, "scores": [{"type": "llm-rubric", "score": 0.95, "pass": true}], "grading_path": "my-test--demo/sample-1/grading.json", "metrics_path": "my-test--demo/sample-1/metrics.json", "error": null}
 ```
 
 ## Step 2 — Optimize
diff --git a/skills-data/agentv-bench/agents/analyzer.md b/skills-data/agentv-bench/agents/analyzer.md
@@ -22,7 +22,7 @@ You are an eval-quality analyst for AgentV. Your job is to read JSONL evaluation
 
 Read every line of the JSONL results file. Each line is a JSON object with:
 - `test_id`, `suite`, `score`, `assertions`, `reasoning`, `target`
-- `scores` (optional): Array of per-grader breakdowns with `name`, `type`, `score`, `weight`, `verdict`, `assertions`, `reasoning`
+- `scores` (optional): Array of per-grader breakdowns with `name`, `type`, `score`, `weight`, `pass`, `assertions`, `reasoning`
 
 If `eval-path` is provided, also read the EVAL.yaml to understand grader configurations.
 
diff --git a/skills-data/agentv-bench/references/eval-yaml-spec.md b/skills-data/agentv-bench/references/eval-yaml-spec.md
@@ -285,7 +285,7 @@ Each line in the results JSONL file is an `EvaluationResult` object. In JSONL, f
 - `score` (number, 0.0-1.0)
 - `assertions` (array of `{text, passed, evidence?}`)
 - `weight` (number, optional)
-- `verdict` (string: `pass` | `fail` | `skip`)
+- `pass` (boolean)
 - `details` (object, optional — structured data from script graders)
 - `reasoning` (string, optional)
 
diff --git a/skills-data/agentv-bench/references/schemas.md b/skills-data/agentv-bench/references/schemas.md
@@ -198,7 +198,7 @@ Output from the executor agent. Located at `<case-dir>/run-N/metrics.json`.
 
 ## timing.json
 
-Wall clock timing for a run attempt. Located at `<case-dir>/run-N/timing.json`.
+Wall clock timing for a run sample. Located at `<case-dir>/sample-N/timing.json`.
 
 **How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.
 
diff --git a/skills-data/agentv-eval-writer/SKILL.md b/skills-data/agentv-eval-writer/SKILL.md
@@ -163,7 +163,7 @@ Use top-level `prompts` plus `tests[].vars` for the Promptfoo-compatible canonic
 input shape. Shared data defaults belong in `default_test.vars`; per-test
 `vars` override those defaults by key. AgentV renders every prompt with each
 test's merged vars, then expands the run across prompts, targets, tests, and
-repeat attempts.
+repeat samples.
 
 ```yaml
 description: Prompt matrix example
@@ -456,7 +456,7 @@ assert:
         weight: 5.0
 ```
 
-If a required grader scores below its threshold, the overall verdict is forced to `fail`.
+If a required grader scores below its threshold, the overall case status is forced to `fail`.
 
 ## Environment Setup/Teardown
 
@@ -586,7 +586,7 @@ Variables: `{{criteria}}`, `{{input}}`, `{{expected_output}}`, `{{output}}`, `{{
       type: llm-rubric
       weight: 0.7
 ```
-Use `assert-set` for Promptfoo-aligned assertion grouping. Without `threshold`, the set passes only when every nonzero-weight child assertion passes. With `threshold`, the weighted aggregate score determines the set verdict. Parent `config` is inherited by children, and child `config` keys override parent keys. Do not use `type: composite`; AgentV rejects it.
+Use `assert-set` for Promptfoo-aligned assertion grouping. Without `threshold`, the set passes only when every nonzero-weight child assertion passes. With `threshold`, the weighted aggregate score determines the set pass/fail result. Parent `config` is inherited by children, and child `config` keys override parent keys. Do not use `type: composite`; AgentV rejects it.
 
 ### Skill And Trajectory Assertions
 ```yaml
diff --git a/skills-data/agentv-eval-writer/references/rubric-evaluator.md b/skills-data/agentv-eval-writer/references/rubric-evaluator.md
@@ -18,7 +18,7 @@ Rubrics are defined as `assert` entries with plain strings or `type: llm-rubric`
 | `outcome` | string | required* | Criterion being evaluated (*optional if `score_ranges` used) |
 | `operator` | string | - | Optional intent: `correctness` or `contradiction` |
 | `weight` | number | 1.0 | Relative importance |
-| `required` | boolean | true | Failing forces verdict to 'fail' (checklist mode) |
+| `required` | boolean | true | Failing forces the case status to `fail` (checklist mode) |
 | `min_score` | number | - | Minimum score (0–1) to pass this criterion |
 | `score_ranges` | map or array | - | Score range definitions for analytic scoring |
 
