Align repeat authoring with Promptfoo numeric repeat (#1724)

* fix(eval): require numeric repeat authoring

* docs(eval): document numeric repeat authoring

* chore(eval): sync numeric repeat schema

Author: christso

.agents/verification.md

@@ -153,7 +153,7 @@ Use live dogfood before marking PRs ready when they affect eval execution, exper
 - Prefer the smallest realistic eval: one or two cases, bounded timeouts, and `workers: 1` for heavyweight agent providers.
 - For artifact/result contract changes, prefer letting AgentV choose the canonical run directory and capture the printed `Artifact workspace written to:` and `Results written to:` paths for evidence. Do not precompute `--output` unless the test specifically needs a fixed path.
 - For native experiment changes, run through `agentv eval run ... --experiment <experiment.yaml|ts>` so resolution, setup, scripts, provider selection, run knobs, and artifact metadata are exercised together.
-- For repeat-run changes, use `evaluate_options.repeat.count >= 2` when validating repeated executions. Inspect `.internal/index.jsonl`, root `summary.json`, and the repeated case folder. Use `repeat` for authored configuration and `sample_index`/`retry_index` for produced executions. The repeated case folder should carry aggregate `summary.json`; sample-specific outputs, transcripts, grading, and metrics live under `sample-N/`. Each `sample-N/` folder should contain `result.json`, `grading.json`, `metrics.json`, `transcript.json`, `transcript-raw.jsonl`, and `outputs/answer.md` when answer output is available. `result.json` should point at `./grading.json`, `./metrics.json`, `./transcript.json`, and `./transcript-raw.jsonl` through the corresponding path fields.
+- For repeat-run changes, use `evaluate_options.repeat: 2` or higher when validating repeated executions. Inspect `.internal/index.jsonl`, root `summary.json`, and the repeated case folder. Use numeric `repeat` for authored configuration and `sample_index`/`retry_index` for produced executions. The repeated case folder should carry aggregate `summary.json`; sample-specific outputs, transcripts, grading, and metrics live under `sample-N/`. Each `sample-N/` folder should contain `result.json`, `grading.json`, `metrics.json`, `transcript.json`, `transcript-raw.jsonl`, and `outputs/answer.md` when answer output is available. `result.json` should point at `./grading.json`, `./metrics.json`, `./transcript.json`, and `./transcript-raw.jsonl` through the corresponding path fields.
 - For local OpenAI-compatible grading through the OAuth proxy, use `base_url: http://127.0.0.1:10531/v1`, but still route `api_key` and `model` through environment references such as `{{ env.LOCAL_OPENAI_PROXY_API_KEY }}` and `{{ env.LOCAL_OPENAI_PROXY_MODEL }}`. Literal secrets and literal model values are intentionally rejected by provider validation unless a resolver explicitly allows them.
 - For `codex`/Codex SDK live dogfood through the same local proxy, configure the agent provider with `id`, `label`, `runtime`, and `config`, and put backend settings such as `base_url`, `api_key`, `model`, and `api_format` under `config`. Configure the reusable grader in the same `providers` catalog, then select it with `defaults.grader` or assertion-level `provider`; do not put a grader selector on the system-under-test provider. A minimal run should use `bun apps/cli/src/cli.ts eval run <eval.yaml> --providers <providers.yaml> --provider <codex-label> --workers 1`.
 - If the local proxy returns `401 token_expired`, the blocker is stale Codex OAuth, not AgentV target configuration. Refresh from a trusted local terminal with `codex logout`, `codex login --device-auth`, then restart `openai-oauth` and rerun the same eval command.

CONCEPTS.md

@@ -102,15 +102,15 @@ env:
 
 **Repeat run** — A configured request to execute the same eval case and target more than once in the same run bundle. Repeat runs measure stochastic reliability, verifier stability, and drift; they are not the default CI path.
 
-**Attempt** — One concrete execution inside a repeat run. Attempts keep their own score, status, metrics, trace, transcript, logs, and artifacts so aggregate results never hide individual evidence.
+**Repeat sample** — One concrete execution inside a repeat run. Samples keep their own score, status, metrics, trace, transcript, logs, and artifacts so aggregate results never hide individual evidence.
 
-**Pass rate** — Assertion or expectation pass rate inside a grading result: passed assertions or expectations divided by total assertions or expectations. AgentV does not use `pass_rate` for repeat-attempt success frequency.
+**Pass rate** — Assertion or expectation pass rate inside a grading result: passed assertions or expectations divided by total assertions or expectations. AgentV does not use `pass_rate` for repeat-sample success frequency.
 
-**Attempt success rate** — Repeat-run reliability metric equal to successful counted attempts divided by counted attempts. This is distinct from `pass_rate`, which is reserved for assertion or expectation pass rate within a grading result.
+**Sample success rate** — Repeat-run reliability metric equal to successful counted samples divided by counted samples. This is distinct from `pass_rate`, which is reserved for assertion or expectation pass rate within a grading result.
 
-**Gate policy** — The explicit rule that decides whether repeated attempts pass CI, such as `all_attempts_successful`, `any_attempt_successful`, `attempt_success_rate_at_least`, or `mean_pass_rate_at_least`. Without a repeat-run gate policy, AgentV preserves the normal single-run gate behavior and treats repeat statistics as report data.
+**Aggregate gate policy** — Future fatal post-run policy surface for deciding whether repeated samples pass CI. Until AgentV deliberately designs that surface, repeat configuration only sets the numeric sample count and users can parse run bundles externally for aggregate gating.
 
-**Flaky eval outcome** — A repeat-run aggregate whose attempts disagree, or whose failure classification points at verifier, infrastructure, or timeout instability rather than a stable model-quality failure.
+**Flaky eval outcome** — A repeat-run aggregate whose samples disagree, or whose failure classification points at verifier, infrastructure, or timeout instability rather than a stable model-quality failure.
 
 ## Release Channels
 

README.md

@@ -19,7 +19,7 @@ Test AI providers on real repo tasks and measure what actually works.
 - **Environment / fixtures / graders** are task-owned context: host or Docker setup, repos, setup scripts, files, fixtures, deterministic checks, and LLM grading prompts.
 - **Provider** is the configured system under test: an agent, model provider, gateway, replay provider, CLI wrapper, transcript provider, or future app/service wrapper. Each provider entry uses `id` for the backend/spec and optional `label` for the stable AgentV selection and result identity.
 - **Tags** are run/result grouping labels. `tags.experiment` is the default experiment namespace, such as `with-skills` or `without-skills`; keep suite/category and provider/model names out of that tag.
-- **Evaluate options** configure eval run behavior such as `max_concurrency`, repeat policy, and budgets.
+- **Evaluate options** configure eval run behavior such as `max_concurrency`, repeat sample count, and budgets.
 - **Default test** configures inherited per-test defaults such as score `threshold`.
 - **Run** is one concrete execution of a tagged eval against a resolved provider that writes portable artifacts for readers such as Dashboard, compare, and trend.
 
@@ -133,9 +133,7 @@ providers:
       api_key: "{{ env.LOCAL_OPENAI_PROXY_API_KEY }}"
       model: gpt-5.4-mini
 evaluate_options:
-  repeat:
-    count: 2
-    strategy: pass_any
+  repeat: 2
 
 default_test:
   threshold: 0.85
@@ -253,11 +251,7 @@ const config: EvalConfig = {
     extends: 'copilot-sdk',
     model: 'claude-sonnet-4.6',
   },
-  repeat: {
-    count: 3,
-    strategy: 'pass_any',
-    earlyExit: false,
-  },
+  repeat: 3,
   threshold: 0.8,
   prompts: ['{{ input }}'],
   environment: {

apps/cli/src/commands/eval/run-eval.ts

@@ -1049,10 +1049,6 @@ function buildExperimentTrialsConfig(experiment: ExperimentConfig): TrialsConfig
     return {
       count: experiment.repeat.count,
       strategy: experiment.repeat.strategy,
-      ...(experiment.repeat.costLimitUsd !== undefined && {
-        costLimitUsd: experiment.repeat.costLimitUsd,
-      }),
-      ...(experiment.repeat.earlyExit !== undefined && { earlyExit: experiment.repeat.earlyExit }),
     };
   }
   return undefined;
@@ -1074,8 +1070,6 @@ function buildRunOverrideTrialsConfig(run: EvalRunOverride | undefined): TrialsC
   return {
     count: repeat.count,
     strategy: repeat.strategy,
-    ...(repeat.costLimitUsd !== undefined && { costLimitUsd: repeat.costLimitUsd }),
-    ...(repeat.earlyExit !== undefined && { earlyExit: repeat.earlyExit }),
   };
 }
 

apps/cli/test/eval.integration.test.ts

@@ -725,10 +725,7 @@ describe('agentv eval CLI', () => {
           'threshold: 0.8',
           'evaluate_options:',
           '  budget_usd: 3',
-          '  repeat:',
-          '    count: 2',
-          '    strategy: pass_any',
-          '    early_exit: true',
+          '  repeat: 2',
           'tests:',
           '  - include: sample.test.yaml',
           '    type: suite',
@@ -737,10 +734,7 @@ describe('agentv eval CLI', () => {
           '      threshold: 1.0',
           '      timeout_seconds: 5',
           '      budget_usd: 0.75',
-          '      repeat:',
-          '        count: 3',
-          '        strategy: pass_all',
-          '        early_exit: true',
+          '      repeat: 3',
           '',
         ].join('\n'),
         'utf8',
@@ -765,8 +759,7 @@ describe('agentv eval CLI', () => {
         threshold: 1,
         trials: {
           count: 3,
-          strategy: 'pass_all',
-          earlyExit: true,
+          strategy: 'pass_any',
         },
       });
 

apps/cli/test/fixtures/mock-run-evaluation.ts

@@ -24,8 +24,6 @@ interface RunEvaluationOptionsLike {
   readonly trials?: {
     readonly count: number;
     readonly strategy: string;
-    readonly costLimitUsd?: number;
-    readonly earlyExit?: boolean;
   };
   readonly threshold?: number;
   readonly budgetUsd?: number;

apps/web/src/content/docs/docs/next/evaluation/eval-files.mdx

@@ -119,9 +119,7 @@ providers:
   - codex-gpt5
 evaluate_options:
   max_concurrency: 3
-  repeat:
-    count: 2
-    strategy: pass_any
+  repeat: 2
 
 tests:
   - file://../evals/cases/refund-smoke.cases.yaml
@@ -179,7 +177,7 @@ tests:
 | `tags` | Optional metadata map. Use `tags.experiment` as the run/result grouping label. |
 | `prompts` | Optional top-level prompt matrix. Entries can be strings, chat message arrays, files, or generated prompt functions rendered with `tests[].vars` and `default_test.vars`. |
 | `providers` | System-under-test matrix. Entries can be Promptfoo-shaped provider strings, complete package provider strings such as `package:@agentv/promptfoo-providers:CodexCliProvider` or `package:@agentv/promptfoo-providers/codex-cli:Provider`, provider option objects, or provider maps; `id` names the backend/spec and `label` is the stable AgentV identity. |
-| `evaluate_options.repeat` | Optional repeat policy as a positive integer shorthand or object with `count`, `strategy`, `early_exit`, and `cost_limit_usd` |
+| `evaluate_options.repeat` | Optional positive integer sample count for repeat runs |
 | `evaluate_options` | Optional evaluation runtime options such as `budget_usd`, `repeat`, and `max_concurrency` |
 | `timeout_seconds` | Optional per-case timeout |
 | `threshold` | Optional suite quality threshold |

apps/web/src/content/docs/docs/next/evaluation/experiments.mdx

@@ -27,9 +27,7 @@ providers:
       reasoning_effort: high
 timeout_seconds: 720
 evaluate_options:
-  repeat:
-    count: 4
-    strategy: pass_any
+  repeat: 4
   budget_usd: 2.00
   max_concurrency: 3
 
@@ -120,9 +118,7 @@ providers:
   - agent
 threshold: 0.8
 evaluate_options:
-  repeat:
-    count: 3
-    strategy: pass_any
+  repeat: 3
 timeout_seconds: 300
 tags:
   area: agentic
@@ -161,7 +157,7 @@ target-specific runner state.
 | Configure an agent runner or provider variant | `providers` entry or `providers.yaml` |
 | Choose the provider | top-level `providers` or CLI `--provider` |
 | Override the provider's default model | `providers[].config.model` |
-| Configure repeat policy, budget, concurrency, timeout, threshold | `evaluate_options.repeat`, `evaluate_options.budget_usd`, `evaluate_options.max_concurrency`, `timeout_seconds`, `threshold` |
+| Configure repeat sample count, budget, concurrency, timeout, threshold | `evaluate_options.repeat`, `evaluate_options.budget_usd`, `evaluate_options.max_concurrency`, `timeout_seconds`, `threshold` |
 | Bind an existing local workspace directory | `--workspace-path` or `.agentv/config.local.yaml` |
 
 ```yaml
@@ -176,9 +172,7 @@ providers:
       before_each:
         command: ["sh", "-c", "cp -R skills \"{{workspace_path}}/.codex/skills\""]
 evaluate_options:
-  repeat:
-    count: 3
-    strategy: pass_any
+  repeat: 3
 ```
 
 Existing local workspace paths are machine-local bindings: pass
@@ -197,25 +191,10 @@ evaluate_options:
   repeat: 3
 ```
 
-Use object form when you need richer AgentV behavior:
-
-```yaml
-evaluate_options:
-  repeat:
-    count: 3
-    strategy: pass_any
-    early_exit: true
-    cost_limit_usd: 1.00
-```
-
-`evaluate_options.repeat.strategy` controls sample aggregation. `pass_any`
-treats the case as successful when any completed sample passes; `pass_all`
-requires every completed sample to pass. `mean` and `confidence_interval`
-aggregate scores where supported today. `evaluate_options.repeat.early_exit` is
-only a scheduling and cost optimization: `pass_any` may stop at the first pass,
-and `pass_all` may stop at the first fail. Leave it unset or `false` when you
-want complete variance data. Per-case `tests[].options.repeat` overrides the
-global repeat count or object for that case.
+Repeat authoring only sets the sample count. A fatal post-run aggregate policy
+surface is future work; until that exists, parse the run bundle externally when
+CI needs aggregate gating across samples. Per-case `tests[].options.repeat`
+overrides the global repeat count for that case.
 
 ## Result Layout
 

apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-files.mdx

@@ -13,7 +13,7 @@ controls for an evaluation run. The reserved `tags.experiment` key is the
 run/result grouping label, top-level `target` identifies the system under test,
 and fields such as `evaluate_options.repeat`, `threshold`, `timeout_seconds`,
 `evaluate_options.budget_usd`, and `evaluate_options.max_concurrency` control repeated
-attempts and gates. Coding-agent testbeds, workdirs, Docker config, repository
+samples and gates. Coding-agent testbeds, workdirs, Docker config, repository
 materialization, setup, and reset policy belong in `environment`. Provider
 environment-variable overrides belong in top-level `env`. Lifecycle hooks belong
 in `extensions`; runner-specific setup belongs in the `target` object, in
@@ -85,9 +85,7 @@ name: refunds-codex
 target: codex-gpt5
 evaluate_options:
   max_concurrency: 3
-  repeat:
-    count: 2
-    strategy: pass_any
+  repeat: 2
 
 tests:
   - file://../evals/cases/refund-smoke.cases.yaml
@@ -133,7 +131,7 @@ tests:
 | `tags` | Optional promptfoo-style metadata map. Use `tags.experiment` as the run/result grouping label. |
 | `prompts` | Optional top-level prompt matrix. Entries can be strings, chat message arrays, files, or generated prompt functions. |
 | `targets` | Optional target matrix. Entries reference target ids or inline target objects. |
-| `evaluate_options.repeat` | Optional repeat policy as a positive integer shorthand or object with `count`, `strategy`, `early_exit`, and `cost_limit_usd` |
+| `evaluate_options.repeat` | Optional positive integer sample count for repeat runs |
 | `timeout_seconds` | Optional per-case timeout |
 | `evaluate_options` | Optional evaluation runtime options such as `budget_usd`, `repeat`, and `max_concurrency` |
 | `threshold` | Optional suite quality threshold |

docs/adr/0006-separate-experiments-from-eval-definitions.md

@@ -21,6 +21,9 @@ Superseded for the current eval authoring contract by
 [ADR 0016](0016-promptfoo-superset-eval-authoring-contract.md): `tests`,
 `prompts`/`vars`, `assert`, direct `input`, `target`/`targets`, and
 `evaluate_options` are the current promptfoo-aligned authoring surface.
+The repeat object examples in this ADR are also superseded by Bead `av-s96i`:
+current public authoring uses Promptfoo-style numeric `evaluate_options.repeat`
+only.
 
 The "Suite And Test Import Surface" section below is historical. Its
 `imports.suites` / `imports.tests` design is superseded by av-kfik.43 and the
@@ -85,10 +88,7 @@ target:
 timeout_seconds: 900
 threshold: 0.8
 evaluate_options:
-  repeat:
-    count: 3
-    strategy: pass_any
-    early_exit: false
+  repeat: 3
   budget_usd: 2.00
 
 tests:
@@ -122,7 +122,7 @@ The old experiment runtime fields are ported into the parent eval file:
 
 - target or target matrix
 - thresholds
-- repeated run count through `evaluate_options.repeat.count`
+- repeated sample count through numeric `evaluate_options.repeat`
 - timeout
 - budget
 - other run-time controls that do not define the task itself
@@ -166,8 +166,8 @@ must stay with the lifecycle surface that actually owns that work:
   discovery files, provider-specific config, and target-specific harness setup
   belong here.
 - Top-level `target` selects the system under test. Top-level `policy` selects
-  runtime and gating controls: repeat strategy, threshold, timeout, budget, and
-  early-exit behavior. Workspace lifetime stays under
+  runtime and gating controls such as repeat count, threshold, timeout, and
+  budget. Workspace lifetime stays under
   `workspace.scope`, and Docker/container binding stays under
   `workspace.docker`.