feat(config)!: hard-deprecate the graders: config block — a grader is just a target

Peer-framework research this session (Margin-Lab/evals, Harbor) found neither
maintains a separate grader/judge schema type distinct from their target/agent
config — Margin-Lab's own docs say it outright: "there is no separate grader
abstraction." AgentV's own design principles already prefer config-level
grader selection through `defaults.grader` over target-level `grader_target`,
and the live `.agentv/targets.yaml` execution path already resolves
`grader_target`/`defaults.grader` against the same target-id namespace. The
only place a separate "grader" entity still existed was the newer, dead
`graders:` top-level config-graph schema (`NormalizedGraderConfig`) — zero
implementation depth, nothing ever turned it into a Provider. A grader can
also need grading itself (oracle/calibration checks), which only works for
free if it's addressable the same way as any other target.

- config-graph.ts: remove `NormalizedGraderConfig`/`parseGraders`/the
  `graders` field from `ComposableConfigGraph`; authoring a `graders:` block
  now throws a clear error pointing at folding entries into `targets`.
  `defaults.grader` now validates against the same `graph.targets` pool as
  `defaults.target`, instead of the removed `graph.graders`.
- eval-file.schema.ts: same hard-rejection for the Zod schema backing the
  published eval.schema.json (IDE-autocomplete surface), matching the existing
  pattern for other removed fields (`eval_cases`, `providerPromptMap`).
- config-loader.ts: drop the `graders` pass-through.
- Updated the one place in docs, README.md, and the readme-quickstart example
  that authored a `graders:` block, folding each grader into `targets:`.
  readme-quickstart's targets.yaml/graders.yaml also still had the legacy
  `${{ VAR }}` env syntax (pre-existing, unrelated breakage) — fixed to
  `{{ env.VAR }}` in the same pass since the file needed touching anyway.
- Added a CONCEPTS.md entry disambiguating grader-selection (a target) from
  grader-method (an assertion type) and grader-prompt (criteria/rubric).

Verified: full workspace typecheck and lint clean; 935 tests across the
touched loader/validation/orchestrator suites pass; all 109 example eval
files validate; live CLI check confirms the error fires end-to-end and the
folded-into-targets shape validates cleanly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: christso

CONCEPTS.md

@@ -10,6 +10,8 @@ Shared domain vocabulary for this project — entities, named processes, and sta
 
 **Target runtime** — The placement/transport mode for invoking a target provider, such as host execution, sandbox/container placement, CLI subprocess, app-server protocol, RPC, or SDK child runner. Runtime describes how the selected agent is invoked. Advanced home/env/profile-style overlays are provider or runtime configuration details, not the authored testbed recipe. Runtime is separate from the environment that prepares files, services, and cwd.
 
+**Grader (selection)** — Not a distinct entity type. A grader is a regular target, listed under `targets` like any other and selected for a grading role via `defaults.grader`, CLI `--grader-target`, or an assertion-level target override. Authoring a separate `graders:` list is a hard error: move each entry into `targets`. This also means a grader can itself be the target of an ordinary eval (oracle/calibration checks), which a separate schema would block. Do not confuse this with **grader (method)** — an assertion `type` such as `llm-rubric`, `code-grader`, or `g-eval` that names *how* scoring works, independent of *which* target performs it; or with a rubric's `criteria`/`value`, which is the grading *prompt*. AgentV once called grader-selection "judge" (`judge_target`) before a deliberate, full rename to "grader" — do not reintroduce "judge" without a stronger reason than surface-level clarity, since peer frameworks (Margin-Lab/evals, Harbor) don't treat judge/grader as a first-class schema concept either.
+
 **Provider runtime boundary** — The process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; SDK providers should run through an AgentV-owned child-runner boundary when runtime teardown can threaten run artifact finalization. This boundary does not own repository/testbed setup by default.
 
 ## Evaluation Model

README.md

@@ -44,9 +44,9 @@ targets:
       api_key: "{{ env.LOCAL_OPENAI_PROXY_API_KEY }}"
       model: "{{ env.LOCAL_OPENAI_PROXY_MODEL }}"
 
-graders:
   - id: local-openai-grader
     provider: openai
+    runtime: host
     config:
       api_format: chat
       base_url: "{{ env.LOCAL_OPENAI_PROXY_BASE_URL }}"

apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx

@@ -474,7 +474,9 @@ default.
 ### YAML config (`config.yaml` plus optional `config.local.yaml`)
 
 Project config and eval YAML share the same eval-definition graph for targets,
-graders, tests, defaults, and supported execution defaults. Small projects can keep that
+tests, defaults, and supported execution defaults. A grader is not a separate
+part of that graph — it is a target, listed under `targets` like any other,
+selected for a grading role via `defaults.grader`. Small projects can keep the
 graph inline:
 
 ```yaml
@@ -486,9 +488,9 @@ targets:
       command: ["codex", "app-server"]
       model: gpt-5-codex
 
-graders:
   - id: openai-grader
     provider: openai
+    runtime: host
     config:
       model: gpt-5-mini
 
@@ -511,7 +513,6 @@ directly:
 
 ```yaml
 targets: file://targets.yaml
-graders: file://graders.yaml
 tests: file://tests.yaml
 defaults: file://defaults.yaml
 ```
@@ -523,6 +524,12 @@ defaults: file://defaults.yaml
   runtime: host
   config:
     command: ["codex", "app-server"]
+
+- id: openai-grader
+  provider: openai
+  runtime: host
+  config:
+    model: gpt-5-mini
 ```
 
 ```yaml

apps/web/src/content/docs/docs/next/targets/cli-provider.mdx

@@ -21,9 +21,9 @@ targets:
     config:
       command: python agent.py --prompt {PROMPT} --out {OUTPUT_FILE}
 
-graders:
   - id: azure-grader
     provider: azure
+    runtime: host
     config:
       endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}"
       api_key: "{{ env.AZURE_OPENAI_API_KEY }}"
@@ -133,9 +133,9 @@ targets:
     config:
       command: cp fixtures/{EVAL_ID}.expected.txt {OUTPUT_FILE}
 
-graders:
   - id: azure-grader
     provider: azure
+    runtime: host
     config:
       endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}"
       api_key: "{{ env.AZURE_OPENAI_API_KEY }}"

apps/web/src/content/docs/docs/next/targets/coding-agents.mdx

@@ -15,7 +15,9 @@ settings live under `config`.
 Use `defaults.grader`, CLI `--grader` / `--grader-target`, or an
 evaluator-specific target override for LLM-based grading. Grader selection is
 separate from the coding-agent target, so target definitions do not carry a
-grader field.
+grader field. A grader is not a distinct kind of entity either — it is a
+regular target, listed under `targets` like any other, selected for a grading
+role.
 
 ```yaml
 targets:
@@ -27,9 +29,9 @@ targets:
       model: gpt-5-codex
       reasoning_effort: high
 
-graders:
   - id: openai-grader
     provider: openai
+    runtime: host
     config:
       model: gpt-5-mini
 

apps/web/src/content/docs/docs/next/targets/configuration.mdx

@@ -10,8 +10,8 @@ Targets define which agent or LLM provider to evaluate. AgentV uses one
 composable config graph across project manifests and eval files:
 
 - `.agentv/config.yaml` is the project-local discovery and composition root. It
-  can hold targets, graders, tests, defaults, execution policy, results
-  settings, and repo-local project policy.
+  can hold targets, tests, defaults, execution policy, results settings, and
+  repo-local project policy.
 - `$AGENTV_HOME/config.yaml` is the user/operator config. Use it for defaults
   that apply across projects, project registry data, default result locations,
   and provider defaults that should not be copied into each repo.
@@ -43,9 +43,9 @@ targets:
       command: ["codex", "app-server"]
       model: gpt-5-codex
 
-graders:
   - id: openai-grader
     provider: openai
+    runtime: host
     config:
       model: gpt-5-mini
 
@@ -61,6 +61,11 @@ the shorthand for the current machine, or object form when you need
 settings belong under `config`. Process-backed coding-agent providers use
 `config.command` as a non-empty argv array.
 
+A grader is not a separate kind of entity — it is a target selected for a
+grading role, either through `defaults.grader` (shown above) or an
+assertion-level target override. There is no separate `graders:` list;
+authoring one is a hard error telling you to move each entry into `targets`.
+
 ## Runtime Modes
 
 Use `runtime: host` when you want AgentV to run the target exactly as it is
@@ -128,9 +133,9 @@ targets:
       command: ["codex", "app-server"]
       model: gpt-5-codex
 
-graders:
   - id: openai-grader
     provider: openai
+    runtime: host
     config:
       model: gpt-5-mini
 
@@ -151,7 +156,6 @@ can be decomposed like this:
 
 ```yaml
 targets: file://targets.yaml
-graders: file://graders.yaml
 tests: file://tests.yaml
 defaults: file://defaults.yaml
 ```
@@ -167,6 +171,12 @@ a bare array, not an object wrapped in `targets:`:
   config:
     command: ["codex", "app-server"]
     model: gpt-5-codex
+
+- id: openai-grader
+  provider: openai
+  runtime: host
+  config:
+    model: gpt-5-mini
 ```
 
 ```yaml

apps/web/src/content/docs/docs/next/targets/custom-providers.mdx

@@ -184,9 +184,9 @@ targets:
     config:
       base_url: http://localhost:8080
 
-graders:
   - id: azure-grader
     provider: azure
+    runtime: host
     config:
       endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}"
       api_key: "{{ env.AZURE_OPENAI_API_KEY }}"

apps/web/src/content/docs/docs/next/targets/llm-providers.mdx

@@ -66,9 +66,10 @@ For smoke tests and dogfood runs against a local OpenAI-compatible proxy, keep
 the endpoint, model, and placeholder key in environment variables:
 
 ```yaml
-graders:
+targets:
   - id: local-openai-grader
     provider: openai
+    runtime: host
     config:
       api_format: chat
       base_url: "{{ env.LOCAL_OPENAI_PROXY_BASE_URL }}"

examples/features/readme-quickstart/.agentv/config.yaml

@@ -1,4 +1,3 @@
 targets: file://targets.yaml
-graders: file://graders.yaml
 tests: file://tests.yaml
 defaults: file://defaults.yaml