fix(evaluation): remove public llm-grader assertion

Remove the public llm-grader assertion surface in favor of llm-rubric and agent-rubric guidance.

Author: christso

.agents/conventions.md

@@ -141,6 +141,7 @@ Grader types use kebab-case everywhere.
 - Internal TypeScript may keep shared LLM grader implementation names, but new authored evals should not use `llm-grader` as a public type.
 - Output `scores[].type`: use the authored grader type when available, such as `"llm-rubric"`, `"g-eval"`, or `"is-json"`.
 - Registry keys: `registry.register('llm-rubric', ...)` for the authored rubric surface, with `llm-grader` retained as internal/shared implementation plumbing.
+- Public SDK assertion helpers and authored YAML schemas must not expose `llm-grader`; use `llmRubricGrader()` / `type: llm-rubric` or `type: agent-rubric`.
 
 Source of truth: `GRADER_KIND_VALUES` in `packages/core/src/evaluation/types.ts`.
 
@@ -149,10 +150,10 @@ Backward compatibility:
 - Snake_case is accepted in YAML by `normalizeGraderType()` in `grader-parser.ts`, for example `llm_rubric` -> `llm-rubric`.
 - Single-word types such as `contains`, `equals`, `regex`, `latency`, and `cost` are unchanged.
 
-Two type definitions exist and must stay in sync:
+Public and internal type definitions intentionally differ for internal-only grader plumbing:
 
-- `EvaluatorKind` in `packages/core/src/evaluation/types.ts`
-- `AssertionType` in `packages/sdk/src/assertion.ts`
+- `GraderKind` in `packages/core/src/evaluation/types.ts` includes internal/runtime keys such as `llm-grader`.
+- `AssertionType` in `packages/sdk/src/assertion.ts` includes public authored assertion types only.
 
 ## Python Scripts
 

apps/cli/test/commands/eval/pipeline/fixtures/input-test.eval.yaml

@@ -12,8 +12,8 @@ tests:
         command: node grader-score-1.js
         weight: 1
       - metric: relevance
-        type: llm-grader
-        prompt: Did the response echo the input?
+        type: llm-rubric
+        value: Did the response echo the input?
         weight: 2
     vars:
       input: hello world

apps/web/src/content/docs/docs/next/graders/llm-graders.mdx

@@ -50,8 +50,8 @@ Use `provider:` when you want different `llm-rubric` entries in the same eval to
 
 Use `type: agent-rubric` when a Promptfoo-style rubric needs an agentic grader
 that can inspect the workspace instead of only judging the final answer text.
-AgentV routes `agent-rubric` through the same `llm-rubric` / `llm-grader`
-scoring path, so results still appear as normal `EvaluationScore` entries.
+AgentV routes `agent-rubric` through the same shared LLM rubric scoring path,
+so results still appear as normal `EvaluationScore` entries.
 
 ```yaml
 assert:

apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-cases.mdx

@@ -86,7 +86,7 @@ tests:
       target: gpt4_target
     assertions:
       - name: depth_check
-        type: llm-grader
+        type: llm-rubric
         prompt: ./graders/depth.md
 ```
 
@@ -111,7 +111,7 @@ tests:
       skip_defaults: true
     assertions:
       - name: custom_eval
-        type: llm-grader
+        type: llm-rubric
     # Does NOT get latency_check
 ```
 
@@ -330,14 +330,14 @@ The `criteria` field is a **data field** that describes what the response should
 
 ### No `assertions` — implicit LLM grader
 
-When a test has no `assertions` field, a default `llm-grader` grader runs automatically and uses `criteria` as the evaluation prompt:
+When a test has no `assertions` field, a default `llm-rubric` grader runs automatically and uses `criteria` as the evaluation prompt:
 
 ```yaml
 tests:
   - id: simple-eval
     criteria: Assistant correctly explains the bug and proposes a fix
     input: "Debug this function..."
-    # No assertions → default llm-grader evaluates against criteria
+    # No assertions → default llm-rubric evaluates against criteria
 ```
 
 Suite-level `preprocessors` also apply to this implicit grader. That matters when the agent output is a `ContentFile` block rather than plain text:
@@ -355,13 +355,13 @@ tests:
 
 ### `assertions` present — explicit graders only
 
-When `assertions` is defined, only the declared graders run. No implicit grader is added. Graders that are declared (such as `llm-grader`, `code-grader`, or `rubrics`) receive `criteria` as input automatically.
+When `assertions` is defined, only the declared graders run. No implicit grader is added. Graders that are declared (such as `llm-rubric`, `script`, or `rubrics`) receive `criteria` as input automatically.
 
 If `assertions` contains only deterministic graders (like `contains` or `regex`), the `criteria` field is not evaluated and a warning is emitted:
 
 ```
 Warning: Test 'my-test': criteria is defined but no grader in assertions
-will evaluate it. Add 'type: llm-grader' to assertions, or remove criteria
+will evaluate it. Add 'type: llm-rubric' to assertions, or remove criteria
 if it is documentation-only.
 ```
 
@@ -373,7 +373,7 @@ tests:
     criteria: Response is helpful and mentions the fix
     input: "Debug this function..."
     assertions:
-      - type: llm-grader        # explicit — receives criteria automatically
+      - type: llm-rubric        # explicit — receives criteria automatically
       - type: contains
         value: "fix"
 ```
@@ -390,7 +390,7 @@ tests:
     criteria: Response is helpful and mentions the fix
     input: "Debug this function..."
     assertions:
-      - type: llm-grader
+      - type: llm-rubric
         preprocessors:
           - type: xlsx
             command: ["bun", "run", "scripts/preprocessors/xlsx-to-json.ts"]

apps/web/src/content/docs/docs/v4.42.4/evaluation/examples.mdx

@@ -91,7 +91,7 @@ tests:
         command: [uv, run, validate_json.py]
         cwd: ./graders
       - name: content_evaluator
-        type: llm-grader
+        type: llm-rubric
         prompt: ./graders/semantic_correctness.md
 
     input: |-
@@ -108,7 +108,7 @@ tests:
 
 ## File Output Preprocessing
 
-Convert a binary file output into text before the `llm-grader` sees it:
+Convert a binary file output into text before the `llm-rubric` sees it:
 
 ```yaml
 description: Grade spreadsheet output via a preprocessor
@@ -183,15 +183,15 @@ assertions:
     threshold: 0.6
     assertions:
       - name: grader-gpt-5-mini
-        type: llm-grader
+        type: llm-rubric
         target: grader_gpt_5_mini
         prompt: ../prompts/grader-pass-fail-v1.md
       - name: grader-claude-haiku
-        type: llm-grader
+        type: llm-rubric
         target: grader_claude_haiku
         prompt: ../prompts/grader-pass-fail-v1.md
       - name: grader-gemini-flash
-        type: llm-grader
+        type: llm-rubric
         target: grader_gemini_flash
         prompt: ../prompts/grader-pass-fail-v1.md
 ```

apps/web/src/content/docs/docs/v4.42.4/graders/execution-metrics.mdx

@@ -129,7 +129,7 @@ tests:
     assertions:
       # Semantic quality
       - name: quality
-        type: llm-grader
+        type: llm-rubric
         prompt: ./prompts/code-quality.md
 
       # Efficiency constraints

apps/web/src/content/docs/docs/v4.42.4/graders/llm-graders.mdx

@@ -12,14 +12,14 @@ LLM graders use a language model to evaluate agent responses against custom crit
 
 ## Default Grader
 
-When a test defines `criteria` but has **no `assertions` field**, a default `llm-grader` runs automatically. The built-in prompt evaluates the response against your `criteria` and `expected_output`:
+When a test defines `criteria` but has **no `assertions` field**, a default `llm-rubric` runs automatically. The built-in prompt evaluates the response against your `criteria` and `expected_output`:
 
 ```yaml
 tests:
   - id: simple-eval
     criteria: Correctly explains the bug and proposes a fix
     input: "Debug this function..."
-    # No assertions needed — default llm-grader evaluates against criteria
+    # No assertions needed — default llm-rubric evaluates against criteria
 ```
 
 When `assertions` **is** present, no default grader is added. To use an LLM grader alongside other graders, declare it explicitly. See [How criteria and assertions interact](/docs/v4.42.4/evaluation/eval-cases/#how-criteria-and-assertions-interact).
@@ -31,12 +31,12 @@ Reference an LLM grader in your eval file:
 ```yaml
 assertions:
   - name: semantic_check
-    type: llm-grader
+    type: llm-rubric
     prompt: file://graders/correctness.md
     target: grader_gpt_5_mini   # optional: route this grader to a named LLM target
 ```
 
-Use `target:` when you want different `llm-grader` entries in the same eval to run on different grader models. This is useful for grader panels, majority-vote ensembles, and grader A/B benchmarks.
+Use `target:` when you want different `llm-rubric` entries in the same eval to run on different grader models. This is useful for grader panels, majority-vote ensembles, and grader A/B benchmarks.
 
 ## Prompt Files
 
@@ -100,26 +100,26 @@ tests:
       row: 1
     assertions:
       - name: dexter_semantic
-        type: llm-grader
+        type: llm-rubric
         prompt: file://prompts/dexter-grader.md
-        rubrics:
+        value:
           - operator: correctness
-            criteria: Uses the provided ticker and company.
+            outcome: Uses the provided ticker and company.
 ```
 
 ## Per-Grader Target
 
-By default, an `llm-grader` uses `defaults.grader` from the resolved config graph.
+By default, an `llm-rubric` uses `defaults.grader` from the resolved config graph.
 Override it per grader when you need multiple grader models in one run:
 
 ```yaml
 assertions:
   - name: grader-gpt
-    type: llm-grader
+    type: llm-rubric
     target: grader_gpt_5_mini
     prompt: ./prompts/pass-fail.md
   - name: grader-haiku
-    type: llm-grader
+    type: llm-rubric
     target: grader_claude_haiku
     prompt: ./prompts/pass-fail.md
 ```
@@ -178,7 +178,7 @@ When using TypeScript templates, configure them in YAML with optional `config` d
 ```yaml
 assertions:
   - name: custom-eval
-    type: llm-grader
+    type: llm-rubric
     prompt:
       command: [bun, run, ../prompts/custom-grader.ts]
       config:
@@ -190,7 +190,7 @@ The `config` object is available as `ctx.config` inside the template function.
 
 ## Preprocessing File Outputs
 
-If an agent returns a `ContentFile` block instead of plain text, you can preprocess that file into text before `llm-grader` builds the candidate prompt.
+If an agent returns a `ContentFile` block instead of plain text, you can preprocess that file into text before `llm-rubric` builds the candidate prompt.
 
 AgentV always tries a default UTF-8 text read first. That is enough for text-based formats such as CSV, JSON, SQL, Markdown, YAML, HTML, XML, and plain text. For binary formats such as `.xlsx`, `.pdf`, or `.docx`, add a preprocessor command:
 
@@ -205,7 +205,7 @@ tests:
     input: Generate the spreadsheet report
     assertions:
       - name: spreadsheet-check
-        type: llm-grader
+        type: llm-rubric
         prompt: |
           Check whether the transformed spreadsheet text contains the revenue rows:
 
@@ -220,7 +220,7 @@ Resolution order:
 - if no preprocessor matches, AgentV falls back to a UTF-8 text read
 - if the fallback read looks binary or invalid, the grader receives a warning note instead of failing the test run
 
-The implicit default `llm-grader` also inherits suite-level `preprocessors`, so you can omit `assertions` and still preprocess file outputs before grading.
+The implicit default `llm-rubric` also inherits suite-level `preprocessors`, so you can omit `assertions` and still preprocess file outputs before grading.
 
 See [`examples/features/preprocessors/`](../../../../../examples/features/preprocessors/) for a runnable example with a file-producing target and a custom preprocessor script.
 

apps/web/src/content/docs/docs/v4.42.4/guides/benchmark-provenance.mdx

@@ -212,7 +212,7 @@ description: Generated finance research cases with row-level provenance.
 
 assertions:
   - name: answer-quality
-    type: llm-grader
+    type: llm-rubric
     prompt: ./graders/finance-answer.md
     required: true
 

apps/web/src/content/docs/docs/v4.42.4/guides/evaluation-types.mdx

@@ -67,7 +67,7 @@ AgentV's eval tooling is designed for **execution quality**:
 - **`EVAL.yaml`** — define test cases with inputs, expected outputs, and assertions
 - **`evals.json`** — lightweight skill evaluation format (prompt/expected-output pairs)
 - **`agentv eval`** — execute evaluations and collect results
-- **Graders** — `llm-grader`, `code-grader`, `tool-trajectory`, `rubrics`, `contains`, `regex`, and others all measure execution behavior
+- **Graders** — `llm-rubric`, `script`, `tool-trajectory`, `rubrics`, `contains`, `regex`, and others all measure execution behavior
 
 These tools assume the skill is already loaded and invoked. They measure what happens *after* routing, not the routing decision itself.
 

apps/web/src/content/docs/docs/v4.42.4/guides/human-review.mdx

@@ -96,7 +96,7 @@ The `feedback.json` file is a structured annotation of a single eval run. It rec
       "notes": "Missing coverage of multi-document queries.",
       "evaluator_overrides": {
         "code-grader:format-check": "Too strict — penalized valid output with trailing newline",
-        "llm-grader:quality": "Score 0.6 seems fair, answer was incomplete"
+        "llm-rubric:quality": "Score 0.6 seems fair, answer was incomplete"
       },
       "workspace_notes": "Workspace had stale cached files from previous run — may have affected retrieval results."
     },
@@ -148,7 +148,7 @@ For workspace evaluations with multiple graders (code graders, LLM graders, tool
   "verdict": "needs_improvement",
   "evaluator_overrides": {
     "code-grader:test-pass": "Tests pass but the refactored code has a subtle race condition the tests don't cover",
-    "llm-grader:quality": "Score 0.9 is too high — the agent left dead code behind",
+    "llm-rubric:quality": "Score 0.9 is too high — the agent left dead code behind",
     "tool-trajectory:efficiency": "Used 12 tool calls where 5 would suffice, but the result is correct"
   },
   "workspace_notes": "Agent cloned the repo correctly but didn't clean up temp files."