rename to evaluators

Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>

Author: peterj

.github/workflows/build-index.yaml

@@ -1,10 +1,10 @@
-name: Build grader index
+name: Build evaluator index
 
 on:
   push:
     branches: [main]
     paths:
-      - "graders/**/grader.yaml"
+      - "evaluators/**/evaluator.yaml"
 
 permissions:
   contents: write
@@ -20,7 +20,7 @@ jobs:
           sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
           sudo chmod +x /usr/local/bin/yq
 
-      - name: Build index.yaml from grader manifests
+      - name: Build index.yaml from evaluator manifests
         run: |
           set -euo pipefail
 
@@ -31,10 +31,10 @@ jobs:
           # Source: .github/workflows/build-index.yaml
           # Generated: ${TIMESTAMP}
 
-          graders:
+          evaluators:
           EOF
 
-          for manifest in graders/*/grader.yaml; do
+          for manifest in evaluators/*/evaluator.yaml; do
             dir=$(dirname "$manifest")
             name=$(yq '.name' "$manifest")
             description=$(yq '.description' "$manifest")

.github/workflows/validate-evaluators.yaml

@@ -0,0 +1,34 @@
+name: Validate evaluators
+
+on:
+  pull_request:
+    paths:
+      - "evaluators/**"
+      - "scripts/validate_evaluator.py"
+      - "scripts/test_input.json"
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          pip install pyyaml
+          # TODO: switch to `pip install agentevals-grader-sdk` once published to PyPI
+          pip install "agentevals-grader-sdk @ git+https://github.com/agentevals-dev/agentevals.git#subdirectory=packages/grader-sdk-py"
+
+      - name: Discover and validate all evaluators
+        run: |
+          evaluator_dirs=$(find evaluators -mindepth 1 -maxdepth 1 -type d | sort)
+          if [ -z "$evaluator_dirs" ]; then
+            echo "No evaluator directories found."
+            exit 0
+          fi
+          python scripts/validate_evaluator.py $evaluator_dirs

.github/workflows/validate-graders.yaml

@@ -1,18 +1,18 @@
-# agentevals Community Graders
+# agentevals Community Evaluators
 
-Community-maintained graders for [agentevals](https://github.com/agentevals-dev/agentevals) -- the agent evaluation framework built on Google ADK.
+Community-maintained evaluators for [agentevals](https://github.com/agentevals-dev/agentevals) -- the agent evaluation framework built on Google ADK.
 
-Graders are standalone scoring programs that evaluate agent traces. They read `EvalInput` JSON from stdin and write `EvalResult` JSON to stdout. This repository is the official index of community-contributed graders.
+Evaluators are standalone scoring programs that evaluate agent traces. They read `EvalInput` JSON from stdin and write `EvalResult` JSON to stdout. This repository is the official index of community-contributed evaluators.
 
-## Using community graders
+## Using community evaluators
 
-### Browse available graders
+### Browse available evaluators
 
 ```bash
-agentevals grader list --source github
+agentevals evaluator list --source github
 ```
 
-### Reference a community grader in your eval config
+### Reference a community evaluator in your eval config
 
 Add a `type: remote` entry to your `eval_config.yaml`:
 
@@ -23,15 +23,15 @@ metrics:
   - name: response_quality
     type: remote
     source: github
-    ref: graders/response_quality/response_quality.py
+    ref: evaluators/response_quality/response_quality.py
     threshold: 0.7
     config:
       min_response_length: 20
 
   - name: tool_coverage
     type: remote
     source: github
-    ref: graders/tool_coverage/tool_coverage.py
+    ref: evaluators/tool_coverage/tool_coverage.py
     threshold: 1.0
     config:
       min_tool_calls: 1
@@ -45,34 +45,34 @@ agentevals run traces/my_trace.json \
   --eval-set eval_set.json
 ```
 
-The grader is downloaded automatically and cached in `~/.cache/agentevals/graders/`.
+The evaluator is downloaded automatically and cached in `~/.cache/agentevals/evaluators/`.
 
-## Contributing a grader
+## Contributing an evaluator
 
-### 1. Scaffold a new grader
+### 1. Scaffold a new evaluator
 
 ```bash
 pip install agentevals
-agentevals grader init my_grader
+agentevals evaluator init my_evaluator
 ```
 
 This creates a directory ready to be added to this repo:
 
 ```
-my_grader/
-├── my_grader.py     # your scoring logic
-└── grader.yaml      # metadata manifest
+my_evaluator/
+├── my_evaluator.py     # your scoring logic
+└── evaluator.yaml      # metadata manifest
 ```
 
 ### 2. Implement your scoring logic
 
-Edit `my_grader.py`. Your function receives an `EvalInput` with the agent's invocations and returns an `EvalResult` with a score between 0.0 and 1.0.
+Edit `my_evaluator.py`. Your function receives an `EvalInput` with the agent's invocations and returns an `EvalResult` with a score between 0.0 and 1.0.
 
 ```python
 from agentevals_grader_sdk import grader, EvalInput, EvalResult
 
 @grader
-def my_grader(input: EvalInput) -> EvalResult:
+def my_evaluator(input: EvalInput) -> EvalResult:
     scores = []
     for inv in input.invocations:
         # Your scoring logic here
@@ -82,19 +82,22 @@ def my_grader(input: EvalInput) -> EvalResult:
         score=sum(scores) / len(scores) if scores else 0.0,
         per_invocation_scores=scores,
     )
+
+if __name__ == "__main__":
+    my_evaluator.run()
 ```
 
 Install the SDK standalone with `pip install agentevals-grader-sdk` (no heavy dependencies).
 
 ### 3. Update the manifest
 
-Edit `grader.yaml` with a description, tags, and your name:
+Edit `evaluator.yaml` with a description, tags, and your name:
 
 ```yaml
-name: my_grader
-description: What this grader checks
+name: my_evaluator
+description: What this evaluator checks
 language: python
-entrypoint: my_grader.py
+entrypoint: my_evaluator.py
 tags: [quality, tools]
 author: your-github-username
 ```
@@ -105,21 +108,21 @@ Run the validation script to catch issues before submitting:
 
 ```bash
 pip install agentevals-grader-sdk pyyaml
-python scripts/validate_grader.py graders/my_grader
+python scripts/validate_evaluator.py evaluators/my_evaluator
 ```
 
 This checks:
 - **Manifest schema** -- required fields, entrypoint exists, name matches directory
 - **Syntax and imports** -- compiles cleanly, uses `@grader` decorator
-- **Smoke run** -- runs the grader with synthetic input and validates the `EvalResult` output (correct types for `score`, `details`, `status`, etc.)
+- **Smoke run** -- runs the evaluator with synthetic input and validates the `EvalResult` output (correct types for `score`, `details`, `status`, etc.)
 
 You can also test with a full eval run:
 
 ```yaml
 metrics:
-  - name: my_grader
+  - name: my_evaluator
     type: code
-    path: ./graders/my_grader/my_grader.py
+    path: ./evaluators/my_evaluator/my_evaluator.py
     threshold: 0.5
 ```
 
@@ -130,13 +133,13 @@ agentevals run traces/sample.json --config eval_config.yaml --eval-set eval_set.
 ### 5. Submit a pull request
 
 1. Fork this repository
-2. Copy your grader directory into `graders/`:
+2. Copy your evaluator directory into `evaluators/`:
 
 ```
-graders/
-├── my_grader/
-│   ├── grader.yaml
-│   └── my_grader.py
+evaluators/
+├── my_evaluator/
+│   ├── evaluator.yaml
+│   └── my_evaluator.py
 ├── response_quality/
 │   └── ...
 └── tool_coverage/
@@ -145,16 +148,16 @@ graders/
 
 3. Open a PR against `main`
 
-CI will automatically validate your grader (manifest, syntax, and smoke run). Once merged, a separate workflow regenerates `index.yaml`, and your grader becomes available to everyone via `agentevals grader list`.
+CI will automatically validate your evaluator (manifest, syntax, and smoke run). Once merged, a separate workflow regenerates `index.yaml`, and your evaluator becomes available to everyone via `agentevals evaluator list`.
 
 ## Supported languages
 
-Graders can be written in any language that reads JSON from stdin and writes JSON to stdout.
+Evaluators can be written in any language that reads JSON from stdin and writes JSON to stdout.
 
 | Language | Extension | SDK available |
 |---|---|---|
 | Python | `.py` | `pip install agentevals-grader-sdk` |
 | JavaScript | `.js` | No SDK yet -- just read stdin, write stdout |
 | TypeScript | `.ts` | No SDK yet -- just read stdin, write stdout |
 
-See the [custom graders documentation](https://github.com/agentevals-dev/agentevals/blob/main/docs/custom-graders.md) for the full protocol reference.
+See the [custom evaluators documentation](https://github.com/agentevals-dev/agentevals/blob/main/docs/custom-evaluators.md) for the full protocol reference.

README.md

@@ -0,0 +1,6 @@
+name: peters_evaluator
+description: 'sample evaluator that returns a 0.123 score'
+language: python
+entrypoint: peters_evaluator.py
+tags: ["test", "example"]
+author: 'peterj'

evaluators/peters_evaluator/evaluator.yaml

@@ -0,0 +1,21 @@
+"""Custom evaluator: peters_evaluator
+
+Usage in eval_config.yaml:
+
+    metrics:
+      - name: peters_evaluator
+        type: code
+        path: ./peters_evaluator/peters_evaluator.py
+        threshold: 0.5
+"""
+
+from agentevals_grader_sdk import grader, EvalInput, EvalResult
+
+
+@grader
+def peters_evaluator(input: EvalInput) -> EvalResult:
+    return EvalResult(score=0.123, details={"message": "All good"})
+
+
+if __name__ == "__main__":
+    peters_evaluator.run()

evaluators/peters_evaluator/peters_evaluator.py

@@ -1,4 +1,4 @@
-"""Community grader: response_quality
+"""Community evaluator: response_quality
 
 Checks that every invocation has a non-empty response, meets a configurable
 minimum length, and doesn't just parrot back the user input.
@@ -48,3 +48,7 @@ def response_quality(input: EvalInput) -> EvalResult:
         per_invocation_scores=scores,
         details={"issues": issues} if issues else None,
     )
+
+
+if __name__ == "__main__":
+    response_quality.run()

graders/response_quality/grader.yaml …aluators/response_quality/evaluator.yamlgraders/response_quality/grader.yaml renamed to evaluators/response_quality/evaluator.yaml graders/response_quality/grader.yaml renamed to evaluators/response_quality/evaluator.yaml

@@ -1,4 +1,4 @@
-"""Community grader: tool_coverage
+"""Community evaluator: tool_coverage
 
 Verifies that each invocation made at least a minimum number of tool calls.
 Useful for ensuring agents actually use their tools rather than hallucinating
@@ -34,3 +34,7 @@ def tool_coverage(input: EvalInput) -> EvalResult:
         per_invocation_scores=scores,
         details={"missing_tools": details} if details else None,
     )
+
+
+if __name__ == "__main__":
+    tool_coverage.run()