diff --git a/docs/integration/ai/.pages b/docs/integration/ai/.pages index b1b5b013..a72256ec 100644 --- a/docs/integration/ai/.pages +++ b/docs/integration/ai/.pages @@ -1,5 +1,6 @@ nav: - LLM Applications: llm-applications.md + - LLM Evaluations: llm-evaluations.md - Claude Code Tracing: claude-code-tracing.md - Frameworks: frameworks - Model Providers: providers diff --git a/docs/integration/ai/images/online-evaluations-1.png b/docs/integration/ai/images/online-evaluations-1.png new file mode 100644 index 00000000..a605ebd7 Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-1.png differ diff --git a/docs/integration/ai/images/online-evaluations-10.png b/docs/integration/ai/images/online-evaluations-10.png new file mode 100644 index 00000000..c1848af2 Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-10.png differ diff --git a/docs/integration/ai/images/online-evaluations-2.png b/docs/integration/ai/images/online-evaluations-2.png new file mode 100644 index 00000000..89bbacf9 Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-2.png differ diff --git a/docs/integration/ai/images/online-evaluations-3.png b/docs/integration/ai/images/online-evaluations-3.png new file mode 100644 index 00000000..ccd909b0 Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-3.png differ diff --git a/docs/integration/ai/images/online-evaluations-4.png b/docs/integration/ai/images/online-evaluations-4.png new file mode 100644 index 00000000..c9dcc669 Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-4.png differ diff --git a/docs/integration/ai/images/online-evaluations-5.png b/docs/integration/ai/images/online-evaluations-5.png new file mode 100644 index 00000000..1e769202 Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-5.png differ diff --git a/docs/integration/ai/images/online-evaluations-6.png b/docs/integration/ai/images/online-evaluations-6.png new file mode 100644 index 00000000..decc6569 Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-6.png differ diff --git a/docs/integration/ai/images/online-evaluations-7.png b/docs/integration/ai/images/online-evaluations-7.png new file mode 100644 index 00000000..41645b7d Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-7.png differ diff --git a/docs/integration/ai/images/online-evaluations-8.png b/docs/integration/ai/images/online-evaluations-8.png new file mode 100644 index 00000000..c7b45e9c Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-8.png differ diff --git a/docs/integration/ai/images/online-evaluations-9.png b/docs/integration/ai/images/online-evaluations-9.png new file mode 100644 index 00000000..c677b20d Binary files /dev/null and b/docs/integration/ai/images/online-evaluations-9.png differ diff --git a/docs/integration/ai/llm-evaluations.md b/docs/integration/ai/llm-evaluations.md new file mode 100644 index 00000000..57ca0cec --- /dev/null +++ b/docs/integration/ai/llm-evaluations.md @@ -0,0 +1,276 @@ +# LLM Evaluations + +Online Evaluations let you continuously score your LLM application's traces and spans using configurable evaluators - either LLM-as-a-judge powered by your own AI providers, or external remote scoring endpoints. + +## Overview + +The Online Evaluations system has four core resources, each building on the previous: + +| Resource | What it does | +|---|---| +| **Provider** | An LLM API configuration (OpenAI, Anthropic, etc.) with credentials and available models. Used by LLM Judge scorers to call an LLM. | +| **Score Config** | Defines the shape of a score - its data type (numeric, categorical, boolean), valid range or categories, and a healthy/unhealthy threshold. | +| **Scorer** | The evaluation logic: a template with `{{variables}}`, parameters for execution, and a link to a score config that describes the output it produces. Two types exist: **LLM Judge** (calls an LLM via a provider) and **Remote** (calls an external HTTP endpoint). | +| **Eval Job** | A running evaluation pipeline: binds one or more scorers to a specific stream, defines which traces/spans to evaluate (filter), how many to sample, and manages the lifecycle (draft, active, paused, archived). | + +When you activate an eval job, the system creates a system-managed evaluation pipeline that runs your scorers against incoming data. Scores flow into the `_llm_scores` stream; evaluator telemetry flows into the `_evaluator` traces stream. + +![the Online Evaluations dashboard listing eval jobs](images/online-evaluations-1.png) + +## Enable Online Evaluations + +Online Evaluations is an enterprise feature. Set the configuration flag to enable it: + +```env +ZO_ONLINE_EVALS_ENABLED=true +``` + +When enabled, the **Evaluations** top-level navigation appears in the UI. When disabled, all evaluation pages and settings are hidden; backend API endpoints remain reachable. + +## Providers + +A Provider stores the connection details for an LLM API. You configure one provider per AI service you want your LLM Judge scorers to use. + +### Create a provider + +Navigate to **Evaluations > Providers** and click **Add Provider**. + +![the Providers list page](images/online-evaluations-2.png) + +| Field | Description | +|---|---| +| **Name** | Display name for the provider. | +| **Provider Type** | The provider kind (`openai`, `anthropic`, `azure`, `gemini`, etc.). Determines the API protocol. | +| **Endpoint** | Override the default API base URL. Leave empty to use the provider's standard endpoint. | +| **Default Model** | The model used when no model is specified on the scorer. | +| **Available Models** | List of model IDs this provider supports. Used for model selection in scorers. | +| **Auth Config** | Credentials in JSON format (e.g., `{"api_key": "sk-..."}`). Masked in API responses. | +| **Is Default** | When set, this provider is preselected when creating new LLM Judge scorers. | + +![the Add Provider form](images/online-evaluations-3.png) + +### Test a provider + +From the provider detail page, use the **Test** button to verify connectivity. The system sends a test request using the configured endpoint and credentials. + +### Manage providers + +- **Update**: Edit any field. The provider is updated in-place. +- **Delete**: Removes the provider. Scorers referencing a deleted provider will fail until reassigned. + +## Score Configs + +A Score Config describes what a score looks like and when it is considered healthy. + +### Create a score config + +Navigate to **Evaluations > Score Configs** and click **Add Score Config**. + +| Field | Description | +|---|---| +| **Name** | A label for this config (e.g., "Faithfulness", "Accuracy"). | +| **Data Type** | `numeric`, `categorical`, or `boolean` - the type of score value. | +| **Description** | Optional description of what the score measures. | +| **Numeric Range** | For numeric scores: `{"min": 0.0, "max": 1.0}`. | +| **Categories** | For categorical scores: a list of valid category labels. | +| **Healthy Threshold** | Defines the boundary for healthy scores (e.g., `{"direction": "gte", "value": 0.7}` means scores ≥ 0.7 are healthy). | + +![the Score Configs list page](images/online-evaluations-4.png) + +### Versioning + +Score configs are versioned. Each config has a stable **entity ID** that stays the same across versions, and a unique **ID** per version. Updating a score config creates a new version and bumps the version number. Scorers that reference a score config can pin to a specific version or always use the latest. + +## Scorers + +A Scorer is the executable evaluation unit. It contains a prompt **template** with `{{variable}}` placeholders, execution **parameters**, and an optional link to a **score config** that describes its output. + +### Scorer types + +- **LLM Judge**: Sends the rendered template to an LLM via a configured provider. Supports temperature, max tokens, timeout, output parsing, and optional reasoning. +- **Remote**: Sends the rendered template as an HTTP request to an external evaluation service. Supports bearer token auth, API keys, basic auth, custom headers, timeouts, and retries. + +### Create a scorer + +Navigate to **Evaluations > Scorers** and click **Add Scorer**. + +![the Scorers list page](images/online-evaluations-5.png) + +| Field | Description | +|---|---| +| **Name** | Display name. | +| **Description** | Optional description. | +| **Scorer Type** | Choose **LLM Judge** or **Remote**. | +| **Produces Score Config** | (Optional) Link to a score config that describes this scorer's output. | +| **Template** | The evaluation prompt with `{{variable}}` placeholders that will be populated at runtime. | +| **Output Schema** | (LLM Judge only) JSON Schema for structured output parsing. | + +For **LLM Judge**, you also configure: + +| Field | Description | +|---|---| +| **Provider** | The provider to use for the LLM call. | +| **Model** | Override the provider's default model. | +| **Temperature** | LLM temperature (0-2). | +| **Max Tokens** | Maximum completion tokens. | +| **Timeout** | Request timeout in milliseconds. | +| **Include Reasoning** | When enabled, the LLM is prompted to include reasoning alongside the score. | +| **Extra Metadata Fields** | Additional fields the LLM should return beyond the score (e.g., failure mode classification). | + +![the Add Scorer form for LLM Judge](images/online-evaluations-6.png) + +For **Remote**, you configure: + +| Field | Description | +|---|---| +| **Endpoint** | The URL of the remote scoring service. | +| **HTTP Method** | `POST` or `PUT`. | +| **Auth** | `none`, `bearer` (token), `basic` (username/password), or `api_key` (token + header name). | +| **Custom Headers** | Additional HTTP headers to send. | +| **Content Type** | Request content type (defaults to `application/json`). | +| **Timeout** | Request timeout in milliseconds. | +| **Max Retries** | Number of retry attempts on failure. | + +### Test a scorer + +From the scorer detail page, use the **Test** button. Provide values for the template variables, and the system executes a one-off evaluation. The response shows the score, reasoning, model used, latency, and token usage. + +![the Scorer Test dialog showing results](images/online-evaluations-7.png) + +### Preview output schema + +For LLM Judge scorers, the **Preview Schema** endpoint shows the derived output schema based on the score config and extra metadata fields, helping you understand what structure the LLM will return. + +### Versioning + +Like score configs, scorers are versioned. Each scorer has a stable **entity ID** and a version number. Updating a scorer creates a new version. Eval jobs can reference a scorer by entity ID (always latest) or pin to a specific version. + +## Eval Jobs + +An Eval Job is the execution unit that runs scorers against incoming traces. It manages the lifecycle of a system-managed evaluation pipeline. + +### Create a job + +Navigate to **Evaluations > Eval Jobs** and click **Add Job**. + +![the Eval Jobs list page](images/online-evaluations-8.png) + +| Field | Description | +|---|---| +| **Name** | Display name for the job. | +| **Description** | Optional description. | +| **Stream** | The stream whose traces or spans to evaluate. | +| **Stream Type** | `logs`, `traces`, or `metrics`. | +| **Filter Condition** | A JSON filter expression. Only spans matching this filter are evaluated. | +| **Scorers** | One or more scorer references (by entity ID). The system evaluates each matching span against every listed scorer. | +| **Input Mapping** | Per-scorer mapping of template variables to span attribute paths (e.g., `"input": "{{gen_ai_input_messages}}", "output": "{{gen_ai_output_messages}}"`). | +| **Sampling Mode** | `all` (evaluate everything), `rate` (evaluate a percentage, e.g., `0.1` for 10%), or `fixed` (evaluate N per time window). | +| **Sampling Value** | The sampling parameter value corresponding to the selected mode. | + +![the Add Eval Job form](images/online-evaluations-9.png) + +### Job lifecycle + +Jobs follow a defined state machine: + +``` +draft → active ⇄ paused + ↓ + degraded → active + ↓ + archived +``` + +| Action | Description | +|---|---| +| **Activate** | Creates the underlying evaluation pipeline and starts scoring. Allowed from `draft`, `paused`, or `degraded`. | +| **Pause** | Temporarily stops evaluation. The pipeline is preserved. Allowed from `active` or `degraded`. | +| **Resume** | Restarts evaluation from `paused` or `degraded` state. | +| **Archive** | Permanently stops evaluation. The job is retained for audit but no longer processes data. | + +Use the action buttons on the job detail page to manage lifecycle transitions. + +![the Eval Job detail page showing status and actions](images/online-evaluations-10.png) + +### Update a job + +Edit any field on a draft or active job. Updating bumps the job's version. If the job is active, the underlying pipeline is automatically reconciled with the new configuration. + +### Scoring pipeline + +When a job is activated, the system creates a `PipelineKind::Evaluation` pipeline behind the scenes. This pipeline is: + +- **Hidden** from the main Pipeline UI - it is managed exclusively by the eval jobs subsystem. +- **Coexisting** with user pipelines on the same stream (no "one pipeline per stream" conflict). +- **Automatically reconciled** when the job is updated. + +Evaluated scores are written to the `_llm_scores` system stream as `LlmScoreRecord` entries, and evaluator telemetry (latency, tokens, status) is recorded as OTLP spans in the `_evaluator` traces stream. + +## RBAC + +Online Evaluations resources have their own OFGA permissions: + +| Resource | OFGA Type | Permissions | +|---|---|---| +| Providers | `provider` | GET, LIST, POST, PUT, DELETE | +| Score Configs | `score_config` | GET, LIST, POST, PUT, DELETE | +| Scorers | `scorer` | GET, LIST, POST, PUT, DELETE | +| Eval Jobs | `eval_job` | GET, LIST, POST, PUT, DELETE | + +Assign the appropriate roles in **Identity & Access Management > Roles** to control access to evaluation resources. + +## API Reference + +All endpoints are prefixed with `/api/{org_id}`. + +### Providers + +| Method | Path | Description | +|---|---|---| +| `GET` | `/providers` | List all providers | +| `POST` | `/providers` | Create a provider | +| `GET` | `/providers/{id}` | Get a provider | +| `PUT` | `/providers/{id}` | Update a provider | +| `DELETE` | `/providers/{id}` | Delete a provider | +| `POST` | `/providers/{id}/test` | Test provider connectivity | + +### Score Configs + +| Method | Path | Description | +|---|---|---| +| `GET` | `/score_configs` | List score configs | +| `POST` | `/score_configs` | Create a score config | +| `GET` | `/score_configs/{id}` | Get a score config | +| `PUT` | `/score_configs/{id}` | Update a score config (version bump) | +| `DELETE` | `/score_configs/{id}` | Delete a score config | +| `GET` | `/score_configs/{id}/versions` | List all versions | + +### Scorers + +| Method | Path | Description | +|---|---|---| +| `GET` | `/scorers?scorer_type=llm_judge` | List scorers (optionally filtered by type) | +| `POST` | `/scorers` | Create a scorer | +| `GET` | `/scorers/{id}` | Get a scorer | +| `PUT` | `/scorers/{id}` | Update a scorer (version bump) | +| `DELETE` | `/scorers/{id}` | Delete a scorer | +| `POST` | `/scorers/{id}/test` | Test a scorer with input variables | +| `GET` | `/scorers/{id}/versions` | List all versions | + +### Eval Jobs + +| Method | Path | Description | +|---|---|---| +| `GET` | `/eval_jobs?status=active` | List jobs (optionally filtered by status) | +| `POST` | `/eval_jobs` | Create a job (draft) | +| `GET` | `/eval_jobs/{id}` | Get a job | +| `PUT` | `/eval_jobs/{id}` | Update a job | +| `DELETE` | `/eval_jobs/{id}` | Delete a job and its pipeline | +| `POST` | `/eval_jobs/{id}/activate` | Activate the job | +| `POST` | `/eval_jobs/{id}/pause` | Pause the job | +| `POST` | `/eval_jobs/{id}/resume` | Resume the job | +| `POST` | `/eval_jobs/{id}/archive` | Archive the job | + +## Super Cluster + +In multi-node deployments, evaluation resources are synchronized across the super cluster via dedicated queue topics (`eval_provider`, `eval_score_config`, `eval_scorer`, `eval_job`). Changes made on any node propagate automatically.