Skip to content

Commit 734de39

Browse files
authored
docs(integrations): document Phoenix adapter
1 parent 4e3d685 commit 734de39

1 file changed

Lines changed: 106 additions & 0 deletions

File tree

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
---
2+
title: Phoenix Adapter
3+
description: Convert AgentV eval YAML into Phoenix datasets and experiments while keeping AgentV as the source of truth.
4+
sidebar:
5+
order: 4
6+
---
7+
8+
The Phoenix adapter converts AgentV eval YAML suites into Phoenix dataset and
9+
experiment payloads. Use it when your team already reviews experiments in
10+
Phoenix but wants AgentV eval files, graders, result JSONL, and run artifacts to
11+
remain the canonical source.
12+
13+
The adapter is intentionally narrow. It supports deterministic assertions that
14+
map cleanly to Phoenix CODE evaluators and reports unsupported AgentV families
15+
instead of silently dropping semantics.
16+
17+
## Quick Start
18+
19+
From the AgentV repository root:
20+
21+
```bash
22+
bun --filter @agentv/phoenix-adapter phoenix:assert-smoke
23+
```
24+
25+
This runs a dry-run smoke conversion for the deterministic assertion example and
26+
writes a structural report to `/tmp/agentv-phoenix-assert-smoke.json`.
27+
28+
Run a broader dry run:
29+
30+
```bash
31+
bun --filter @agentv/phoenix-adapter phoenix:dry-run
32+
```
33+
34+
Run one eval source directly:
35+
36+
```bash
37+
bun packages/phoenix-adapter/src/cli.ts run \
38+
--dry-run \
39+
--agentv-root . \
40+
--eval-file examples/features/assert/evals/dataset.eval.yaml \
41+
--out reports/phoenix-assert.json
42+
```
43+
44+
## Supported Evaluators
45+
46+
| AgentV assertion family | Phoenix adapter behavior |
47+
| --- | --- |
48+
| `contains` | Converts to deterministic Phoenix evaluator logic |
49+
| `regex` | Converts to deterministic Phoenix evaluator logic |
50+
| `equals` | Converts to deterministic Phoenix evaluator logic |
51+
| `is-json` | Converts to deterministic Phoenix evaluator logic |
52+
| `llm-grader`, rubrics, `code-grader`, `tool-trajectory`, composite, metrics, and custom families | Reported as unsupported in the adapter report |
53+
54+
Unsupported families do not fail conversion by default. Add
55+
`--fail-on-unsupported` when a parity report should fail CI if any suite needs a
56+
manual Phoenix-specific evaluator.
57+
58+
```bash
59+
bun packages/phoenix-adapter/src/cli.ts run \
60+
--dry-run \
61+
--agentv-root . \
62+
--filter examples/features/assert \
63+
--fail-on-unsupported
64+
```
65+
66+
## When to Use the Adapter
67+
68+
Use the Phoenix adapter for:
69+
70+
- deterministic assertion suites that should appear as Phoenix datasets and
71+
experiments
72+
- parity checks that prove Phoenix row IDs match AgentV test IDs
73+
- integration smoke tests before writing a custom Phoenix evaluator
74+
75+
Keep the eval in AgentV when you need:
76+
77+
- workspace setup, lifecycle hooks, Docker workspaces, or repo materialization
78+
- code graders that execute commands in the AgentV workspace
79+
- tool trajectory, trace, cost, latency, or composite scoring
80+
- rich rubric semantics that need AgentV's assertion objects in result JSONL
81+
82+
Those features can still be represented in Phoenix with custom task and
83+
evaluator code, but the adapter does not attempt a lossy automatic conversion.
84+
85+
## Traces vs Datasets
86+
87+
The Phoenix adapter creates dataset and experiment payloads. It is separate from
88+
AgentV's OpenTelemetry trace export.
89+
90+
For trace export, use AgentV's standard OTel options:
91+
92+
```bash
93+
agentv eval evals/my-eval.yaml --otel-file traces/eval.otlp.json
94+
```
95+
96+
For live OTel export to a configured backend, use the options documented in
97+
[Running Evaluations](/docs/evaluation/running-evals/#live-otel-export/).
98+
99+
## Package Docs
100+
101+
The adapter package includes the implementation README, support matrix, and
102+
verification notes:
103+
104+
- [`packages/phoenix-adapter/README.md`](https://github.com/EntityProcess/agentv/tree/main/packages/phoenix-adapter)
105+
- [`packages/phoenix-adapter/docs/support-matrix.md`](https://github.com/EntityProcess/agentv/blob/main/packages/phoenix-adapter/docs/support-matrix.md)
106+
- [`packages/phoenix-adapter/docs/e2e-verification.md`](https://github.com/EntityProcess/agentv/blob/main/packages/phoenix-adapter/docs/e2e-verification.md)

0 commit comments

Comments
 (0)