|
| 1 | +--- |
| 2 | +title: Phoenix Adapter |
| 3 | +description: Convert AgentV eval YAML into Phoenix datasets and experiments while keeping AgentV as the source of truth. |
| 4 | +sidebar: |
| 5 | + order: 4 |
| 6 | +--- |
| 7 | + |
| 8 | +The Phoenix adapter converts AgentV eval YAML suites into Phoenix dataset and |
| 9 | +experiment payloads. Use it when your team already reviews experiments in |
| 10 | +Phoenix but wants AgentV eval files, graders, result JSONL, and run artifacts to |
| 11 | +remain the canonical source. |
| 12 | + |
| 13 | +The adapter is intentionally narrow. It supports deterministic assertions that |
| 14 | +map cleanly to Phoenix CODE evaluators and reports unsupported AgentV families |
| 15 | +instead of silently dropping semantics. |
| 16 | + |
| 17 | +## Quick Start |
| 18 | + |
| 19 | +From the AgentV repository root: |
| 20 | + |
| 21 | +```bash |
| 22 | +bun --filter @agentv/phoenix-adapter phoenix:assert-smoke |
| 23 | +``` |
| 24 | + |
| 25 | +This runs a dry-run smoke conversion for the deterministic assertion example and |
| 26 | +writes a structural report to `/tmp/agentv-phoenix-assert-smoke.json`. |
| 27 | + |
| 28 | +Run a broader dry run: |
| 29 | + |
| 30 | +```bash |
| 31 | +bun --filter @agentv/phoenix-adapter phoenix:dry-run |
| 32 | +``` |
| 33 | + |
| 34 | +Run one eval source directly: |
| 35 | + |
| 36 | +```bash |
| 37 | +bun packages/phoenix-adapter/src/cli.ts run \ |
| 38 | + --dry-run \ |
| 39 | + --agentv-root . \ |
| 40 | + --eval-file examples/features/assert/evals/dataset.eval.yaml \ |
| 41 | + --out reports/phoenix-assert.json |
| 42 | +``` |
| 43 | + |
| 44 | +## Supported Evaluators |
| 45 | + |
| 46 | +| AgentV assertion family | Phoenix adapter behavior | |
| 47 | +| --- | --- | |
| 48 | +| `contains` | Converts to deterministic Phoenix evaluator logic | |
| 49 | +| `regex` | Converts to deterministic Phoenix evaluator logic | |
| 50 | +| `equals` | Converts to deterministic Phoenix evaluator logic | |
| 51 | +| `is-json` | Converts to deterministic Phoenix evaluator logic | |
| 52 | +| `llm-grader`, rubrics, `code-grader`, `tool-trajectory`, composite, metrics, and custom families | Reported as unsupported in the adapter report | |
| 53 | + |
| 54 | +Unsupported families do not fail conversion by default. Add |
| 55 | +`--fail-on-unsupported` when a parity report should fail CI if any suite needs a |
| 56 | +manual Phoenix-specific evaluator. |
| 57 | + |
| 58 | +```bash |
| 59 | +bun packages/phoenix-adapter/src/cli.ts run \ |
| 60 | + --dry-run \ |
| 61 | + --agentv-root . \ |
| 62 | + --filter examples/features/assert \ |
| 63 | + --fail-on-unsupported |
| 64 | +``` |
| 65 | + |
| 66 | +## When to Use the Adapter |
| 67 | + |
| 68 | +Use the Phoenix adapter for: |
| 69 | + |
| 70 | +- deterministic assertion suites that should appear as Phoenix datasets and |
| 71 | + experiments |
| 72 | +- parity checks that prove Phoenix row IDs match AgentV test IDs |
| 73 | +- integration smoke tests before writing a custom Phoenix evaluator |
| 74 | + |
| 75 | +Keep the eval in AgentV when you need: |
| 76 | + |
| 77 | +- workspace setup, lifecycle hooks, Docker workspaces, or repo materialization |
| 78 | +- code graders that execute commands in the AgentV workspace |
| 79 | +- tool trajectory, trace, cost, latency, or composite scoring |
| 80 | +- rich rubric semantics that need AgentV's assertion objects in result JSONL |
| 81 | + |
| 82 | +Those features can still be represented in Phoenix with custom task and |
| 83 | +evaluator code, but the adapter does not attempt a lossy automatic conversion. |
| 84 | + |
| 85 | +## Traces vs Datasets |
| 86 | + |
| 87 | +The Phoenix adapter creates dataset and experiment payloads. It is separate from |
| 88 | +AgentV's OpenTelemetry trace export. |
| 89 | + |
| 90 | +For trace export, use AgentV's standard OTel options: |
| 91 | + |
| 92 | +```bash |
| 93 | +agentv eval evals/my-eval.yaml --otel-file traces/eval.otlp.json |
| 94 | +``` |
| 95 | + |
| 96 | +For live OTel export to a configured backend, use the options documented in |
| 97 | +[Running Evaluations](/docs/evaluation/running-evals/#live-otel-export/). |
| 98 | + |
| 99 | +## Package Docs |
| 100 | + |
| 101 | +The adapter package includes the implementation README, support matrix, and |
| 102 | +verification notes: |
| 103 | + |
| 104 | +- [`packages/phoenix-adapter/README.md`](https://github.com/EntityProcess/agentv/tree/main/packages/phoenix-adapter) |
| 105 | +- [`packages/phoenix-adapter/docs/support-matrix.md`](https://github.com/EntityProcess/agentv/blob/main/packages/phoenix-adapter/docs/support-matrix.md) |
| 106 | +- [`packages/phoenix-adapter/docs/e2e-verification.md`](https://github.com/EntityProcess/agentv/blob/main/packages/phoenix-adapter/docs/e2e-verification.md) |
0 commit comments