feat(evals): add Promptfoo-based AI tool call evaluation suite (#2351)

* feat(evals): add Promptfoo-based AI tool call evaluation suite

Add automated evaluation infrastructure for validating LLM tool call
quality across SuperDoc's Document Engine API. Tests whether models
select the correct tools and construct valid arguments when given
document editing tasks.

The suite extracts 6 essential tool definitions from the SDK and
runs them against multiple OpenAI models and cross-provider comparisons
(Anthropic, Google). Includes deterministic assertions for tool
selection, argument accuracy, and production correctness rules learned
from the labs agent implementation.

* docs(evals): simplify README

* feat(evals): enhance GDPval benchmark configuration and test assertions

Updated the GDPval benchmark configuration to include distinct prompts for SuperDoc tool-augmented and baseline models. Enhanced the test assertions in the GDPval workflows to provide clearer scoring criteria for model responses, focusing on the specificity and executable nature of the responses. Adjusted thresholds for scoring to better reflect the quality of tool calls and text descriptions in document editing tasks.

* chore(evals): update GPT model version in GDPval configuration

Changed the model identifier from GPT-4o to GPT-5.4 in the GDPval benchmark configuration for both SuperDoc tool-augmented and baseline prompts, ensuring alignment with the latest model updates.

* feat(evals): add execution tests and enhance configuration for SuperDoc agent

Introduced a new execution test suite for the SuperDoc agent, validating real document editing capabilities through the CLI. Added a new script command for executing these tests and updated the GDPval configuration to reflect the latest GPT model version. Included necessary dependencies and created a new provider for the SuperDoc agent to facilitate the execution of tool calls against DOCX files.

* feat(evals): enhance SuperDoc agent with document copy and round-trip validation

Updated the SuperDoc agent to create temporary copies of documents for editing, ensuring original fixtures remain unaltered. Implemented round-trip validation to verify that edits persist after saving and re-opening DOCX files. Added a new memorandum fixture and expanded execution tests to cover various document editing scenarios, enhancing overall test coverage and reliability.

* feat(evals): add keepFile option to SuperDoc agent for document preservation

Enhanced the SuperDoc agent to include a `keepFile` option, allowing users to save edited documents to a specified output directory. Updated the logic to create the output directory if it doesn't exist and modified the cleanup process to conditionally copy the edited document based on this new option. Adjusted execution tests to validate the new functionality, ensuring comprehensive coverage of document editing scenarios.

* feat(evals): increase maxConcurrency for SuperDoc agent tests and update execution logic

Enhanced the SuperDoc agent's execution configuration by increasing the `maxConcurrency` from 1 to 5, allowing for more efficient concurrent test execution. Updated the cleanup process to ensure isolated state directories are properly managed, improving resource handling during tests. Adjusted execution tests to reflect these changes, ensuring robust validation of document editing capabilities.

* feat(evals): refactor SuperDoc agent evaluation scripts and enhance tool configuration

Refactored the SuperDoc agent's evaluation scripts to streamline the execution process and improve clarity. Removed the deprecated cross-provider configuration and consolidated tool evaluation logic into a unified structure. Introduced new assertion checks for tool quality and argument accuracy, ensuring comprehensive validation of document editing tasks. Updated the test suite to reflect these changes, enhancing overall test coverage and reliability.

* feat(evals): add AI Gateway support and new execution configuration for SuperDoc agent

Introduced the AI Gateway API key in the environment configuration to enable optional integration with Vercel AI Gateway. Added a new script command for executing evaluations through the gateway, enhancing the SuperDoc agent's capabilities. Created a new YAML configuration file for execution tests via the AI Gateway, allowing for testing across multiple models. Updated the package dependencies to include the necessary SDK for AI Gateway functionality.

* feat(evals): enhance SuperDoc agent with usage tracking and new customer prompt tests

Updated the SuperDoc agent to include tracking of total usage and steps during text generation, improving performance insights. Added a series of customer prompt tests in YAML format to validate various document editing tasks, ensuring comprehensive coverage of real-world scenarios. This enhancement aims to bolster the agent's capabilities and testing framework.

* feat(evals): streamline evaluation configuration and remove deprecated files

Removed the JavaScript assertion file and context builder, simplifying the evaluation framework. Updated the prompt configuration to eliminate unused metrics and added new document fixtures for testing. Enhanced execution tests to validate document editing capabilities with the new fixtures, ensuring comprehensive coverage of various scenarios.

* fix(evals): update model labels and refine execution test descriptions

Updated the model labels in the execution gateway configuration for clarity and accuracy. Refined the execution test descriptions to better reflect the specific tasks being validated, enhancing the readability and intent of the tests. Commented out deprecated Google provider configurations to streamline the YAML files.

* chore(evals): clean up evaluation configurations and remove obsolete files

Updated the .gitignore to exclude temporary files and removed deprecated YAML configuration files related to GDPval and execution tests. Streamlined the package.json by eliminating unused evaluation scripts, enhancing overall project organization and clarity.

* chore(evals): update pnpm-lock.yaml and .gitignore for dependency management

Updated pnpm-lock.yaml to reflect new versions of dependencies, including @types/node and added new SDK entries for SuperDoc. Modified .gitignore to exclude additional temporary files and states, improving project cleanliness and organization.

* feat(evals): implement caching mechanism for SuperDoc agent evaluations

Added a caching system to the SuperDoc agent and gateway providers to improve performance by storing and retrieving results based on a generated cache key. Updated the utility functions to handle cache operations, ensuring efficient reuse of previous evaluation results. Modified the evaluation logic to check for cached results before executing tasks, enhancing overall efficiency in the evaluation framework. Additionally, updated the package.json to reflect changes in evaluation scripts and added a new YAML configuration for end-to-end tests via the AI Gateway.

* feat(evals): expand evaluation framework with two-level testing and enhanced documentation

Updated the evaluation framework to include two levels of testing: tool quality and execution. Enhanced the README to clarify testing processes, commands, and configurations. Introduced new YAML files for tool quality and execution tests, detailing the number of tests and providers involved. Improved command descriptions for better usability and added new document fixtures for comprehensive testing of document editing capabilities.

* feat(evals): add Vercel tools provider and enhance evaluation scripts

Introduced a new Vercel tools provider for the SuperDoc evaluation framework, enabling structured tool calls with the Vercel AI SDK. Updated the package.json to include a new script for evaluating tools with the Vercel configuration. Enhanced the prompt configuration by adding a new YAML file for tool evaluations and refined existing evaluation scripts to support the new provider. Additionally, made minor adjustments to the presentation HTML for improved accessibility and clarity.

* chore(deps): update pnpm-lock.yaml to remove naive-ui and add @superdoc/common dependency

Removed outdated naive-ui entries and added @superdoc/common as a workspace dependency in pnpm-lock.yaml, ensuring the project reflects the latest dependency structure.

* feat(evals): enhance evaluation scripts and update Vercel tools provider

Refined evaluation scripts in package.json to output results to specific JSON files for better organization. Updated the Vercel tools provider to support live discovery of tools and improved error handling. Enhanced YAML configurations for tool evaluations, including clearer descriptions and adjustments to thresholds for tool-call metrics. Added caching functionality to optimize performance and ensure efficient reuse of evaluation results.

* fix: remove unused Vercel AI SDK evaluation configuration file and update tool quality test descriptions for clarity and consistency

* feat: add analysis functionality for eval results and update tool quality tests

- Introduced a new script `analyze-results.mjs` for generating a visual HTML dashboard from evaluation results using the Claude Agent SDK.
- Added new npm scripts: `analyze` and `eval:analyze` for easier result analysis.
- Updated `package.json` to include the `@anthropic-ai/claude-agent-sdk` dependency.
- Enhanced tool quality tests to allow for node search using either `query_match` or `blocks_list` for improved flexibility in evaluations.

* chore(deps): update @types/node version across multiple dependencies in pnpm-lock.yaml

- Updated the version of @types/node from 25.5.0 to 22.19.2 for various dependencies to ensure compatibility and reduce potential conflicts.
- Added the @anthropic-ai/claude-agent-sdk dependency with version 0.2.76.
- Adjusted the version of esbuild in the vitest dependency to 0.27.2.

* fix(evals): address code review findings in eval assertions and caching

- correctFormatArgs: validate args.inline on all format.apply steps, not just bold
- nodeSearchOrBlocksList: enforce expected nodeType match for both query_match and blocks_list
- compare-baselines: match tests by identity (description+provider+prompt) instead of array index
- cacheKey: include prompt hash so prompt changes invalidate cached results
- Remove unused eval:tools script referencing deleted config

Author: tupizz

.gitignore

@@ -71,6 +71,7 @@ perf-baseline-results.json
 .claude
 plans/
 
+tests/layout-snapshots
 tests/layout/candidate/
 tests/layout/reference/
 tests/layout/reports/

CLAUDE.md

@@ -112,6 +112,24 @@ Many packages use `.js` files with JSDoc `@typedef` for type definitions (e.g.,
 - `pnpm dev` - Start dev server (from examples/)
 - `pnpm run generate:all` - Generate all derived artifacts (schemas, SDK clients, tool catalogs, reference docs)
 
+## AI Eval Suite
+
+The `evals/` directory contains a Promptfoo-based evaluation suite for validating AI tool call quality.
+
+| Command | What it does | Cost |
+|---------|-------------|------|
+| `pnpm --filter @superdoc-testing/evals run eval` | Run deterministic evals (reading + argument tests) | ~$0.30 |
+| `pnpm --filter @superdoc-testing/evals run eval:reading` | Run reading tool tests only | ~$0.15 |
+| `pnpm --filter @superdoc-testing/evals run eval:gdpval` | Run GDPval benchmark (Model+SuperDoc vs Model-Only) | ~$1-2 |
+| `pnpm --filter @superdoc-testing/evals run eval:view` | Open Promptfoo web UI with results | Free |
+| `pnpm --filter @superdoc-testing/evals run baseline:save <label>` | Save versioned results snapshot | Free |
+
+Tool definitions are extracted from `packages/sdk/tools/` via `evals/tools/extract.mjs`. Run `pnpm run generate:all` first if SDK artifacts are missing.
+
+Test files are YAML in `evals/tests/`. Each test has a `vars.task` prompt and JavaScript assertions that check tool call structure (Level 1: tool selection + argument accuracy, not execution).
+
+The system prompt at `evals/prompts/agent.txt` is a copy of the proven prompt from `examples/eval-demo/lib/agent.ts`. Update both when changing the prompt.
+
 ## Generated Artifacts
 
 These directories are produced by `pnpm run generate:all`:

evals/.env.example

@@ -0,0 +1,4 @@
+OPENAI_API_KEY=sk-...
+ANTHROPIC_API_KEY=sk-ant-...
+GOOGLE_API_KEY=...
+AI_GATEWAY_API_KEY=...

evals/.gitignore

@@ -0,0 +1,9 @@
+lib/essential.json
+results/
+.promptfoo/
+node_modules/
+.env
+.env.local
+__*
+fixtures/.state*
+fixtures/tmp-*

evals/README.md

@@ -0,0 +1,175 @@
+# SuperDoc AI Eval Suite
+
+Tests whether LLMs correctly use SuperDoc's 193 document editing tools. Two levels: tool quality (does the model pick the right tool?) and execution (does the document actually change?).
+
+## Quick start
+
+```bash
+cp .env.example .env           # add AI_GATEWAY_API_KEY (+ optional OPENAI_API_KEY)
+pnpm run extract-tools         # extract tool definitions from SDK (run once after clone)
+pnpm run eval                  # Level 1: tool quality (6 providers, 47 tests)
+pnpm run eval:e2e              # Level 2: execution (3 providers, 21 real DOCX tests)
+pnpm run eval:view             # open results in browser
+```
+
+## Two levels of testing
+
+### Level 1: Tool Quality
+
+Give the LLM a task and tool definitions. Check the response: did it pick the right tool with valid arguments? No document execution. Fast, cheap.
+
+- **47 tests** across 9 categories (reading, mutations, formatting, structure, tables, comments, tracked changes, lists, hygiene)
+- **6 providers** via Vercel AI Gateway: GPT-4o, GPT-4.1, GPT-4.1-mini, GPT-5.4, Claude Haiku 4.5, Gemini 2.5 Flash
+- Config: `promptfooconfig.yaml`
+
+### Level 2: Execution (E2E)
+
+Run the full agent loop on real .docx files. Open document, LLM picks tools, CLI executes them. Assert the document content changed correctly.
+
+- **21 tests** on 3 fixture documents (document.docx, memorandum.docx, table-doc.docx)
+- **3 providers** via Vercel AI SDK + AI Gateway: GPT-5.4, Claude Haiku 4.5, Gemini 2.5 Pro
+- Config: `promptfooconfig.e2e.yaml`
+- Provider: `providers/superdoc-agent-gateway.mjs` (uses `generateText()` + `jsonSchema()` + `stepCountIs(10)`)
+
+## Commands
+
+| Command | What it does |
+|---------|-------------|
+| `pnpm run eval` | Level 1: tool quality (all providers) |
+| `pnpm run eval:e2e` | Level 2: execution via AI Gateway |
+| `pnpm run eval:openai` | Level 1 filtered to OpenAI models only |
+| `pnpm run eval:view` | Open Promptfoo web UI with results |
+| `pnpm run eval:export` | Run eval + save to `results/latest.json` |
+| `pnpm run eval:repeat` | Run 3x, no cache (variance testing) |
+| `pnpm run extract-tools` | Re-extract tools from SDK |
+| `pnpm run baseline:save` | Snapshot current results for comparison |
+| `pnpm run baseline:compare` | Compare two snapshots for regressions |
+
+## Structure
+
+```
+evals/
+  promptfooconfig.yaml              Level 1: tool quality (6 providers via AI Gateway)
+  promptfooconfig.e2e.yaml          Level 2: execution (3 providers via AI Gateway)
+  prompts/
+    agent.txt                       System prompt (127 lines, tool categories + API guide)
+    minimal.txt                     Minimal baseline prompt (for GDPval)
+  tests/
+    tool-quality.yaml               47 tests: tool selection + argument validation
+    execution.yaml                  21 tests: real DOCX editing + content assertions
+  providers/
+    superdoc-agent-gateway.mjs      Vercel AI SDK provider (any model via config.modelId)
+    superdoc-agent.mjs              OpenAI-only provider (legacy, direct API)
+    utils.mjs                       Shared: SDK loading, file management, caching
+  lib/
+    checks.cjs                      18 assertion functions (noHallucinatedParams, validOpNames, etc.)
+    normalize.cjs                   Cross-provider tool call format normalization
+    extract.mjs                     SDK tool extraction script
+    essential.json                  Extracted tool definitions (6 essential + discover_tools)
+    save-baseline.mjs               Save versioned result snapshot
+    compare-baselines.mjs           Compare two snapshots for regressions
+  fixtures/
+    document.docx                   Bullet lists (read, replace, insert)
+    memorandum.docx                 Legal memo (dates, amounts, party names)
+    table-doc.docx                  Tables (headers, cell content)
+    contract.docx                   Long contract (future stress tests)
+    comments-doc.docx               Document with comments (future)
+  results/
+    latest.json                     Most recent eval output
+    .cache/                         Response cache (keyed by model+fixture+task)
+    baselines/                      Versioned snapshots
+    output/                         Saved DOCX files from keepFile tests
+```
+
+## Writing tests
+
+### Tool quality test (Level 1)
+
+```yaml
+- description: 'Replace uses text.rewrite, not bare replace'
+  metadata: { category: mutation }
+  vars:
+    task: 'Replace "old title" with "new title" in the document.'
+  assert:
+    - type: tool-call-f1
+      value: [query_match, apply_mutations]
+      threshold: 0.5
+      metric: tool_selection
+    - type: javascript
+      value: file://lib/checks.cjs:validOpNames
+      metric: argument_accuracy
+    - type: javascript
+      value: file://lib/checks.cjs:noHallucinatedParams
+      metric: argument_accuracy
+```
+
+`tool-call-f1` checks tool selection (F1 score). `file://lib/checks.cjs:functionName` runs a specific assertion function.
+
+### Execution test (Level 2)
+
+```yaml
+- description: 'Replace: $25M to $50M, $150M untouched'
+  vars:
+    fixture: memorandum.docx
+    task: 'Replace "$25,000,000" with "$50,000,000".'
+  assert:
+    - type: contains
+      value: '$50,000,000'
+    - type: not-contains
+      value: '$25,000,000'
+    - type: contains
+      value: '$150,000,000'
+```
+
+Every execution test asserts: new content exists, old content gone, unrelated content intact.
+
+## Assertion functions (`lib/checks.cjs`)
+
+| Function | What it checks |
+|----------|---------------|
+| `noHallucinatedParams` | No `doc` or `sessionId` in tool args |
+| `validOpNames` | Ops are `text.rewrite`/`text.insert`/`text.delete`, not bare `replace`/`insert`/`delete` |
+| `stepFields` | Every step has `op` and `where` |
+| `noRequireAny` | Mutations use `require: "first"`, not `"any"` |
+| `noMixedBatch` | No `text.rewrite` + `format.apply` in same batch |
+| `correctFormatArgs` | `format.apply` uses `{inline: {bold: true}}`, not `{bold: true}` |
+| `textSearchArgs` | `query_match` has `select.type: "text"` + `select.pattern` |
+| `nodeSearchArgs` | `query_match` has `select.type: "node"` + correct `nodeType` |
+| `noTextInsertForStructure` | Headings/paragraphs use standalone tools, not `text.insert` |
+| `validDiscoverGroups` | `discover_tools` groups are valid names |
+| `isTrackedMode` | Tracked changes set `changeMode: "tracked"` |
+| `isNotTrackedMode` | Direct edits do not set `changeMode: "tracked"` |
+| `atomicMultiStep` | Multi-step has `atomic: true` + 2+ steps |
+
+Each function: `(output, context) => { pass, score, reason }` or `true` (skip).
+
+## Adding a new model
+
+Add a provider to any config YAML:
+
+```yaml
+# In promptfooconfig.yaml (Level 1)
+- id: vercel:anthropic/claude-sonnet-4.6
+  label: Claude Sonnet 4.6
+  delay: 1000
+  config:
+    temperature: 0
+    tools: file://lib/essential.json
+    maxTokens: 1024
+
+# In promptfooconfig.e2e.yaml (Level 2)
+- id: file://providers/superdoc-agent-gateway.mjs
+  label: Claude Sonnet 4.6 (Gateway)
+  config:
+    modelId: anthropic/claude-sonnet-4.6
+```
+
+## Notes
+
+- All providers route through **Vercel AI Gateway** (`AI_GATEWAY_API_KEY`). One key, all models.
+- Run `pnpm run generate:all` from repo root if `extract-tools` fails (SDK artifacts need regenerating).
+- `prompts/agent.txt` is the canonical system prompt. Update it when changing tool documentation.
+- Promptfoo caches responses. Changing assertions re-runs on cached data for free. Clear: `npx promptfoo cache clear`.
+- `normalize.cjs` converts Anthropic `tool_use` and Google `functionCall` formats to OpenAI format so all assertions work across providers.
+- Execution provider caches results in `results/.cache/` (keyed by model+fixture+task). Disable: `PROMPTFOO_CACHE_ENABLED=false`.
+- Files prefixed with `__` (e.g. `__promptfooconfig.gdpval.yaml`) are disabled/legacy configs kept for reference.