test(evals): Migrate to vitest-evals harness - #1132
Draft
dcramer wants to merge 8 commits into
Draft
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
dcramer
force-pushed
the
dcramer/test/migrate-vitest-evals-harness
branch
from
June 27, 2026 18:49
8507ed7 to
da42106
Compare
Move the eval suite to the current vitest-evals harness and judge APIs, using the existing embedded-agent provider factory so evals run against the configured provider credentials. Update the eval docs and mocks for the OpenRouter-backed path. Co-Authored-By: GPT-5 Codex <codex@openai.com>
Keep agent evals on the same default judge threshold as the other eval helpers and make explicit Sentry syntax deterministic without requiring a discovery call. Co-Authored-By: GPT-5 Codex <codex@openai.com>
Describe the whoami value as an email address instead of a prequoted Sentry search token. This keeps issue event search guidance consistent with the rule to avoid quoting simple email values. Co-Authored-By: GPT-5 Codex <codex@openai.com>
Allow MSW test setup to bypass OpenAI, Anthropic, OpenRouter, and Azure OpenAI provider calls while continuing to fail on unmocked product API requests. Co-Authored-By: GPT-5 Codex <codex@openai.com>
Restore targeted datasetAttributes filters for custom field discovery while keeping common-field search paths tool-free. Update stdio testing docs to use the supported Node 22.13 minimum and a newer Node 24 check instead of Node 20. Co-Authored-By: Codex <noreply@openai.com>
Handle the AI SDK streaming response union before asserting the non-streaming agent tool result. Co-Authored-By: OpenAI Codex <noreply@openai.com>
dcramer
force-pushed
the
dcramer/test/migrate-vitest-evals-harness
branch
from
July 28, 2026 21:47
609a077 to
6a699d0
Compare
Treat malformed or relative request URLs as non-provider traffic so MSW can report the intended unhandled request error. Co-Authored-By: OpenAI Codex <noreply@openai.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 39d5d8d. Configure here.
Expect substring and numeric-type filters in search-event agent discovery calls so strict eval matching rewards the current prompt guidance. Co-Authored-By: OpenAI Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The eval suite now uses the current vitest-evals harness and judge APIs instead of the old scorer-style wiring. The shared helpers keep the eval files small while preserving the existing expected tool calls and structured output checks.
Eval model setup now goes through the existing embedded-agent provider factory, so OpenRouter-backed credentials work the same way as the runtime AI tools. The PR also tightens the affected search prompts from eval traces and updates the eval docs, mocks, and suite-wide timeout configuration to match the new harness.