Skip to content

ci(evals): run eval workflow on OpenRouter - #1220

Merged
HazAT merged 2 commits into
mainfrom
ci/eval-openrouter-key
Jul 31, 2026
Merged

ci(evals): run eval workflow on OpenRouter#1220
HazAT merged 2 commits into
mainfrom
ci/eval-openrouter-key

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Moves the eval CI job and eval harness off a direct OpenAI key onto OpenRouter, so GitHub Actions no longer needs OPENAI_API_KEY for evals.

Changes

  • .github/workflows/eval.yml now injects OPENROUTER_API_KEY and EMBEDDED_AGENT_PROVIDER=openrouter
  • eval scorers/task runners default to getOpenRouterModel() instead of hardcoding openai("gpt-4o")
  • MSW unhandled-request filters also allow openrouter.ai traffic during evals
  • docs/comments updated to mention OPENROUTER_API_KEY

Follow-up

Test plan

  • pnpm --filter @sentry/mcp-server-evals exec tsc -b --force
  • After secret is set, confirm Eval workflow can call OpenRouter successfully

Requested by Daniel Griesser.

--

View Junior Session Sentry

Switch the GitHub Actions eval job and eval harness defaults from
OPENAI_API_KEY/gpt-4o to OPENROUTER_API_KEY so CI no longer depends on
a direct OpenAI key.

Co-Authored-By: Daniel Griesser <dgriesser@sentry.io>
@HazAT
HazAT marked this pull request as ready for review July 31, 2026 11:48

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 30f9fe2. Configure here.

Comment thread packages/mcp-server-evals/src/evals/utils/toolPredictionScorer.ts
@HazAT
HazAT merged commit 6ee17bf into main Jul 31, 2026
20 checks passed
@HazAT
HazAT deleted the ci/eval-openrouter-key branch July 31, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant