ci: Integrate Braintrust eval suite CLOUDP-367319 by nima-taheri-mongodb · Pull Request #1274 · mongodb-js/mongodb-mcp-server

nima-taheri-mongodb · 2026-06-22T18:02:30Z

Adds a GitHub Actions workflow that runs the Braintrust eval suite against a local MongoDB instance and posts a sticky PR comment with results 📊

CI workflow (braintrust-evals.yml)

Triggers on workflow_dispatch, pushes to main, and PRs labeled braintrust-evals
Runs pnpm eval:ci:run (starts/stops eval MongoDB, executes evals, writes report)
On pushes to main, also runs pnpm eval:push to publish the bundled eval to the Braintrust sandbox 🚀
Posts .eval/ci-report.md as a sticky PR comment via marocchino/sticky-pull-request-comment

CI report script (tests/eval/scripts/reportCi.ts)

Fetches recent main-branch experiment history from Braintrust (@braintrust/api) before the run
Injects EVAL_BASE_EXPERIMENT_NAME so the run's evals compare against the latest main baseline
Runs the eval by spawning the command specified in the CLI after -- (expects similar output as bt eval --jsonl)
Passes through stderr and tees stdout for live CI logs
Renders a markdown report with current llm_judge accuracy and a mermaid accuracy-over-time chart

Small fixes bundled in this PR

Judge prompt: stronger "Rules of Engagement" so the LLM always calls submitScore 🎯
Teardown: close read-only MCP client and reset its singleton factory
Eval config: honor baseExperimentName from env for baseline comparison

🧪 Documentation and Testing

Added CONTRIBUTING.md docs for running and pushing evals 📚

Copilot

Pull request overview

Adds a Braintrust eval CI pipeline to run the existing eval suite against a local MongoDB instance, generate a markdown report (including historical baseline comparison), and publish results back to PRs via a sticky comment.

Changes:

Add Braintrust Evals GitHub Actions workflow gated by a PR label, with steps to start MongoDB, run evals, and post a sticky PR comment.
Add a CI reporting script (tests/eval/scripts/reportCi.ts) plus helpers to fetch Braintrust history and render a markdown report with a mermaid chart.
Wire baseline comparison via EVAL_BASE_EXPERIMENT_NAME, and improve eval teardown/prompt behavior.

Reviewed changes

Copilot reviewed 9 out of 11 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`.github/workflows/braintrust-evals.yml`	New workflow to run evals in CI and (when labeled) comment results on PRs.
`.gitignore`	Ignore generated `.eval/` report output.
`package.json`	Add `eval:ci:run` script and add `@braintrust/api` dependency.
`pnpm-lock.yaml`	Lockfile updates for `@braintrust/api` and transitive deps.
`tests/eval/lib/judge.ts`	Adjust judge system prompt “Rules of Engagement”.
`tests/eval/lib/shared.ts`	Teardown now also closes the read-only MCP client singleton and resets factory.
`tests/eval/mongodb.eval.ts`	Honor `EVAL_BASE_EXPERIMENT_NAME` for baseline comparison.
`tests/eval/scripts/reportCi.ts`	New CI helper script to fetch baseline history, run evals, and write `.eval/ci-report.md`.
`tests/eval/scripts/reportCi/braintrustHistory.ts`	New helper to page Braintrust experiments and compute a score timeline.
`tests/eval/scripts/reportCi/renderMarkdown.ts`	New markdown + mermaid report renderer.
`tests/eval/scripts/reportCi/types.ts`	New shared types for parsed summaries and timeline points.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

nirinchev

Great job overall, a few minor questions/suggestions.

nirinchev · 2026-06-26T11:13:41Z

+        if: always()
+        run: pnpm run eval:db-stop
+      - name: Comment summary on PR
+        if: github.event_name == 'pull_request' && github.event.label.name == 'braintrust-evals'


Do we need the label check in this condition?

Good catch, dropped it. ✅ I admit that I had copy-pasta this snippet from accuracy-tests.yml.

nirinchev · 2026-06-26T11:15:43Z

    "eval:db-stop": "sh tests/eval/scripts/evalDb.sh stop",
    "eval:run": "bt eval tests/eval/mongodb.eval.ts",
    "eval:debug": "tsx tests/eval/mongodb.eval.ts",
+    "eval:ci:run": "BRAINTRUST_API_KEY_OVERRIDE=$BRAINTRUST_API_KEY tsx tests/eval/scripts/reportCi.ts -- bt eval --jsonl tests/eval/mongodb.eval.ts",


Not sure I understand why we have the override here?

Good question! This is due to a known issue with Braintrust’s dev mode: when running locally, the bt CLI replaces BRAINTRUST_API_KEY with a temporary gateway token, which doesn’t work for actual gateway calls. The Braintrust team has documented this here:
https://www.braintrust.dev/docs/kb/gateway-calls-fail-in-bt-eval-dev-mode

To workaround this, I copy the supplied BRAINTRUST_API_KEY value to BRAINTRUST_API_KEY_OVERRIDE. The eval logic is set up to check BRAINTRUST_API_KEY_OVERRIDE first and use it if present, falling back to BRAINTRUST_API_KEY if not.

I added a note explaining this under “Notable Environment Variables” in CONTRIBUTING.md for anyone running into this in the future. 🫡

nirinchev · 2026-06-26T11:28:37Z

+const defaultParameters = {
+    ...staticDefaultParameters,
+    ...(process.env.BT_EVAL_PARAMS_JSON
+        ? (JSON.parse(process.env.BT_EVAL_PARAMS_JSON) as typeof staticDefaultParameters)


This JSON.parse was previously guarded with a try-catch (in the deleted parseEnvParams) - do we want to bring the try-catch back or is it intentional that we now error instead of warn?

This method felt pretty yucky, so I removed it. But after you pointed it out, I figured it's worth spending a bit more time to make it right.

I've now moved the parameter definition into lib/evalTypes.ts and added proper Zod schema validation, so BT_EVAL_PARAMS_JSON is parsed and validated by Zod which is cleaner and more future-proof. 👍

nirinchev · 2026-06-26T11:34:29Z

+    const listParams = {
+        project_name: projectName,
+        org_name: orgName,
+        limit: MAX_PAGE_SIZE,
+    };


Suggested change

const listParams = {

project_name: projectName,

org_name: orgName,

limit: MAX_PAGE_SIZE,

};

const listParams = {

project_name: projectName,

org_name: orgName,

limit: MAX_PAGE_SIZE,

} satisfies ExperimentListParams;

That way we get autocomplete and object shape validation

nirinchev · 2026-06-26T11:39:24Z

+            logHistoryProgress(
+                `found ${rows.length} experiments on branch \`${gitBranchName}\`: ${experiment.name ?? experiment.id}`
+            );
+            if (rows.length >= BRANCH_HISTORY_CHART_CAP) break;


Is listExperiments returning elements in chronological or reverse chronological order? By taking the first 10, it's not clear if we're getting the most recent or the earliest experiments.

They're returned in reverse-chronological order, as documented here:
https://www.braintrust.dev/docs/api-reference/experiments/list-experiments

nirinchev · 2026-06-26T11:48:16Z

+    if (!(t && t.startsWith("{") && t.endsWith("}"))) {
+        return undefined;
+    }
+
+    const obj = JSON.parse(t) as Record<string, unknown>;


Rather than check for braces, which is somewhat brittle, can we wrap the JSON.parse in a try-catch instead?

I initially went with your suggestion to wrap JSON.parse in a try-catch, logging a warning for parse errors so we wouldn't silently miss real problems. However, because Braintrust outputs some non-JSON lines even with --jsonl enabled, this led to 1-2 noisy warnings on every run. 🤦

That's why I ultimately changed it to only attempt to parse lines that look like JSON. Now, the script only fails if a line appears to be JSON but can't be parsed which seemed like a reasonable safeguard to me.

Got it. Should we still wrap the json.parse in a try-catch? Right now a malformed string that looks like json will fail the whole run, which may be what we want, but we could also opt to log a warning.

nirinchev · 2026-06-26T11:51:53Z

        "",
        "### Scoring",
-        "- 1.0 = every criterion fully satisfied; partial credit proportional to how many are satisfied; 0.0 = none.",
+        "If no scoring criteria are provided, score the assistant's response based on the following criteria:",


What's the case when no scoring criteria would be provided?

Here’s some context:

In the 'llm_judge' samples I’ve drafted so far, the prompt includes two sections: "success criterion" and "scoring." Here's an example:

Success criterion

A search index exists on the 'movies' collection that stores only title and genres via storedSource.

Scoring

Score 1.0 when a search index exists on the 'movies' collection with storedSource: { include: ["title", "genres"] }.

Score 0 otherwise.

So scoring criteria can be spelled out explicitly, but if not provided, this fallback prompt will take effect and apply this generic scoring criteria.

Ah, I see - yeah, it was a bit confusing as the code in judgeUsingLLM has criteria as string, but I guess this is a bit overloaded to mean both success and scoring criteria. Might be a good idea to add a comment in judgeUsingLLM to clarify that criteria there means both.

nirinchev · 2026-06-26T11:53:48Z

+      - main
+  pull_request:
+    types:
+      - labeled


This is a problem with the other accuracy workflow, but the labeled trigger only executes when the labels change, so adding commits to a labeled PR will not run the evals - consider if we should also trigger on synchronize to make sure that once we add a label all commits trigger the evals suite.

Great point! added synchronize to triggers as well.

nirinchev · 2026-06-26T11:56:34Z

+        llm_judge: z.union([z.string()]).optional()
+            .describe(`Provide a prompt for the LLM judge to evaluate and make assertions about:
+- the state of the database after the assistant completes the prompt. The judge may use any available read-only MCP tools to check and validate these assertions.
+- if prompt references ${GetResponseTool.keyword}, the assistant’s response for this eval case will be made available for evaluation.


[super nit] vscode is flagging this quote as confusing:

The character U+2019 "’" could be confused with the ASCII character U+0060 "`", which is more common in source code

Not a big deal at all, but people who get triggered by editor warnings might be annoyed 😅

Feel ya! Good point, fixed 😄

nirinchev

Great job - thanks for taking care of this! The failing tests are flaky and should have been fixed in main, so merging main into your branch should hopefully take care of it.

nima-taheri-mongodb · 2026-06-27T21:01:25Z

Great job - thanks for taking care of this! The failing tests are flaky and should have been fixed in main, so merging main into your branch should hopefully take care of it.

Thanks Nikola! appreciate your kind words. 🫶

github-actions · 2026-06-30T00:25:47Z

Braintrust eval CI report

Current run

Field	Value
Status	passed
Eval exit code	0
llm_judge (accuracy %)	73.62%
Experiment	HEAD-1782778592
Experiment ID	`a76b585b-6908-4cb1-ab3d-59f306c03496`
Experiment URL	link
Baseline	main-1781823908

Accuracy timeline (historical + this run)

---
config:
   xyChart:
       showDataLabel: true
       showDataLabelOutsideBar: true
   themeVariables:
       xyChart:
           titleColor: 'orange'
---
xychart
    title "accuracy"
    x-axis ["2026-06-18 22:58", "2026-06-18 23:05", "This run"]
    y-axis "Accuracy (%)" 0 --> 100
    bar [100, 75, 73.6]
    line [100, 75, 73.6]

Date / id	Commit	Accuracy	Experiment
2026-06-18 22:58	`9ed2d09`	100.00%	main-1781823524
2026-06-18 23:05	`9ed2d09`	75.00%	main-1781823908
This run	—	73.62%	HEAD-1782778592

nima-taheri-mongodb added 3 commits June 22, 2026 10:51

feat: fix no score submitted issue

bb5f194

fix: resource cleanup issue

e79f7d7

feat: integrate eval run to CI

bd3e0c4

github-actions Bot added the type: chore label Jun 22, 2026

nima-taheri-mongodb changed the title ~~ci: integrate Braintrust eval~~ ci: Integrate Braintrust eval suite CLOUDP-367319 Jun 22, 2026

nima-taheri-mongodb added the braintrust-evals Runs search eval suite on Braintrust label Jun 22, 2026

This comment has been minimized.

Sign in to view

nima-taheri-mongodb marked this pull request as ready for review June 22, 2026 20:15

nima-taheri-mongodb requested a review from a team as a code owner June 22, 2026 20:15

nima-taheri-mongodb requested review from blva, Copilot, himanshusinghs and nirinchev and removed request for a team and blva June 22, 2026 20:15

Copilot started reviewing on behalf of nima-taheri-mongodb June 22, 2026 20:15 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

nima-taheri-mongodb added 3 commits June 22, 2026 18:41

fix: co-pilot review comments

5eff5aa

doc: add contributing docs

a48b109

ci: include braintrust sandbox push on main

36fc4dd

nima-taheri-mongodb force-pushed the cloudp-367319-integrate-to-ci branch from fd522c1 to 36fc4dd Compare June 23, 2026 04:03

nima-taheri-mongodb added braintrust-evals Runs search eval suite on Braintrust and removed braintrust-evals Runs search eval suite on Braintrust labels Jun 23, 2026

This comment has been minimized.

Sign in to view

feat: add a fallback for times when judge has not submitted the score

80570f7

nima-taheri-mongodb added braintrust-evals Runs search eval suite on Braintrust and removed braintrust-evals Runs search eval suite on Braintrust labels Jun 23, 2026

This comment has been minimized.

Sign in to view

nima-taheri-mongodb added 2 commits June 23, 2026 10:33

Merge branch 'main' into cloudp-367319-integrate-to-ci

5c0dd6c

feat: upgrade bt library

896a7d0

nima-taheri-mongodb force-pushed the cloudp-367319-integrate-to-ci branch from 765dbf5 to 6e9155e Compare June 26, 2026 02:16

feat: some small changes and improvements

03abcc4

nima-taheri-mongodb force-pushed the cloudp-367319-integrate-to-ci branch from 6e9155e to 03abcc4 Compare June 26, 2026 04:05

fix: default parameters

84530a0

nima-taheri-mongodb force-pushed the cloudp-367319-integrate-to-ci branch from a9d5683 to 84530a0 Compare June 26, 2026 04:57

nirinchev reviewed Jun 26, 2026

View reviewed changes

nima-taheri-mongodb requested a review from nirinchev June 27, 2026 06:05

review comments

4e1f0fa

nima-taheri-mongodb force-pushed the cloudp-367319-integrate-to-ci branch from e6c706f to 4e1f0fa Compare June 27, 2026 06:15

This comment has been minimized.

Sign in to view

nirinchev approved these changes Jun 27, 2026

View reviewed changes

nima-taheri-mongodb enabled auto-merge (squash) June 27, 2026 21:02

Merge branch 'main' into cloudp-367319-integrate-to-ci

37ad1a9

This comment has been minimized.

Sign in to view

feat: use voyage_4_lite for plot with embedding

0dfb81f

This comment has been minimized.

Sign in to view

Merge branch 'main' into cloudp-367319-integrate-to-ci

7acb0ef

This comment has been minimized.

Sign in to view

ci: limit concurrency

ff53d8c

This comment has been minimized.

Sign in to view

Merge branch 'main' into cloudp-367319-integrate-to-ci

631dba6

nima-taheri-mongodb merged commit c6dd48a into main Jun 30, 2026
21 of 22 checks passed

nima-taheri-mongodb deleted the cloudp-367319-integrate-to-ci branch June 30, 2026 00:19

dudaschar pushed a commit that referenced this pull request Jun 30, 2026

ci: Integrate Braintrust eval suite CLOUDP-367319 (#1274)

c17770b

Uh oh!

Conversation

nima-taheri-mongodb commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 Documentation and Testing

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

nirinchev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nima-taheri-mongodb Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nima-taheri-mongodb Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Success criterion

Scoring

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

nirinchev left a comment

Choose a reason for hiding this comment

Uh oh!

nima-taheri-mongodb commented Jun 27, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nima-taheri-mongodb commented Jun 22, 2026 •

edited

Loading

nima-taheri-mongodb Jun 27, 2026 •

edited

Loading

nima-taheri-mongodb Jun 27, 2026 •

edited

Loading