Skip to content

fix!: build judge input as string and strip legacy judge config messages#1364

Merged
jsonbailey merged 1 commit into
feat/next-ai-releasefrom
jb/fix-judge-string-input
May 7, 2026
Merged

fix!: build judge input as string and strip legacy judge config messages#1364
jsonbailey merged 1 commit into
feat/next-ai-releasefrom
jb/fix-judge-string-input

Conversation

@jsonbailey

@jsonbailey jsonbailey commented May 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Aligns the JS AI SDK with python-server-sdk-ai#165 and sdk-specs#160.
  • Judges now pass a single formatted string (MESSAGE HISTORY:\n...\n\nRESPONSE TO EVALUATE:\n...) to the runner instead of an interpolated message list.
  • Legacy judge configs with {{message_history}} / {{response_to_evaluate}} placeholder messages are stripped at config construction so old and new flag values continue to work.
  • Runner.run is narrowed to (input: string, outputType?) everywhere — server-ai, server-ai-openai, server-ai-langchain, server-ai-vercel.

Breaking changes

  • Runner.run(input: string | LDMessage[], outputType?)Runner.run(input: string, outputType?). Provider runners (OpenAI, LangChain, Vercel) no longer accept a pre-built LDMessage[]. They always prepend any config messages and append the prompt as a user turn.
  • Judge no longer Mustache-interpolates {{message_history}} / {{response_to_evaluate}} into config messages. The SDK builds the input string and passes it verbatim to the runner.
  • Judge.evaluate no longer early-returns with "Judge configuration must include messages" — a judge with no messages now proceeds to invoke the runner with the formatted input string.

Test plan

  • yarn workspace @launchdarkly/server-sdk-ai test — 239 passing
  • yarn workspace @launchdarkly/server-sdk-ai-openai test — 39 passing
  • yarn workspace @launchdarkly/server-sdk-ai-langchain test — 38 passing
  • yarn workspace @launchdarkly/server-sdk-ai-vercel test — 20 passing
  • lint clean for all four packages
  • yarn workspaces foreach -pR --topological-dev --from '@launchdarkly/server-sdk-ai' run build — all pass
  • Run chat-judge example end-to-end and confirm judge events emit correctly

🤖 Generated with Claude Code


Note

Medium Risk
Medium risk due to a breaking API change (Runner.run no longer accepts LDMessage[]) and behavioral changes in judge evaluation/config processing that could affect downstream integrations.

Overview
Breaking API change: Runner.run is narrowed from (string | LDMessage[]) to (string) across server-ai and provider packages, and provider runners (OpenAI/LangChain/Vercel) now always build the message list by prepending config messages and appending the prompt as a user turn.

Judge evaluation update: Judge stops Mustache-interpolating legacy templates into message arrays and instead builds a single evaluation string (MESSAGE HISTORY... / RESPONSE TO EVALUATE...) that it passes to the runner; it also no longer errors when judgeConfig.messages is missing.

Backward compatibility: LDAIClientImpl.judgeConfig now strips legacy judge placeholder messages containing {{message_history}}/{{response_to_evaluate}} before returning configs, with tests updated/added to cover the new input format and stripping behavior.

Reviewed by Cursor Bugbot for commit f4960cc. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor

@launchdarkly/js-sdk-common size report
This is the brotli compressed size of the ESM build.
Compressed size: 26281 bytes
Compressed size limit: 29000
Uncompressed size: 128971 bytes

@github-actions

github-actions Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor

@launchdarkly/js-client-sdk size report
This is the brotli compressed size of the ESM build.
Compressed size: 31867 bytes
Compressed size limit: 34000
Uncompressed size: 113634 bytes

@github-actions

github-actions Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor

@launchdarkly/browser size report
This is the brotli compressed size of the ESM build.
Compressed size: 179543 bytes
Compressed size limit: 200000
Uncompressed size: 830815 bytes

@github-actions

github-actions Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor

@launchdarkly/js-client-sdk-common size report
This is the brotli compressed size of the ESM build.
Compressed size: 38473 bytes
Compressed size limit: 39000
Uncompressed size: 211104 bytes

@jsonbailey jsonbailey force-pushed the jb/fix-judge-string-input branch 3 times, most recently from e4e5069 to f4960cc Compare May 7, 2026 14:52
@jsonbailey jsonbailey marked this pull request as ready for review May 7, 2026 15:03
@jsonbailey jsonbailey requested a review from a team as a code owner May 7, 2026 15:03
@jsonbailey jsonbailey changed the title fix!: build judge input as string and narrow Runner.run signature fix!: build judge input as string and strip legacy judge config messages May 7, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f4960cc. Configure here.

Comment thread packages/sdk/server-ai/src/LDAIClientImpl.ts
Aligns the JS AI SDK with the spec change implemented in
launchdarkly/python-server-sdk-ai#165 and launchdarkly/sdk-specs#160.

Judges now build a single formatted string ("MESSAGE HISTORY:\n...\n\n
RESPONSE TO EVALUATE:\n...") and pass it to the runner instead of an
interpolated message list. Legacy judge configs that contain
{{message_history}} or {{response_to_evaluate}} placeholders in
non-system messages are stripped at config-construction time so old and
new flag values both work without behavioral surprises.

BREAKING CHANGE: Runner.run is narrowed from
`run(input: string | LDMessage[], outputType?)` to
`run(input: string, outputType?)`. The OpenAI, LangChain, and Vercel
provider runners no longer accept a pre-built message array; they
always prepend any config messages and append the prompt as a user
turn. The Judge no longer interpolates {{message_history}} or
{{response_to_evaluate}} into config messages — the SDK builds the
input string directly and the runner receives that string verbatim.
The "Judge configuration must include messages" early-return was
removed; a judge with no messages now proceeds to invoke the runner
with the formatted input.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@jsonbailey jsonbailey merged commit 12a90c9 into feat/next-ai-release May 7, 2026
39 checks passed
@jsonbailey jsonbailey deleted the jb/fix-judge-string-input branch May 7, 2026 16:05
@github-actions github-actions Bot mentioned this pull request May 7, 2026
jsonbailey pushed a commit that referenced this pull request May 7, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>server-sdk-ai: 0.20.0</summary>

##
[0.20.0](server-sdk-ai-v0.19.1...server-sdk-ai-v0.20.0)
(2026-05-07)


### ⚠ BREAKING CHANGES

* Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens
([#1366](#1366))
* Remove AIProvider deprecated methods and create*/init* aliases
(AIC-2388)
([#1363](#1363))
* Build judge input as string and strip legacy judge config messages
([#1364](#1364))
* Use LDAIGraphMetricSummary for graph metric summary
([#1362](#1362))

### Features

* add Evaluator class for judge orchestration
([#1331](#1331))
([54faa69](54faa69))
* add ManagedAgent with evaluations support
([#1334](#1334))
([7f09c46](7f09c46))
* add ManagedGraphResult, GraphMetricSummary, and ManagedAgentGraph
([#1335](#1335))
([09fa1db](09fa1db))
* introduce ManagedResult, RunnerResult, and LDAIMetricSummary
([#1332](#1332))
([5040122](5040122))
* Remove AIProvider deprecated methods and create*/init* aliases
(AIC-2388)
([#1363](#1363))
([ad66314](ad66314))
* Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens
([#1366](#1366))
([ff932b7](ff932b7))
* Replace OpenAIProvider with Runner protocol implementation (AIC-2388)
([#1337](#1337))
([e32a955](e32a955))


### Bug Fixes

* Build judge input as string and strip legacy judge config messages
([#1364](#1364))
([c90034b](c90034b))
* Use LDAIGraphMetricSummary for graph metric summary
([#1362](#1362))
([76a4bf2](76a4bf2))
</details>

<details><summary>server-sdk-ai-langchain: 0.7.0</summary>

##
[0.7.0](server-sdk-ai-langchain-v0.6.3...server-sdk-ai-langchain-v0.7.0)
(2026-05-07)


### ⚠ BREAKING CHANGES

* Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens
([#1366](#1366))
* Remove AIProvider deprecated methods and create*/init* aliases
(AIC-2388)
([#1363](#1363))
* Build judge input as string and strip legacy judge config messages
([#1364](#1364))

### Features

* Remove AIProvider deprecated methods and create*/init* aliases
(AIC-2388)
([#1363](#1363))
([ad66314](ad66314))
* Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens
([#1366](#1366))
([ff932b7](ff932b7))
* Replace LangChainProvider with Runner protocol implementation
(AIC-2388)
([#1338](#1338))
([113a0d2](113a0d2))


### Bug Fixes

* Build judge input as string and strip legacy judge config messages
([#1364](#1364))
([c90034b](c90034b))


### Dependencies

* The following workspace dependencies were updated
  * devDependencies
    * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0
  * peerDependencies
    * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0
</details>

<details><summary>server-sdk-ai-openai: 0.6.0</summary>

##
[0.6.0](server-sdk-ai-openai-v0.5.10...server-sdk-ai-openai-v0.6.0)
(2026-05-07)


### ⚠ BREAKING CHANGES

* Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens
([#1366](#1366))
* Remove AIProvider deprecated methods and create*/init* aliases
(AIC-2388)
([#1363](#1363))
* Build judge input as string and strip legacy judge config messages
([#1364](#1364))

### Features

* Remove AIProvider deprecated methods and create*/init* aliases
(AIC-2388)
([#1363](#1363))
([ad66314](ad66314))
* Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens
([#1366](#1366))
([ff932b7](ff932b7))
* Replace OpenAIProvider with Runner protocol implementation (AIC-2388)
([#1337](#1337))
([e32a955](e32a955))


### Bug Fixes

* Build judge input as string and strip legacy judge config messages
([#1364](#1364))
([c90034b](c90034b))


### Dependencies

* The following workspace dependencies were updated
  * devDependencies
    * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0
  * peerDependencies
    * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0
</details>

<details><summary>server-sdk-ai-vercel: 0.6.0</summary>

##
[0.6.0](server-sdk-ai-vercel-v0.5.10...server-sdk-ai-vercel-v0.6.0)
(2026-05-07)


### ⚠ BREAKING CHANGES

* Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens
([#1366](#1366))
* Remove AIProvider deprecated methods and create*/init* aliases
(AIC-2388)
([#1363](#1363))
* Build judge input as string and strip legacy judge config messages
([#1364](#1364))

### Features

* Remove AIProvider deprecated methods and create*/init* aliases
(AIC-2388)
([#1363](#1363))
([ad66314](ad66314))
* Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens
([#1366](#1366))
([ff932b7](ff932b7))
* replace VercelProvider with Runner protocol implementation (AIC-2388)
([#1339](#1339))
([d5a62de](d5a62de))


### Bug Fixes

* add zod devDependency to Vercel provider (peer dep of ai v5)
([aab6226](aab6226))
* Build judge input as string and strip legacy judge config messages
([#1364](#1364))
([c90034b](c90034b))


### Dependencies

* The following workspace dependencies were updated
  * devDependencies
    * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0
  * peerDependencies
    * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> This is primarily a versioning/release metadata update, but it ships
breaking API changes in `@launchdarkly/server-sdk-ai` and provider
packages that may require consumer code updates (e.g., metrics field
rename and deprecated method removals).
> 
> **Overview**
> Publishes a new release for `@launchdarkly/server-sdk-ai` (`0.19.1` →
`0.20.0`) and the provider packages (`server-sdk-ai-langchain` `0.6.3` →
`0.7.0`, `server-sdk-ai-openai`/`server-sdk-ai-vercel` `0.5.10` →
`0.6.0`), updating `.release-please-manifest.json`, package versions,
and `sdkInfo.ts`.
> 
> Updates provider `peerDependencies`/`devDependencies` to require
`@launchdarkly/server-sdk-ai@^0.20.0`, and bumps all `server-ai`
examples to depend on the new SDK/provider versions. Changelogs
highlight *breaking changes* including renaming metrics `usage` →
`tokens`, removing deprecated `AIProvider` methods/aliases, and
judge-input/config cleanup (plus additional `server-sdk-ai`
graph/managed result summary changes).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
cb3aebc. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants