feat: introduce ManagedResult, RunnerResult, and LDAIMetricSummary#1332
Draft
jsonbailey wants to merge 3 commits intonext-ai-releasefrom
Draft
feat: introduce ManagedResult, RunnerResult, and LDAIMetricSummary#1332jsonbailey wants to merge 3 commits intonext-ai-releasefrom
jsonbailey wants to merge 3 commits intonext-ai-releasefrom
Conversation
…-1657) Introduces `Evaluator` wrapping judges and JudgeConfiguration. The evaluator runs all configured judges in parallel, warns+skips on missing judge keys, and intentionally does NOT call tracker.trackJudgeResult — that responsibility belongs in the managed layer. Attaches Evaluator to LDAICompletionConfig and LDAIAgentConfig via createChat/createAgent. Adds Evaluator.noop() static factory. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
46ab0a4 to
c751ce6
Compare
…IC-2388) Adds RunnerResult (provider-level result type without evaluations), ManagedResult (managed-layer result with async evaluations promise), and LDAIMetricSummary (flat metric summary including resumptionToken). Adds toolCalls and durationMs to LDAIMetrics. TrackedChat.run() replaces invoke() returning ManagedResult with LDAIMetricSummary built from tracker. Adds createModel() to LDAIClient/LDAIClientImpl as the preferred replacement for createChat(). Updates chat-judge example. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fe6948b to
192315f
Compare
jsonbailey
commented
Apr 30, 2026
Contributor
Author
There was a problem hiding this comment.
We need to rename the TrackedChat type to ManagedModel naming. We should also introduce the runner interface as defined in the spec and follow it. This will be a breaking change to move to the .run method and we should drop the old invoke method.
… conversation management - Add `Runner` and `AgentGraphRunner` interfaces in api/providers/Runner.ts. Runner.run takes a prompt string + optional output schema and returns a RunnerResult. AgentGraphRunner.run takes a string and returns an AgentGraphRunnerResult. Re-export both from api/providers/index.ts. - Add the supporting `GraphMetrics` and `AgentGraphRunnerResult` types to api/graph/types.ts so AgentGraphRunner has its result shape on this branch. - Rename `TrackedChat` -> `ManagedModel` (file + class). The constructor now takes a `Runner` instead of an `AIProvider`. The class is stateless: it owns no conversation history, and `run(prompt)` forwards the prompt directly to the runner. Drop `invoke()`, `_evaluateWithJudges`, `appendMessages`, `getMessages`, `getJudges`, `getProvider`, and the internal `messages` field. - Update `LDAIClientImpl.createModel` to construct a `ManagedModel` with a `Runner`. The factory still produces a (deprecated) `AIProvider`, so a small `runnerFromAIProvider` adapter wraps it: it prepends the AIConfig's configured messages to the user prompt to preserve existing system-prompt behavior under the stateless contract. - Mark `createChat` `@deprecated` (now delegates to `createModel`); keep `initChat` deprecated. Update the `LDAIClient` interface accordingly. - Mark the `AIProvider` abstract class `@deprecated` in favor of `Runner`. - Update `tracked-chat` and `chat-observability` examples to call `createModel` + `model.run()` instead of `createChat` + `chat.invoke()`. - Rewrite the test suite for the stateless ManagedModel: prompt is passed through verbatim, no history is retained, ManagedResult is built from the RunnerResult plus the tracker's resumption token. Drop the old tests for `appendMessages`/`getMessages`/`invoke`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
jsonbailey
added a commit
that referenced
this pull request
May 1, 2026
… (AIC-2388) Adds RunnerProtocol.test.ts to verify that the Runner and AgentGraphRunner interfaces can be implemented as plain objects. The Runner, AgentGraphRunner interfaces, AIProvider deprecation, and providers/index.ts re-exports landed in the parent PR (#1332). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RunnerResultinterface (provider-level result: content, metrics, raw?, parsed? — no evaluations)ManagedResultinterface (managed-layer result withevaluations: Promise<JudgeResult[]>)LDAIMetricSummary(flat summary: success, usage?, toolCalls?, durationMs?, resumptionToken?)toolCalls?anddurationMs?fields toLDAIMetricsTrackedChat.run()replaces/supplementsinvoke(), returningManagedResultwith metric summary built from trackercreateModel()toLDAIClientandLDAIClientImplas the preferred replacement forcreateChat()chat-judgeexample to usecreateModel()andrun()Test plan
chat-judgeexample updated to use new API🤖 Generated with Claude Code