feat: add Runner, RunnerResult, Judge, and Evaluator

[mattrmc1]

Summary

Adds the AIEVALS types — Runner, RunnerResult, Judge, and Evaluator — and wires Evaluator.noop() into all config types. Callers can now implement a Runner to wrap any model provider, construct a Judge to evaluate AI outputs against a judge prompt with structured {score, reasoning} output, and coordinate multiple judges through an Evaluator.

New types

public interface Runner {
  RunnerResult run(String input, Map<String, Object> outputType) throws Exception;
  default RunnerResult run(String input) throws Exception;
}

Wraps a model provider SDK. outputType carries a JSON-Schema-like map when structured output is needed. Single-arg overload delegates with outputType = null.

RunnerResult.builder(String content, AIMetrics metrics)
    .raw(Object raw)
    .parsed(Map<String, Object> parsed)
    .build();

Immutable result of a Runner invocation. parsed is defensively copied and returned as unmodifiable.

public Judge(AIJudgeConfig config, Runner runner, LDLogger logger);

JudgeResult evaluate(String input, String output);
JudgeResult evaluate(String input, String output, double samplingRate);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response, double samplingRate);

Evaluates AI output by invoking a runner with a formatted evaluation prompt and parsing the structured response. Sampling gate runs first — below the rate, returns sampled=false immediately. Creates a fresh tracker per evaluation via config.createTracker(). Parses score (Number, [0.0, 1.0]) and reasoning (String, optional). Runner exceptions are caught and returned as JudgeResult(success=false) — judge failures are results, not exceptions. Does not call trackJudgeResult.

public static Evaluator noop();
public Evaluator(Map<String, Judge> judges, JudgeConfiguration judgeConfiguration, LDLogger logger);

CompletableFuture<List<JudgeResult>> evaluate(String input, String output);

Coordinates sequential execution of judges. Missing judges skipped with a warning. Evaluator.noop() returns a singleton whose evaluate immediately returns an empty list. For v1.0, all configs receive Evaluator.noop().

Config type changes

AIConfig base class gains an Evaluator field and getEvaluator() accessor. AICompletionConfig and AIAgentConfig constructors accept an Evaluator. AIJudgeConfig always wires Evaluator.noop() internally — judges do not evaluate themselves.

Test plan

• ./gradlew :lib:sdk:server-ai:test passes
• JudgeTest — successful evaluation, score boundary validation (0 and 1), reasoning optional, runner exception handling (caught not rethrown), null/missing parsed output, score out of range, sampling rates (0 always skips, 1 always runs), message formatting, getter accessors
• EvaluatorTest — noop returns empty list, noop singleton identity, single/multiple judge execution, missing judge skipped, evaluator does not call trackJudgeResult, returned future is already complete
• RunnerResultTest — builder field assignment, immutability, defensive copy of parsed map

---

Note

Medium Risk
New public SDK types and constructor changes on internal config builders; runtime judge paths invoke user-supplied Runners and affect metrics tracking, though client retrieval still uses noop evaluators.

Overview
Adds AI evaluation plumbing to server-ai: a Runner abstraction for provider calls, immutable RunnerResult, Judge for scored {score, reasoning} evals via structured output, and Evaluator to run configured judges sequentially (with sampling and skip-on-missing).

Config surface: AIConfig now carries an Evaluator and exposes getEvaluator(). Completion and agent configs take an evaluator at construction; LDAIClientImpl always passes Evaluator.noop() for v1.0. Judge configs hard-wire noop internally.

Behavior notes: Judge applies sampling before calling the runner, validates parsed scores in [0, 1], wraps runner work in trackMetricsOf, and returns failures as JudgeResult rather than throwing. Evaluator does not call trackJudgeResult—callers own tracking.

^{Reviewed by Cursor Bugbot for commit 792d33c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 792d33c. Configure here.}

[cursor]

Exception path omits judge identity

Medium Severity

When Judge.evaluate catches a runner or tracker failure, the returned JudgeResult sets sampled, success, and errorMessage but not judgeConfigKey or metricKey. Other sampled failure paths populate those fields. Callers and Evaluator runs that handle multiple judges cannot reliably tell which judge failed except by list index.

 

^{Reviewed by Cursor Bugbot for commit 792d33c. Configure here.}