feat: add Runner, RunnerResult, Judge, and Evaluator (#180)

## Summary

Adds the **AIEVALS types** — `Runner`, `RunnerResult`, `Judge`, and
`Evaluator` — and wires `Evaluator.noop()` into all config types.
Callers can now implement a `Runner` to wrap any model provider,
construct a `Judge` to evaluate AI outputs against a judge prompt with
structured `{score, reasoning}` output, and coordinate multiple judges
through an `Evaluator`.

### New types

```java
public interface Runner {
  RunnerResult run(String input, Map<String, Object> outputType) throws Exception;
  default RunnerResult run(String input) throws Exception;
}
```

Wraps a model provider SDK. `outputType` carries a JSON-Schema-like map
when structured output is needed. Single-arg overload delegates with
`outputType = null`.

```java
RunnerResult.builder(String content, AIMetrics metrics)
    .raw(Object raw)
    .parsed(Map<String, Object> parsed)
    .build();
```

Immutable result of a `Runner` invocation. `parsed` is defensively
copied and returned as unmodifiable.

```java
public Judge(AIJudgeConfig config, Runner runner, LDLogger logger);

JudgeResult evaluate(String input, String output);
JudgeResult evaluate(String input, String output, double samplingRate);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response, double samplingRate);
```

Evaluates AI output by invoking a runner with a formatted evaluation
prompt and parsing the structured response. Sampling gate runs first —
below the rate, returns `sampled=false` immediately. Creates a fresh
tracker per evaluation via `config.createTracker()`. Parses `score`
(Number, [0.0, 1.0]) and `reasoning` (String, optional). Runner
exceptions are caught and returned as `JudgeResult(success=false)` —
judge failures are results, not exceptions. Does **not** call
`trackJudgeResult`.

```java
public static Evaluator noop();
public Evaluator(Map<String, Judge> judges, JudgeConfiguration judgeConfiguration, LDLogger logger);

CompletableFuture<List<JudgeResult>> evaluate(String input, String output);
```

Coordinates sequential execution of judges. Missing judges skipped with
a warning. `Evaluator.noop()` returns a singleton whose `evaluate`
immediately returns an empty list. For v1.0, all configs receive
`Evaluator.noop()`.

### Config type changes

`AIConfig` base class gains an `Evaluator` field and `getEvaluator()`
accessor. `AICompletionConfig` and `AIAgentConfig` constructors accept
an `Evaluator`. `AIJudgeConfig` always wires `Evaluator.noop()`
internally — judges do not evaluate themselves.

## Test plan

- [ ] `./gradlew :lib:sdk:server-ai:test` passes
- [ ] `JudgeTest` — successful evaluation, score boundary validation (0
and 1), reasoning optional, runner exception handling (caught not
rethrown), null/missing parsed output, score out of range, sampling
rates (0 always skips, 1 always runs), message formatting, getter
accessors
- [ ] `EvaluatorTest` — noop returns empty list, noop singleton
identity, single/multiple judge execution, missing judge skipped,
evaluator does not call `trackJudgeResult`, returned future is already
complete
- [ ] `RunnerResultTest` — builder field assignment, immutability,
defensive copy of parsed map

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> New public API and config constructor changes in server-ai; runtime
behavior stays noop until custom evaluators are wired, but Judge paths
will invoke external runners when used.
> 
> **Overview**
> Introduces the **AI evaluation stack** for `server-ai`: a pluggable
**`Runner`** / **`RunnerResult`** pair for model calls (including
optional structured `outputType`), a **`Judge`** that runs judge prompts
via the runner and parses `{score, reasoning}` with sampling and
defensive failure handling, and an **`Evaluator`** that runs configured
judges sequentially and returns a completed `CompletableFuture`.
> 
> **`AIConfig`** and completion/agent configs now carry an
**`Evaluator`** exposed via **`getEvaluator()`**; **`LDAIClientImpl`**
wires **`Evaluator.noop()`** for all retrieved configs in v1.0 (judge
configs always noop internally). Unit tests cover judge scoring,
sampling, evaluator orchestration, and `RunnerResult` immutability.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
aa028c1. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: mattrmc1

lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/AIAgentConfig.java

@@ -30,8 +30,9 @@ public final class AIAgentConfig extends AIConfig {
       String instructions,
       JudgeConfiguration judgeConfiguration,
       Map<String, Tool> tools,
-      Supplier<LDAIConfigTracker> trackerFactory) {
-    super(key, enabled, Mode.AGENT, model, provider, trackerFactory);
+      Supplier<LDAIConfigTracker> trackerFactory,
+      Evaluator evaluator) {
+    super(key, enabled, Mode.AGENT, model, provider, trackerFactory, evaluator);
     this.instructions = instructions;
     this.judgeConfiguration = judgeConfiguration;
     this.tools = tools == null ? null : Collections.unmodifiableMap(tools);

lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/AICompletionConfig.java

@@ -32,8 +32,9 @@ public final class AICompletionConfig extends AIConfig {
       List<Message> messages,
       JudgeConfiguration judgeConfiguration,
       Map<String, Tool> tools,
-      Supplier<LDAIConfigTracker> trackerFactory) {
-    super(key, enabled, Mode.COMPLETION, model, provider, trackerFactory);
+      Supplier<LDAIConfigTracker> trackerFactory,
+      Evaluator evaluator) {
+    super(key, enabled, Mode.COMPLETION, model, provider, trackerFactory, evaluator);
     this.messages = messages == null ? null : Collections.unmodifiableList(messages);
     this.judgeConfiguration = judgeConfiguration;
     this.tools = tools == null ? null : Collections.unmodifiableMap(tools);

lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/AIConfig.java

@@ -24,20 +24,23 @@ public abstract class AIConfig {
   private final Model model;
   private final Provider provider;
   private final Supplier<LDAIConfigTracker> trackerFactory;
+  private final Evaluator evaluator;
 
   AIConfig(
       String key,
       boolean enabled,
       Mode mode,
       Model model,
       Provider provider,
-      Supplier<LDAIConfigTracker> trackerFactory) {
+      Supplier<LDAIConfigTracker> trackerFactory,
+      Evaluator evaluator) {
     this.key = key;
     this.enabled = enabled;
     this.mode = mode;
     this.model = model;
     this.provider = provider;
     this.trackerFactory = Objects.requireNonNull(trackerFactory, "trackerFactory");
+    this.evaluator = Objects.requireNonNull(evaluator, "evaluator");
   }
 
   /**
@@ -102,4 +105,17 @@ public Provider getProvider() {
   public LDAIConfigTracker createTracker() {
     return trackerFactory.get();
   }
+
+  /**
+   * Returns the evaluator that coordinates judge execution for this configuration.
+   * <p>
+   * For {@link AIJudgeConfig} this is always {@link Evaluator#noop()}. For
+   * {@link AICompletionConfig} and {@link AIAgentConfig} it is the evaluator supplied at
+   * construction time (also {@link Evaluator#noop()} unless a custom one is wired in).
+   *
+   * @return the evaluator, never {@code null}
+   */
+  public Evaluator getEvaluator() {
+    return evaluator;
+  }
 }

lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/AIJudgeConfig.java

@@ -29,7 +29,7 @@ public final class AIJudgeConfig extends AIConfig {
       List<Message> messages,
       String evaluationMetricKey,
       Supplier<LDAIConfigTracker> trackerFactory) {
-    super(key, enabled, Mode.JUDGE, model, provider, trackerFactory);
+    super(key, enabled, Mode.JUDGE, model, provider, trackerFactory, Evaluator.noop());
     this.messages = messages == null ? null : Collections.unmodifiableList(messages);
     this.evaluationMetricKey = evaluationMetricKey;
   }

lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/Evaluator.java

@@ -0,0 +1,93 @@
+package com.launchdarkly.sdk.server.ai;
+
+import com.launchdarkly.logging.LDLogger;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAIConfigTypes.JudgeConfiguration;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.JudgeResult;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Coordinates evaluation of an AI Config output by running a set of {@link Judge} instances.
+ * <p>
+ * An {@code Evaluator} is attached to an {@link AICompletionConfig} or {@link AIAgentConfig} and
+ * invoked by managed AI types (plan 4). In v1.0, the evaluator returned by the config retrieval
+ * methods is always a noop that returns an empty list immediately.
+ * <p>
+ * Instances are immutable and thread-safe.
+ */
+public final class Evaluator {
+  private static final Evaluator NOOP = new Evaluator();
+
+  private final Map<String, Judge> judges;
+  private final JudgeConfiguration judgeConfiguration;
+  private final LDLogger logger;
+
+  private Evaluator() {
+    this.judges = Collections.emptyMap();
+    this.judgeConfiguration = null;
+    this.logger = null;
+  }
+
+  /**
+   * Constructs an evaluator with the given judges and configuration.
+   *
+   * @param judges a map from judge config key to {@link Judge} instance; a {@code null} value is
+   *     treated as an empty map
+   * @param judgeConfiguration the judge configuration listing which judges to run and their sampling
+   *     rates
+   * @param logger the logger
+   */
+  public Evaluator(Map<String, Judge> judges, JudgeConfiguration judgeConfiguration, LDLogger logger) {
+    this.judges = judges != null
+        ? Collections.unmodifiableMap(new HashMap<>(judges))
+        : Collections.emptyMap();
+    this.judgeConfiguration = judgeConfiguration;
+    this.logger = logger;
+  }
+
+  /**
+   * Returns the shared noop evaluator, which immediately returns an empty result list without
+   * logging any warnings.
+   *
+   * @return the noop singleton, never {@code null}
+   */
+  public static Evaluator noop() {
+    return NOOP;
+  }
+
+  /**
+   * Runs all configured judges against the given input/output pair and returns their results.
+   * <p>
+   * Judges are run sequentially in the order specified by the {@link JudgeConfiguration}.
+   * Returns an empty list immediately when no judge configuration is present.
+   * Judges referenced in the configuration but absent from the judges map are skipped with a
+   * warning; this is not an error.
+   * <p>
+   * This method does NOT call {@code trackJudgeResult} — that is the caller's responsibility.
+   *
+   * @param input the message history or prompt that was sent to the model
+   * @param output the model's response to evaluate
+   * @return a completed future holding the list of judge results; never {@code null}
+   */
+  public CompletableFuture<List<JudgeResult>> evaluate(String input, String output) {
+    if (judgeConfiguration == null) {
+      return CompletableFuture.completedFuture(Collections.emptyList());
+    }
+
+    List<JudgeResult> results = new ArrayList<>();
+    for (JudgeConfiguration.Judge entry : judgeConfiguration.getJudges()) {
+      Judge judge = judges.get(entry.getKey());
+      if (judge == null) {
+        if (logger != null) logger.warn("Evaluator: no judge found for key '{}', skipping", entry.getKey());
+        continue;
+      }
+      results.add(judge.evaluate(input, output, entry.getSamplingRate()));
+    }
+    return CompletableFuture.completedFuture(results);
+  }
+}

lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/Judge.java

@@ -0,0 +1,222 @@
+package com.launchdarkly.sdk.server.ai;
+
+import com.launchdarkly.logging.LDLogger;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAIConfigTypes.Message;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.JudgeResult;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.Collectors;
+
+/**
+ * Evaluates an AI model output against a judge prompt, returning a scored {@link JudgeResult}.
+ * <p>
+ * A {@code Judge} wraps an {@link AIJudgeConfig} and a {@link Runner}. Each call to
+ * {@link #evaluate} or {@link #evaluateMessages} invokes the runner with a formatted evaluation
+ * prompt and parses the structured {@code {score, reasoning}} response. Evaluation can be sampled
+ * to reduce cost: pass a {@code samplingRate} of {@code 0.0} to always skip, or {@code 1.0} to
+ * always run.
+ * <p>
+ * Instances are immutable and thread-safe.
+ */
+public final class Judge {
+  /**
+   * JSON-Schema fragment sent to the runner as the {@code outputType}, requesting structured
+   * {@code {score, reasoning}} output.
+   */
+  private static final Map<String, Object> EVALUATION_SCHEMA;
+  static {
+    Map<String, Object> scoreSchema = new HashMap<>();
+    scoreSchema.put("type", "number");
+    scoreSchema.put("minimum", 0);
+    scoreSchema.put("maximum", 1);
+    scoreSchema.put("description", "Score between 0.0 and 1.0.");
+
+    Map<String, Object> reasoningSchema = new HashMap<>();
+    reasoningSchema.put("type", "string");
+    reasoningSchema.put("description", "Reasoning behind the score.");
+
+    Map<String, Object> properties = new HashMap<>();
+    properties.put("score", Collections.unmodifiableMap(scoreSchema));
+    properties.put("reasoning", Collections.unmodifiableMap(reasoningSchema));
+
+    Map<String, Object> schema = new HashMap<>();
+    schema.put("type", "object");
+    schema.put("properties", Collections.unmodifiableMap(properties));
+    schema.put("required", Arrays.asList("score", "reasoning"));
+    schema.put("additionalProperties", false);
+
+    EVALUATION_SCHEMA = Collections.unmodifiableMap(schema);
+  }
+
+  private final AIJudgeConfig config;
+  private final Runner runner;
+  private final LDLogger logger;
+
+  /**
+   * Constructs a judge.
+   *
+   * @param config the judge AI Config
+   * @param runner the runner to invoke
+   * @param logger the logger
+   */
+  public Judge(AIJudgeConfig config, Runner runner, LDLogger logger) {
+    this.config = config;
+    this.runner = runner;
+    this.logger = logger;
+  }
+
+  /**
+   * Evaluates the given input/output pair, always running (sampling rate {@code 1.0}).
+   *
+   * @param input the message history or prompt that was sent to the model
+   * @param output the model's response to evaluate
+   * @return the evaluation result; never {@code null}
+   */
+  public JudgeResult evaluate(String input, String output) {
+    return evaluate(input, output, 1.0);
+  }
+
+  /**
+   * Evaluates the given input/output pair, subject to the given sampling rate.
+   *
+   * @param input the message history or prompt that was sent to the model
+   * @param output the model's response to evaluate
+   * @param samplingRate the fraction of evaluations to actually run; {@code 0.0} always skips,
+   *     {@code 1.0} always runs
+   * @return the evaluation result; never {@code null}
+   */
+  public JudgeResult evaluate(String input, String output, double samplingRate) {
+    if (samplingRate <= 0.0) {
+      return JudgeResult.builder()
+          .sampled(false)
+          .success(false)
+          .judgeConfigKey(config.getKey())
+          .metricKey(config.getEvaluationMetricKey())
+          .build();
+    }
+    if (ThreadLocalRandom.current().nextDouble() > samplingRate) {
+      return JudgeResult.builder()
+          .sampled(false)
+          .success(false)
+          .judgeConfigKey(config.getKey())
+          .metricKey(config.getEvaluationMetricKey())
+          .build();
+    }
+
+    try {
+      String formatted = "MESSAGE HISTORY:\n" + input + "\n\nRESPONSE TO EVALUATE:\n" + output;
+      LDAIConfigTracker tracker = config.createTracker();
+
+      RunnerResult result = tracker.trackMetricsOf(RunnerResult::getMetrics, () -> runner.run(formatted, EVALUATION_SCHEMA));
+
+      Map<String, Object> parsed = result.getParsed();
+      if (parsed == null) {
+        if (logger != null) logger.warn("Judge {}: runner returned null parsed output", config.getKey());
+        return JudgeResult.builder()
+            .sampled(true)
+            .success(false)
+            .judgeConfigKey(config.getKey())
+            .metricKey(config.getEvaluationMetricKey())
+            .build();
+      }
+
+      Object scoreRaw = parsed.get("score");
+      if (!(scoreRaw instanceof Number)) {
+        if (logger != null) logger.warn("Judge {}: parsed output missing numeric score", config.getKey());
+        return JudgeResult.builder()
+            .sampled(true)
+            .success(false)
+            .judgeConfigKey(config.getKey())
+            .metricKey(config.getEvaluationMetricKey())
+            .build();
+      }
+      double score = ((Number) scoreRaw).doubleValue();
+      if (!Double.isFinite(score) || score < 0.0 || score > 1.0) {
+        if (logger != null) logger.warn("Judge {}: score {} is outside [0.0, 1.0]", config.getKey(), score);
+        return JudgeResult.builder()
+            .sampled(true)
+            .success(false)
+            .judgeConfigKey(config.getKey())
+            .metricKey(config.getEvaluationMetricKey())
+            .build();
+      }
+
+      JudgeResult.Builder resultBuilder = JudgeResult.builder()
+          .sampled(true)
+          .success(true)
+          .judgeConfigKey(config.getKey())
+          .metricKey(config.getEvaluationMetricKey())
+          .score(score);
+
+      Object reasoningRaw = parsed.get("reasoning");
+      if (reasoningRaw instanceof String) {
+        resultBuilder.reasoning((String) reasoningRaw);
+      } else if (reasoningRaw != null) {
+        if (logger != null) logger.warn("Judge {}: reasoning is not a string, ignoring", config.getKey());
+      }
+
+      return resultBuilder.build();
+    } catch (Exception ex) {
+      return JudgeResult.builder()
+          .sampled(true)
+          .success(false)
+          .judgeConfigKey(config.getKey())
+          .metricKey(config.getEvaluationMetricKey())
+          .errorMessage(ex.getMessage())
+          .build();
+    }
+  }
+
+  /**
+   * Evaluates a message list and runner response, always running (sampling rate {@code 1.0}).
+   * <p>
+   * Messages are formatted as {@code role: content} lines, joined by newlines.
+   *
+   * @param messages the messages that were sent to the model
+   * @param response the runner result whose {@link RunnerResult#getContent() content} is evaluated
+   * @return the evaluation result; never {@code null}
+   */
+  public JudgeResult evaluateMessages(List<Message> messages, RunnerResult response) {
+    return evaluateMessages(messages, response, 1.0);
+  }
+
+  /**
+   * Evaluates a message list and runner response, subject to the given sampling rate.
+   * <p>
+   * Messages are formatted as {@code role: content} lines, joined by newlines.
+   *
+   * @param messages the messages that were sent to the model
+   * @param response the runner result whose {@link RunnerResult#getContent() content} is evaluated
+   * @param samplingRate the fraction of evaluations to actually run
+   * @return the evaluation result; never {@code null}
+   */
+  public JudgeResult evaluateMessages(List<Message> messages, RunnerResult response, double samplingRate) {
+    String formattedMessages = messages == null ? "" : messages.stream()
+        .map(m -> m.getRole().getWireValue() + ": " + m.getContent())
+        .collect(Collectors.joining("\n"));
+    return evaluate(formattedMessages, response == null ? "" : response.getContent(), samplingRate);
+  }
+
+  /**
+   * Returns the judge AI Config this instance was constructed with.
+   *
+   * @return the judge config
+   */
+  public AIJudgeConfig getConfig() {
+    return config;
+  }
+
+  /**
+   * Returns the runner this instance was constructed with.
+   *
+   * @return the runner
+   */
+  public Runner getRunner() {
+    return runner;
+  }
+}