feat: Add AIConfigTracker with at-most-once tracking and resumption tokens (#179)

## Summary

Implements the full `LDAIConfigTracker` interface — previously a stub.
Callers can now record AI operation metrics (duration, tokens,
success/error, feedback, tool calls, judge results) with at-most-once
enforcement, extract metrics from runner operations via
`trackMetricsOf`, and reconstruct trackers across processes via
resumption tokens.

### Tracking methods

```java
void trackDuration(Duration duration);
<T> T trackDurationOf(Callable<T> operation) throws Exception;
```

Records wall-clock duration. Null silently dropped (debug log);
negatives clamped to zero. `trackDurationOf` wraps a `Callable`,
measures via `System.nanoTime()`, records duration in `finally` even on
exception.

```java
<T> T trackMetricsOf(Function<? super T, AIMetrics> metricsExtractor, Callable<T> operation) throws Exception;
```

All-in-one wrapper: starts timer, invokes operation, stops clock before
calling the extractor (slow extractors don't inflate duration). On
success: prefers runner-reported `durationMs` over wall-clock, then
delegates to `trackSuccess`/`trackError`, `trackTokens`,
`trackToolCalls`. On exception: records wall-clock duration, calls
`trackError`, rethrows. If the extractor itself throws, operation
duration is still recorded before propagating — `trackError` is NOT
called since the AI operation succeeded.

```java
void trackSuccess();
void trackError();
```

Share a single `AtomicReference<Boolean>` guard — only the first to fire
wins.

```java
void trackFeedback(FeedbackKind kind);
```

Validates and resolves the event name before claiming the at-most-once
guard, so null/invalid input doesn't burn the slot.

```java
void trackTokens(TokenUsage tokens);
```

Emits events for each positive count (total, input, output). All-zero
usage does not consume the at-most-once slot.

```java
void trackToolCall(String toolKey);
void trackToolCalls(List<String> toolKeys);
```

Multi-fire (not at-most-once). Each call emits a separate
`$ld:ai:tool_call` event.

```java
void trackJudgeResult(JudgeResult result);
```

Silently dropped when not sampled, not successful, or when `metricKey`
is blank/null or `score` is null/non-finite. Multi-fire.

```java
void trackTimeToFirstToken(Duration duration);
```

Records time-to-first-token duration. At-most-once.

### Resumption tokens

```java
String getResumptionToken();                                            // on LDAIConfigTracker
LDAIConfigTracker createTracker(String resumptionToken, LDContext context); // on LDAIClient
```

`getResumptionToken()` returns URL-safe Base64 (no padding) JSON
containing `{ runId, configKey, variationKey, version, graphKey }`.
`variationKey` and `graphKey` omitted when null. No length cap — large
config keys are supported. Empty `runId` / `configKey` are rejected on
decode.

### Tracker factory wiring

`LDAIClientImpl` now creates real `LDAIConfigTrackerImpl` instances. A
private `trackerFactory` method captures config identity and returns a
`Supplier<LDAIConfigTracker>` producing a fresh tracker with a new
`runId` on each call. Default configs also get real trackers. Default
version is `1`.

`NoOpAIConfigTracker` deleted — no longer needed.

### New types

**`FeedbackKind`** — enum: `POSITIVE`, `NEGATIVE`.

**`TokenUsage`** — immutable record: `total`, `input`, `output`.

**`AIMetrics`** — immutable builder: `success`, optional `tokens`,
`durationMs`, `toolCalls`.

**`JudgeResult`** — immutable builder: `metricKey`, `score`, `sampled`,
`success`, optional `judgeConfigKey`, `reasoning`, `errorMessage`.

**`MetricSummary`** — snapshot of all tracked metrics plus resumption
token.

**`TrackData`** — run identity fields with `toLDValue()`.

### Thread safety

All at-most-once slots use `AtomicReference<T>.compareAndSet(null,
value)` — single atomic guard+value, no race window. Tool calls use
`CopyOnWriteArrayList`.

## Test plan

- [ ] `./gradlew :lib:sdk:server-ai:test` passes
- [ ] `LDAIConfigTrackerImplTest` — duration (emit, clamp, at-most-once,
null), durationOf (success + exception), success/error (emit, shared
guard both directions), feedback (emit, at-most-once, null slot
preservation), tokens (positive counts, zero skip, slot preservation),
tool calls (multi-fire, null), judge result
(sampled/success/metricKey/score guards, multi-fire), trackMetricsOf
(success path, error path, extractor failure duration tracking, null
AIMetrics guard), variationKey/graphKey in payload, concurrency
(20-thread contention), constructor null rejection
- [ ] `ResumptionTokensTest` — encode/decode round-trips, large keys,
special character escaping, null/malformed rejection, empty
runId/configKey rejection

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> New public tracking API and telemetry emission change observability
behavior; resumption tokens embed flag-targeting metadata if exposed to
clients.
> 
> **Overview**
> Replaces the **no-op** `LDAIConfigTracker` stub with a full
implementation that emits LaunchDarkly custom metrics for AI runs
(duration, time-to-first-token, success/error, feedback, tokens, tool
calls, and judge scores).
> 
> **`LDAIClientImpl`** now supplies a per-config `Supplier` that creates
**`LDAIConfigTrackerImpl`** instances (new UUID `runId` per
`createTracker()`), including when falling back to caller defaults.
**`NoOpAIConfigTracker`** is removed.
**`LDAIClient#createTracker(String, LDContext)`** decodes a resumption
token to continue the same run across requests.
> 
> The expanded **`LDAIConfigTracker`** API adds **`trackMetricsOf`**,
**`getSummary`**, **`getTrackData`**, and **`getResumptionToken`**, with
**at-most-once** semantics on most metrics (tool calls and judge results
are multi-fire). **`LDAITrackingTypes`** holds the new immutable value
types; **`ResumptionTokens`** encodes/decodes URL-safe Base64 JSON for
run identity (docs warn tokens can expose **variation key** / version
and should stay server-side).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
121b140. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Author: mattrmc1

lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/LDAIClient.java

@@ -81,4 +81,23 @@ AIJudgeConfig judgeConfig(
       LDContext context,
       AIJudgeConfigDefault defaultValue,
       Map<String, Object> variables);
+
+  /**
+   * Reconstructs a tracker from a resumption token, preserving the original run's identity.
+   * <p>
+   * Use this when a multi-turn or streaming AI interaction spans multiple requests. The caller
+   * stores the resumption token from a previous tracker (via
+   * {@link LDAIConfigTracker#getResumptionToken()}) and passes it back here to continue tracking
+   * against the same run.
+   * <p>
+   * <strong>Security note:</strong> resumption tokens embed flag-evaluation details such as the
+   * variation key and config version. Keep tokens server-side and do not round-trip them through
+   * untrusted clients where they could leak flag-targeting information.
+   *
+   * @param resumptionToken the token returned by a previous tracker; must not be {@code null}
+   * @param context the evaluation context for the new request; must not be {@code null}
+   * @return a tracker with the decoded run identity, never {@code null}
+   * @throws IllegalArgumentException if the token is malformed
+   */
+  LDAIConfigTracker createTracker(String resumptionToken, LDContext context);
 }

lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/LDAIClientImpl.java

@@ -8,20 +8,21 @@
 import com.launchdarkly.sdk.LDContext;
 import com.launchdarkly.sdk.LDValue;
 import com.launchdarkly.sdk.LDValueType;
-import com.launchdarkly.sdk.server.ai.datamodel.LDAIConfigTypes.Mode;
 import com.launchdarkly.sdk.server.ai.datamodel.LDAIConfigTypes.Message;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAIConfigTypes.Mode;
 import com.launchdarkly.sdk.server.ai.internal.AIConfigFlagValue;
 import com.launchdarkly.sdk.server.ai.internal.AIConfigParser;
 import com.launchdarkly.sdk.server.ai.internal.AISdkInfo;
 import com.launchdarkly.sdk.server.ai.internal.Interpolator;
-import com.launchdarkly.sdk.server.ai.internal.NoOpAIConfigTracker;
+import com.launchdarkly.sdk.server.ai.internal.LDAIConfigTrackerImpl;
 import com.launchdarkly.sdk.server.interfaces.LDClientInterface;
 
 import java.util.ArrayList;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
+import java.util.UUID;
 import java.util.function.Supplier;
 
 /**
@@ -51,8 +52,6 @@ public final class LDAIClientImpl implements LDAIClient {
       .anonymous(true)
       .build();
 
-  // Tracking is implemented in a later step; until then every config hands out the no-op tracker.
-  private static final Supplier<LDAIConfigTracker> TRACKER_FACTORY = () -> NoOpAIConfigTracker.INSTANCE;
 
   private final LDClientInterface client;
   private final LDLogger logger;
@@ -187,6 +186,9 @@ private AIConfig buildConfig(
       AIConfigFlagValue parsed,
       LDContext context,
       Map<String, Object> variables) {
+    Supplier<LDAIConfigTracker> factory = trackerFactory(
+        key, parsed.getVariationKey(), parsed.getVersion(),
+        parsed.getModel(), parsed.getProvider(), context);
     switch (mode) {
       case AGENT:
         return new AIAgentConfig(
@@ -197,7 +199,7 @@ private AIConfig buildConfig(
             interpolate(parsed.getInstructions(), variables, context),
             parsed.getJudgeConfiguration(),
             parsed.getTools(),
-            TRACKER_FACTORY);
+            factory);
       case JUDGE:
         return new AIJudgeConfig(
             key,
@@ -206,7 +208,7 @@ private AIConfig buildConfig(
             parsed.getProvider(),
             interpolateMessages(parsed.getMessages(), variables, context),
             parsed.getEvaluationMetricKey(),
-            TRACKER_FACTORY);
+            factory);
       case COMPLETION:
       default:
         return new AICompletionConfig(
@@ -217,7 +219,7 @@ private AIConfig buildConfig(
             interpolateMessages(parsed.getMessages(), variables, context),
             parsed.getJudgeConfiguration(),
             parsed.getTools(),
-            TRACKER_FACTORY);
+            factory);
     }
   }
 
@@ -231,6 +233,9 @@ private AIConfig buildConfigFromDefault(
       AIConfigDefault defaultValue,
       LDContext context,
       Map<String, Object> variables) {
+    // Default configs still get real trackers — the configKey was requested even if no flag was found.
+    // variationKey is null because no flag evaluation occurred.
+    Supplier<LDAIConfigTracker> factory = trackerFactory(key, null, null, null, null, context);
     switch (mode) {
       case AGENT: {
         AIAgentConfigDefault agent = (AIAgentConfigDefault) defaultValue;
@@ -242,7 +247,7 @@ private AIConfig buildConfigFromDefault(
             interpolate(agent.getInstructions(), variables, context),
             agent.getJudgeConfiguration(),
             agent.getTools(),
-            TRACKER_FACTORY);
+            factory);
       }
       case JUDGE: {
         AIJudgeConfigDefault judge = (AIJudgeConfigDefault) defaultValue;
@@ -253,7 +258,7 @@ private AIConfig buildConfigFromDefault(
             judge.getProvider(),
             interpolateMessages(judge.getMessages(), variables, context),
             judge.getEvaluationMetricKey(),
-            TRACKER_FACTORY);
+            factory);
       }
       case COMPLETION:
       default: {
@@ -266,11 +271,43 @@ private AIConfig buildConfigFromDefault(
             interpolateMessages(completion.getMessages(), variables, context),
             completion.getJudgeConfiguration(),
             completion.getTools(),
-            TRACKER_FACTORY);
+            factory);
       }
     }
   }
 
+  /**
+   * Creates a per-evaluation tracker factory. Each call to the returned {@link Supplier} produces
+   * a fresh {@link LDAIConfigTrackerImpl} with a new {@code runId}.
+   */
+  private Supplier<LDAIConfigTracker> trackerFactory(
+      String configKey,
+      String variationKey,
+      Integer version,
+      com.launchdarkly.sdk.server.ai.datamodel.LDAIConfigTypes.Model model,
+      com.launchdarkly.sdk.server.ai.datamodel.LDAIConfigTypes.Provider provider,
+      LDContext context) {
+    String modelName = model != null && model.getName() != null ? model.getName() : "";
+    String providerName = provider != null && provider.getName() != null ? provider.getName() : "";
+    int ver = version != null ? version : 1;
+    return () -> new LDAIConfigTrackerImpl(
+        client,
+        UUID.randomUUID().toString(),
+        configKey,
+        variationKey,
+        ver,
+        modelName,
+        providerName,
+        context,
+        null, // graphKey — set by agentGraph() in Plan 3
+        logger);
+  }
+
+  @Override
+  public LDAIConfigTracker createTracker(String resumptionToken, LDContext context) {
+    return LDAIConfigTrackerImpl.fromResumptionToken(resumptionToken, client, context, logger);
+  }
+
   private List<Message> interpolateMessages(
       List<Message> messages, Map<String, Object> variables, LDContext context) {
     if (messages == null) {

lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/LDAIConfigTracker.java

@@ -1,16 +1,169 @@
 package com.launchdarkly.sdk.server.ai;
 
+import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.AIMetrics;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.FeedbackKind;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.JudgeResult;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.MetricSummary;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.TokenUsage;
+import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.TrackData;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.function.Function;
+
 /**
  * Reports events related to a single AI run of an {@link AIConfig}.
  * <p>
- * A tracker is obtained from a retrieved config via {@link AIConfig#createTracker()}. Each tracker
- * corresponds to one AI run and is used to record metrics such as model usage, duration, and
- * feedback against the AI Config it was created from.
+ * A tracker is obtained from a retrieved config via {@link AIConfig#createTracker()}, or
+ * reconstructed from a resumption token via {@link LDAIClient#createTracker(String, com.launchdarkly.sdk.LDContext)}.
+ * Each tracker corresponds to one AI run and is used to record metrics such as model usage,
+ * duration, and feedback against the AI Config it was created from.
+ * <p>
+ * Most tracking methods are at-most-once: a second call to the same method on the same tracker
+ * is silently dropped. {@link #trackToolCall(String)} and {@link #trackJudgeResult(JudgeResult)}
+ * are multi-fire — each call records a distinct event.
  * <p>
- * <strong>This interface is an intentional placeholder.</strong> The metric- and feedback-reporting
- * methods (and resumption-token support) are introduced in a later step of the AI SDK build-out; it
- * is defined here so that the public config types expose a stable {@code createTracker()} surface.
- * The only implementation in this release is an internal no-op.
+ * Implementations are thread-safe.
  */
 public interface LDAIConfigTracker {
+
+  /**
+   * Returns the correlation metadata for this tracker's run.
+   *
+   * @return the track data, never {@code null}
+   */
+  TrackData getTrackData();
+
+  /**
+   * Returns the resumption token for this run.
+   * <p>
+   * The resumption token encodes the run's identity and can be passed to
+   * {@link LDAIClient#createTracker(String, com.launchdarkly.sdk.LDContext)} to reconstruct a
+   * tracker on a subsequent request (for example, in a streaming scenario).
+   * <p>
+   * <strong>Security note:</strong> resumption tokens embed flag-evaluation details such as the
+   * variation key and config version. Keep tokens server-side and do not round-trip them through
+   * untrusted clients where they could leak flag-targeting information.
+   *
+   * @return the resumption token, or {@code null} if not available
+   */
+  String getResumptionToken();
+
+  /**
+   * Records the duration of the AI generation.
+   * <p>
+   * At-most-once: subsequent calls on the same tracker are silently dropped.
+   *
+   * @param duration the duration; ignored if {@code null}
+   */
+  void trackDuration(Duration duration);
+
+  /**
+   * Executes the given operation and records its wall-clock duration.
+   * <p>
+   * The duration is recorded even if the operation throws. Equivalent to wrapping the operation
+   * in a try/finally that calls {@link #trackDuration(Duration)}.
+   *
+   * @param <T> the return type of the operation
+   * @param operation the operation to execute and time; must not be {@code null}
+   * @return the result of the operation
+   * @throws Exception if the operation throws
+   */
+  <T> T trackDurationOf(Callable<T> operation) throws Exception;
+
+  /**
+   * Records the time from request start to receipt of the first token.
+   * <p>
+   * At-most-once: subsequent calls on the same tracker are silently dropped.
+   *
+   * @param duration the time to first token; ignored if {@code null}
+   */
+  void trackTimeToFirstToken(Duration duration);
+
+  /**
+   * Records that the AI generation succeeded.
+   * <p>
+   * At-most-once and mutually exclusive with {@link #trackError()}: whichever is called first wins.
+   */
+  void trackSuccess();
+
+  /**
+   * Records that the AI generation failed.
+   * <p>
+   * At-most-once and mutually exclusive with {@link #trackSuccess()}: whichever is called first wins.
+   */
+  void trackError();
+
+  /**
+   * Records user feedback for this AI generation.
+   * <p>
+   * At-most-once: subsequent calls on the same tracker are silently dropped.
+   *
+   * @param kind the feedback kind; ignored if {@code null}
+   */
+  void trackFeedback(FeedbackKind kind);
+
+  /**
+   * Records token usage for this AI generation.
+   * <p>
+   * At-most-once: subsequent calls on the same tracker are silently dropped. Calls where all
+   * counts are zero do not consume the at-most-once slot.
+   *
+   * @param tokens the token usage; ignored if {@code null}
+   */
+  void trackTokens(TokenUsage tokens);
+
+  /**
+   * Records a single tool call made during this AI generation.
+   * <p>
+   * Multi-fire: every call emits an event.
+   *
+   * @param toolKey the tool key; ignored if {@code null}
+   */
+  void trackToolCall(String toolKey);
+
+  /**
+   * Records multiple tool calls made during this AI generation.
+   * <p>
+   * Equivalent to calling {@link #trackToolCall(String)} for each key.
+   *
+   * @param toolKeys the tool keys; ignored if {@code null}
+   */
+  void trackToolCalls(List<String> toolKeys);
+
+  /**
+   * Records the result of a judge evaluation.
+   * <p>
+   * Multi-fire per judge metric key. The result is silently skipped if it was not sampled, if
+   * the evaluation did not succeed, or if the metric key or score is absent.
+   *
+   * @param result the judge result; ignored if {@code null}
+   */
+  void trackJudgeResult(JudgeResult result);
+
+  /**
+   * Executes the given operation and tracks its metrics using the extracted {@link AIMetrics}.
+   * <p>
+   * Tracks duration (preferring runner-reported duration when present), success or error, tokens,
+   * and tool calls. If the operation throws, {@link #trackError()} is called and the exception
+   * is re-thrown.
+   *
+   * @param <T> the return type of the operation
+   * @param metricsExtractor a function that extracts {@link AIMetrics} from the operation result;
+   *     exceptions from the extractor propagate to the caller
+   * @param operation the AI operation to execute; must not be {@code null}
+   * @return the result of the operation
+   * @throws Exception if the operation or the metrics extractor throws
+   */
+  <T> T trackMetricsOf(
+      Function<? super T, AIMetrics> metricsExtractor,
+      Callable<T> operation) throws Exception;
+
+  /**
+   * Returns a snapshot of all metrics tracked so far on this tracker.
+   *
+   * @return the metric summary, never {@code null}
+   */
+  MetricSummary getSummary();
 }