feat(llmobs): sampling decisions, rates, and propagation (#9030)

* feat(llmobs): add sampling decisions, rates, and propagation

Compute a keep/drop decision + rate once on the root LLMObs span and
inherit it across spans and services (via x-datadog-tags). Spans are
always shipped; the decision is recorded in the event _dd block so the
backend can honor it. Mirrors dd-trace-py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(llmobs): address sampling PR review comments

- Match dd-trace-py: only the true trace root makes a fresh sampling
  decision; a span under a propagated LLMObs parent inherits whatever
  was propagated (possibly none) instead of starting a divergent one.
- Reuse the shared formatKnuthRate (hoisted to sampler.js) instead of a
  bespoke llmobs formatRate.
- Rename handleLLMObsParentIdInjection -> handleLLMObsInjection.
- Declare llmobs.sampleRate in the public index.d.ts (fixes config-names
  lint) and tag sample_rate on enablement telemetry.
- Drop redundant comments flagged in review.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(llmobs): regenerate config types after merging master

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(llmobs): address follow-up sampling review comments

- Rebuild the sampler lazily from the live config rate so a disable/
  re-enable (or any rate change) is reflected, instead of caching one
  sampler at construction.
- Tweak the injection comment wording.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(llmobs): export formatKnuthRate from priority_sampler.js

Keep the formatter in priority_sampler.js (export it) instead of
hoisting to sampler.js, per review.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(llmobs): build the sampler once in the tagger constructor

The sample rate is set at init and not mutable at runtime, so the
per-span lazy rebuild was unnecessary work on a hot path. Build it once
at construction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Update packages/dd-trace/src/llmobs/tagger.js

* refactor(llmobs): address sampling review comments and make sampler config-reactive

- Mirror `llmobs.sampleRate` in the v5 type declarations (index.d.v5.ts), per
  the repo type policy requiring non-v6-only public APIs in both declaration
  surfaces (PR review).
- Register DD_LLMOBS_SAMPLE_RATE as implementation "A" in
  supported-configurations.json (PR review); generated types are unchanged.
- Build the LLMObs sampler lazily and rebuild it when config.llmobs.sampleRate
  changes, instead of fixing it for the tagger's lifetime, so a future remote
  config update to the sample rate takes effect without re-instantiation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(llmobs): match DD_LLMOBS_SAMPLE_RATE default to config registry (1.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(llmobs): address remaining sampling review comments

- Drop the redundant #samplerRate field; use sampler.rate() for the rebuild check.
- Move formatKnuthRate to the shared src/util.js so the LLMObs tagger no longer
  imports priority_sampler just for a string formatter.
- Reword the sampleRate JSDoc in index.d.ts and index.d.v5.ts ("honored at
  ingestion time").
- Test rates that exercise the formatter: 1/3 (6-decimal cap) and 0.25
  (trailing-zero stripping).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: 2 people

index.d.ts

@@ -4033,6 +4033,15 @@ declare namespace tracer {
        * Programmatic configuration takes precedence over the environment variables listed above.
        */
       agentlessEnabled?: boolean,
+
+      /**
+       * The proportion of LLM Observability traces to sample, between `0` and `1` (inclusive).
+       * The decision is computed once per trace, propagated across services, and recorded on every
+       * span; spans are always sent and the decision is honored at ingestion time. Defaults to `1`.
+       * @env DD_LLMOBS_SAMPLE_RATE
+       * Programmatic configuration takes precedence over the environment variables listed above.
+       */
+      sampleRate?: number,
     }
 
     /** @hidden */

index.d.v5.ts

@@ -4234,6 +4234,15 @@ declare namespace tracer {
        * Programmatic configuration takes precedence over the environment variables listed above.
        */
       agentlessEnabled?: boolean,
+
+      /**
+       * The proportion of LLM Observability traces to sample, between `0` and `1` (inclusive).
+       * The decision is computed once per trace, propagated across services, and recorded on every
+       * span; spans are always sent and the decision is honored at ingestion time. Defaults to `1`.
+       * @env DD_LLMOBS_SAMPLE_RATE
+       * Programmatic configuration takes precedence over the environment variables listed above.
+       */
+      sampleRate?: number,
     }
     /** @hidden */
     type spanKind = 'agent' | 'workflow' | 'task' | 'tool' | 'retrieval' | 'embedding' | 'llm'

packages/dd-trace/src/config/generated-config-types.d.ts

@@ -477,6 +477,7 @@ export interface GeneratedConfig {
     agentlessEnabled: boolean | undefined;
     enabled: boolean;
     mlApp: string | undefined;
+    sampleRate: number;
   };
   logInjection: boolean;
   logLevel: "debug" | "info" | "warn" | "error";
@@ -724,6 +725,7 @@ export interface GeneratedEnvVarConfig {
   DD_LLMOBS_AGENTLESS_ENABLED: boolean | undefined;
   DD_LLMOBS_ENABLED: boolean;
   DD_LLMOBS_ML_APP: string | undefined;
+  DD_LLMOBS_SAMPLE_RATE: number;
   DD_LOG_LEVEL: "debug" | "info" | "warn" | "error";
   DD_LOGS_INJECTION: boolean;
   DD_LOGS_OTEL_ENABLED: boolean;

packages/dd-trace/src/config/supported-configurations.json

@@ -1238,6 +1238,16 @@
         "default": null
       }
     ],
+    "DD_LLMOBS_SAMPLE_RATE": [
+      {
+        "implementation": "A",
+        "type": "decimal",
+        "configurationNames": [
+          "llmobs.sampleRate"
+        ],
+        "default": "1.0"
+      }
+    ],
     "DD_LOGS_INJECTION": [
       {
         "implementation": "B",

packages/dd-trace/src/llmobs/constants/tags.js

@@ -14,6 +14,12 @@ module.exports = {
   PROPAGATED_PARENT_ID_KEY: '_dd.p.llmobs_parent_id',
   PROPAGATED_ML_APP_KEY: '_dd.p.llmobs_ml_app',
   PARENT_ID_KEY: '_ml_obs.llmobs_parent_id',
+  PROPAGATED_SAMPLE_RATE_KEY: '_dd.p.llmobs_sr',
+  PROPAGATED_SAMPLING_DECISION_KEY: '_dd.p.llmobs_sd',
+  SAMPLE_RATE: '_ml_obs.sample_rate',
+  SAMPLING_DECISION: '_ml_obs.sampling_decision',
+  SAMPLING_DECISION_SAMPLED: '1',
+  SAMPLING_DECISION_DROPPED: '0',
   TAGS: '_ml_obs.tags',
   NAME: '_ml_obs.name',
   TRACE_ID: '_ml_obs.trace_id',

packages/dd-trace/src/llmobs/index.js

@@ -9,6 +9,10 @@ const {
   ML_APP,
   PROPAGATED_ML_APP_KEY,
   PROPAGATED_PARENT_ID_KEY,
+  SAMPLE_RATE,
+  SAMPLING_DECISION,
+  PROPAGATED_SAMPLE_RATE_KEY,
+  PROPAGATED_SAMPLING_DECISION_KEY,
 } = require('./constants/tags')
 const { storage } = require('./storage')
 const telemetry = require('./telemetry')
@@ -68,7 +72,7 @@ function enable (config) {
   spanFinishCh.subscribe(handleSpanProcess)
 
   // distributed tracing for llmobs
-  injectCh.subscribe(handleLLMObsParentIdInjection)
+  injectCh.subscribe(handleLLMObsInjection)
 
   setAgentStrategy(config, useAgentless => {
     if (useAgentless && !(config.apiKey && config.site)) {
@@ -92,7 +96,7 @@ function disable () {
   if (evalMetricAppendCh.hasSubscribers) evalMetricAppendCh.unsubscribe(handleEvalMetricAppend)
   if (flushCh.hasSubscribers) flushCh.unsubscribe(handleFlush)
   if (spanFinishCh.hasSubscribers) spanFinishCh.unsubscribe(handleSpanProcess)
-  if (injectCh.hasSubscribers) injectCh.unsubscribe(handleLLMObsParentIdInjection)
+  if (injectCh.hasSubscribers) injectCh.unsubscribe(handleLLMObsInjection)
   if (registerUserSpanProcessorCh.hasSubscribers) registerUserSpanProcessorCh.unsubscribe(handleRegisterProcessor)
 
   spanWriter?.destroy()
@@ -106,8 +110,8 @@ function disable () {
 }
 
 // since LLMObs traces can extend between services and be the same trace,
-// we need to propagate the parent id and mlApp.
-function handleLLMObsParentIdInjection ({ carrier }) {
+// we need to propagate the parent id, mlApp, and sampling rate/decision.
+function handleLLMObsInjection ({ carrier }) {
   // Respect the standard propagator's gate: when trace tag propagation is
   // disabled, don't write `x-datadog-tags` for LLMObs either.
   if (globalTracerConfig.DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH === 0) return
@@ -122,14 +126,21 @@ function handleLLMObsParentIdInjection ({ carrier }) {
     parentContext?._trace?.tags?.[PROPAGATED_ML_APP_KEY] ||
     globalTracerConfig.llmobs.mlApp
 
-  if (!parentId && !mlApp) return
+  const sampleRate =
+    mlObsSpanTags?.[SAMPLE_RATE] ?? parentContext?._trace?.tags?.[PROPAGATED_SAMPLE_RATE_KEY]
+  const samplingDecision =
+    mlObsSpanTags?.[SAMPLING_DECISION] ?? parentContext?._trace?.tags?.[PROPAGATED_SAMPLING_DECISION_KEY]
+
+  if (!parentId && !mlApp && samplingDecision == null) return
 
   // `_injectTags` only writes `x-datadog-tags` when the trace has `_dd.p.*`
   // tags, so it may be undefined here — coalesce before appending.
   const existing = carrier['x-datadog-tags']
   let tags = existing || ''
   if (parentId) tags += `${tags ? ',' : ''}${PROPAGATED_PARENT_ID_KEY}=${parentId}`
   if (mlApp) tags += `${tags ? ',' : ''}${PROPAGATED_ML_APP_KEY}=${mlApp}`
+  if (sampleRate != null) tags += `${tags ? ',' : ''}${PROPAGATED_SAMPLE_RATE_KEY}=${sampleRate}`
+  if (samplingDecision != null) tags += `${tags ? ',' : ''}${PROPAGATED_SAMPLING_DECISION_KEY}=${samplingDecision}`
   if (tags !== existing) carrier['x-datadog-tags'] = tags
 }
 

packages/dd-trace/src/llmobs/span_processor.js

@@ -33,6 +33,8 @@ const {
   ROUTING_API_KEY,
   ROUTING_SITE,
   LLMOBS_SUBMITTED_TAG_KEY,
+  SAMPLE_RATE,
+  SAMPLING_DECISION,
 } = require('./constants/tags')
 const { UNSERIALIZABLE_VALUE_TEXT } = require('./constants/text')
 const telemetry = require('./telemetry')
@@ -248,6 +250,8 @@ class LLMObsSpanProcessor {
       _dd: {
         span_id: span.context().toSpanId(),
         trace_id: span.context().toTraceId(true),
+        sample_rate: mlObsTags[SAMPLE_RATE],
+        sampling_decision: mlObsTags[SAMPLING_DECISION],
       },
     }
 

packages/dd-trace/src/llmobs/tagger.js

@@ -1,6 +1,8 @@
 'use strict'
 
 const log = require('../log')
+const Sampler = require('../sampler')
+const { formatKnuthRate } = require('../util')
 const {
   MODEL_NAME,
   MODEL_PROVIDER,
@@ -41,6 +43,12 @@ const {
   ROUTING_SITE,
   PROMPT_TRACKING_INSTRUMENTATION_METHOD,
   INSTRUMENTATION_METHOD_ANNOTATED,
+  SAMPLE_RATE,
+  SAMPLING_DECISION,
+  SAMPLING_DECISION_SAMPLED,
+  SAMPLING_DECISION_DROPPED,
+  PROPAGATED_SAMPLE_RATE_KEY,
+  PROPAGATED_SAMPLING_DECISION_KEY,
 } = require('./constants/tags')
 const { storage } = require('./storage')
 const { findGenAIAncestorSpanId, validateCostTags, writeBridgeTags, validateToolDefinitions } = require('./util')
@@ -53,12 +61,31 @@ class LLMObsTagger {
   /** @type {import('../config/config-base')} */
   #config
 
+  /** @type {import('../sampler') | null} */
+  #sampler = null
+
   constructor (config, softFail = false) {
     this.#config = config
 
     this.softFail = softFail
   }
 
+  /**
+   * The sampler reads its rate from `config.llmobs.sampleRate`, which can change
+   * at runtime (e.g. via remote config). Rebuild the sampler whenever the rate
+   * changes so decisions reflect the current config, while reusing the existing
+   * sampler when it hasn't.
+   *
+   * @returns {import('../sampler')}
+   */
+  #getSampler () {
+    const rate = this.#config.llmobs?.sampleRate ?? 1
+    if (this.#sampler === null || rate !== this.#sampler.rate()) {
+      this.#sampler = new Sampler(rate)
+    }
+    return this.#sampler
+  }
+
   static get tagMap () {
     return registry
   }
@@ -124,6 +151,8 @@ class LLMObsTagger {
       ROOT_PARENT_ID
     this._setTag(span, PARENT_ID_KEY, parentId)
 
+    this.#tagSamplingDecision(span, parent)
+
     // apply annotation context
     const annotationContext = storage.getStore()?.annotationContext
 
@@ -153,6 +182,32 @@ class LLMObsTagger {
     }
   }
 
+  #tagSamplingDecision (span, parent) {
+    const traceTags = span.context()._trace.tags
+    const parentTags = registry.get(parent)
+
+    let sampleRate, samplingDecision
+    if (parentTags) {
+      // Local LLMObs parent: inherit its decision.
+      sampleRate = parentTags[SAMPLE_RATE]
+      samplingDecision = parentTags[SAMPLING_DECISION]
+    } else if (traceTags[PROPAGATED_PARENT_ID_KEY]) {
+      // Distributed LLMObs parent: inherit whatever was propagated. This may be
+      // absent if the upstream service predates sampling propagation, in which
+      // case we make no decision here rather than starting a divergent one.
+      sampleRate = traceTags[PROPAGATED_SAMPLE_RATE_KEY]
+      samplingDecision = traceTags[PROPAGATED_SAMPLING_DECISION_KEY]
+    } else {
+      // Root span: make the trace's one sampling decision.
+      const sampler = this.#getSampler()
+      sampleRate = formatKnuthRate(sampler.rate())
+      samplingDecision = sampler.isSampled(span) ? SAMPLING_DECISION_SAMPLED : SAMPLING_DECISION_DROPPED
+    }
+
+    if (sampleRate != null) this._setTag(span, SAMPLE_RATE, sampleRate)
+    if (samplingDecision != null) this._setTag(span, SAMPLING_DECISION, samplingDecision)
+  }
+
   // TODO: similarly for the following `tag` methods,
   // how can we transition from a span weakmap to core API functionality
   tagLLMIO (span, inputData, outputData) {

packages/dd-trace/src/llmobs/telemetry.js

@@ -87,6 +87,7 @@ function recordLLMObsEnabled (startTime, config, value = 1) {
     site: config.site,
     auto: Number(autoEnabled),
     ml_app: config.llmobs.mlApp,
+    sample_rate: config.llmobs.sampleRate,
   }
   llmobsMetrics.count('product_enabled', tags).inc(value)
   llmobsMetrics.distribution('init_time', tags).track(initTimeMs)

packages/dd-trace/src/priority_sampler.js

@@ -19,6 +19,7 @@ const RateLimiter = require('./rate_limiter')
 const Sampler = require('./sampler')
 const { setSamplingRules } = require('./startup-log')
 const SamplingRule = require('./sampling_rule')
+const { formatKnuthRate } = require('./util')
 
 const {
   SAMPLING_MECHANISM_DEFAULT,
@@ -36,19 +37,6 @@ const {
 
 const DEFAULT_KEY = 'service:,env:'
 
-/**
- * Formats a sampling rate as a string with up to 6 decimal digits and no trailing zeros.
- *
- * @param {number} rate
- */
-function formatKnuthRate (rate) {
-  const string = Number(rate).toFixed(6)
-  for (let i = string.length - 1; i > 0; i--) {
-    if (string[i] === '0') continue
-    return string.slice(0, i + (string[i] === '.' ? 0 : 1))
-  }
-}
-
 const defaultSampler = new Sampler(AUTO_KEEP)
 
 /**