Implement Client-Side Stats (CSS) 1.2.0 (#8420)

## Summary of changes

Updates the existing client-side-stats implementation to match version
1.2.0 [as defined in the
RFC](https://datadoghq.atlassian.net/wiki/spaces/APM/pages/6378947571/Client-Side+Stats+v1.2.0)
and as [implemented in the
agent](https://github.com/DataDog/datadog-agent/blob/c1d67a906f4c594654600760da1eea4c8037471a/pkg/proto/datadog/trace/stats.proto#L83).

## Reason for change

Our implementation is severely lagging the latest implementation in the
agent. This hasn't been a big deal, as it's not documented and not
enabled by default, but we'd like to fix the implementation to make it
usable.

## Implementation details

This was driven almost entirely by 🤖, by comparing our existing
implementation to the RFC, and also taking the agent/go implementation
as the canonical implementation.

> Implementing this in .NET highlighted a number of missing aspects in
the RFC, which I've raised elsewhere, and aim to get incorporated into
the RFC.

At a high level, the PR contains the following changes:

- Stats Wire Format & Serialization
- **Added new aggregation dimensions**: `SpanKind`, `IsTraceRoot` (as
Trilean), `HTTPMethod`, `HTTPEndpoint`, `GRPCStatusCode`,
`ServiceSource`, `PeerTags` to both the aggregation key and the msgpack
wire format
- **Fixed `GRPCStatusCode` type**: Changed from `int` to `string` to
match Go agent's wire format (agent was returning 400 Bad Request in
system tests)
- **gRPC status code extraction**: Checks 4 tag names in priority order
(`rpc.grpc.status_code`, `grpc.code`, `rpc.grpc.status.code`,
`grpc.status.code`)
- **Stochastic rounding**: `Hits`, `Errors`, `Duration`, `TopLevelHits`
accumulated as `double` (weighted by sampling rate) then rounded
probabilistically to `int64` for serialization
- **Duration weighting**: Durations are now multiplied by sampling
weight (`1/rate`), matching Go agent behavior
- **Default env**: Serializes `"unknown-env"` when environment is not
configured
  - **`git_commit_sha`**: Added as optional field in stats payload
  - **`Service`**: Added as top-level field in stats payload
- **Empty bucket suppression**: `HasHits()` check prevents sending
payloads with zero-hit buckets (stale keys retained for sketch reuse)
- Bucket Timing
- **10-second alignment**: Bucket `Start` timestamps aligned to
10-second boundaries (`ts - ts % 10_000_000_000`) matching Go tracer's
`alignTs`
- **Removed unused `StartTime`** property (only `Start` as aligned
nanoseconds)
- Agent Discovery (`/info` Endpoint)
- **`peer_tags`**: Parsed, sorted, deduplicated; used for peer tag
extraction on client/producer/consumer spans
- **`span_kinds_stats_computed`**: Parsed to override eligible span
kinds
  - **`obfuscation_version`**: Parsed for obfuscation negotiation
- **Trace filters**: Parsed `filter_tags`, `filter_tags_regex`,
`ignore_resources` from `/info`
- Trace Filtering
- **`TraceFilter` implementation**: Evaluates agent-configured filters
(exact tags, regex tags, resource patterns) against root spans before
stats computation
- **Tag-only filters**: Handles filter entries that match on tag key
presence without a specific value
- SQL Obfuscation
- **Operator splitters**: Added `* / = < > ! & ^ % ~ ? @ : #` as token
splitters (matching Go agent's `go-sqllexer` `isOperator()`) so queries
like `WHERE id='1'` are properly obfuscated
- **Whitespace normalization**: Post-processing pass adds spaces around
comparison operators (`=`, `<`, `>`, `!`) adjacent to `?` placeholders,
matching Go agent's normalizer output (e.g., `id='1'` → `id = ?`)
- **Obfuscation gating**: Only runs when `obfuscation_version` is
negotiated with agent; sends `Datadog-Obfuscation-Version` header
- Peer Tags
- **IP quantization**: Peer tag values run through
`IpAddressObfuscationUtil.QuantizePeerIpAddresses()` replacing
non-allowed IPs with `"blocked-ip-address"`
- **Base service handling**: Internal/missing-kind spans with
non-default service name use `_dd.base_service:{serviceName}` as sole
peer tag
- **FNV-1a hashing**: Peer tags hashed with null-byte separators for
aggregation key
- Other
- **No retries for stats**: Stats sends are fire-and-forget (retry limit
= 0)
- **Synthetics detection**: Uses `StartsWith("synthetics")` prefix
matching (not exact match)
- **Mock agent updates**: Test mock agent returns `obfuscation_version`
in `/info` response

## Test coverage

There's a _lot_ going on in this PR, because we were so far behind. I
_could_ split this into implementing individual features, but there
would be a lot of duplication between PRs, and it didn't seem like it
would be that easy to track. At least with this big bang we can compare
directly against the system tests etc.

The existing system tests for stats computation were checked, and made
to pass (which identified a number of hidden expectations which will be
added to the RFC). I'll create a PR to enable these in the system-tests
repo

## Other details

Part of a stack

- #8417
- #8418

There are still some _theoretical_ gaps between the go implementation
and the .NET implementation, but I _think_ these are non-issues in
_most_ cases:


  | Area | Gap | Impact |
  |------|-----|--------|
| gRPC status code normalization | Go agent normalizes string statuses
(e.g., `"CANCELLED"` → `"1"`); .NET passes raw tag value | Stats
mismatch if gRPC library uses string-form status codes |
| HTTP status code tags | Go agent checks both `http.status_code` and
`http.response.status_code` (OTel convention); .NET only checks
`http.status_code` | OTel spans using newer convention would get `0` in
.NET |
| Duration precision truncation | Go agent uses float bit masking; .NET
uses integer shifting — both target ~10 bits but may produce slightly
different values | Minor histogram differences |
| SQL obfuscation | Go agent uses full tokenizer + normalizer; .NET uses
character-level splitter with targeted normalization around comparison
operators only | Complex SQL with unusual formatting may produce
different resource strings |
| `HTTP_method` / `HTTP_endpoint` | .NET always populates from span
tags; Go agent only populates from newer OTel pipeline paths | Creates
different aggregation keys — Go groups all methods/routes together in
default path |
| `_top_level` metric | Go checks both `_top_level` and `_dd.top_level`;
.NET only checks `_dd.top_level` | Spans using older metric name would
be missed by .NET |
| `span_derived_primary_tags` | Still in Go agent code but implemented
in v1.3.0 RFC, which was reverted; removed from .NET | No current
impact; may need to re-add if spec reverts back |

There is another big elephant in the room, which is perf. The peer tags,
in particular, currently requires a _bunch_ of allocation. I'd rather
defer trying to fight against that to another PR if possible, unless
anyone has some clear ideas 😄

Another aspect I'm not sure about is how this interacts with
@zacharycmontoya's recent work to publish OTLP stats. I took a random
guess and fought the refactoring, but need to verify it.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Pimentel <lucas.pimentel@datadoghq.com>

Author: 3 people

tracer/missing-nullability-files.csv

@@ -43,7 +43,6 @@ src/Datadog.Trace/Agent/IStreamFactory.cs
 src/Datadog.Trace/Agent/MovingAverageKeepRateCalculator.cs
 src/Datadog.Trace/Agent/NullStatsAggregator.cs
 src/Datadog.Trace/Agent/SpanBuffer.cs
-src/Datadog.Trace/Agent/StatsAggregationKey.cs
 src/Datadog.Trace/Agent/StatsAggregator.cs
 src/Datadog.Trace/Agent/StatsBucket.cs
 src/Datadog.Trace/Agent/StatsBuffer.cs

tracer/src/Datadog.Trace/Agent/Api.cs

@@ -1,11 +1,12 @@
-// <copyright file="Api.cs" company="Datadog">
+// <copyright file="Api.cs" company="Datadog">
 // Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
 // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
 // </copyright>
 
 using System;
 using System.Collections.Generic;
 using System.Diagnostics.CodeAnalysis;
+using System.Globalization;
 using System.IO;
 using System.Net.Sockets;
 using System.Threading;
@@ -98,13 +99,14 @@ public void ToggleTracerHealthMetrics(bool enabled)
 
         public Task<bool> Ping() => SendTracesAsync(EmptyPayload, 0, false, 0, 0);
 
-        public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration)
+        public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion)
         {
             _log.Debug("Sending stats to the Datadog Agent.");
 
-            var state = new SendStatsState(stats, bucketDuration);
+            var state = new SendStatsState(stats, bucketDuration, tracerObfuscationVersion);
 
-            return SendWithRetry(_statsEndpoint, _sendStats, state);
+            // We are supposed to be fire and forget for these stats, with no retries
+            return SendWithRetry(_statsEndpoint, _sendStats, state, retryLimit: 0);
         }
 
         public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces, bool statsComputationEnabled, long numberOfDroppedP0Traces, long numberOfDroppedP0Spans, bool apmTracingEnabled = true)
@@ -138,10 +140,9 @@ internal bool LogPartialFlushWarningIfRequired(string agentVersion)
             return false;
         }
 
-        private async Task<bool> SendWithRetry<T>(Uri endpoint, SendCallback<T> callback, T state)
+        private async Task<bool> SendWithRetry<T>(Uri endpoint, SendCallback<T> callback, T state, int retryLimit = 5)
         {
             // retry up to 5 times with exponential back-off
-            var retryLimit = 5;
             var retryCount = 1;
             var sleepDuration = 100; // in milliseconds
 
@@ -218,6 +219,11 @@ private async Task<SendResult> SendStatsAsyncImpl(IApiRequest request, bool isFi
 
             request.AddContainerMetadataHeaders(_containerMetadata);
 
+            if (state.TracerObfuscationVersion > 0)
+            {
+                request.AddHeader("Datadog-Obfuscation-Version", state.TracerObfuscationVersion.ToString(CultureInfo.InvariantCulture));
+            }
+
             using var stream = new MemoryStream();
             state.Stats.Serialize(stream, state.BucketDuration);
 
@@ -441,11 +447,13 @@ private readonly struct SendStatsState
         {
             public readonly StatsBuffer Stats;
             public readonly long BucketDuration;
+            public readonly int TracerObfuscationVersion;
 
-            public SendStatsState(StatsBuffer stats, long bucketDuration)
+            public SendStatsState(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion)
             {
                 Stats = stats;
                 BucketDuration = bucketDuration;
+                TracerObfuscationVersion = tracerObfuscationVersion;
             }
         }
     }

tracer/src/Datadog.Trace/Agent/ApiOtlp.cs

@@ -74,13 +74,14 @@ private enum SendResult
         // return true.
         public Task<bool> Ping() => Task.FromResult(true);
 
-        public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration)
+        public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion)
         {
             _log.Debug("Sending trace stats to the OTLP Metrics endpoint.");
 
-            var state = new SendStatsState(stats, bucketDuration);
+            var state = new SendStatsState(stats, bucketDuration, tracerObfuscationVersion);
 
-            return SendWithRetry(_statsEndpoint, _sendStats, state);
+            // We are supposed to be fire and forget for these stats, with no retries
+            return SendWithRetry(_statsEndpoint, _sendStats, state, retryLimit: 0);
         }
 
         public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces, bool statsComputationEnabled, long numberOfDroppedP0Traces, long numberOfDroppedP0Spans, bool apmTracingEnabled = true)
@@ -92,10 +93,9 @@ public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces,
             return SendWithRetry(_tracesEndpoint, _sendTraces, state);
         }
 
-        private async Task<bool> SendWithRetry<T>(Uri endpoint, SendCallback<T> callback, T state)
+        private async Task<bool> SendWithRetry<T>(Uri endpoint, SendCallback<T> callback, T state, int retryLimit = 5)
         {
             // retry up to 5 times with exponential back-off
-            var retryLimit = 5;
             var retryCount = 1;
             var sleepDuration = 100; // in milliseconds
 
@@ -293,11 +293,13 @@ private readonly struct SendStatsState
         {
             public readonly StatsBuffer Stats;
             public readonly long BucketDuration;
+            public readonly int TracerObfuscationVersion;
 
-            public SendStatsState(StatsBuffer stats, long bucketDuration)
+            public SendStatsState(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion)
             {
                 Stats = stats;
                 BucketDuration = bucketDuration;
+                TracerObfuscationVersion = tracerObfuscationVersion;
             }
         }
     }

tracer/src/Datadog.Trace/Agent/ClientStatsPayload.cs

@@ -24,8 +24,8 @@ public void UpdateDetails(MutableSettings settings)
             => Interlocked.Exchange(ref _settings, CreateSettings(settings));
 
         private static AppSettings CreateSettings(MutableSettings settings)
-            => new(settings.Environment, settings.ServiceVersion, settings.ProcessTags);
+            => new(settings.Environment, settings.ServiceVersion, settings.DefaultServiceName, settings.ProcessTags, settings.GitCommitSha);
 
-        internal sealed record AppSettings(string? Environment, string? Version, ProcessTags? ProcessTags);
+        internal sealed record AppSettings(string? Environment, string? Version, string DefaultServiceName, ProcessTags? ProcessTags, string? GitCommitSha);
     }
 }

tracer/src/Datadog.Trace/Agent/DiscoveryService/AgentConfiguration.cs

@@ -5,6 +5,8 @@
 
 #nullable enable
 
+using System.Collections.Generic;
+
 namespace Datadog.Trace.Agent.DiscoveryService;
 
 internal sealed record AgentConfiguration
@@ -24,7 +26,10 @@ public AgentConfiguration(
         string? containerTagsHash,
         bool clientDropP0,
         bool spanMetaStructs,
-        bool? spanEvents)
+        bool? spanEvents,
+        List<string>? peerTags = null,
+        int obfuscationVersion = 0,
+        AgentTraceFilterConfig? traceFilterConfig = null)
     {
         ConfigurationEndpoint = configurationEndpoint;
         DebuggerEndpoint = debuggerEndpoint;
@@ -41,6 +46,9 @@ public AgentConfiguration(
         ClientDropP0s = clientDropP0;
         SpanMetaStructs = spanMetaStructs;
         SpanEvents = spanEvents ?? false;
+        PeerTags = peerTags;
+        ObfuscationVersion = obfuscationVersion;
+        TraceFilterConfig = traceFilterConfig ?? AgentTraceFilterConfig.Empty;
     }
 
     public string? ConfigurationEndpoint { get; }
@@ -84,4 +92,10 @@ public AgentConfiguration(
     public bool SpanMetaStructs { get; }
 
     public bool SpanEvents { get; }
+
+    public List<string>? PeerTags { get; }
+
+    public int ObfuscationVersion { get; }
+
+    public AgentTraceFilterConfig TraceFilterConfig { get; }
 }

tracer/src/Datadog.Trace/Agent/DiscoveryService/AgentTraceFilterConfig.cs

@@ -0,0 +1,31 @@
+// <copyright file="AgentTraceFilterConfig.cs" company="Datadog">
+// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
+// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
+// </copyright>
+
+#nullable enable
+
+using System.Collections.Generic;
+
+namespace Datadog.Trace.Agent.DiscoveryService;
+
+/// <summary>
+/// Trace-level filtering configuration received from the agent's /info endpoint.
+/// Filters are applied to the root span before stats computation.
+/// </summary>
+internal sealed record AgentTraceFilterConfig(
+    List<string>? FilterTagsRequire,
+    List<string>? FilterTagsReject,
+    List<string>? FilterTagsRegexRequire,
+    List<string>? FilterTagsRegexReject,
+    List<string>? IgnoreResources)
+{
+    public static readonly AgentTraceFilterConfig Empty = new(null, null, null, null, null);
+
+    public bool HasFilters =>
+        FilterTagsRequire is { Count: > 0 } ||
+        FilterTagsReject is { Count: > 0 } ||
+        FilterTagsRegexRequire is { Count: > 0 } ||
+        FilterTagsRegexReject is { Count: > 0 } ||
+        IgnoreResources is { Count: > 0 };
+}

tracer/src/Datadog.Trace/Agent/DiscoveryService/DiscoveryService.cs

@@ -348,6 +348,23 @@ private async Task ProcessDiscoveryResponse(IApiResponse response)
             var clientDropP0 = jObject["client_drop_p0s"]?.Value<bool>() ?? false;
             var spanMetaStructs = jObject["span_meta_structs"]?.Value<bool>() ?? false;
             var spanEvents = jObject["span_events"]?.Value<bool>() ?? false;
+            var peerTags = (jObject["peer_tags"] as JArray)?.Values<string>().Where(x => !string.IsNullOrEmpty(x)).Distinct().OrderBy(x => x).ToList();
+            var obfuscationVersion = jObject["obfuscation_version"]?.Value<int>() ?? 0;
+
+            // Parse trace filter configuration
+            var filterTags = jObject["filter_tags"];
+            var filterTagsRegex = jObject["filter_tags_regex"];
+            var ignoreResources = (jObject["ignore_resources"] as JArray)?.Values<string>().Where(x => !string.IsNullOrEmpty(x)).ToList();
+            var filterTagsRequire = (filterTags?["require"] as JArray)?.Values<string>().Where(x => !string.IsNullOrEmpty(x)).ToList();
+            var filterTagsReject = (filterTags?["reject"] as JArray)?.Values<string>().Where(x => !string.IsNullOrEmpty(x)).ToList();
+            var filterTagsRegexRequire = (filterTagsRegex?["require"] as JArray)?.Values<string>().Where(x => !string.IsNullOrEmpty(x)).ToList();
+            var filterTagsRegexReject = (filterTagsRegex?["reject"] as JArray)?.Values<string>().Where(x => !string.IsNullOrEmpty(x)).ToList();
+
+            AgentTraceFilterConfig? traceFilterConfig = null;
+            if (ignoreResources is not null || filterTagsRequire is not null || filterTagsReject is not null || filterTagsRegexRequire is not null || filterTagsRegexReject is not null)
+            {
+                traceFilterConfig = new AgentTraceFilterConfig(filterTagsRequire!, filterTagsReject!, filterTagsRegexRequire!, filterTagsRegexReject!, ignoreResources!);
+            }
 
             var discoveredEndpoints = (jObject["endpoints"] as JArray)?.Values<string>().ToArray();
             string? configurationEndpoint = null;
@@ -436,7 +453,10 @@ private async Task ProcessDiscoveryResponse(IApiResponse response)
                 containerTagsHash: _serviceRemappingHash.ContainerTagsHash, // either the value just received, or the one we stored before (prevents overriding with null)
                 clientDropP0: clientDropP0,
                 spanMetaStructs: spanMetaStructs,
-                spanEvents: spanEvents);
+                spanEvents: spanEvents,
+                peerTags: peerTags!,
+                obfuscationVersion: obfuscationVersion,
+                traceFilterConfig: traceFilterConfig);
 
             // Save the hash, whether the details we care about changed or not
             _configurationHash = HexString.ToHexString(sha256.Hash);

tracer/src/Datadog.Trace/Agent/IApi.cs

@@ -16,6 +16,6 @@ internal interface IApi
 
         Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces, bool statsComputationEnabled, long numberOfDroppedP0Traces, long numberOfDroppedP0Spans, bool apmTracingEnabled = true);
 
-        Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration);
+        Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion);
     }
 }

tracer/src/Datadog.Trace/Agent/IStatsAggregator.cs

@@ -4,6 +4,7 @@
 // </copyright>
 
 using System;
+using System.Collections.Generic;
 using System.Threading.Tasks;
 using Datadog.Trace.SourceGenerators;
 
@@ -50,5 +51,7 @@ internal interface IStatsAggregator
         SpanCollection ProcessTrace(in SpanCollection trace);
 
         Task DisposeAsync();
+
+        StatsAggregationKey BuildKey(Span span, out List<byte[]> utf8PeerTags);
     }
 }

tracer/src/Datadog.Trace/Agent/ManagedApi.cs

@@ -1,4 +1,4 @@
-// <copyright file="ManagedApi.cs" company="Datadog">
+// <copyright file="ManagedApi.cs" company="Datadog">
 // Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
 // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
 // </copyright>
@@ -62,6 +62,6 @@ void UpdateApi(ExporterSettings exporterSettings, bool healthMetricsEnabled)
     public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces, bool statsComputationEnabled, long numberOfDroppedP0Traces, long numberOfDroppedP0Spans, bool apmTracingEnabled = true)
         => Volatile.Read(ref _api).SendTracesAsync(traces, numberOfTraces, statsComputationEnabled, numberOfDroppedP0Traces, numberOfDroppedP0Spans, apmTracingEnabled);
 
-    public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration)
-        => Volatile.Read(ref _api).SendStatsAsync(stats, bucketDuration);
+    public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion)
+        => Volatile.Read(ref _api).SendStatsAsync(stats, bucketDuration, tracerObfuscationVersion);
 }