[Tracing] Add experimental http/protobuf support for OTLP traces export (#8645)

## Summary of changes
Adds `http/protobuf` support to the OTLP traces exporter introduced in
#8211, which introduced the exporter and `http/json` support.

This feature is enabled by setting `OTEL_TRACES_EXPORTER=otlp` and
`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf` (or the parent
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`).

## Reason for change
When #8211 landed, `http/json` was the only functional OTLP traces
protocol — setting any other protocol would fall back (with a log
message) to the Datadog MessagePack encoding.

`http/protobuf` is the OTLP default protocol per the OpenTelemetry
specification and is what most OTel collectors, Datadog Agent included,
expect to receive, so we need to support it to be useful out-of-the-box
for users adopting the OTLP traces export.

## Implementation details

### Serializer
Adds `Datadog.Trace.OpenTelemetry.Traces.OtlpTracesProtobufSerializer`,
a new `ISpanBufferSerializer` that emits the [OTLP
`ExportTraceServiceRequest`](https://github.com/open-telemetry/opentelemetry-proto/blob/v1.2.0/opentelemetry/proto/trace/v1/trace.proto)
protobuf payload directly into `SpanBuffer`'s output buffer:
- The `resource_spans` and `scope_spans` envelopes are opened on the
first `SerializeSpans` call, span entries are appended per chunk, and
`FinishBody` patches the two length-prefix placeholders using absolute
positions into the destination buffer.
- The serializer respects the same per-span / per-event / per-link
attribute count limits (`128`) as `OtlpTracesJsonSerializer`, and
re-uses the `OtlpMapper` logic to map resource attributes and tags onto
the serialized spans.
- `OtlpTracesProtobufSerializer` is **stateful** (it tracks the two
length-placeholder positions), so `AgentWriter` now creates a fresh
instance for the front and back buffers rather than sharing one — the
previous shared `OtlpTracesJsonSerializer` was stateless and didn't need
this.

### Vendored OpenTelemetry SDK
Builds on the protobuf primitives vendored from the OpenTelemetry .NET
SDK in #7653 (`ProtobufSerializer`, `ProtobufWireType`):
- Adds new field-number constant files:
`ProtobufOtlpTraceFieldNumberConstants` and
`ProtobufOtlpResourceFieldNumberConstants` (split out so the OTLP trace
serializer only depends on what it needs).
- Removes the `#if NETCOREAPP3_1_OR_GREATER` guard from the vendored
`ProtobufSerializer` — now that we vendor a `Span<>` polyfill the
serializer compiles and runs on all supported runtimes (.NET Framework
4.6.1+).
- Drops the matching `#if NET6_0_OR_GREATER` guard around the
`SubmitsOtlpTraces` integration test (`http/protobuf` and `http/json`
both run on .NET Framework now too).

### Wiring
- `ExporterSettings`: `OtlpProtocol.HttpProtobuf` now resolves to
`TracesEncoding.OtlpProtobuf` instead of falling back to `DatadogV0_4`.
- `AgentWriter.GetSpanSerializer`: selects
`OtlpTracesProtobufSerializer` when `TracesEncoding == OtlpProtobuf`.
- `ApiOtlp.SendTracesAsync`: chooses `application/x-protobuf` vs
`application/json` for the request `Content-Type` based on
`_tracesEncoding`. Adds the `application/x-protobuf` MIME type constant
to `Datadog.Trace.Agent.Transports.MimeTypes`.
- `supported-configurations.yaml`: documents `http/protobuf` as a valid
value for `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` and regenerates
`ConfigurationKeys.OpenTelemetry.g.cs` for all TFMs.

## Test coverage
- **Unit tests**: New `OtlpTracesProtobufSerializerTests` exercises the
serializer end-to-end using `Google.Protobuf` and a little custom
parsing. Test cases include:
  - Empty buffer (`FinishBody` returns 0)
  - Multiple traces reported in a single-chunk
  - Serializing span events/links/status
  - Exceeding buffer `maxSize` honoring behavior
  - Exceeding span limits
- **Integration tests**: `OpenTelemetrySdkTests.SubmitsOtlpTraces` now
runs both `http/json` and `http/protobuf` cases for the Datadog SDK on
all TFMs.
- **Snapshot unification**: To keep one Datadog-SDK snapshot regardless
of OTLP protocol, snapshots for `http/protobuf` are normalized to the
`http/json` shape via scrubbers that translate the test agent's protobuf
rendering (snake_case field names, string-form enum values like
`SPAN_KIND_SERVER`) into the JSON serializer's shape (camelCase field
names, integer enum values).
- **Snapshot rename**: `SubmitsOtlpTraces_DD_http_json.verified.txt`
renamed to `SubmitsOtlpTraces_DD.verified.txt` (one snapshot now covers
both protocols).

## Other details
With this PR, `http/json` and `http/protobuf` are both functional;
`grpc` still falls back to Datadog MessagePack encoding with a startup
warning.

### Follow-up work
- Implement the `grpc` OTLP traces protocol
- Add OTLP traces telemetry counts (the `TODO: Telemetry - Record OTLP
Traces API submissions` in `ApiOtlp.SendTracesAsync` still applies)
- Honor `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` in `ApiOtlp` (still unused)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: zacharycmontoya

tracer/src/Datadog.Trace/Agent/AgentWriter.cs

@@ -94,16 +94,18 @@ internal AgentWriter(IApi api, IStatsAggregator? statsAggregator, IStatsdManager
             _batchInterval = batchInterval;
             _traceKeepRateCalculator = traceKeepRateCalculator;
 
-            ISpanBufferSerializer spanBufferSerializer = api.TracesEncoding switch
+            ISpanBufferSerializer CreateSpanSerializer() => api.TracesEncoding switch
             {
                 TracesEncoding.OtlpJson => new OtlpTracesJsonSerializer(),
+                TracesEncoding.OtlpProtobuf => new OtlpTracesProtobufSerializer(),
                 _ => new SpanBufferMessagePackSerializer(SpanFormatterResolver.Instance),
             };
 
             _forceFlush = new TaskCompletionSource<bool>(TaskOptions);
 
-            _frontBuffer = new SpanBuffer(maxBufferSize, spanBufferSerializer);
-            _backBuffer = new SpanBuffer(maxBufferSize, spanBufferSerializer);
+            // OtlpTracesProtobufSerializer is stateful, so we need to create a new instance for each buffer.
+            _frontBuffer = new SpanBuffer(maxBufferSize, CreateSpanSerializer());
+            _backBuffer = new SpanBuffer(maxBufferSize, CreateSpanSerializer());
             _activeBuffer = _frontBuffer;
 
             _apmTracingEnabled = apmTracingEnabled;

tracer/src/Datadog.Trace/Agent/ApiOtlp.cs

@@ -3,8 +3,6 @@
 // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
 // </copyright>
 
-#if NET6_0_OR_GREATER
-
 using System;
 using System.Collections.Generic;
 using System.IO;
@@ -37,7 +35,6 @@ internal sealed class ApiOtlp : IApi
         private readonly Uri _statsEndpoint; // This endpoint is passed for the _sendStats callback, but otherwise unused
         private readonly SendCallback<SendStatsState> _sendStats;
         private readonly SendCallback<SendTracesState> _sendTraces;
-        private readonly Datadog.Trace.OpenTelemetry.Metrics.OtlpExporter _metricsExporter;
 
         public ApiOtlp(IApiRequestFactory apiRequestFactory, TracerSettings settings, ExporterSettings exporterSettings, IDatadogLogger log = null)
         {
@@ -53,8 +50,6 @@ public ApiOtlp(IApiRequestFactory apiRequestFactory, TracerSettings settings, Ex
             _tracesHeaders = exporterSettings.OtlpTracesHeaders ?? [];
             _statsEndpoint = exporterSettings.OtlpMetricsEndpoint;
             _log.Debug("Using traces endpoint {TracesEndpoint}", _tracesEndpoint.ToString());
-
-            _metricsExporter = new Datadog.Trace.OpenTelemetry.Metrics.OtlpExporter(settings, exporterSettings);
         }
 
         private delegate Task<SendResult> SendCallback<T>(IApiRequest request, bool isFinalTry, T state);
@@ -196,8 +191,13 @@ private async Task<SendResult> SendTracesAsyncImpl(IApiRequest request, bool fin
                 try
                 {
                     // TODO: Telemetry - Record OTLP Traces API submissions
-                    // TODO: Add more precise logic for "application/x-protobuf" vs "application/json"
-                    response = await request.PostAsync(traces, MimeTypes.Json).ConfigureAwait(false);
+                    var contentType = _tracesEncoding switch
+                    {
+                        TracesEncoding.OtlpProtobuf => MimeTypes.XProtobuf,
+                        _ => MimeTypes.Json,
+                    };
+
+                    response = await request.PostAsync(traces, contentType).ConfigureAwait(false);
                 }
                 catch (Exception)
                 {
@@ -304,4 +304,3 @@ public SendStatsState(StatsBuffer stats, long bucketDuration, int tracerObfuscat
         }
     }
 }
-#endif

tracer/src/Datadog.Trace/Agent/ManagedApiOtlp.cs

@@ -3,8 +3,6 @@
 // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
 // </copyright>
 
-#if NET6_0_OR_GREATER
-
 #nullable enable
 
 using System;
@@ -50,5 +48,3 @@ public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces,
     public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion)
         => Volatile.Read(ref _api).SendStatsAsync(stats, bucketDuration, tracerObfuscationVersion);
 }
-
-#endif

tracer/src/Datadog.Trace/Agent/Transports/MimeTypes.cs

@@ -9,6 +9,7 @@ internal static class MimeTypes
     {
         public const string MsgPack = "application/msgpack";
         public const string Json = "application/json";
+        public const string XProtobuf = "application/x-protobuf";
         public const string MultipartFormData = "multipart/form-data";
         public const string PlainText = "text/plain";
         public const string Gzip = "application/gzip";

tracer/src/Datadog.Trace/Configuration/ExporterSettings.cs

@@ -112,8 +112,7 @@ internal ExporterSettings(Raw rawSettings, Func<string, bool> fileExists, IConfi
             {
                 TracesEncoding = otlpTraceSettings.OtlpProtocol switch
                 {
-                    OtlpProtocol.Grpc => TracesEncoding.DatadogV0_4,
-                    OtlpProtocol.HttpProtobuf => TracesEncoding.DatadogV0_4,
+                    OtlpProtocol.HttpProtobuf => TracesEncoding.OtlpProtobuf,
                     OtlpProtocol.HttpJson => TracesEncoding.OtlpJson,
                     _ => TracesEncoding.DatadogV0_4,
                 };

tracer/src/Datadog.Trace/Configuration/supported-configurations.yaml

@@ -2676,7 +2676,7 @@ supportedConfigurations:
     documentation: |-
       Configuration key to set the OTLP protocol for traces export.
       Takes precedence over <see cref="ExporterOtlpProtocol"/>.
-      Valid values: http/json. Default: http/json
+      Valid values: http/json, http/protobuf. Default: http/json
   OTEL_EXPORTER_OTLP_TRACES_TIMEOUT:
   - implementation: B
     type: int

tracer/src/Datadog.Trace/Generated/net461/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator/ConfigurationKeys.OpenTelemetry.g.cs

@@ -136,7 +136,7 @@ internal static class OpenTelemetry
         /// <summary>
         /// Configuration key to set the OTLP protocol for traces export.
         /// Takes precedence over <see cref="ExporterOtlpProtocol"/>.
-        /// Valid values: http/json. Default: http/json
+        /// Valid values: http/json, http/protobuf. Default: http/json
         /// </summary>
         public const string ExporterOtlpTracesProtocol = "OTEL_EXPORTER_OTLP_TRACES_PROTOCOL";
 

tracer/src/Datadog.Trace/Generated/net6.0/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator/ConfigurationKeys.OpenTelemetry.g.cs

@@ -136,7 +136,7 @@ internal static class OpenTelemetry
         /// <summary>
         /// Configuration key to set the OTLP protocol for traces export.
         /// Takes precedence over <see cref="ExporterOtlpProtocol"/>.
-        /// Valid values: http/json. Default: http/json
+        /// Valid values: http/json, http/protobuf. Default: http/json
         /// </summary>
         public const string ExporterOtlpTracesProtocol = "OTEL_EXPORTER_OTLP_TRACES_PROTOCOL";
 

tracer/src/Datadog.Trace/Generated/netcoreapp3.1/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator/ConfigurationKeys.OpenTelemetry.g.cs

@@ -136,7 +136,7 @@ internal static class OpenTelemetry
         /// <summary>
         /// Configuration key to set the OTLP protocol for traces export.
         /// Takes precedence over <see cref="ExporterOtlpProtocol"/>.
-        /// Valid values: http/json. Default: http/json
+        /// Valid values: http/json, http/protobuf. Default: http/json
         /// </summary>
         public const string ExporterOtlpTracesProtocol = "OTEL_EXPORTER_OTLP_TRACES_PROTOCOL";
 

tracer/src/Datadog.Trace/Generated/netstandard2.0/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator/ConfigurationKeys.OpenTelemetry.g.cs

@@ -136,7 +136,7 @@ internal static class OpenTelemetry
         /// <summary>
         /// Configuration key to set the OTLP protocol for traces export.
         /// Takes precedence over <see cref="ExporterOtlpProtocol"/>.
-        /// Valid values: http/json. Default: http/json
+        /// Valid values: http/json, http/protobuf. Default: http/json
         /// </summary>
         public const string ExporterOtlpTracesProtocol = "OTEL_EXPORTER_OTLP_TRACES_PROTOCOL";