Conversation
…n_extension is enabled When `compute_trace_stats_on_extension` is true, traces with a sampling priority of <= 0 (`AUTO_DROP` or `USER_DROP`) are now dropped and not flushed to the Datadog backend. Stats computation is unaffected: dropped traces are still processed and their stats are sent to the concentrator. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_processed_traces Instead of partitioning raw traces and calling process_traces twice, the sampling-out filtering now happens inside ServerlessTraceProcessor::process_traces. The payloads_for_stats clone (for stats) is taken before the filter, so stats always include sampled-out traces. The backend payload has sampled-out chunks removed in-place. send_processed_traces is simplified to a single process_traces call, with stats computed before the backend send. The OwnedTracerHeaderTags upfront conversion is also removed since only one process_traces call is needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When compute_trace_stats_on_extension is true and all trace chunks are filtered out due to sampling priority, process_traces now returns Option::None for the SendDataBuilderInfo instead of constructing an empty payload. Callers skip the backend send when None is received. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
||
| /// Sentinel value used by `collect_pb_trace_chunks` when `_sampling_priority_v1` is absent. | ||
| const CHUNK_PRIORITY_NOT_SET: i32 = i8::MIN as i32; | ||
|
|
There was a problem hiding this comment.
For the meaning of sampling priority values, see https://github.com/DataDog/datadog-agent/blob/87b54753cff79a75fa9c68390d838735b46023b0/pkg/trace/sampler/sampler.go#L50-L63
There was a problem hiding this comment.
Do we have these available in libdatadog by any chance?
There was a problem hiding this comment.
Good point. Imported the const from libdatadog.
lym953
changed the title
There was a problem hiding this comment.
Pull request overview
This PR updates the trace processing pipeline so that when compute_trace_stats_on_extension is enabled, sampled-out traces (sampling priority <= 0) are excluded from the backend flush while still being included in the local stats computation.
Changes:
- Update
TraceProcessor::process_tracesto optionally return no backend payload when all chunks are sampled out. - Filter sampled-out trace chunks from the backend payload when
compute_trace_stats_on_extensionis enabled, while preserving unfiltered payloads for stats generation. - Add unit tests covering partial and full sampled-out dropping behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
bottlecap/src/traces/trace_processor.rs |
Filters sampled-out chunks from the backend payload (when configured), returns None when nothing remains to flush, and adds unit tests for these behaviors. |
bottlecap/src/otlp/agent.rs |
Adapts OTLP ingestion path to handle an optional backend payload (skip enqueue when None). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| Some(SendDataBuilderInfo::new( | ||
| builder, | ||
| body_size, | ||
| owned_header_tags, | ||
| )), |
There was a problem hiding this comment.
After filtering sampled-out chunks, body_size may significantly overestimate the actual payload that will be queued and batched (it’s still based on the pre-filter trace vector). Since SendDataBuilderInfo.size is used by TraceAggregator to enforce the 3.2MB batch limit, this can lead to unnecessarily small batches / extra flushes when many chunks are dropped. Consider recomputing/adjusting the size based on the filtered payload (or at least based on kept chunk/span counts) when compute_trace_stats_on_extension filtering is applied.
There was a problem hiding this comment.
Good point. Will address it in a separate PR.
…bdd-trace-normalization Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Summary
compute_trace_stats_on_extensionistrue, traces with sampling priority<= 0(AUTO_DROP/USER_DROP) are now dropped and not flushed to the Datadog backend.compute_trace_stats_on_extensionisfalse(the current default), traces are not dropped because all traces need to be sent to Datadog so trace stats can be computed.process_tracesand their stats are forwarded to the stats concentrator.Test plan
Manual test
Steps
Result
Logs show that
aws.lambdaspan for some of the 6 invocations were not flushed to Datadog.🤖 Partially generated with Claude Code