Skip to content

Commit 74a5638

Browse files
committed
feat(trace-sampler): add shared datadog-agent-trace-sampler crate
Add a dependency-free leaf crate that ports the Go trace agent error sampler (ScoreSampler targeting ErrorTPS) so serverless agents can rescue error traces from an agent-side P0 drop, keeping error visibility under aggressive sampling. The crate takes primitives in (SpanView/TraceView) and returns a SampleDecision out, exposing no protobuf Span type, so consumers pinning different libdatadog revisions can share it. bottlecap consumes it via the existing serverless-components git dependency; SCL wiring follows later. Ports FNV-1a signatures, the 6x5s rolling-bucket TPS budget with cascade and 20% rate-increase cap, deterministic sample-by-rate, and cardinality shrink, with unit tests mirroring the Go table tests. APMSVLS-469 🤖
1 parent d05932a commit 74a5638

6 files changed

Lines changed: 1109 additions & 0 deletions

File tree

Cargo.lock

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Copyright 2025-Present Datadog, Inc. https://www.datadoghq.com/
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
[package]
5+
name = "datadog-agent-trace-sampler"
6+
version = "0.1.0"
7+
edition.workspace = true
8+
license.workspace = true
9+
homepage.workspace = true
10+
repository.workspace = true
11+
12+
[dependencies]
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Datadog Agent Trace Sampler
2+
3+
Agent-side trace sampling shared across the serverless agents (bottlecap and the
4+
Serverless Compatibility Layer).
5+
6+
This crate is a dependency-free 1:1 port of the Go trace agent's **error sampler**
7+
(`ScoreSampler` targeting `ErrorTPS`, from `pkg/trace/sampler/` in
8+
`DataDog/datadog-agent`). The error sampler is a *rescue* sampler: after an agent
9+
decides to drop a trace, the trace gets a second look, and if it contains an
10+
error it is kept, up to a budget of `target_tps` error traces per second
11+
distributed fairly across distinct trace signatures. This guarantees error
12+
visibility even under aggressive sampling.
13+
14+
## Why dependency-free
15+
16+
The public API takes primitives in (`SpanView` / `TraceView`) and returns a
17+
`SampleDecision` out; it never exposes a protobuf `Span` type. This lets
18+
consumers that pin different `libdatadog` revisions share the crate without
19+
compiling incompatible `pb::Span` types into their build graphs.
20+
21+
## Usage
22+
23+
```rust
24+
use datadog_agent_trace_sampler::{
25+
ErrorSamplerConfig, ErrorsSampler, SampleDecision, SpanView, TraceView,
26+
};
27+
28+
let mut sampler = ErrorsSampler::new(ErrorSamplerConfig::default());
29+
30+
let spans = [SpanView {
31+
service: "web",
32+
name: "web.request",
33+
resource: "GET /",
34+
error: true,
35+
http_status_code: Some("500"),
36+
error_type: None,
37+
}];
38+
let trace = TraceView {
39+
env: "prod",
40+
trace_id: 0xdead_beef,
41+
root_index: 0,
42+
root_global_sample_rate: 1.0,
43+
spans: &spans,
44+
};
45+
46+
// `now_unix_secs` drives the rolling window and is passed in (not read from a
47+
// clock) so the crate stays dependency-free and deterministically testable.
48+
match sampler.sample(1_700_000_000, &trace) {
49+
SampleDecision::Keep { errors_sr } => {
50+
// caller stamps `_dd.errors_sr = errors_sr` on the root span
51+
}
52+
SampleDecision::Drop => {
53+
// the pending agent-side drop proceeds
54+
}
55+
}
56+
```
57+
58+
`ErrorsSampler::sample` takes `&mut self` (the rolling buffer and rate map mutate
59+
on every call). Consumers that share one sampler across threads wrap it in
60+
`Arc<Mutex<ErrorsSampler>>`.
61+
62+
Setting `target_tps` to `0.0` disables the sampler: every candidate returns
63+
`SampleDecision::Drop` (i.e. nothing is rescued).
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
// Copyright 2025-Present Datadog, Inc. https://www.datadoghq.com/
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
#![cfg_attr(not(test), deny(clippy::panic))]
5+
#![cfg_attr(not(test), deny(clippy::unwrap_used))]
6+
#![cfg_attr(not(test), deny(clippy::expect_used))]
7+
#![cfg_attr(not(test), deny(clippy::todo))]
8+
#![cfg_attr(not(test), deny(clippy::unimplemented))]
9+
10+
//! Agent-side trace sampling shared across serverless agents (bottlecap and the
11+
//! Serverless Compatibility Layer).
12+
//!
13+
//! This crate is a dependency-free 1:1 port of the Go trace agent's error
14+
//! sampler (`ScoreSampler` targeting `ErrorTPS`). The error sampler is a
15+
//! *rescue* sampler: after an agent decides to drop a trace, the trace gets a
16+
//! second look, and if it contains an error it is kept, up to a budget of
17+
//! `target_tps` error traces per second distributed fairly across distinct trace
18+
//! signatures. This guarantees error visibility even under aggressive sampling.
19+
//!
20+
//! The public API takes primitives in and returns a decision out (no protobuf
21+
//! `Span` type), so consumers pinning different `libdatadog` revisions can share
22+
//! it without compiling incompatible span types into their build graphs.
23+
//!
24+
//! # Example
25+
//!
26+
//! ```
27+
//! use datadog_agent_trace_sampler::{
28+
//! ErrorSamplerConfig, ErrorsSampler, SampleDecision, SpanView, TraceView,
29+
//! };
30+
//!
31+
//! let mut sampler = ErrorsSampler::new(ErrorSamplerConfig::default());
32+
//! let spans = [SpanView {
33+
//! service: "web",
34+
//! name: "web.request",
35+
//! resource: "GET /",
36+
//! error: true,
37+
//! http_status_code: Some("500"),
38+
//! error_type: None,
39+
//! }];
40+
//! let trace = TraceView {
41+
//! env: "prod",
42+
//! trace_id: 0xdead_beef,
43+
//! root_index: 0,
44+
//! root_global_sample_rate: 1.0,
45+
//! spans: &spans,
46+
//! };
47+
//! match sampler.sample(/* now_unix_secs */ 1_700_000_000, &trace) {
48+
//! SampleDecision::Keep { errors_sr } => {
49+
//! // caller stamps `_dd.errors_sr = errors_sr` on the root span
50+
//! let _ = errors_sr;
51+
//! }
52+
//! SampleDecision::Drop => { /* the pending drop proceeds */ }
53+
//! }
54+
//! ```
55+
56+
mod score_sampler;
57+
mod signature;
58+
59+
pub use score_sampler::ErrorsSampler;
60+
pub use signature::Signature;
61+
62+
/// A read-only view of a single span, holding only the fields the sampler needs.
63+
///
64+
/// `http_status_code` and `error_type` come from the span's `meta` map keys
65+
/// `http.status_code` and `error.type` respectively.
66+
#[derive(Debug, Clone, Copy)]
67+
pub struct SpanView<'a> {
68+
pub service: &'a str,
69+
pub name: &'a str,
70+
pub resource: &'a str,
71+
pub error: bool,
72+
pub http_status_code: Option<&'a str>,
73+
pub error_type: Option<&'a str>,
74+
}
75+
76+
/// A read-only view of a trace chunk to be sampled.
77+
#[derive(Debug, Clone, Copy)]
78+
pub struct TraceView<'a> {
79+
pub env: &'a str,
80+
pub trace_id: u64,
81+
/// Index of the root span within `spans`.
82+
pub root_index: usize,
83+
/// The root span's global sample rate (`metrics["_sample_rate"]`), default 1.0.
84+
pub root_global_sample_rate: f64,
85+
pub spans: &'a [SpanView<'a>],
86+
}
87+
88+
/// Configuration for the error sampler.
89+
#[derive(Debug, Clone, Copy)]
90+
pub struct ErrorSamplerConfig {
91+
/// Target error traces per second (`ErrorTPS`). `0.0` disables the sampler
92+
/// (every candidate is dropped, i.e. never rescued).
93+
pub target_tps: f64,
94+
/// Extra raw sampling rate applied on top of the computed rate.
95+
pub extra_sample_rate: f64,
96+
}
97+
98+
impl Default for ErrorSamplerConfig {
99+
/// Matches the Go agent defaults: `ErrorTPS = 10`, `ExtraSampleRate = 1.0`.
100+
fn default() -> Self {
101+
ErrorSamplerConfig {
102+
target_tps: 10.0,
103+
extra_sample_rate: 1.0,
104+
}
105+
}
106+
}
107+
108+
/// The outcome of sampling a trace.
109+
#[derive(Debug, PartialEq)]
110+
pub enum SampleDecision {
111+
/// Keep (rescue) the trace. The caller should stamp `_dd.errors_sr` on the
112+
/// root span with `errors_sr`.
113+
Keep { errors_sr: f64 },
114+
/// Drop the trace (do not rescue it); the pending agent-side drop proceeds.
115+
Drop,
116+
}

0 commit comments

Comments
 (0)