Skip to content

Commit 533dc1f

Browse files
dougqhclaude
andcommitted
Hoist stats whole_key collapse counter off the per-dropped-span map lookup
The collapsed:whole_key counter is incremented once per dropped span on the aggregator thread (Aggregator, findOrInsert == null branch), unlike every other collapse reason which is batched once per reporting cycle. Under a cardinality explosion the aggregate table pins at cap and every arriving span is dropped, turning that path hot on an already-bottlenecked thread -- yet it went through a ConcurrentHashMap.computeIfAbsent lookup on StatsMetrics per span. Pre-create the whole_key TaggedCounter at construction (still registered in the reason map so the telemetry drain picks it up unchanged) and add a dedicated onWholeKeyCollapse() that increments the cached reference directly. The hot path no longer touches the map; the batched reasons still route through onCollapsedSpans as before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 302b882 commit 533dc1f

2 files changed

Lines changed: 20 additions & 5 deletions

File tree

dd-trace-core/src/main/java/datadog/trace/common/metrics/Aggregator.java

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,6 @@ final class Aggregator implements Runnable {
1616

1717
private static final long DEFAULT_SLEEP_MILLIS = 10;
1818

19-
// Telemetry collapse reason for a whole-key drop (aggregate table at cap, no evictable entry);
20-
// mirrors the statsd "collapsed:whole_key" tag on datadog.tracer.stats.collapsed_spans.
21-
private static final String COLLAPSED_WHOLE_KEY_TAG = "collapsed:whole_key";
22-
2319
private static final Logger log = LoggerFactory.getLogger(Aggregator.class);
2420

2521
private final MessagePassingQueue<InboxItem> inbox;
@@ -161,7 +157,7 @@ public void accept(InboxItem item) {
161157
} else {
162158
// table at cap with no stale entry available to evict
163159
healthMetrics.onStatsAggregateDropped();
164-
StatsMetrics.getInstance().onCollapsedSpans(COLLAPSED_WHOLE_KEY_TAG, 1);
160+
StatsMetrics.getInstance().onWholeKeyCollapse();
165161
}
166162
}
167163
}

internal-api/src/main/java/datadog/trace/api/metrics/StatsMetrics.java

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,23 @@
2424
*/
2525
public final class StatsMetrics {
2626
static final String COLLAPSED_SPANS = "stats.collapsed_spans";
27+
static final String COLLAPSED_WHOLE_KEY = "collapsed:whole_key";
2728

2829
private static final StatsMetrics INSTANCE = new StatsMetrics();
2930

3031
// reason tag (e.g. "collapsed:additional_metric_tags") -> counter. Created on first collapse for
3132
// that reason; the reason set is bounded, so this never grows unboundedly.
3233
private final ConcurrentMap<String, TaggedCounter> collapsedByReason = new ConcurrentHashMap<>();
3334

35+
// The one reason counted per dropped span on the hot aggregator path (aggregate table at cap);
36+
// every other reason is batched once per reporting cycle. Pre-created and cached so the per-span
37+
// increment is a direct counter hit rather than a map lookup -- this matters precisely when a
38+
// cardinality explosion pins the table at cap and every arriving span is dropped, turning a cold
39+
// path hot. Still registered in the map above, so the telemetry drain sees it with the rest.
40+
private final TaggedCounter wholeKeyCollapses =
41+
this.collapsedByReason.computeIfAbsent(
42+
COLLAPSED_WHOLE_KEY, tag -> new TaggedCounter(COLLAPSED_SPANS, tag));
43+
3444
public static StatsMetrics getInstance() {
3545
return INSTANCE;
3646
}
@@ -51,6 +61,15 @@ public void onCollapsedSpans(String reason, long count) {
5161
.addAndGet(count);
5262
}
5363

64+
/**
65+
* Records a single whole-key collapse: a span dropped because the aggregate table was at cap with
66+
* no entry to evict. Increments the pre-created {@link #COLLAPSED_WHOLE_KEY} counter directly,
67+
* keeping the per-dropped-span aggregator path off the reason map.
68+
*/
69+
public void onWholeKeyCollapse() {
70+
this.wholeKeyCollapses.counter.addAndGet(1);
71+
}
72+
5473
public Collection<TaggedCounter> getTaggedCounters() {
5574
return this.collapsedByReason.values();
5675
}

0 commit comments

Comments
 (0)