Skip to content

Commit 50cecf3

Browse files
dougqhclaude
andcommitted
Document the client-side-stats package design (package-info)
Add a package-info.java for datadog.trace.common.metrics giving a two-minute mental model of the re-architected client-side-stats (CSS) pipeline: an at-a-glance ASCII flow diagram plus prose on the single-writer threading model, the per-interval aggregate/flush cycle, the bound->sentinel->report-per-cycle cardinality invariant, and the three cardinality-handler families. The per-class javadocs carry the mechanics; this is the top-level shape for a newcomer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 27bc29c commit 50cecf3

1 file changed

Lines changed: 80 additions & 0 deletions

File tree

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
/**
2+
* Client-side trace statistics (CSS): the tracer computes per-interval aggregate stats -- hit
3+
* counts, error counts, and latency histograms grouped by a tuple of span labels (resource,
4+
* service, operation, span kind, HTTP/gRPC status, peer tags, ...) -- and ships them to the Datadog
5+
* agent every reporting interval. This lets the backend show accurate metrics even when individual
6+
* spans are sampled away.
7+
*
8+
* <h2>At a glance</h2>
9+
*
10+
* <pre>{@code
11+
* App / producer threads Single aggregator thread
12+
* ---------------------- ------------------------
13+
*
14+
* span finishes (1) drain inbox
15+
* | |
16+
* v MPSC inbox v
17+
* build immutable (lock-free hand-off) (2) canonicalize each label through its
18+
* SpanSnapshot -----------------------> cardinality handler (Core / PeerTag /
19+
* | AdditionalTags); overflow -> sentinel
20+
* v |
21+
* (no locks, no v
22+
* mutable state) (3) fold into AggregateTable, keyed by the
23+
* canonical labels (counts + histograms)
24+
* |
25+
* v every ~10s (reporting interval)
26+
* (4) flush table via MetricWriter (msgpack
27+
* or OTLP) -> agent, then clear
28+
* }</pre>
29+
*
30+
* <h2>Threading model (the load-bearing rule)</h2>
31+
*
32+
* Producer application threads do almost nothing: each captures an immutable {@link
33+
* datadog.trace.common.metrics.SpanSnapshot} of the finished span and hands it off through a
34+
* lock-free MPSC {@code inbox}. A <em>single</em> aggregator thread ({@link
35+
* datadog.trace.common.metrics.Aggregator}) drains that inbox and owns <em>all</em> mutable state
36+
* -- the {@link datadog.trace.common.metrics.AggregateTable} and every cardinality handler -- as
37+
* its sole writer. None of that state is synchronized; correctness rests entirely on the
38+
* single-writer invariant. Any cross-thread request that must mutate (e.g. clearing the table on
39+
* downgrade) does not touch the state directly -- it is funneled onto the aggregator thread as a
40+
* signal through the same inbox. If you add a mutation path, it goes through the inbox too.
41+
*
42+
* <h2>The cycle</h2>
43+
*
44+
* The aggregator folds each drained snapshot into an {@link
45+
* datadog.trace.common.metrics.AggregateEntry}, keyed by the canonical form of its labels, updating
46+
* that entry's counters and histograms. Once per reporting interval (~10s) it flushes the whole
47+
* table through a {@link datadog.trace.common.metrics.MetricWriter} -- msgpack ({@link
48+
* datadog.trace.common.metrics.SerializingMetricWriter}) or OTLP -- and clears it for the next
49+
* interval. Buckets are per-interval deltas, not cumulative.
50+
*
51+
* <h2>Cardinality is the thing that must stay bounded</h2>
52+
*
53+
* Grouping keys come from span/user data, so distinct values -- and therefore the series count, its
54+
* memory, and its backend cost -- would grow without bound if left alone. Every label is
55+
* canonicalized through a {@link datadog.trace.common.metrics.TagCardinalityHandler} that caps the
56+
* number of distinct values per tag and collapses any overflow into a shared sentinel, so
57+
* over-cardinality folds into one entry rather than exploding the table. The collapses are counted
58+
* per cycle and reported (to {@link datadog.trace.core.monitor.HealthMetrics} and, rate-limited,
59+
* through {@link datadog.trace.common.metrics.CardinalityLimitReporter}) -- so the bounding is
60+
* observable without the bookkeeping itself becoming unbounded.
61+
*
62+
* <h2>Three handler families</h2>
63+
*
64+
* <ul>
65+
* <li>{@link datadog.trace.common.metrics.CoreHandlers} -- the always-present per-field handlers
66+
* (resource, service, operation, ...).
67+
* <li>{@link datadog.trace.common.metrics.PeerTagSchema} -- remote-config / feature-discovery
68+
* driven; reconciled once per cycle on the aggregator thread. An old and a new schema may
69+
* briefly co-exist in the table after a discovery change, so a few spans may not fold quite
70+
* as expected for one cycle -- acceptable, because updating the config was always
71+
* fundamentally racy against in-flight spans.
72+
* <li>{@link datadog.trace.common.metrics.AdditionalTagsSchema} -- local-config span-derived
73+
* tags, fixed once at construction.
74+
* </ul>
75+
*
76+
* Each {@link datadog.trace.common.metrics.SpanSnapshot} captures its own schema reference at
77+
* capture time, so producer and aggregator agree on tag indexing even if the current schema is
78+
* swapped out between capture and consumption.
79+
*/
80+
package datadog.trace.common.metrics;

0 commit comments

Comments
 (0)