|
| 1 | +/** |
| 2 | + * Client-side trace statistics (CSS): the tracer computes per-interval aggregate stats -- hit |
| 3 | + * counts, error counts, and latency histograms grouped by a tuple of span labels (resource, |
| 4 | + * service, operation, span kind, HTTP/gRPC status, peer tags, ...) -- and ships them to the Datadog |
| 5 | + * agent every reporting interval. This lets the backend show accurate metrics even when individual |
| 6 | + * spans are sampled away. |
| 7 | + * |
| 8 | + * <h2>At a glance</h2> |
| 9 | + * |
| 10 | + * <pre>{@code |
| 11 | + * App / producer threads Single aggregator thread |
| 12 | + * ---------------------- ------------------------ |
| 13 | + * |
| 14 | + * span finishes (1) drain inbox |
| 15 | + * | | |
| 16 | + * v MPSC inbox v |
| 17 | + * build immutable (lock-free hand-off) (2) canonicalize each label through its |
| 18 | + * SpanSnapshot -----------------------> cardinality handler (Core / PeerTag / |
| 19 | + * | AdditionalTags); overflow -> sentinel |
| 20 | + * v | |
| 21 | + * (no locks, no v |
| 22 | + * mutable state) (3) fold into AggregateTable, keyed by the |
| 23 | + * canonical labels (counts + histograms) |
| 24 | + * | |
| 25 | + * v every ~10s (reporting interval) |
| 26 | + * (4) flush table via MetricWriter (msgpack |
| 27 | + * or OTLP) -> agent, then clear |
| 28 | + * }</pre> |
| 29 | + * |
| 30 | + * <h2>Threading model (the load-bearing rule)</h2> |
| 31 | + * |
| 32 | + * Producer application threads do almost nothing: each captures an immutable {@link |
| 33 | + * datadog.trace.common.metrics.SpanSnapshot} of the finished span and hands it off through a |
| 34 | + * lock-free MPSC {@code inbox}. A <em>single</em> aggregator thread ({@link |
| 35 | + * datadog.trace.common.metrics.Aggregator}) drains that inbox and owns <em>all</em> mutable state |
| 36 | + * -- the {@link datadog.trace.common.metrics.AggregateTable} and every cardinality handler -- as |
| 37 | + * its sole writer. None of that state is synchronized; correctness rests entirely on the |
| 38 | + * single-writer invariant. Any cross-thread request that must mutate (e.g. clearing the table on |
| 39 | + * downgrade) does not touch the state directly -- it is funneled onto the aggregator thread as a |
| 40 | + * signal through the same inbox. If you add a mutation path, it goes through the inbox too. |
| 41 | + * |
| 42 | + * <h2>The cycle</h2> |
| 43 | + * |
| 44 | + * The aggregator folds each drained snapshot into an {@link |
| 45 | + * datadog.trace.common.metrics.AggregateEntry}, keyed by the canonical form of its labels, updating |
| 46 | + * that entry's counters and histograms. Once per reporting interval (~10s) it flushes the whole |
| 47 | + * table through a {@link datadog.trace.common.metrics.MetricWriter} -- msgpack ({@link |
| 48 | + * datadog.trace.common.metrics.SerializingMetricWriter}) or OTLP -- and clears it for the next |
| 49 | + * interval. Buckets are per-interval deltas, not cumulative. |
| 50 | + * |
| 51 | + * <h2>Cardinality is the thing that must stay bounded</h2> |
| 52 | + * |
| 53 | + * Grouping keys come from span/user data, so distinct values -- and therefore the series count, its |
| 54 | + * memory, and its backend cost -- would grow without bound if left alone. Every label is |
| 55 | + * canonicalized through a {@link datadog.trace.common.metrics.TagCardinalityHandler} that caps the |
| 56 | + * number of distinct values per tag and collapses any overflow into a shared sentinel, so |
| 57 | + * over-cardinality folds into one entry rather than exploding the table. The collapses are counted |
| 58 | + * per cycle and reported (to {@link datadog.trace.core.monitor.HealthMetrics} and, rate-limited, |
| 59 | + * through {@link datadog.trace.common.metrics.CardinalityLimitReporter}) -- so the bounding is |
| 60 | + * observable without the bookkeeping itself becoming unbounded. |
| 61 | + * |
| 62 | + * <h2>Three handler families</h2> |
| 63 | + * |
| 64 | + * <ul> |
| 65 | + * <li>{@link datadog.trace.common.metrics.CoreHandlers} -- the always-present per-field handlers |
| 66 | + * (resource, service, operation, ...). |
| 67 | + * <li>{@link datadog.trace.common.metrics.PeerTagSchema} -- remote-config / feature-discovery |
| 68 | + * driven; reconciled once per cycle on the aggregator thread. An old and a new schema may |
| 69 | + * briefly co-exist in the table after a discovery change, so a few spans may not fold quite |
| 70 | + * as expected for one cycle -- acceptable, because updating the config was always |
| 71 | + * fundamentally racy against in-flight spans. |
| 72 | + * <li>{@link datadog.trace.common.metrics.AdditionalTagsSchema} -- local-config span-derived |
| 73 | + * tags, fixed once at construction. |
| 74 | + * </ul> |
| 75 | + * |
| 76 | + * Each {@link datadog.trace.common.metrics.SpanSnapshot} captures its own schema reference at |
| 77 | + * capture time, so producer and aggregator agree on tag indexing even if the current schema is |
| 78 | + * swapped out between capture and consumption. |
| 79 | + */ |
| 80 | +package datadog.trace.common.metrics; |
0 commit comments