diff --git a/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java new file mode 100644 index 00000000000..f248825e6be --- /dev/null +++ b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java @@ -0,0 +1,94 @@ +/** + * Client-side stats aggregate eligible completed spans by resource, service, operation, span kind, + * status, and configured tags. Each reporting interval produces hit and error counts and latency + * distributions. The aggregates are computed independently of trace sampling, so they include spans + * that are not sent individually. + * + *

A {@link datadog.trace.common.metrics.MetricWriter} publishes each interval: msgpack to the + * Datadog Agent or OTLP to the configured OTLP metrics endpoint. + * + *

At a glance

+ * + *
{@code
+ * App / Producer threads              Control threads
+ * ----------------------              ------------------------------
+ * eligible completed span             report schedule, tracer close,
+ *          |                          clear needed
+ * capture values and create                     |
+ * immutable SpanSnapshot                        |
+ *          |                                    |
+ *          +----------------+-------------------+
+ *                           v
+ *                   lock-free MPSC inbox
+ *                           |
+ *                           v
+ *                    Aggregator thread
+ *               (sole writer of aggregate state)
+ *                      /              \
+ *            SpanSnapshot              SignalItem
+ *                 |                        |
+ * canonicalize bounded fields   ReportSignal: reconcile peer schema,
+ * and tags; find/create entry                 expunge stale entries,
+ *                 |                           write interval deltas,
+ * update counts and histograms                reset active entries
+ *                                ClearSignal: clear the table
+ *                                 StopSignal: final report, then stop
+ * }
+ * + *

Threading model (the load-bearing rule)

+ * + * Producer application threads do almost nothing: each captures an immutable {@link + * datadog.trace.common.metrics.SpanSnapshot} of the finished span and hands it off through a + * lock-free MPSC {@code inbox}. A single aggregator thread ({@link + * datadog.trace.common.metrics.Aggregator}) drains that inbox and owns all mutable state + * -- the {@link datadog.trace.common.metrics.AggregateTable} and every cardinality handler -- as + * its sole writer. None of that state is synchronized; correctness rests entirely on the + * single-writer invariant. Any cross-thread request that must mutate (e.g. clearing the table on + * downgrade) does not touch the state directly -- it is funneled onto the aggregator thread as a + * signal through the same inbox. If you add a mutation path, it goes through the inbox too. + * + *

The cycle

+ * + * The aggregator folds each drained snapshot into an {@link + * datadog.trace.common.metrics.AggregateEntry}, keyed by the canonical form of its labels, updating + * that entry's counters and histograms. Once per reporting interval (~10s) it flushes the whole + * table through a {@link datadog.trace.common.metrics.MetricWriter} -- msgpack ({@link + * datadog.trace.common.metrics.SerializingMetricWriter}) or OTLP -- then resets each entry's + * counters for the next interval rather than discarding the table: entries (and their cross-cycle + * caches) are retained for reuse, and stale ones are expunged over subsequent cycles. Buckets are + * per-interval deltas, not cumulative. + * + *

Cardinality is the thing that must stay bounded

+ * + * Grouping keys come from span/user data, so distinct values -- and therefore the series count, its + * memory, and its backend cost -- would grow without bound if left alone. High-cardinality labels + * are canonicalized through a cardinality handler -- {@link + * datadog.trace.common.metrics.PropertyCardinalityHandler} for the core string fields (resource, + * service, operation, ...) and {@link datadog.trace.common.metrics.TagCardinalityHandler} for peer + * and additional tags -- that caps the number of distinct values per label and collapses any + * overflow into a shared sentinel, so over-cardinality folds into one entry rather than exploding + * the table. (Primitive key fields such as HTTP/gRPC status and the boolean flags are inherently + * low-cardinality and copied directly.) The collapses are counted per cycle and reported (to {@link + * datadog.trace.core.monitor.HealthMetrics} and, rate-limited, through {@link + * datadog.trace.common.metrics.CardinalityLimitReporter}) -- so the bounding is observable without + * the bookkeeping itself becoming unbounded. + * + *

Three handler families

+ * + * + * + * Each {@link datadog.trace.common.metrics.SpanSnapshot} captures its own schema reference at + * capture time, so producer and aggregator agree on tag indexing even if the current schema is + * swapped out between capture and consumption. + */ +package datadog.trace.common.metrics;