Commit 19d68e3
[Debugger] Add memory pressure monitoring telemetry for Dynamic Instrumentation (observe-only) (#7834)
## Summary of changes
- Adds an observe-only memory pressure monitor for Dynamic
Instrumentation.
- Samples runtime/system memory pressure around DI activity (no
background timer).
- Emits low-cardinality telemetry only when entering or exiting high
memory pressure, plus a one-shot counter if the monitor disables itself.
- Does not block, throttle, skip, or otherwise change probe behavior.
## Reason for change
Dynamic Instrumentation can perform memory-sensitive work when capturing
variables, creating snapshots, serializing payloads, and
enqueueing/uploading data.
Before using memory pressure to affect DI behavior, we need production
data showing how often high pressure occurs, what drives it, how severe
each signal is at entry, how long episodes last, where the signal is
even available, and whether it is stable enough for future throttling
decisions. This PR adds exactly that observability and nothing else.
## Implementation details
The monitor detects high pressure from concrete runtime/system signals
only:
- Memory load ratio (fraction of available memory in use, so the value
is comparable across platforms):
- On .NET Core 3.1+, uses `GC.GetGCMemoryInfo()` (`MemoryLoadBytes /
TotalAvailableMemoryBytes`). This is container/cgroup-aware, and is
system/container-wide load rather than process-private bytes - the right
scope for "are we close to an OOM".
- On .NET Framework/Windows, uses `GlobalMemoryStatusEx`
(`dwMemoryLoad`, clamped to `[0,1]`). This is machine-wide and **not**
container/job-object aware.
- On runtimes/platforms where neither is available, the memory signal is
reported as unsupported.
- Gen2 collections per second:
- Calculated from `GC.CollectionCount(2)` deltas over elapsed time.
Identical and process-specific on every platform.
Sampling is activity-driven rather than always-on. DI calls into the
monitor around capture/snapshot/upload-relevant work, and the monitor
refreshes only when the last sample is stale (default min interval
`1s`). If DI is idle, the monitor does not continuously sample memory.
Time is read from `Environment.TickCount64` on .NET Core 3.0+ (a
monotonic `Stopwatch`-based fallback is used on net461/netstandard2.0),
so the hot `RefreshIfStale()` fast path is a couple of volatile reads
with no clock abstraction.
If neither memory nor GC signals are available on the host, or a
provider throws, the monitor **permanently disables itself** (it never
samples again), writes a single log line (debug for "no signals", error
for an exception), and records a one-shot `disabled` counter tagged by
reason so a silent zero in the transition metrics is explainable rather
than ambiguous.
Telemetry is emitted on transitions only (enter high pressure / exit
high pressure):
- Transition count tagged by `state` (`enter`/`exit`) and `trigger`
(`memory`/`gen2`/`both` on enter; `none` on exit) - i.e. which signal
drove entry.
- Severity values on transition: memory usage percent and Gen2
collections/sec (tagged by `state`).
- Duration of the high-pressure period, recorded on exit.
No high-cardinality tags such as probe ID, service name, file path,
exception type, or method name are added.
```mermaid
flowchart TD
diStart["DynamicInstrumentation starts"] --> createMonitor["Create MemoryPressureMonitor"]
createMonitor --> idleMonitor["No background sampling while idle"]
probeActivity["DI capture/snapshot/upload activity"] --> refreshIfStale["RefreshIfStale (min interval)"]
refreshIfStale --> signalsAvailable{"Memory or GC signal available?"}
signalsAvailable -->|"No / provider error"| disable["Disable monitor (logged) + disabled{reason}"]
signalsAvailable -->|"Yes"| sampleSignals["Sample memory load and Gen2/sec"]
sampleSignals --> pressureDecision{"High pressure transition?"}
pressureDecision -->|"ENTER"| enterTelemetry["transitions{state:enter, trigger:*} + severity"]
pressureDecision -->|"EXIT"| exitTelemetry["transitions{state:exit, trigger:none} + severity + duration"]
pressureDecision -->|"No transition"| noOp["No telemetry emission"]
enterTelemetry --> observeOnly["No DI behavior change"]
exitTelemetry --> observeOnly
noOp --> observeOnly
disable --> observeOnly
diStart --> dispose["DynamicInstrumentation.Dispose"]
dispose --> stopMonitor["Dispose monitor (lock-free)"]
```
Risk is kept low by avoiding per-second telemetry gauges, background
sampling while idle, probe-level cardinality and allocation on hot
paths. Refreshes are guarded by a non-blocking single-writer CAS so
concurrent DI activity does not run overlapping samples.
## Metrics implementation
All metrics are instrumentation-telemetry metrics, defined as **common**
metrics (`isCommon: true`) under the `live_debugger` namespace. They
surface in the backend as
`dd.instrumentation_telemetry_data.live_debugger.memory_pressure.*`.
Tags are fixed, low-cardinality enums.
| Metric | Type | Tags | Emitted |
|---|---|---|---|
| `memory_pressure.transitions` | count | `state` = `enter`\|`exit`,
`trigger` = `none`\|`memory`\|`gen2`\|`both` | Once per high-pressure
state change. `trigger` is the entry cause on `enter`; `none` on `exit`.
|
| `memory_pressure.disabled` | count | `reason` = `no_signals`\|`error`
| Once, when the monitor permanently disables itself (no signals
available, or a provider threw). |
| `memory_pressure.memory_usage_pct` | count | `state` =
`enter`\|`exit`, `bucket` = `lt_70`\|`70_80`\|`80_85`\|`85_90`\|`gte_90`
| On each transition - count bucketed by memory load percent at that
moment. |
| `memory_pressure.gen2_per_sec` | count | `state` = `enter`\|`exit`,
`bucket` = `lt_1`\|`1_2`\|`2_5`\|`gte_5` | On each transition - count
bucketed by Gen2 collections/sec at that moment. |
| `memory_pressure.duration_ms` | count | `bucket` =
`lt_1s`\|`1_5s`\|`5_30s`\|`gte_30s` | On `exit` - count bucketed by
length of the high-pressure episode. |
Runtime/platform segmentation (runtime name, OS, architecture, tracer
version) is **not** added as per-metric tags; it comes from the
telemetry payload's `application`/`host` metadata and is applied at
query time in the backend. This keeps series count bounded while still
allowing per-runtime analysis.
The severity and duration metrics use normal telemetry `count` metrics
with a fixed `bucket` tag instead of distributions. Distribution
telemetry is not supported for this internal telemetry path, and
bucketed counts preserve the decision-making signal with bounded
cardinality.
The `trigger` tag lives only on the transition count, not on the
severity bucket counts: if `trigger:memory`, the memory bucket already
captures how severe the memory signal was, and splitting every bucket by
trigger would mostly restate the transition breakdown with extra series.
The transition count's `trigger` plus the `state`-tagged bucket counts
answer "what drove it" and "how severe / how long".
## Dashboards and how we will use the data
The goal is a single dashboard that answers, per runtime/OS, whether DI
should ever gate capture under memory pressure and, if so, on which
signal and at what threshold.
| Widget | Source metric | Question | Decision it informs |
|---|---|---|---|
| Coverage timeseries / count by `reason` | `memory_pressure.disabled` |
Where does the monitor not run at all? | Defines the observed population
- distinguishes "no pressure" from "no data". |
| Enter rate timeseries, split by `trigger` |
`memory_pressure.transitions{state:enter}` | How often does high
pressure happen, and is it memory- or GC-driven? | Whether gating is
worth building at all, and which signal to gate on. |
| Trigger breakdown (top-list/pie) |
`memory_pressure.transitions{state:enter}` | `memory` vs `gen2` vs
`both` mix | Which threshold actually matters. |
| Memory severity bucket breakdown |
`memory_pressure.memory_usage_pct{state:enter}` | How deep is memory
pressure at entry? | Where to set a memory gating threshold. |
| GC severity bucket breakdown |
`memory_pressure.gen2_per_sec{state:enter}` | How hot is GC at entry? |
Where to set a gen2 gating threshold. |
| Episode duration bucket breakdown | `memory_pressure.duration_ms` |
How long do episodes last? | Whether gating needs hysteresis/cooldown or
episodes are too short to act on. |
Every widget can be grouped by the dimensions the telemetry intake
stamps onto each series from the payload's `application`/`host`
metadata.
How the data is gathered:
1. These metrics flow through the existing tracer telemetry pipeline
(generate-metrics payloads) to the Datadog backend - no new transport.
2. Because emission is transition-only and activity-driven, idle apps
cost nothing and noisy apps cannot spam per-second points.
3. We collect across runtimes for a few weeks, then read the dashboard
top-down: `memory_pressure.disabled` (coverage) →
`memory_pressure.transitions{state:enter}` by `trigger` (frequency +
cause) → severity distributions (thresholds) → duration (persistence).
4. The resulting numbers define concrete gating thresholds (or show that
gating is unnecessary) for a follow-up PR.
### Scope: what this telemetry answers (and what it doesn't)
This is a fleet-aggregate, pre-decision dataset: it tells us whether
high memory pressure occurs during DI work often, severely, and long
enough to justify gating capture, and on which signal at what threshold.
It is deliberately **not** a tool for measuring the impact of a future
gate.
## Test coverage
Added/updated tests cover:
- Memory threshold enter/exit behavior and hysteresis.
- Gen2/sec threshold and rate behavior.
- Consecutive high/low cycle (debounce) behavior.
- Unavailable memory/GC signals → self-disable with `reason:no_signals`,
no further sampling.
- Provider exception handling → self-disable with `reason:error`, no
crash.
- Refresh overlap handling (non-blocking single writer) and dispose
during an in-flight refresh.
- Dispose/lifecycle behavior (no further sampling or emission after
dispose).
- Activity-driven stale refresh (no sampling while idle; samples only
when stale).
- Transition telemetry: enter emitted once, exit emitted once, severity
recorded on transition, duration recorded on exit, no repeat while state
is unchanged.
- `trigger` classification: `gen2` when only GC drives entry, `both`
when memory and GC cross together, `memory`/`none` on the enter/exit
pair.
- Extreme/edge configuration values are clamped to safe minimums.
- Windows `GlobalMemoryStatusEx` interop returns valid values.
- DI integration: the monitor is sampled only through the dedicated hook
and is disposed by `DynamicInstrumentation`.
- Telemetry aggregation: transitions tagged by `state`+`trigger`, the
`disabled` counter, and the three bucketed count metrics aggregate into
the expected series/tags.
## Other details
Initial thresholds are observation thresholds, not throttling
thresholds:
- Memory pressure threshold: `0.85`.
- Memory exit threshold with default hysteresis: `0.80`.
- Gen2 threshold: `2` collections/sec.
- Gen2 exit threshold with default hysteresis: `1` collection/sec.
These values are intentionally conservative starting points for data
collection and can be tuned after production telemetry is available.
This PR intentionally does not include an explicit/manual pressure event
signal. There is no concrete production caller or well-defined meaning
for that signal yet. If a future DI path has a specific condition, such
as snapshot capture exceeding a memory budget, it should add a named API
and metric for that scenario.
Any future DI throttling or blocking based on memory pressure should be
done in a separate PR after reviewing the collected telemetry.
## Note
Incidental DynamicInstrumentation lifecycle hardening (not
memory-pressure specific): Wiring the new monitor in as a disposable
owned by DynamicInstrumentation, exposed pre-existing races in the
start/subscribe/dispose paths. DynamicInstrumentation is disposed at
runtime via RCM, not just at shutdown. Since the monitor's lifecycle is
tied to those paths, we hardened them here.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>1 parent d8273e8 commit 19d68e3
35 files changed
Lines changed: 2240 additions & 211 deletions
File tree
- tracer
- src/Datadog.Trace
- Debugger
- Caching
- Expressions
- RateLimiting
- Generated
- net461/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator
- net6.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator
- netcoreapp3.1/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator
- netstandard2.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator
- Telemetry
- DTOs
- Metrics
- test/Datadog.Trace.Tests
- Debugger
- RateLimiting
- Telemetry
- Collectors
- Metrics
Lines changed: 2 additions & 45 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | 30 | | |
34 | 31 | | |
35 | 32 | | |
36 | 33 | | |
37 | | - | |
38 | 34 | | |
39 | 35 | | |
40 | 36 | | |
| |||
73 | 69 | | |
74 | 70 | | |
75 | 71 | | |
76 | | - | |
| 72 | + | |
77 | 73 | | |
78 | 74 | | |
79 | 75 | | |
| |||
148 | 144 | | |
149 | 145 | | |
150 | 146 | | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | 147 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
45 | 46 | | |
46 | 47 | | |
47 | 48 | | |
| |||
56 | 57 | | |
57 | 58 | | |
58 | 59 | | |
59 | | - | |
| 60 | + | |
| 61 | + | |
60 | 62 | | |
61 | 63 | | |
62 | 64 | | |
| |||
Lines changed: 75 additions & 22 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
44 | 46 | | |
45 | 47 | | |
46 | 48 | | |
| |||
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
| 58 | + | |
56 | 59 | | |
57 | 60 | | |
58 | 61 | | |
| |||
72 | 75 | | |
73 | 76 | | |
74 | 77 | | |
| 78 | + | |
75 | 79 | | |
76 | 80 | | |
77 | 81 | | |
78 | 82 | | |
79 | | - | |
| 83 | + | |
80 | 84 | | |
81 | 85 | | |
82 | 86 | | |
| |||
87 | 91 | | |
88 | 92 | | |
89 | 93 | | |
| 94 | + | |
90 | 95 | | |
91 | 96 | | |
92 | 97 | | |
| |||
145 | 150 | | |
146 | 151 | | |
147 | 152 | | |
148 | | - | |
| 153 | + | |
149 | 154 | | |
150 | 155 | | |
151 | 156 | | |
| |||
154 | 159 | | |
155 | 160 | | |
156 | 161 | | |
157 | | - | |
158 | | - | |
| 162 | + | |
159 | 163 | | |
160 | 164 | | |
161 | 165 | | |
| |||
184 | 188 | | |
185 | 189 | | |
186 | 190 | | |
187 | | - | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
188 | 197 | | |
189 | | - | |
| 198 | + | |
190 | 199 | | |
191 | 200 | | |
192 | 201 | | |
193 | 202 | | |
194 | | - | |
195 | | - | |
196 | | - | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
197 | 240 | | |
198 | 241 | | |
199 | 242 | | |
| |||
836 | 879 | | |
837 | 880 | | |
838 | 881 | | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
839 | 892 | | |
840 | 893 | | |
841 | 894 | | |
| |||
899 | 952 | | |
900 | 953 | | |
901 | 954 | | |
902 | | - | |
| 955 | + | |
903 | 956 | | |
904 | 957 | | |
905 | 958 | | |
| |||
930 | 983 | | |
931 | 984 | | |
932 | 985 | | |
933 | | - | |
934 | 986 | | |
935 | 987 | | |
936 | 988 | | |
937 | 989 | | |
938 | 990 | | |
939 | | - | |
940 | | - | |
941 | | - | |
942 | | - | |
| 991 | + | |
943 | 992 | | |
944 | | - | |
945 | | - | |
946 | 993 | | |
947 | 994 | | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
948 | 999 | | |
949 | 1000 | | |
950 | 1001 | | |
951 | 1002 | | |
952 | 1003 | | |
953 | 1004 | | |
| 1005 | + | |
954 | 1006 | | |
955 | 1007 | | |
956 | 1008 | | |
957 | | - | |
958 | | - | |
959 | | - | |
960 | | - | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
961 | 1014 | | |
962 | 1015 | | |
963 | 1016 | | |
Lines changed: 16 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
109 | 119 | | |
110 | 120 | | |
111 | 121 | | |
| |||
138 | 148 | | |
139 | 149 | | |
140 | 150 | | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
141 | 156 | | |
142 | | - | |
| 157 | + | |
143 | 158 | | |
144 | 159 | | |
145 | 160 | | |
| |||
Lines changed: 65 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
0 commit comments