Skip to content

Commit 19d68e3

Browse files
[Debugger] Add memory pressure monitoring telemetry for Dynamic Instrumentation (observe-only) (#7834)
## Summary of changes - Adds an observe-only memory pressure monitor for Dynamic Instrumentation. - Samples runtime/system memory pressure around DI activity (no background timer). - Emits low-cardinality telemetry only when entering or exiting high memory pressure, plus a one-shot counter if the monitor disables itself. - Does not block, throttle, skip, or otherwise change probe behavior. ## Reason for change Dynamic Instrumentation can perform memory-sensitive work when capturing variables, creating snapshots, serializing payloads, and enqueueing/uploading data. Before using memory pressure to affect DI behavior, we need production data showing how often high pressure occurs, what drives it, how severe each signal is at entry, how long episodes last, where the signal is even available, and whether it is stable enough for future throttling decisions. This PR adds exactly that observability and nothing else. ## Implementation details The monitor detects high pressure from concrete runtime/system signals only: - Memory load ratio (fraction of available memory in use, so the value is comparable across platforms): - On .NET Core 3.1+, uses `GC.GetGCMemoryInfo()` (`MemoryLoadBytes / TotalAvailableMemoryBytes`). This is container/cgroup-aware, and is system/container-wide load rather than process-private bytes - the right scope for "are we close to an OOM". - On .NET Framework/Windows, uses `GlobalMemoryStatusEx` (`dwMemoryLoad`, clamped to `[0,1]`). This is machine-wide and **not** container/job-object aware. - On runtimes/platforms where neither is available, the memory signal is reported as unsupported. - Gen2 collections per second: - Calculated from `GC.CollectionCount(2)` deltas over elapsed time. Identical and process-specific on every platform. Sampling is activity-driven rather than always-on. DI calls into the monitor around capture/snapshot/upload-relevant work, and the monitor refreshes only when the last sample is stale (default min interval `1s`). If DI is idle, the monitor does not continuously sample memory. Time is read from `Environment.TickCount64` on .NET Core 3.0+ (a monotonic `Stopwatch`-based fallback is used on net461/netstandard2.0), so the hot `RefreshIfStale()` fast path is a couple of volatile reads with no clock abstraction. If neither memory nor GC signals are available on the host, or a provider throws, the monitor **permanently disables itself** (it never samples again), writes a single log line (debug for "no signals", error for an exception), and records a one-shot `disabled` counter tagged by reason so a silent zero in the transition metrics is explainable rather than ambiguous. Telemetry is emitted on transitions only (enter high pressure / exit high pressure): - Transition count tagged by `state` (`enter`/`exit`) and `trigger` (`memory`/`gen2`/`both` on enter; `none` on exit) - i.e. which signal drove entry. - Severity values on transition: memory usage percent and Gen2 collections/sec (tagged by `state`). - Duration of the high-pressure period, recorded on exit. No high-cardinality tags such as probe ID, service name, file path, exception type, or method name are added. ```mermaid flowchart TD diStart["DynamicInstrumentation starts"] --> createMonitor["Create MemoryPressureMonitor"] createMonitor --> idleMonitor["No background sampling while idle"] probeActivity["DI capture/snapshot/upload activity"] --> refreshIfStale["RefreshIfStale (min interval)"] refreshIfStale --> signalsAvailable{"Memory or GC signal available?"} signalsAvailable -->|"No / provider error"| disable["Disable monitor (logged) + disabled{reason}"] signalsAvailable -->|"Yes"| sampleSignals["Sample memory load and Gen2/sec"] sampleSignals --> pressureDecision{"High pressure transition?"} pressureDecision -->|"ENTER"| enterTelemetry["transitions{state:enter, trigger:*} + severity"] pressureDecision -->|"EXIT"| exitTelemetry["transitions{state:exit, trigger:none} + severity + duration"] pressureDecision -->|"No transition"| noOp["No telemetry emission"] enterTelemetry --> observeOnly["No DI behavior change"] exitTelemetry --> observeOnly noOp --> observeOnly disable --> observeOnly diStart --> dispose["DynamicInstrumentation.Dispose"] dispose --> stopMonitor["Dispose monitor (lock-free)"] ``` Risk is kept low by avoiding per-second telemetry gauges, background sampling while idle, probe-level cardinality and allocation on hot paths. Refreshes are guarded by a non-blocking single-writer CAS so concurrent DI activity does not run overlapping samples. ## Metrics implementation All metrics are instrumentation-telemetry metrics, defined as **common** metrics (`isCommon: true`) under the `live_debugger` namespace. They surface in the backend as `dd.instrumentation_telemetry_data.live_debugger.memory_pressure.*`. Tags are fixed, low-cardinality enums. | Metric | Type | Tags | Emitted | |---|---|---|---| | `memory_pressure.transitions` | count | `state` = `enter`\|`exit`, `trigger` = `none`\|`memory`\|`gen2`\|`both` | Once per high-pressure state change. `trigger` is the entry cause on `enter`; `none` on `exit`. | | `memory_pressure.disabled` | count | `reason` = `no_signals`\|`error` | Once, when the monitor permanently disables itself (no signals available, or a provider threw). | | `memory_pressure.memory_usage_pct` | count | `state` = `enter`\|`exit`, `bucket` = `lt_70`\|`70_80`\|`80_85`\|`85_90`\|`gte_90` | On each transition - count bucketed by memory load percent at that moment. | | `memory_pressure.gen2_per_sec` | count | `state` = `enter`\|`exit`, `bucket` = `lt_1`\|`1_2`\|`2_5`\|`gte_5` | On each transition - count bucketed by Gen2 collections/sec at that moment. | | `memory_pressure.duration_ms` | count | `bucket` = `lt_1s`\|`1_5s`\|`5_30s`\|`gte_30s` | On `exit` - count bucketed by length of the high-pressure episode. | Runtime/platform segmentation (runtime name, OS, architecture, tracer version) is **not** added as per-metric tags; it comes from the telemetry payload's `application`/`host` metadata and is applied at query time in the backend. This keeps series count bounded while still allowing per-runtime analysis. The severity and duration metrics use normal telemetry `count` metrics with a fixed `bucket` tag instead of distributions. Distribution telemetry is not supported for this internal telemetry path, and bucketed counts preserve the decision-making signal with bounded cardinality. The `trigger` tag lives only on the transition count, not on the severity bucket counts: if `trigger:memory`, the memory bucket already captures how severe the memory signal was, and splitting every bucket by trigger would mostly restate the transition breakdown with extra series. The transition count's `trigger` plus the `state`-tagged bucket counts answer "what drove it" and "how severe / how long". ## Dashboards and how we will use the data The goal is a single dashboard that answers, per runtime/OS, whether DI should ever gate capture under memory pressure and, if so, on which signal and at what threshold. | Widget | Source metric | Question | Decision it informs | |---|---|---|---| | Coverage timeseries / count by `reason` | `memory_pressure.disabled` | Where does the monitor not run at all? | Defines the observed population - distinguishes "no pressure" from "no data". | | Enter rate timeseries, split by `trigger` | `memory_pressure.transitions{state:enter}` | How often does high pressure happen, and is it memory- or GC-driven? | Whether gating is worth building at all, and which signal to gate on. | | Trigger breakdown (top-list/pie) | `memory_pressure.transitions{state:enter}` | `memory` vs `gen2` vs `both` mix | Which threshold actually matters. | | Memory severity bucket breakdown | `memory_pressure.memory_usage_pct{state:enter}` | How deep is memory pressure at entry? | Where to set a memory gating threshold. | | GC severity bucket breakdown | `memory_pressure.gen2_per_sec{state:enter}` | How hot is GC at entry? | Where to set a gen2 gating threshold. | | Episode duration bucket breakdown | `memory_pressure.duration_ms` | How long do episodes last? | Whether gating needs hysteresis/cooldown or episodes are too short to act on. | Every widget can be grouped by the dimensions the telemetry intake stamps onto each series from the payload's `application`/`host` metadata. How the data is gathered: 1. These metrics flow through the existing tracer telemetry pipeline (generate-metrics payloads) to the Datadog backend - no new transport. 2. Because emission is transition-only and activity-driven, idle apps cost nothing and noisy apps cannot spam per-second points. 3. We collect across runtimes for a few weeks, then read the dashboard top-down: `memory_pressure.disabled` (coverage) → `memory_pressure.transitions{state:enter}` by `trigger` (frequency + cause) → severity distributions (thresholds) → duration (persistence). 4. The resulting numbers define concrete gating thresholds (or show that gating is unnecessary) for a follow-up PR. ### Scope: what this telemetry answers (and what it doesn't) This is a fleet-aggregate, pre-decision dataset: it tells us whether high memory pressure occurs during DI work often, severely, and long enough to justify gating capture, and on which signal at what threshold. It is deliberately **not** a tool for measuring the impact of a future gate. ## Test coverage Added/updated tests cover: - Memory threshold enter/exit behavior and hysteresis. - Gen2/sec threshold and rate behavior. - Consecutive high/low cycle (debounce) behavior. - Unavailable memory/GC signals → self-disable with `reason:no_signals`, no further sampling. - Provider exception handling → self-disable with `reason:error`, no crash. - Refresh overlap handling (non-blocking single writer) and dispose during an in-flight refresh. - Dispose/lifecycle behavior (no further sampling or emission after dispose). - Activity-driven stale refresh (no sampling while idle; samples only when stale). - Transition telemetry: enter emitted once, exit emitted once, severity recorded on transition, duration recorded on exit, no repeat while state is unchanged. - `trigger` classification: `gen2` when only GC drives entry, `both` when memory and GC cross together, `memory`/`none` on the enter/exit pair. - Extreme/edge configuration values are clamped to safe minimums. - Windows `GlobalMemoryStatusEx` interop returns valid values. - DI integration: the monitor is sampled only through the dedicated hook and is disposed by `DynamicInstrumentation`. - Telemetry aggregation: transitions tagged by `state`+`trigger`, the `disabled` counter, and the three bucketed count metrics aggregate into the expected series/tags. ## Other details Initial thresholds are observation thresholds, not throttling thresholds: - Memory pressure threshold: `0.85`. - Memory exit threshold with default hysteresis: `0.80`. - Gen2 threshold: `2` collections/sec. - Gen2 exit threshold with default hysteresis: `1` collection/sec. These values are intentionally conservative starting points for data collection and can be tuned after production telemetry is available. This PR intentionally does not include an explicit/manual pressure event signal. There is no concrete production caller or well-defined meaning for that signal yet. If a future DI path has a specific condition, such as snapshot capture exceeding a memory budget, it should add a named API and metric for that scenario. Any future DI throttling or blocking based on memory pressure should be done in a separate PR after reviewing the collected telemetry. ## Note Incidental DynamicInstrumentation lifecycle hardening (not memory-pressure specific): Wiring the new monitor in as a disposable owned by DynamicInstrumentation, exposed pre-existing races in the start/subscribe/dispose paths. DynamicInstrumentation is disposed at runtime via RCM, not just at shutdown. Since the monitor's lifecycle is tied to those paths, we hardened them here. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent d8273e8 commit 19d68e3

35 files changed

Lines changed: 2240 additions & 211 deletions

File tree

tracer/src/Datadog.Trace/Debugger/Caching/DefaultMemoryChecker.cs

Lines changed: 2 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77

88
using System;
99
using System.Runtime.InteropServices;
10+
using Datadog.Trace.Debugger.RateLimiting;
1011
using Datadog.Trace.Logging;
1112

1213
namespace Datadog.Trace.Debugger.Caching;
@@ -26,15 +27,10 @@ private DefaultMemoryChecker()
2627

2728
public bool IsLowResourceEnvironment { get; }
2829

29-
[return: MarshalAs(UnmanagedType.Bool)]
30-
[DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
31-
private static extern bool GlobalMemoryStatusEx([In, Out] MEMORYSTATUSEX lpBuffer);
32-
3330
private bool CheckLowResourceEnvironment()
3431
{
3532
try
3633
{
37-
Logger.Debug("Checking if environment is low on resources");
3834
// Check if we're using more than 75% of available memory or there is less than 1GB of RAM available.
3935
return IsLowResourceEnvironmentGc() || IsLowResourceEnvironmentSystem();
4036
}
@@ -73,7 +69,7 @@ internal bool CheckWindowsMemory()
7369
{
7470
try
7571
{
76-
if (MEMORYSTATUSEX.GetAvailablePhysicalMemory(out var availableMemory))
72+
if (WindowsMemoryInfo.TryGetAvailablePhysicalMemory(out var availableMemory))
7773
{
7874
// If less than 1GB of RAM is available, consider it a low-resource environment
7975
return availableMemory < 1_073_741_824; // 1 GB in bytes
@@ -148,43 +144,4 @@ private ReadOnlySpan<char> ReadMemInfo()
148144

149145
return value.Slice(0, spaceIndex);
150146
}
151-
152-
// Windows API for memory information
153-
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
154-
private sealed class MEMORYSTATUSEX
155-
{
156-
#pragma warning disable IDE0044 // Add readonly modifier
157-
#pragma warning disable CS0169 // Field is never used
158-
private uint dwLength;
159-
private uint dwMemoryLoad;
160-
private ulong ullTotalPhys;
161-
#pragma warning disable CS0649 // Field is never assigned to, and will always have its default value
162-
private ulong ullAvailPhys;
163-
#pragma warning restore CS0649 // Field is never assigned to, and will always have its default value
164-
private ulong ullTotalPageFile;
165-
private ulong ullAvailPageFile;
166-
private ulong ullTotalVirtual;
167-
private ulong ullAvailVirtual;
168-
private ulong ullAvailExtendedVirtual;
169-
#pragma warning restore CS0169 // Field is never used
170-
#pragma warning restore IDE0044 // Add readonly modifier
171-
172-
private MEMORYSTATUSEX()
173-
{
174-
dwLength = (uint)Marshal.SizeOf(typeof(MEMORYSTATUSEX));
175-
}
176-
177-
internal static bool GetAvailablePhysicalMemory(out ulong availableMemory)
178-
{
179-
availableMemory = 0;
180-
MEMORYSTATUSEX memStatus = new MEMORYSTATUSEX();
181-
if (GlobalMemoryStatusEx(memStatus))
182-
{
183-
availableMemory = memStatus.ullAvailPhys;
184-
return true;
185-
}
186-
187-
return false;
188-
}
189-
}
190147
}

tracer/src/Datadog.Trace/Debugger/DebuggerFactory.cs

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ internal static DynamicInstrumentation CreateDynamicInstrumentation(IDiscoverySe
4242
var lineProbeResolver = LineProbeResolver.Create(debuggerSettings.ThirdPartyDetectionExcludes, debuggerSettings.ThirdPartyDetectionIncludes);
4343
var probeStatusPoller = ProbeStatusPoller.Create(diagnosticsSink, debuggerSettings);
4444
var configurationUpdater = ConfigurationUpdater.Create(tracerSettings.Manager.InitialMutableSettings.Environment, tracerSettings.Manager.InitialMutableSettings.ServiceVersion, debuggerSettings.MaxProbesPerType, globalRateLimiter);
45+
var memoryPressureMonitor = new MemoryPressureMonitor(MemoryPressureConfig.Default);
4546

4647
var statsd = GetDogStatsd(tracerSettings);
4748

@@ -56,7 +57,8 @@ internal static DynamicInstrumentation CreateDynamicInstrumentation(IDiscoverySe
5657
probeStatusPoller: probeStatusPoller,
5758
configurationUpdater: configurationUpdater,
5859
dogStats: statsd,
59-
globalRateLimiter: globalRateLimiter);
60+
globalRateLimiter: globalRateLimiter,
61+
memoryPressureMonitor: memoryPressureMonitor);
6062
}
6163

6264
private static IDogStatsd GetDogStatsd(TracerSettings tracerSettings)

tracer/src/Datadog.Trace/Debugger/DynamicInstrumentation.cs

Lines changed: 75 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,9 @@ internal sealed class DynamicInstrumentation : IDisposable
4040
{
4141
private static readonly IDatadogLogger Log = DatadogLogging.GetLoggerFor(typeof(DynamicInstrumentation));
4242

43-
private readonly TaskCompletionSource<bool> _processExit;
43+
// Completed when this DI instance is being disposed (runtime disable via remote config, or process shutdown).
44+
// Used to abort in-flight initialization waits promptly.
45+
private readonly TaskCompletionSource<bool> _disposalSignal;
4446
private readonly IDiscoveryService _discoveryService;
4547
private readonly IRcmSubscriptionManager _subscriptionManager;
4648
private readonly ISubscription _subscription;
@@ -53,6 +55,7 @@ internal sealed class DynamicInstrumentation : IDisposable
5355
private readonly IProbeStatusPoller _probeStatusPoller;
5456
private readonly ConfigurationUpdater _configurationUpdater;
5557
private readonly IDogStatsd _dogStats;
58+
private readonly MemoryPressureMonitor _memoryPressureMonitor;
5659
private readonly DebuggerSettings _settings;
5760
private readonly NativeProbeInstrumentationRequester _instrumentProbes;
5861
private readonly object _instanceLock = new();
@@ -72,11 +75,12 @@ internal DynamicInstrumentation(
7275
ConfigurationUpdater configurationUpdater,
7376
IDogStatsd dogStats,
7477
IDebuggerGlobalRateLimiter? globalRateLimiter = null,
78+
MemoryPressureMonitor? memoryPressureMonitor = null,
7579
NativeProbeInstrumentationRequester? instrumentProbes = null)
7680
{
7781
Log.Information("Initializing Dynamic Instrumentation");
7882
_settings = settings;
79-
_processExit = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
83+
_disposalSignal = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
8084
_discoveryService = discoveryService;
8185
_lineProbeResolver = lineProbeResolver;
8286
_snapshotUploader = snapshotUploader;
@@ -87,6 +91,7 @@ internal DynamicInstrumentation(
8791
_configurationUpdater = configurationUpdater;
8892
_configurationUpdater.SetProbeInstrumentationHandlers(UpdateAddedProbeInstrumentations, UpdateRemovedProbeInstrumentations);
8993
_dogStats = dogStats;
94+
_memoryPressureMonitor = memoryPressureMonitor ?? new MemoryPressureMonitor(MemoryPressureConfig.Default);
9095
_instrumentProbes = instrumentProbes ?? DebuggerNativeMethods.InstrumentProbes;
9196
_unboundProbes = new List<ProbeDefinition>();
9297
_lastReportedUnboundProbeErrors = new Dictionary<string, LineProbeResolveErrorKey>();
@@ -145,7 +150,7 @@ private async Task InitializeAsync()
145150
hasFileProbes = _configurationUpdater.HasAnyEffectiveProbeForFile(probeConfiguration);
146151
if (hasFileProbes)
147152
{
148-
StartRuntimeIfNeeded();
153+
StartRuntimeIfNeeded(subscribeToRcm: false);
149154
}
150155

151156
_configurationUpdater.AcceptFile(probeConfiguration);
@@ -154,8 +159,7 @@ private async Task InitializeAsync()
154159
var isRcmAvailable = await rcmAvailabilityTask.ConfigureAwait(false);
155160
if (isRcmAvailable)
156161
{
157-
StartRuntimeIfNeeded();
158-
_subscriptionManager.SubscribeToChanges(_subscription);
162+
StartRuntimeIfNeeded(subscribeToRcm: true);
159163
}
160164

161165
// Start background processing and register the assembly load callback if either:
@@ -184,16 +188,55 @@ private async Task InitializeAsync()
184188
}
185189
}
186190

187-
private void StartRuntimeIfNeeded()
191+
/// <summary>
192+
/// Starts the runtime (background processing + assembly-load callback) exactly once, and optionally
193+
/// subscribes to RCM. Idempotent and safe to call from both the file-probe path (<paramref name="subscribeToRcm"/>
194+
/// false) and the RCM-available path (<paramref name="subscribeToRcm"/> true).
195+
/// </summary>
196+
private void StartRuntimeIfNeeded(bool subscribeToRcm)
188197
{
189-
if (IsInitialized || IsDisposed)
198+
if (IsDisposed || (IsInitialized && !subscribeToRcm))
190199
{
191200
return;
192201
}
193202

194-
AppDomain.CurrentDomain.AssemblyLoad += CheckUnboundProbes;
195-
StartBackgroundProcess();
196-
Volatile.Write(ref _initializationState, 2);
203+
lock (_instanceLock)
204+
{
205+
if (IsDisposed)
206+
{
207+
return;
208+
}
209+
210+
// Start the runtime exactly once.
211+
if (!IsInitialized)
212+
{
213+
var assemblyLoadSubscribed = false;
214+
try
215+
{
216+
AppDomain.CurrentDomain.AssemblyLoad += CheckUnboundProbes;
217+
assemblyLoadSubscribed = true;
218+
StartBackgroundProcess();
219+
Volatile.Write(ref _initializationState, 2);
220+
}
221+
catch
222+
{
223+
if (assemblyLoadSubscribed)
224+
{
225+
AppDomain.CurrentDomain.AssemblyLoad -= CheckUnboundProbes;
226+
}
227+
228+
throw;
229+
}
230+
}
231+
232+
// Subscribe only on the RCM path, and only if a disable hasn't landed in the meantime.
233+
// Best-effort: correctness against a leaked subscription is guaranteed by Dispose's Unsubscribe
234+
// running under this same lock.
235+
if (subscribeToRcm && !IsDisposed)
236+
{
237+
_subscriptionManager.SubscribeToChanges(_subscription);
238+
}
239+
}
197240
}
198241

199242
private void StartBackgroundProcess()
@@ -836,6 +879,16 @@ internal void AddLog(ProbeInfo probe, string log)
836879
SetProbeStatusToEmitting(probe);
837880
}
838881

882+
internal void RefreshMemoryPressureIfStale()
883+
{
884+
if (IsDisposed)
885+
{
886+
return;
887+
}
888+
889+
_memoryPressureMonitor.RefreshIfStale();
890+
}
891+
839892
internal void SetProbeStatusToEmitting(ProbeInfo probe)
840893
{
841894
if (IsDisposed)
@@ -899,7 +952,7 @@ private async Task<bool> WaitForRcmAvailabilityAsync()
899952
var rcmTimeout = TimeSpan.FromMinutes(5);
900953
var timeoutTask = Task.Delay(rcmTimeout);
901954

902-
var completedTask = await Task.WhenAny(rcmAvailabilityTcs.Task, timeoutTask, _processExit.Task).ConfigureAwait(false);
955+
var completedTask = await Task.WhenAny(rcmAvailabilityTcs.Task, timeoutTask, _disposalSignal.Task).ConfigureAwait(false);
903956
if (completedTask == timeoutTask)
904957
{
905958
Log.Warning("Dynamic Instrumentation could not be enabled because Remote Configuration Management is not available after waiting {Timeout} seconds. Please note that Dynamic Instrumentation is not supported in all environments (e.g. AAS). Ensure that you are using datadog-agent version 7.41.1 or higher, and that Remote Configuration Management is enabled in datadog-agent's yaml configuration file.", rcmTimeout.TotalSeconds);
@@ -930,34 +983,34 @@ void DiscoveryCallback(AgentConfiguration x)
930983

931984
public void Dispose()
932985
{
933-
// Already disposed
934986
if (Interlocked.CompareExchange(ref _disposeState, 1, 0) != 0)
935987
{
936988
return;
937989
}
938990

939-
if (_processExit.Task.IsCompleted)
940-
{
941-
return;
942-
}
991+
_disposalSignal.TrySetResult(true);
943992

944-
_processExit.TrySetResult(true);
945-
AppDomain.CurrentDomain.AssemblyLoad -= CheckUnboundProbes;
946993
lock (_instanceLock)
947994
{
995+
// Must stay under the lock: StartRuntimeIfNeeded subscribes AssemblyLoad under the same
996+
// lock, so unsubscribing here is what prevents a subscribe-after-dispose handler leak.
997+
AppDomain.CurrentDomain.AssemblyLoad -= CheckUnboundProbes;
998+
948999
SafeDisposal.New()
9491000
.Execute(() => _subscriptionManager.Unsubscribe(_subscription), "unsubscribing from RCM")
9501001
.Add(_snapshotUploader)
9511002
.Add(_logUploader)
9521003
.Add(_diagnosticsUploader)
9531004
.Add(_probeStatusPoller)
1005+
.Add(_memoryPressureMonitor)
9541006
.DisposeAll();
9551007
}
9561008

957-
// Cannot await here because Dispose() is synchronous and callers hold locks.
958-
// On master, _dogStats was disposed via SafeDisposal.Add() which called sync
959-
// Dispose() — itself fire-and-forget internally via Task.Run().
960-
_dogStats?.DisposeAsync().ContinueWith(t => Log.Error(t.Exception, "Error waiting for StatsD disposal"), TaskContinuationOptions.OnlyOnFaulted);
1009+
_dogStats?.DisposeAsync().ContinueWith(
1010+
t => Log.Error(t.Exception, "Error waiting for StatsD disposal"),
1011+
CancellationToken.None,
1012+
TaskContinuationOptions.OnlyOnFaulted,
1013+
TaskScheduler.Default);
9611014
}
9621015
}
9631016
}

tracer/src/Datadog.Trace/Debugger/Expressions/ProbeProcessor.cs

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,16 @@ private static CaptureLimitInfo ToCaptureLimitInfo(Capture? capture)
106106
return result.Count == 0 ? null : result.ToArray();
107107
}
108108

109+
private static bool ShouldRefreshMemoryPressureBeforeCapture(MethodState methodState)
110+
{
111+
return methodState is MethodState.BeginLine
112+
or MethodState.BeginLineAsync
113+
or MethodState.EntryStart
114+
or MethodState.EntryAsync
115+
or MethodState.ExitStart
116+
or MethodState.ExitStartAsync;
117+
}
118+
109119
public bool TryBeginProcess(in ProbeData probeData, [NotNullWhen(true)] out IDebuggerSnapshotCreator? snapshotCreator)
110120
{
111121
var state = _state;
@@ -138,8 +148,13 @@ public bool Process<TCapture>(ref CaptureInfo<TCapture> info, IDebuggerSnapshotC
138148
}
139149

140150
var probeInfo = state.ProbeInfo;
151+
var dynamicInstrumentation = DebuggerManager.Instance.DynamicInstrumentation;
152+
if (dynamicInstrumentation is not null && ShouldRefreshMemoryPressureBeforeCapture(info.MethodState))
153+
{
154+
dynamicInstrumentation.RefreshMemoryPressureIfStale();
155+
}
141156

142-
if (DebuggerManager.Instance.DynamicInstrumentation?.IsInitialized == false)
157+
if (dynamicInstrumentation?.IsInitialized == false)
143158
{
144159
Log.Debug("Stop processing probe {ID} because Dynamic Instrumentation has not initialized yet or has been disabled, probably dynamically through Remote Config", probeData.ProbeId);
145160
snapshotCreator.Stop();
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
// <copyright file="MemoryPressureConfig.cs" company="Datadog">
2+
// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
3+
// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
4+
// </copyright>
5+
6+
#nullable enable
7+
8+
using System;
9+
using System.Globalization;
10+
using Datadog.Trace.Util;
11+
12+
namespace Datadog.Trace.Debugger.RateLimiting
13+
{
14+
internal readonly struct MemoryPressureConfig
15+
{
16+
public static MemoryPressureConfig Default { get; } = new()
17+
{
18+
HighPressureThresholdRatio = 0.85,
19+
MaxGen2PerSecond = 2,
20+
MemoryExitMargin = 0.05,
21+
Gen2ExitMargin = 1,
22+
ConsecutiveHighToEnter = 1,
23+
ConsecutiveLowToExit = 1,
24+
RefreshInterval = TimeSpan.FromSeconds(1)
25+
};
26+
27+
// Enter high pressure at this fraction (0.0–1.0) of available memory in use. Machine-wide on .NET Framework.
28+
public double HighPressureThresholdRatio { get; init; }
29+
30+
public int MaxGen2PerSecond { get; init; }
31+
32+
public double MemoryExitMargin { get; init; }
33+
34+
public int Gen2ExitMargin { get; init; }
35+
36+
public int ConsecutiveHighToEnter { get; init; }
37+
38+
public int ConsecutiveLowToExit { get; init; }
39+
40+
public TimeSpan RefreshInterval { get; init; }
41+
42+
public override string ToString()
43+
{
44+
var culture = CultureInfo.InvariantCulture;
45+
var sb = StringBuilderCache.Acquire();
46+
47+
sb.Append("Threshold=");
48+
sb.Append(HighPressureThresholdRatio.ToString("F2", culture));
49+
sb.Append(" (");
50+
sb.Append((HighPressureThresholdRatio * 100).ToString("F1", culture));
51+
sb.Append("%), MaxGen2=");
52+
sb.Append(MaxGen2PerSecond.ToString(culture));
53+
sb.Append("/s, ExitMargin=");
54+
sb.Append(MemoryExitMargin.ToString("F2", culture));
55+
sb.Append(", Gen2ExitMargin=");
56+
sb.Append(Gen2ExitMargin.ToString(culture));
57+
sb.Append(", HighToEnter=");
58+
sb.Append(ConsecutiveHighToEnter.ToString(culture));
59+
sb.Append(", LowToExit=");
60+
sb.Append(ConsecutiveLowToExit.ToString(culture));
61+
62+
return StringBuilderCache.GetStringAndRelease(sb);
63+
}
64+
}
65+
}

0 commit comments

Comments
 (0)