fix(profiling): avoid crashing stack sampler with active foreign SIGSEGV handlers (PROF-15342) (#18798)

| [Next PR](#18947)

User-reported crash: https://datadoghq.atlassian.net/browse/AIPTS-1715

## Description

`safe_memcpy`'s fault recovery only works while the profiler owns the `SIGSEGV`/`SIGBUS`
handlers. Libraries like PyTorch/CUDA (and abseil via vLLM/gRPC) install their own
handler during startup; once a foreign handler owns those signals, a fault on a stale
read is no longer recovered and the process crashes ([PROF-14568](https://datadoghq.atlassian.net/browse/PROF-14568)).

This PR makes the default-on state safe and removes the need for the `_DD_PROFILING_STACK_FAST_COPY=0`
workaround. It handles the handlers we *cannot* wrap (torch/CUDA/abseil) via detect-and-fallback.

## Changes

* Sampler **starts on the safe syscall copy** (`process_vm_readv` / `mach_vm_read_overwrite`)
  for a short warmup, so a fault during crash-prone startup can't crash the process (these
  syscalls return an error instead of faulting).
* After warmup it **upgrades to `safe_memcpy` only if we still own both `SIGSEGV` and
  `SIGBUS`** (`segv_handler_installed()`).
* It **re-checks ownership every cycle** and permanently falls back to the syscall copy if
  a handler is taken over later (e.g. lazy CUDA init). If no safe fallback exists
  (`process_vm_readv` blocked), it **stops sampling** — we degrade to dropped samples,
  never a crash.
* Warmup is a fixed 15s internal constant (not a user knob). Policy is auto-fallback, not
  reinstall-and-chain, so the foreign handler stays authoritative; `init_segv_catcher`
  stays `call_once` to avoid reintroducing handler-chaining races.

## Test plan

* New unit tests
* Manual repro (synthetic + `--torch`) on #18911):
crashes on `main`, runs OK on this branch
* Tested on another internal service in staging: `ai_gateway` ([profile link](https://ddstaging.datadoghq.com/profiling/explorer?query=service%3Aai_gateway%20env%3Astaging%20version%3Afaulthandling-1ac266be&my_code=disabled&profile_type=heap-live-size&refresh_mode=paused&viz=flame_graph&from_ts=1784830809012&to_ts=1784834409012&live=false))

* Py CPU

<img width="1004" height="1024" alt="image" src="https://github.com/user-attachments/assets/5b4311bf-ee2b-4dd0-bb1f-b23f05ca567d" />

* eBPF CPU

<img width="999" height="1024" alt="image" src="https://github.com/user-attachments/assets/ada757ba-8e35-48c0-ba97-e6972f69549b" />

* Live Heap

<img width="1009" height="1024" alt="image" src="https://github.com/user-attachments/assets/c7161fad-e922-43d6-b7c8-5cfbc0706cd0" />



### Testing in User Environment:

* User deployed a binary with this and next PR (#18797) to their service; no crashes observed in 24 hours. (cc @askardog )

[PROF-14568]: https://datadoghq.atlassian.net/browse/PROF-14568?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Co-authored-by: KowalskiThomas <thomas.kowalski@datadoghq.com>
Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>

Author: vlad-scherbich

ddtrace/internal/datadog/profiling/dd_wrapper/include/profiler_stats.hpp

@@ -31,6 +31,15 @@ class ProfilerStats
     // Whether fast_copy_memory (safe_memcpy) is enabled; unset until the sampler starts
     std::optional<bool> fast_copy_memory_enabled;
 
+    // User opted out of fast copy (env var or set_fast_copy(false)); static per process
+    std::optional<bool> fast_copy_memory_user_disabled;
+
+    // Whether safe_memcpy initialized at startup; static per process
+    std::optional<bool> fast_copy_memory_capable;
+
+    // Sticky: fell back to syscall copy (init failure, foreign handler, etc.)
+    std::optional<bool> fast_copy_memory_syscall_fallback;
+
     // Number of copy_memory errors accumulated since the last profile reset (i.e. since the last upload)
     size_t copy_memory_error_count = 0;
 
@@ -69,6 +78,18 @@ class ProfilerStats
     void set_fast_copy_memory_enabled(bool enabled);
     std::optional<bool> get_fast_copy_memory_enabled() const;
 
+    void set_fast_copy_memory_user_disabled(bool disabled);
+    std::optional<bool> get_fast_copy_memory_user_disabled() const;
+
+    void set_fast_copy_memory_capable(bool capable);
+    std::optional<bool> get_fast_copy_memory_capable() const;
+
+    void set_fast_copy_memory_syscall_fallback(bool fallback);
+    std::optional<bool> get_fast_copy_memory_syscall_fallback() const;
+
+    // fast_copy_memory_* are process-static; carry them across ProfilerStats swaps.
+    void copy_fast_copy_metadata_from(const ProfilerStats& other);
+
     void add_copy_memory_error_count(size_t count);
     size_t get_copy_memory_error_count() const;
 

ddtrace/internal/datadog/profiling/dd_wrapper/src/profiler_stats.cpp

@@ -12,6 +12,19 @@ append_to_string(std::string& s, size_t value)
     s.append(buf, ptr);
 }
 
+void
+append_optional_bool(std::string& s, const char* key, const std::optional<bool>& value)
+{
+    if (!value.has_value()) {
+        return;
+    }
+    s += '"';
+    s += key;
+    s += "\": ";
+    s += *value ? "true" : "false";
+    s += ',';
+}
+
 } // namespace
 
 void
@@ -51,7 +64,7 @@ Datadog::ProfilerStats::reset_state()
     asyncio_task_count = std::nullopt;
     greenlet_count = std::nullopt;
     sample_capture_cpu_time_us = 0;
-    // fast_copy_memory_enabled is intentionally not reset: it reflects a static configuration
+    // fast_copy_memory_* static fields are intentionally not reset (see setters).
 }
 
 void
@@ -66,6 +79,59 @@ Datadog::ProfilerStats::get_fast_copy_memory_enabled() const
     return fast_copy_memory_enabled;
 }
 
+void
+Datadog::ProfilerStats::set_fast_copy_memory_user_disabled(bool disabled)
+{
+    fast_copy_memory_user_disabled = disabled;
+}
+
+std::optional<bool>
+Datadog::ProfilerStats::get_fast_copy_memory_user_disabled() const
+{
+    return fast_copy_memory_user_disabled;
+}
+
+void
+Datadog::ProfilerStats::set_fast_copy_memory_capable(bool capable)
+{
+    fast_copy_memory_capable = capable;
+}
+
+std::optional<bool>
+Datadog::ProfilerStats::get_fast_copy_memory_capable() const
+{
+    return fast_copy_memory_capable;
+}
+
+void
+Datadog::ProfilerStats::set_fast_copy_memory_syscall_fallback(bool fallback)
+{
+    fast_copy_memory_syscall_fallback = fallback;
+}
+
+std::optional<bool>
+Datadog::ProfilerStats::get_fast_copy_memory_syscall_fallback() const
+{
+    return fast_copy_memory_syscall_fallback;
+}
+
+void
+Datadog::ProfilerStats::copy_fast_copy_metadata_from(const ProfilerStats& other)
+{
+    if (auto value = other.get_fast_copy_memory_user_disabled()) {
+        set_fast_copy_memory_user_disabled(*value);
+    }
+    if (auto value = other.get_fast_copy_memory_capable()) {
+        set_fast_copy_memory_capable(*value);
+    }
+    if (auto value = other.get_fast_copy_memory_syscall_fallback()) {
+        set_fast_copy_memory_syscall_fallback(*value);
+    }
+    if (auto value = other.get_fast_copy_memory_enabled()) {
+        set_fast_copy_memory_enabled(*value);
+    }
+}
+
 void
 Datadog::ProfilerStats::add_copy_memory_error_count(size_t count)
 {
@@ -203,6 +269,11 @@ Datadog::ProfilerStats::get_internal_metadata_json()
         internal_metadata_json += ",";
     }
 
+    append_optional_bool(internal_metadata_json, "fast_copy_memory_user_disabled", fast_copy_memory_user_disabled);
+    append_optional_bool(internal_metadata_json, "fast_copy_memory_capable", fast_copy_memory_capable);
+    append_optional_bool(
+      internal_metadata_json, "fast_copy_memory_syscall_fallback", fast_copy_memory_syscall_fallback);
+
     auto maybe_heap_tracker_count = get_heap_tracker_size();
     if (maybe_heap_tracker_count) {
         internal_metadata_json += R"("heap_tracker_count": )";

ddtrace/internal/datadog/profiling/dd_wrapper/src/uploader_builder.cpp

@@ -205,6 +205,7 @@ Datadog::UploaderBuilder::build()
         // Swap the ProfilerStats (which replaces the one being written to with an empty state).
         // We do this first as we still want to reset ProfilerStats if the serialization fails.
         std::swap(stats, borrowed.stats());
+        borrowed.stats().copy_fast_copy_metadata_from(stats);
 
         // Try to encode the Profile (which will also reset it)
         encoded = ddog_prof_Profile_serialize(&borrowed.profile(), nullptr, nullptr);

ddtrace/internal/datadog/profiling/stack/__init__.pyi

@@ -42,6 +42,10 @@ def set_interval(new_interval: float) -> None: ...
 # Memory copy strategy
 def set_fast_copy(enabled: bool) -> None: ...
 def is_safe_copy_failed() -> bool: ...
+def fast_copy_memory_active() -> bool: ...  # test introspection: is safe_memcpy active?
+
+# _set_fast_copy_warmup_seconds is test-only; accessed via _stack (import * skips it).
+
 def uninstall_segv_handler() -> None: ...
 def reinstall_segv_handler() -> None:
     """Reinstall SIGSEGV/SIGBUS handlers after another component overwrites them.
@@ -52,6 +56,14 @@ def reinstall_segv_handler() -> None:
     """
     ...
 
+def segv_handler_installed() -> bool:
+    """Return True if our handler is the installed disposition for SIGSEGV and SIGBUS.
+
+    Primarily test introspection: it queries the live disposition via sigaction(2)
+    for both signals on every call, so it is not free. Do not call it on hot paths.
+    """
+    ...
+
 # Pause/resume sampling
 def pause_sampling() -> bool | None:
     """Pause the sampling thread and wait for any in-flight sample to complete.

ddtrace/internal/datadog/profiling/stack/_stack.pyi

@@ -23,6 +23,15 @@ def set_interval(new_interval: float) -> None: ...
 # Memory copy strategy
 def set_fast_copy(enabled: bool) -> None: ...
 def is_safe_copy_failed() -> bool: ...
+def fast_copy_memory_active() -> bool: ...  # test introspection: is safe_memcpy active?
+def _set_fast_copy_warmup_seconds(seconds: float) -> None: ...  # test-only; before start
+def segv_handler_installed() -> bool:
+    """Return True if our handler is the installed disposition for SIGSEGV and SIGBUS.
+
+    Primarily test introspection: it queries the live disposition via sigaction(2)
+    for both signals on every call, so it is costly. Do not call it on hot paths.
+    """
+    ...
 
 # span <-> profile association
 def link_span(

ddtrace/internal/datadog/profiling/stack/echion/echion/danger.h

@@ -27,6 +27,10 @@ init_segv_catcher();
 void
 uninstall_segv_handler();
 
+// Returns true only if our signal handler owns both SIGSEGV and SIGBUS; false on any error.
+bool
+segv_handler_installed();
+
 #if defined PL_LINUX
 ssize_t
 safe_memcpy_wrapper(pid_t,

ddtrace/internal/datadog/profiling/stack/echion/echion/vm.h

@@ -54,8 +54,19 @@ inline kern_return_t (*safe_copy)(vm_map_read_t,
 // Whether safe_copy is currently set to the memcpy-based wrapper.
 inline bool fast_copy_active = false;
 
-// Whether init_segv_catcher succeeded at constructor time. Persists even if
-// fast_copy_active is later toggled off by set_fast_copy_enabled.
+// User opted out via _DD_PROFILING_STACK_FAST_COPY or set_fast_copy(false).
+inline bool fast_copy_user_disabled = false;
+
+// Sticky: fell back to syscall copy (init failure, foreign handler, warmup miss).
+inline bool fast_copy_syscall_fallback = false;
+
+inline void
+mark_fast_copy_syscall_fallback()
+{
+    fast_copy_syscall_fallback = true;
+}
+
+// Set at init; survives toggling fast_copy_active.
 inline bool safe_memcpy_initialized = false;
 
 #if defined PL_LINUX

ddtrace/internal/datadog/profiling/stack/include/sampler.hpp

@@ -45,9 +45,6 @@ class Sampler
     // The mutex + condition variable pair is used to avoid the "lost wake-up" race condition
     // where stop() could miss the notification and hang forever (or until timeout).
     std::atomic<bool> thread_running{ false };
-    std::mutex thread_exit_mutex;
-    std::condition_variable thread_exit_cv;
-
     // Whether the sampler is currently active. Unlike thread_running (which tracks
     // the thread's actual lifecycle) this is set synchronously in start/stop, so
     // it is not subject to the sampling-thread startup race.
@@ -56,6 +53,8 @@ class Sampler
     // prefork reads this to decide whether to restart the sampler after fork,
     // ensuring a sampler that stopped due to an error is not silently restarted.
     std::atomic<bool> sampler_active_{ false };
+    std::mutex thread_exit_mutex;
+    std::condition_variable thread_exit_cv;
 
     // Pause synchronization — allows the faulthandler wrapper to temporarily
     // suspend sampling so signal handlers can be safely swapped without racing
@@ -101,6 +100,9 @@ class Sampler
     // Percentile (0..1) used for p_stable; configurable, default p95.
     double p_stable_percentile_frac = 0.95;
 
+    // Fast-copy startup warmup in seconds.
+    double fast_copy_warmup_seconds = 15.0;
+
     // Rolling window duration in seconds; controls the ring buffer capacity.
     uint32_t p_stable_window_s = 600;
 
@@ -164,6 +166,8 @@ class Sampler
     // Set the percentile (0–100) used to compute p_stable from the rolling window.
     void set_p_stable_percentile(double percentile) { p_stable_percentile_frac = percentile / 100.0; }
 
+    void set_fast_copy_warmup_seconds(double value) { fast_copy_warmup_seconds = value; }
+
     // Delegates to the StackRenderer to clear its caches after fork
     void postfork_child();
 
@@ -178,4 +182,7 @@ class Sampler
     bool restart_after_fork();
 };
 
+void
+seed_fast_copy_profiler_stats();
+
 } // namespace Datadog

ddtrace/internal/datadog/profiling/stack/src/echion/danger.cc

@@ -140,6 +140,24 @@ init_segv_catcher()
     return 0;
 }
 
+bool
+segv_handler_installed()
+{
+    // Recovery needs our handler to own BOTH SIGSEGV and SIGBUS
+    // (a copy fault can arrive as either); anything else means we can't recover.
+    const int signals[] = { SIGSEGV, SIGBUS };
+    for (int signo : signals) {
+        struct sigaction current;
+        if (sigaction(signo, nullptr, &current) != 0) {
+            return false;
+        }
+        if (current.sa_sigaction != segv_handler || (current.sa_flags & SA_SIGINFO) == 0) {
+            return false;
+        }
+    }
+    return true;
+}
+
 void
 uninstall_segv_handler()
 {

ddtrace/internal/datadog/profiling/stack/src/echion/vm.cc

@@ -41,6 +41,7 @@ init_safe_copy()
     // Honor the fast-copy opt-out: when disabled via env var, skip installing
     // the SIGSEGV/SIGBUS handlers and alt stack entirely.
     if (fast_copy_env_disabled()) {
+        fast_copy_user_disabled = true;
         if (process_vm_readv_available) {
             safe_copy = process_vm_readv;
         } else {
@@ -60,6 +61,7 @@ init_safe_copy()
         fprintf(stderr, "Failed to initialize segv catcher. Trying process_vm_readv.\n");
         if (process_vm_readv_available) {
             safe_copy = process_vm_readv;
+            mark_fast_copy_syscall_fallback();
         } else {
             fprintf(stderr, "Failed to initialize safe copy interface\n");
             failed_safe_copy = true;
@@ -72,6 +74,7 @@ init_safe_copy()
 {
     // Honor the fast-copy opt-out: skip installing signal handlers when disabled.
     if (fast_copy_env_disabled()) {
+        fast_copy_user_disabled = true;
         return;
     }
 
@@ -84,6 +87,7 @@ init_safe_copy()
 
     // std::cerr might not be fully initialized at constructor time.
     fprintf(stderr, "Failed to initialize segv catcher. Using mach_vm_read_overwrite instead.\n");
+    mark_fast_copy_syscall_fallback();
 }
 #endif // PL_DARWIN
 
@@ -99,6 +103,7 @@ set_fast_copy_enabled(bool enabled)
         }
         fprintf(stderr,
                 "Warning: fast copy requested but safe_memcpy was not initialized; falling back to process_vm_readv\n");
+        mark_fast_copy_syscall_fallback();
 
         // Fall through to process_vm_readv.
     }