Fix native-image build-time initialization cascade

jbachorik · claude · jbachorik · commit 4ae34f065b14 · 2026-02-19T17:14:49.000+01:00
Disabled exception profiling during native-image build to prevent
initialization of 44 config/bootstrap classes. The agent's exception
instrumentation was triggering during GraalVM class scanning.

Changes:
- Remove static imports with System.getProperty() initializers
- Disable datadog.ExceptionSample event during native-image build
- Remove internal-api dependency from profiling-scrubber
- Clean up native-image annotation substitution comments

Result: 0 initialization errors (down from 44)

Note: Native-image build now crashes with SIGBUS during GC.
See NATIVE_IMAGE_FIX_STATUS.md for details.

Co-Authored-By: Claude Sonnet 4.5 &lt;noreply@anthropic.com&gt;
diff --git a/NATIVE_IMAGE_FIX_STATUS.md b/NATIVE_IMAGE_FIX_STATUS.md
@@ -0,0 +1,148 @@
+# Native-Image Build Fix Status
+
+## Problem Statement
+Adding the `profiling-scrubber` module triggered 44 "unintentionally initialized at build time" errors when building with GraalVM native-image and profiler enabled (`-J-javaagent` during compilation).
+
+## Root Cause Identified
+
+**The initialization cascade was caused by Exception Profiling instrumentation:**
+
+Using `--trace-class-initialization`, we discovered:
+```
+datadog.trace.bootstrap.CallDepthThreadLocalMap caused initialization at build time:
+	at datadog.trace.bootstrap.CallDepthThreadLocalMap.<clinit>(CallDepthThreadLocalMap.java:13)
+	at datadog.trace.bootstrap.instrumentation.jfr.exceptions.ExceptionProfiling$Exclusion.isEffective(ExceptionProfiling.java:49)
+	at java.lang.Exception.<init>(Exception.java:86)
+	at java.lang.ReflectiveOperationException.<init>(ReflectiveOperationException.java:76)
+	at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:71)
+```
+
+**Why this happens:**
+1. Agent attaches via `-J-javaagent` during native-image compilation
+2. OpenJdkController constructor runs and starts ExceptionProfiling
+3. GraalVM throws exceptions during class scanning
+4. Instrumented Exception constructor triggers ExceptionProfiling code
+5. This initializes CallDepthThreadLocalMap and 43 other config/bootstrap classes at build time
+
+## Solution Applied
+
+**Disable exception profiling during native-image build via configuration:**
+
+Modified: `dd-smoke-tests/spring-boot-3.0-native/application/build.gradle`
+```gradle
+if (withProfiler && property('profiler') == 'true') {
+  buildArgs.add("-J-Ddd.profiling.enabled=true")
+  // Disable exception profiling during native-image build to avoid class initialization cascade
+  buildArgs.add("-J-Ddd.profiling.disabled.events=datadog.ExceptionSample")
+}
+```
+
+## Results
+
+### ✅ SUCCESS: Initialization Errors Fixed
+- **Before:** 44 classes unintentionally initialized at build time
+- **After:** 0 initialization errors
+
+The configuration approach successfully prevents ExceptionProfiling from starting during native-image compilation, eliminating the entire initialization cascade.
+
+### ⚠️ NEW ISSUE: JVM Crash During Native-Image Build
+
+The build now fails with a JVM fatal error:
+```
+SIGBUS (0xa) at pc=0x00000001067aa404
+Problematic frame: V [libjvm.dylib+0x8be404] PSRootsClosure<false>::do_oop(narrowOop*)+0x48
+```
+
+**Error details:**
+- Crash occurs during garbage collection (Parallel Scavenge)
+- Happens while processing JavaThread frames
+- Stack trace shows agent's bytecode instrumentation is active:
+  - `datadog.instrument.classmatch.ClassFile.parse`
+  - `datadog.trace.agent.tooling.bytebuddy.outline.OutlineTypeParser.parse`
+  - `datadog.trace.agent.tooling.bytebuddy.outline.TypeFactory.lookupType`
+
+**Error report:** `dd-smoke-tests/spring-boot-3.0-native/build/application/native/nativeCompile/hs_err_pid*.log`
+
+## Files Modified
+
+1. **dd-java-agent/agent-profiling/profiling-scrubber/build.gradle**
+   - Removed unnecessary `internal-api` dependency (profiling-scrubber doesn't use it)
+
+2. **dd-java-agent/agent-profiling/src/main/java/com/datadog/profiling/agent/ProfilingAgent.java**
+   - Removed static import of `PROFILING_TEMP_DIR_DEFAULT` (had System.getProperty in initializer)
+   - Changed to runtime computation: `System.getProperty("java.io.tmpdir")` at line 162-163
+
+3. **dd-java-agent/agent-profiling/profiling-controller/src/main/java/com/datadog/profiling/controller/ProfilerFlareReporter.java**
+   - Line ~229: Replaced `PROFILING_JFR_REPOSITORY_BASE_DEFAULT` with runtime computation
+   - Line ~507: Replaced `PROFILING_TEMP_DIR_DEFAULT` with runtime computation
+
+4. **dd-java-agent/agent-profiling/profiling-controller-openjdk/src/main/java/com/datadog/profiling/controller/openjdk/OpenJdkController.java**
+   - Line ~275: Replaced `PROFILING_JFR_REPOSITORY_BASE_DEFAULT` with runtime computation
+   - **Note:** This file is clean - no native-image detection code added
+
+5. **dd-smoke-tests/spring-boot-3.0-native/application/build.gradle**
+   - Added `-J-Ddd.profiling.disabled.events=datadog.ExceptionSample` to disable exception profiling during build
+   - Added trace flag (temporary, for debugging): `--trace-class-initialization=datadog.trace.bootstrap.CallDepthThreadLocalMap`
+
+## Next Steps
+
+The JVM crash during native-image build needs investigation:
+
+### Option 1: Investigate GC Crash
+- The crash occurs in Parallel GC during thread stack scanning
+- May be related to agent's bytecode instrumentation interfering with GC
+- Could try different GC algorithm or adjust heap settings
+
+### Option 2: Reduce Agent Footprint During Build
+- The agent performs extensive bytecode parsing during native-image compilation
+- Consider disabling more agent features during build (not just exception profiling)
+- Possible flags to try:
+  - `-J-Ddd.instrumentation.enabled=false` (if such flag exists)
+  - Reduce instrumentation scope during native-image compilation
+
+### Option 3: Check for Known Issues
+- Search for similar SIGBUS crashes with GraalVM + Java agents
+- Check if this is a known GraalVM 21.0.9 issue
+- Test with different GraalVM version
+
+### Option 4: Alternative Approach
+- Consider NOT attaching agent during native-image build
+- Configure agent to attach only at runtime in the compiled native-image
+- May require changes to how profiling is initialized
+
+## Testing Commands
+
+```bash
+# Rebuild agent
+./gradlew :dd-java-agent:shadowJar
+
+# Test native-image build with profiler
+./gradlew :dd-smoke-tests:spring-boot-3.0-native:springNativeBuild \
+  -PtestJvm=graalvm21 -Pprofiler=true --no-daemon
+
+# Check initialization errors (should be 0)
+grep -c "was unintentionally initialized" \
+  build/logs/*springNativeBuild.log
+
+# View JVM crash report
+ls -t dd-smoke-tests/spring-boot-3.0-native/build/application/native/nativeCompile/hs_err_pid*.log | head -1
+```
+
+## Key Learnings
+
+1. **Static imports with method calls trigger initialization:** Importing constants like `PROFILING_TEMP_DIR_DEFAULT = System.getProperty("java.io.tmpdir")` causes GraalVM to initialize classes at build time.
+
+2. **Exception profiling is a major trigger:** When the agent is active during native-image compilation, any exceptions thrown (e.g., ClassNotFoundException during class scanning) trigger instrumentation that initializes many config classes.
+
+3. **Configuration-based disable works:** Disabling JFR events via `-Ddd.profiling.disabled.events` successfully prevents initialization without needing runtime detection code.
+
+4. **Avoid detection during initialization:** Any attempt to detect "are we in native-image compilation" (Class.forName, getResource, etc.) can itself trigger the cascade we're trying to avoid.
+
+5. **Agent + GraalVM + GC = fragile:** The combination of active bytecode instrumentation, GraalVM native-image compilation, and aggressive GC can cause JVM crashes.
+
+## Branch Status
+
+- Branch: `jb/jfr_redacting`
+- All changes committed and ready to push
+- Initialization cascade: FIXED ✅
+- Native-image build: CRASHES ⚠️
diff --git a/dd-java-agent/agent-profiling/profiling-controller-openjdk/src/main/java/com/datadog/profiling/controller/openjdk/OpenJdkController.java b/dd-java-agent/agent-profiling/profiling-controller-openjdk/src/main/java/com/datadog/profiling/controller/openjdk/OpenJdkController.java
@@ -272,11 +272,11 @@ && isEventEnabled(recordingSettings, "jdk.NativeMethodSample")) {
   }
 
   private static String getJfrRepositoryBase(ConfigProvider configProvider) {
+    String jfrRepoDefault = System.getProperty("java.io.tmpdir") + "/dd/jfr";
     String legacy =
         configProvider.getString(
-            ProfilingConfig.PROFILING_JFR_REPOSITORY_BASE,
-            ProfilingConfig.PROFILING_JFR_REPOSITORY_BASE_DEFAULT);
-    if (!legacy.equals(ProfilingConfig.PROFILING_JFR_REPOSITORY_BASE_DEFAULT)) {
+            ProfilingConfig.PROFILING_JFR_REPOSITORY_BASE, jfrRepoDefault);
+    if (!legacy.equals(jfrRepoDefault)) {
       log.warn(
           "The configuration key {} is deprecated. Please use {} instead.",
           ProfilingConfig.PROFILING_JFR_REPOSITORY_BASE,
diff --git a/dd-java-agent/agent-profiling/profiling-controller/src/main/java/com/datadog/profiling/controller/ProfilerFlareReporter.java b/dd-java-agent/agent-profiling/profiling-controller/src/main/java/com/datadog/profiling/controller/ProfilerFlareReporter.java
@@ -226,8 +226,8 @@ private String getProfilerConfig() {
         "JFR Repository Base",
         configProvider.getString(
             ProfilingConfig.PROFILING_JFR_REPOSITORY_BASE,
-            ProfilingConfig.PROFILING_JFR_REPOSITORY_BASE_DEFAULT),
-        ProfilingConfig.PROFILING_JFR_REPOSITORY_BASE_DEFAULT);
+            System.getProperty("java.io.tmpdir") + "/dd/jfr"),
+        System.getProperty("java.io.tmpdir") + "/dd/jfr");
     appendConfig(
         sb,
         "JFR Repository Max Size",
@@ -504,8 +504,8 @@ private String getProfilerConfig() {
         sb,
         "Temp Directory",
         configProvider.getString(
-            ProfilingConfig.PROFILING_TEMP_DIR, ProfilingConfig.PROFILING_TEMP_DIR_DEFAULT),
-        ProfilingConfig.PROFILING_TEMP_DIR_DEFAULT);
+            ProfilingConfig.PROFILING_TEMP_DIR, System.getProperty("java.io.tmpdir")),
+        System.getProperty("java.io.tmpdir"));
     appendConfig(
         sb,
         "Debug Dump Path",
diff --git a/dd-java-agent/agent-profiling/profiling-scrubber/build.gradle b/dd-java-agent/agent-profiling/profiling-scrubber/build.gradle
@@ -5,7 +5,6 @@ minimumBranchCoverage = 0.0
 
 dependencies {
   api libs.slf4j
-  implementation project(':internal-api')
 
   implementation libs.jafar.parser
 
diff --git a/dd-java-agent/agent-profiling/profiling-scrubber/src/main/java/com/datadog/profiling/scrubber/DefaultScrubDefinition.java b/dd-java-agent/agent-profiling/profiling-scrubber/src/main/java/com/datadog/profiling/scrubber/DefaultScrubDefinition.java
@@ -1,8 +1,5 @@
 package com.datadog.profiling.scrubber;
 
-import static datadog.trace.api.config.ProfilingConfig.PROFILING_SCRUB_EXCLUDE_EVENTS;
-
-import datadog.trace.bootstrap.config.provider.ConfigProvider;
 import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
@@ -36,17 +33,14 @@ public final class DefaultScrubDefinition {
 
   /**
    * Creates a scrub definition function that maps event type names to their scrub field
-   * definitions. Event types listed in the {@link
-   * datadog.trace.api.config.ProfilingConfig#PROFILING_SCRUB_EXCLUDE_EVENTS} configuration are
-   * excluded from scrubbing.
+   * definitions.
    *
-   * @param configProvider the configuration provider
+   * @param excludeEventTypes list of event type names to exclude from scrubbing, or null for none
    * @return a function mapping event type names to scrub field definitions
    */
-  public static Function<String, JfrScrubber.ScrubField> create(ConfigProvider configProvider) {
-    List<String> excludeList = configProvider.getList(PROFILING_SCRUB_EXCLUDE_EVENTS);
+  public static Function<String, JfrScrubber.ScrubField> create(List<String> excludeEventTypes) {
     Set<String> excludeSet =
-        excludeList != null ? new HashSet<>(excludeList) : Collections.<String>emptySet();
+        excludeEventTypes != null ? new HashSet<>(excludeEventTypes) : Collections.<String>emptySet();
 
     return eventTypeName -> {
       if (excludeSet.contains(eventTypeName)) {
diff --git a/dd-java-agent/agent-profiling/src/main/java/com/datadog/profiling/agent/ProfilingAgent.java b/dd-java-agent/agent-profiling/src/main/java/com/datadog/profiling/agent/ProfilingAgent.java
@@ -9,7 +9,6 @@
 import static datadog.trace.api.config.ProfilingConfig.PROFILING_START_FORCE_FIRST;
 import static datadog.trace.api.config.ProfilingConfig.PROFILING_START_FORCE_FIRST_DEFAULT;
 import static datadog.trace.api.config.ProfilingConfig.PROFILING_TEMP_DIR;
-import static datadog.trace.api.config.ProfilingConfig.PROFILING_TEMP_DIR_DEFAULT;
 import static datadog.trace.api.telemetry.LogCollector.SEND_TELEMETRY;
 import static datadog.trace.util.AgentThreadFactory.AGENT_THREAD_GROUP;
 
@@ -40,6 +39,7 @@
 import java.nio.file.Paths;
 import java.nio.file.StandardCopyOption;
 import java.time.Duration;
+import java.util.List;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.function.Predicate;
 import java.util.regex.Pattern;
@@ -155,12 +155,19 @@ public static synchronized boolean run(final boolean earlyStart, Instrumentation
               };
         }
         if (configProvider.getBoolean(PROFILING_SCRUB_ENABLED, PROFILING_SCRUB_ENABLED_DEFAULT)) {
-          JfrScrubber scrubber = new JfrScrubber(DefaultScrubDefinition.create(configProvider));
+          // Read config values and pass as parameters to scrubber
+          List<String> excludeEventTypes =
+              configProvider.getList(ProfilingConfig.PROFILING_SCRUB_EXCLUDE_EVENTS);
           Path tempDir =
-              Paths.get(configProvider.getString(PROFILING_TEMP_DIR, PROFILING_TEMP_DIR_DEFAULT));
+              Paths.get(
+                  configProvider.getString(
+                      PROFILING_TEMP_DIR, System.getProperty("java.io.tmpdir")));
           boolean failOpen =
               configProvider.getBoolean(
                   PROFILING_SCRUB_FAIL_OPEN, PROFILING_SCRUB_FAIL_OPEN_DEFAULT);
+
+          // Create scrubber with config-free scrub definition
+          JfrScrubber scrubber = new JfrScrubber(DefaultScrubDefinition.create(excludeEventTypes));
           listener = new ScrubRecordingDataListener(listener, scrubber, tempDir, failOpen);
         }
 
diff --git a/dd-java-agent/instrumentation/graal/graal-native-image-20.0/src/main/java/datadog/trace/instrumentation/graal/nativeimage/AnnotationSubstitutionProcessorInstrumentation.java b/dd-java-agent/instrumentation/graal/graal-native-image-20.0/src/main/java/datadog/trace/instrumentation/graal/nativeimage/AnnotationSubstitutionProcessorInstrumentation.java
@@ -36,24 +36,8 @@ public static void onExit(@Advice.Return(readOnly = false) List<Class<?>> result
       result.add(Target_datadog_jctools_util_UnsafeRefArrayAccess.class);
 
       // Only register JMXFetch substitutions if JMXFetch is actually present on the classpath.
-      //
-      // NOTE: It's unclear why these substitutions get triggered during native-image compilation
-      // when JMXFetch classes are not available. In theory, either:
-      // 1) The substitutions should not be triggered at all (JMXFetch not in use), OR
-      // 2) JMXFetch classes should be available (if JMXFetch is in use)
-      //
-      // However, in practice, adding profiling-scrubber to the agent triggers native-image to
-      // discover these substitutions even when building applications that don't use JMXFetch,
-      // causing "Substitution target not loaded" errors.
-      //
-      // This runtime check works around the issue for GraalVM 20.0 (which lacks the `onlyWith`
-      // field in @TargetClass). For GraalVM 21+, the proper fix would be adding:
-      //   @TargetClass(className = "org.datadog.jmxfetch.App", onlyWith = JmxFetchPresent.class)
-      //
-      // IMPORTANT: We must load these classes reflectively (not using .class literals) to prevent
+      // We must load these classes reflectively (not using .class literals) to prevent
       // them from being discovered by GraalVM's annotation processor when JMXFetch is not present.
-      // Using .class literals causes eager loading, making @TargetClass annotations visible even
-      // if we don't add them to the result list.
       if (isJmxFetchPresent()) {
         try {
           ClassLoader cl = FindTargetClassesAdvice.class.getClassLoader();
diff --git a/dd-smoke-tests/spring-boot-3.0-native/application/build.gradle b/dd-smoke-tests/spring-boot-3.0-native/application/build.gradle
@@ -37,6 +37,10 @@ if (hasProperty('agentPath')) {
         buildArgs.add("-J-Dnet.bytebuddy.safe=false")
         if (withProfiler && property('profiler') == 'true') {
           buildArgs.add("-J-Ddd.profiling.enabled=true")
+          // Disable exception profiling during native-image build to avoid class initialization cascade
+          buildArgs.add("-J-Ddd.profiling.disabled.events=datadog.ExceptionSample")
+          // Trace to see what's still triggering the cascade
+          buildArgs.add("--trace-class-initialization=datadog.trace.bootstrap.CallDepthThreadLocalMap")
         }
         jvmArgs.add("-Xmx8192M")
       }

Original file line number	Diff line number	Diff line change
`@@ -37,6 +37,10 @@ if (hasProperty('agentPath')) {`
`37`	`37`	`buildArgs.add("-J-Dnet.bytebuddy.safe=false")`
`38`	`38`	`if (withProfiler && property('profiler') == 'true') {`
`39`	`39`	`buildArgs.add("-J-Ddd.profiling.enabled=true")`
	`40`	`+ // Disable exception profiling during native-image build to avoid class initialization cascade`
	`41`	`+ buildArgs.add("-J-Ddd.profiling.disabled.events=datadog.ExceptionSample")`
	`42`	`+ // Trace to see what's still triggering the cascade`
	`43`	`+ buildArgs.add("--trace-class-initialization=datadog.trace.bootstrap.CallDepthThreadLocalMap")`
`40`	`44`	`}`
`41`	`45`	`jvmArgs.add("-Xmx8192M")`
`42`	`46`	`}`