Skip to content

Commit 94686da

Browse files
rkennkeclaudejbachorik
authored
fix(context): scope OTEL ThreadContext storage to the carrier thread (PROF-15271) (#625)
* fix(context): scope OTEL ThreadContext storage to the carrier thread (PROF-15271) The ThreadContext DirectByteBuffer is a window into the OTEP record embedded in the *carrier's* native ProfiledThread — the record the (carrier-bound) sampler reads. It was cached in a plain ThreadLocal, keying it by the *virtual* thread and pinning it to whichever carrier was mounted at first use. That is wrong once the vthread migrates (writes land on the old carrier, so a sampler on the new carrier sees stale/empty context) and unsafe once the old carrier's OS thread exits: the record is freed while the buffer keeps being written — a use-after-free that can corrupt JVM-owned native memory (observed as a crash in ThreadsSMRSupport::free_list). Introduce a context-storage mode. In CARRIER mode, storage is backed by jdk.internal.misc.CarrierThreadLocal (JDK 21+), whose get()/set()/remove() operate on the current carrier's map even from a mounted virtual thread, so a vthread always resolves to its current carrier's live record. Storage lifetime then matches the native record's lifetime, eliminating the dangling-buffer window. - OtelContextStorage: factory + Mode enum + kill-switch. The internal type is built reflectively and held as its ThreadLocal supertype, so calls dispatch virtually with no per-call reflection and no compile-time dependency (the lib is a Java 8 baseline). Degrades to a plain ThreadLocal — today's behavior — when the type is missing (older JDK) or inaccessible (export not granted), never failing hard. Selected via -Dddprof.context.storage.mode=auto|carrier|thread. - JavaProfiler: field uses the factory; a currentContext() get-or-init helper replaces ThreadLocal.withInitial (a reflectively-built CarrierThreadLocal cannot carry a supplier); the 8 context write sites route through it; contextStorageMode() added for diagnostics/tests. - Build: add --add-exports java.base/jdk.internal.misc=ALL-UNNAMED to the test JVM, gated on testJvmMajorVersion() >= 21 (the flag aborts a Java 8 JVM). Production grants the export from the agent (follow-up). Tests: - CarrierContextStorageTest: 2000 vthreads resolve to exactly one ThreadContext per carrier (far fewer than the vthread count). - OtelContextStorageTest: kill-switch forces plain ThreadLocal; auto selects carrier when available; explicit carrier falls back without throwing. - context_uaf_ut.cpp: ASan proof of the unguarded write-after-free primitive. - Existing OtelContextStorageModeTest unregressed. Follow-ups: export grant in dd-trace-java; reapply-on-mount / clear-on-unmount coherence (PROF / #11646), which this change makes safe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(context): fail hard on mode=carrier when CarrierThreadLocal is unavailable Silently falling back to thread-scoped storage on a JDK 21+ JVM re-exposes the exact virtual-thread context use-after-free this change removes. Make the policy explicit at the mode level: - carrier: carrier scoping is required — create() now throws if jdk.internal.misc.CarrierThreadLocal is not accessible, with an actionable message (add the --add-exports, or opt into mode=thread). This is the fail-fast callers get once the runtime export is guaranteed. - auto (default): still degrades so profiling loads even without the export (the export may be granted by an agent that loads after this library, and the fallback is safe for non-Loom apps), but now logs a loud WARN on JDK 21+ where the fallback is unsafe under virtual threads. Silent on JDK < 21 (expected). - thread: unchanged (legacy, silent). Default stays graceful because at init we only know the JVM is Loom-capable, not whether virtual threads route context here; failing hard by default would break profiler startup for every JDK 21+ deployment until the agent-side export grant lands. The tracer opts into mode=carrier once it guarantees the export. Tests: carrierModeThrowsWhenUnavailable (fail-fast) and carrierModeUsesCarrierWhenAvailable replace the previous graceful-fallback-on-carrier test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(context): address PR review — remove motivation-only cpp test, harden test & parsing - Remove ddprof-lib/src/test/cpp/context_uaf_ut.cpp: it proved the native write-after-free *primitive*, not this PR's Java carrier-scoping fix (its red->green flip referenced a hypothetical native guard). The fix mechanism is validated by CarrierContextStorageTest; a deterministic end-to-end UAF test isn't feasible, so the danger stays documented in the PR/commit rationale. - CarrierContextStorageTest: fix the stale "two touches with a park" comment to match the single-touch reality, and relax the per-carrier assertion from ==1 to >=1. The map is keyed by carrier *name*, so a retired-and-replaced ForkJoinPool worker reusing a slot name could legitimately expose a second context; the did-NOT-key-by-vthread guarantee is now carried by the aggregate bound (context count well below vthread count), which is name-stability-free. - OtelContextStorage: parse the mode property with toLowerCase(Locale.ROOT) so "CARRIER" is not mangled to "carrıer" under tr_TR (which would silently drop the fail-fast semantics). - JavaProfiler.getThreadContext(): document that the returned ThreadContext must not be cached across a possible carrier migration (unmount/remount) — the buffer targets the carrier mounted at call time; setContext* re-fetch per use. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(build): gate carrier --add-exports on the test JVM at execution time The musl split-JDK CI (build JDK 21 via JAVA_HOME, test JDK 8 via JAVA_TEST_HOME) failed with "Unrecognized option: --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED" on the JDK 8 test JVM. Root cause: the flag was gated on PlatformUtils.testJvmMajorVersion() inside ProfilerTestExtension.init{}, i.e. at Gradle *configuration* time. JAVA_TEST_HOME is not resolvable then (that is precisely why the task executables are assigned in doFirst, "so environment variables are read correctly"), so testJavaHome() fell back to the build JDK (JAVA_HOME=21) and added a JDK-21-only flag that then aborted the JDK-8 launcher. Move the gate to task execution time: a carrierExportJvmArgs() helper evaluated in both test doFirst blocks (glibc Test task and musl Exec task), where the real test JVM (JAVA_TEST_HOME) is resolvable. When it is < 21 the flag is omitted and the profiler degrades to thread-scoped storage (carrier tests skip) — safe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(build): detect test-JVM version from the release file, not by exec The previous execution-time gate still put --add-exports on the JDK 8 musl test JVM: PlatformUtils.testJvmMajorVersion() launches `$JAVA_TEST_HOME/bin/java -version`, and in the split-JDK musl matrix that probe reported >= 21 even though the resolved executable (and the JVM that then rejected the flag) was JDK 8. Read the major version from the test JDK's `release` file (JAVA_VERSION=...) instead of executing the launcher: a pure file read of the same JAVA_TEST_HOME the executable is resolved from — deterministic, no subprocess, no exec-format/PATH hazards. Returns 0 when it can't be determined (missing/old `release`), so the flag is omitted and carrier tests skip — fail-safe, never an abort. Adds a lifecycle log of the resolved testJavaHome + detected major to make the gate decision visible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(build): lower carrier --add-exports gate log from lifecycle to info The gate decision was logged at lifecycle level during stabilization to make the musl/ibm CI behavior visible. The fix is proven (musl-8 green, ibm-8 flaky and green on rerun), so drop it to info to avoid printing on every test invocation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(context): address review — typed ContextStorageMode, ddprof.debug.* property, tidy create() Addresses Jaroslav's review on PR #625: - contextStorageMode() returns a typed ContextStorageMode (new public enum) instead of a String; the OtelContextStorage factory stays package-private. - Rename the selector to ddprof.debug.context.storage.mode (ddprof.debug.*, like ddprof.debug.malloc_arena_max) to signal it is an internal knob, not supported config. - create() parses the selector once into locals and documents that it runs once per JavaProfiler instance (the tlsContextStorage field initializer), not per thread — clarifying the review's per-thread concern. Preserves the prior commits' work (Locale.ROOT parsing, getThreadContext caching note, build-side --add-exports gating, removed motivation-only cpp test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Jaroslav Bachorik <jaroslav.bachorik@datadoghq.com>
1 parent 934f110 commit 94686da

6 files changed

Lines changed: 564 additions & 14 deletions

File tree

build-logic/conventions/src/main/kotlin/com/datadoghq/profiler/ProfilerTestPlugin.kt

Lines changed: 67 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import org.gradle.api.provider.Property
1515
import org.gradle.api.tasks.Exec
1616
import org.gradle.api.tasks.SourceSetContainer
1717
import org.gradle.api.tasks.testing.Test
18+
import java.io.File
1819
import java.time.Duration
1920
import javax.inject.Inject
2021

@@ -76,6 +77,61 @@ import javax.inject.Inject
7677
* ```
7778
*/
7879
class ProfilerTestPlugin : Plugin<Project> {
80+
81+
/**
82+
* Major version of the *test* JVM, read from its `release` file (`JAVA_VERSION="..."`) rather
83+
* than by executing the launcher.
84+
*
85+
* Executing `$JAVA_TEST_HOME/bin/java -version` (PlatformUtils.testJvmMajorVersion()) is
86+
* unreliable here: in the musl split-JDK matrix it has been observed to report the build JDK
87+
* (21) even when the test JVM is JDK 8, which put a JDK-21-only `--add-exports` onto a JDK-8
88+
* launcher and aborted it. Reading the `release` file is a pure file read of the same
89+
* JAVA_TEST_HOME the executable is resolved from — deterministic, no subprocess, no exec-format
90+
* or PATH hazards. Returns 0 when it cannot be determined (missing/old `release`), so callers
91+
* fail safe: they omit the flag, the profiler degrades to thread-scoped storage, and the
92+
* carrier-scoping tests skip — never an abort.
93+
*/
94+
private fun testJvmMajorVersionFromRelease(): Int = try {
95+
val release = File(PlatformUtils.testJavaHome(), "release")
96+
val version = release.takeIf { it.isFile }
97+
?.readLines()
98+
?.firstOrNull { it.startsWith("JAVA_VERSION=") }
99+
?.substringAfter('=')?.trim()?.trim('"')
100+
// "1.8.0_452" -> 8 ; "21.0.5" -> 21
101+
val parts = version?.split('.').orEmpty()
102+
val majorToken = when {
103+
parts.isEmpty() -> ""
104+
parts[0] == "1" && parts.size > 1 -> parts[1]
105+
else -> parts[0]
106+
}
107+
majorToken.takeWhile { it.isDigit() }.toIntOrNull() ?: 0
108+
} catch (e: Exception) {
109+
0
110+
}
111+
112+
/**
113+
* JVM args required to enable carrier-scoped OTEL context storage
114+
* (`OtelContextStorage.Mode.CARRIER`), or an empty list when the test JVM does not support it.
115+
*
116+
* Carrier scoping resolves `jdk.internal.misc.CarrierThreadLocal`, which lives in a
117+
* non-exported package, so it needs `--add-exports java.base/jdk.internal.misc=ALL-UNNAMED`.
118+
* That type only exists on JDK 21+, and the flag *aborts* a Java 8 JVM ("Unrecognized option"),
119+
* so it is gated on the version of the actual test JVM.
120+
*
121+
* MUST be evaluated at task execution time (inside doFirst), not configuration time: the test
122+
* JVM is selected via JAVA_TEST_HOME, which the CI only makes resolvable at execution time (see
123+
* the `executable` assignments below).
124+
*/
125+
private fun carrierExportJvmArgs(project: Project): List<String> {
126+
val major = testJvmMajorVersionFromRelease()
127+
val enabled = major >= 21
128+
project.logger.info(
129+
"ddprof: carrier --add-exports gate — testJavaHome={}, detected major={}, flag {}",
130+
PlatformUtils.testJavaHome(), major, if (enabled) "ADDED" else "omitted"
131+
)
132+
return if (enabled) listOf("--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED") else emptyList()
133+
}
134+
79135
override fun apply(project: Project) {
80136
val extension = project.extensions.create(
81137
"profilerTest",
@@ -238,6 +294,8 @@ class ProfilerTestPlugin : Plugin<Project> {
238294
testTask.doFirst {
239295
val allArgs = mutableListOf<String>()
240296
allArgs.addAll(testConfig.standardJvmArgs)
297+
// Version-gated at execution time, when the real test JVM is resolvable.
298+
allArgs.addAll(carrierExportJvmArgs(project))
241299

242300
if (extension.nativeLibDir.isPresent) {
243301
allArgs.add("-Djava.library.path=${extension.nativeLibDir.get().asFile.absolutePath}")
@@ -302,6 +360,8 @@ class ProfilerTestPlugin : Plugin<Project> {
302360

303361
// JVM args
304362
allArgs.addAll(testConfig.standardJvmArgs)
363+
// Version-gated at execution time, when the real test JVM (JAVA_TEST_HOME) is resolvable.
364+
allArgs.addAll(carrierExportJvmArgs(project))
305365
if (extension.nativeLibDir.isPresent) {
306366
allArgs.add("-Djava.library.path=${extension.nativeLibDir.get().asFile.absolutePath}")
307367
}
@@ -661,7 +721,13 @@ abstract class ProfilerTestExtension @Inject constructor(
661721
abstract val applicationMainClass: Property<String>
662722

663723
init {
664-
// Standard JVM arguments for profiler testing
724+
// Standard JVM arguments for profiler testing.
725+
// NOTE: JDK-version-gated flags (e.g. the carrier-scoping --add-exports) must NOT be
726+
// added here. This convention is computed at configuration time, where JAVA_TEST_HOME
727+
// is not yet resolvable and PlatformUtils.testJavaHome() falls back to the *build* JDK
728+
// (JAVA_HOME) — which misdetects in the musl split-JDK CI (build JDK 21, test JDK 8) and
729+
// would emit a JDK-21 flag onto a JDK-8 test JVM. Version-gated flags are added at
730+
// execution time in the task doFirst blocks instead (see ProfilerTestPlugin).
665731
standardJvmArgs.convention(listOf(
666732
"-Djdk.attach.allowAttachSelf", // Allow profiler to attach to self
667733
"-Djol.tryWithSudo=true", // JOL memory layout analysis
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
/*
2+
* Copyright 2026, Datadog, Inc
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
package com.datadoghq.profiler;
17+
18+
/**
19+
* Scope of the OTEL context {@link ThreadContext} storage actually in effect, as reported by
20+
* {@link JavaProfiler#contextStorageMode()}. See {@link OtelContextStorage} for how it is
21+
* selected.
22+
*/
23+
public enum ContextStorageMode {
24+
/** Carrier-scoped via {@code jdk.internal.misc.CarrierThreadLocal} (JDK 21+). */
25+
CARRIER,
26+
/** Legacy virtual-thread-scoped plain {@link ThreadLocal}. */
27+
THREAD
28+
}

ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java

Lines changed: 58 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,31 @@ static final class TSCFrequencyHolder {
3939
}
4040
private static JavaProfiler instance;
4141

42-
// Thread-local storage for profiling context
43-
private final ThreadLocal<ThreadContext> tlsContextStorage = ThreadLocal.withInitial(JavaProfiler::initializeThreadContext);
42+
// Storage for profiling context. Scoped to the carrier thread when available so a
43+
// mounted virtual thread resolves to its current carrier's OTEP record (the record the
44+
// sampler reads); falls back to plain thread-local storage otherwise. See
45+
// OtelContextStorage for the mode selection and the rationale.
46+
private final ThreadLocal<ThreadContext> tlsContextStorage = OtelContextStorage.create();
47+
48+
/**
49+
* Returns the calling thread's (or, in carrier mode, its current carrier's)
50+
* {@link ThreadContext}, creating and caching it on first use. Replaces the previous
51+
* {@code ThreadLocal.withInitial(...)} supplier: a carrier-scoped storage instance is
52+
* built reflectively and cannot carry a supplier, so lazy initialization is done here.
53+
*
54+
* <p>Race-free without synchronization: a carrier runs at most one mounted virtual
55+
* thread at a time and this method has no blocking point, so no unmount can occur
56+
* mid-call. A redundant re-init could at worst produce a second {@link ThreadContext}
57+
* over the same carrier record, which is harmless.
58+
*/
59+
private ThreadContext currentContext() {
60+
ThreadContext ctx = tlsContextStorage.get();
61+
if (ctx == null) {
62+
ctx = initializeThreadContext();
63+
tlsContextStorage.set(ctx);
64+
}
65+
return ctx;
66+
}
4467

4568
private JavaProfiler() {
4669
}
@@ -191,7 +214,7 @@ public void removeThread() {
191214
*/
192215
@Deprecated
193216
public void setContext(long spanId, long rootSpanId) {
194-
tlsContextStorage.get().put(spanId, rootSpanId);
217+
currentContext().put(spanId, rootSpanId);
195218
}
196219

197220
/**
@@ -203,15 +226,15 @@ public void setContext(long spanId, long rootSpanId) {
203226
* @param traceIdLow Lower 64 bits of the 128-bit trace ID
204227
*/
205228
public void setContext(long localRootSpanId, long spanId, long traceIdHigh, long traceIdLow) {
206-
tlsContextStorage.get().put(localRootSpanId, spanId, traceIdHigh, traceIdLow);
229+
currentContext().put(localRootSpanId, spanId, traceIdHigh, traceIdLow);
207230
}
208231

209232
/**
210233
* Resets the current thread's context to zero (traceId=0, spanId=0, localRootSpanId=0).
211234
* Custom context attributes are also cleared.
212235
*/
213236
public void clearContext() {
214-
tlsContextStorage.get().put(0, 0, 0, 0);
237+
currentContext().put(0, 0, 0, 0);
215238
}
216239

217240
/**
@@ -226,7 +249,7 @@ public void clearContext() {
226249
* for this slot
227250
*/
228251
public boolean setContextAttribute(int offset, String value) {
229-
return tlsContextStorage.get().setContextAttribute(offset, value);
252+
return currentContext().setContextAttribute(offset, value);
230253
}
231254

232255
/**
@@ -236,7 +259,7 @@ public boolean setContextAttribute(int offset, String value) {
236259
* @param offset slot index (0-based, in [0, 9]); out-of-range values are silently ignored
237260
*/
238261
public void clearContextAttribute(int offset) {
239-
tlsContextStorage.get().clearContextAttribute(offset);
262+
currentContext().clearContextAttribute(offset);
240263
}
241264

242265
/**
@@ -263,11 +286,11 @@ public void clearContextAttribute(int offset) {
263286
* or any active {@code utf8[i]} exceeds 255 bytes
264287
*/
265288
public boolean setContextAttributesByIdAndBytes(int[] constantIds, byte[][] utf8) {
266-
return tlsContextStorage.get().setContextAttributesByIdAndBytes(constantIds, utf8);
289+
return currentContext().setContextAttributesByIdAndBytes(constantIds, utf8);
267290
}
268291

269292
void copyTags(int[] snapshot) {
270-
tlsContextStorage.get().copyCustoms(snapshot);
293+
currentContext().copyCustoms(snapshot);
271294
}
272295

273296
/**
@@ -444,8 +467,29 @@ private static ThreadContext initializeThreadContext() {
444467
*/
445468
private static native ByteBuffer initializeContextTLS0(long[] metadata);
446469

470+
/**
471+
* Returns the {@link ThreadContext} for the current storage slot (the calling thread, or in
472+
* {@link ContextStorageMode#CARRIER} its current carrier).
473+
*
474+
* <p><b>Do not cache the returned instance across a point where the calling thread may be
475+
* unmounted and remounted on a different carrier</b> (any blocking operation on a virtual
476+
* thread). In carrier mode the returned context's buffer targets the carrier that was mounted
477+
* at call time; after migration it no longer corresponds to the current carrier's record — the
478+
* sampler reads the new carrier, and once the old carrier's OS thread exits the buffer dangles.
479+
* Callers that write context (span/attributes) should re-fetch per use — the {@code setContext*}
480+
* methods already do this internally via {@code currentContext()}.
481+
*/
447482
public ThreadContext getThreadContext() {
448-
return tlsContextStorage.get();
483+
return currentContext();
484+
}
485+
486+
/**
487+
* Diagnostics/tests: the resolved OTEL context storage mode, as selected by
488+
* {@code -D}{@value OtelContextStorage#MODE_PROPERTY} and the availability of
489+
* {@code jdk.internal.misc.CarrierThreadLocal}.
490+
*/
491+
public ContextStorageMode contextStorageMode() {
492+
return OtelContextStorage.modeOf(tlsContextStorage);
449493
}
450494

451495
// --- test and debug utility methods
@@ -459,9 +503,10 @@ public ThreadContext getThreadContext() {
459503
public static native void dumpContext();
460504

461505
/**
462-
* Resets the cached ThreadContext for the current thread.
463-
* The next call to {@link #getThreadContext()} or any {@code setContext} overload
464-
* will re-create it with fresh OTEL TLS buffers.
506+
* Resets the cached ThreadContext for the current storage slot — the calling thread in
507+
* {@link ContextStorageMode#THREAD}, or its current carrier in
508+
* {@link ContextStorageMode#CARRIER}. The next call to {@link #getThreadContext()}
509+
* or any {@code setContext} overload will re-create it with fresh OTEL TLS buffers.
465510
*/
466511
public void resetThreadContext() {
467512
tlsContextStorage.remove();

0 commit comments

Comments
 (0)