JVM Crash (SIGSEGV) in UnsafeStringEncoder.isLatin1 on JDK 21 (Agent v2.26.1 ignores disable-unsafe flags)

[Raushan-Sapiens]

Describe the bug

The JVM experiences a fatal crash (SIGSEGV) at Unsafe_GetByte when the OpenTelemetry Java agent attempts to export spans via the BatchSpanProcessor. The crash occurs within io.opentelemetry.exporter.internal.marshal.UnsafeStringEncoder.isLatin1.

Additionally, attempting to work around the issue by disabling Unsafe optimizations via system properties (-Dotel.javaagent.experimental.disable-unsafe=true and -Dio.opentelemetry.exporter.internal.marshal.UnsafeStringEncoder.enabled=false) fails. The agent appears to ignore these flags for this specific OTLP attribute marshalling execution path and continues to call sun.misc.Unsafe.getByte, resulting in the same JVM crash.

Steps to reproduce

Run WildFly v34 using OpenTelemetry Java Agent v2.26.1 on JDK 21.

Configure the agent to export via OTLP (-Dotel.metrics.exporter=otlp, -Dotel.logs.exporter=otlp, etc.).

Generate application load to trigger span batch exports.

The JVM crashes during BatchSpanProcessor$Worker.exportCurrentBatch(). Added the following flags to the JVM startup script, but the crash still occurred in the exact same frame:
-Dotel.javaagent.experimental.disable-unsafe=true -Dio.opentelemetry.exporter.internal.marshal.UnsafeStringEncoder.enabled=false

Expected behavior

The agent should safely marshal strings without causing memory segmentation faults. Furthermore, if the disable-unsafe flags are explicitly provided, the agent should strictly fall back to standard String.getBytes() APIs and completely avoid sun.misc.Unsafe calls.

Actual behavior

The JVM terminates abruptly with a SIGSEGV (memory segmentation fault) in libjvm.so at the Unsafe_GetByte frame. This crash occurs on the BatchSpanProcessor_WorkerThread-1 thread while the agent is marshalling OTLP attributes via io.opentelemetry.exporter.internal.marshal.UnsafeStringEncoder.isLatin1.

Crucially, the agent ignores explicit configuration flags intended to disable Unsafe string optimizations (-Dotel.javaagent.experimental.disable-unsafe=true and -Dio.opentelemetry.exporter.internal.marshal.UnsafeStringEncoder.enabled=false). Despite these flags being successfully parsed by the JVM on startup, the agent still executes the Unsafe.getByte() path, failing to fall back to safe string encoding and resulting in the exact same fatal crash

Javaagent or library instrumentation version

v2.26.1

Environment

Java Agent Version: 2.26.1

OS: RHEL 9.7 (running in WSL2)

JDK: OpenJDK 64-Bit Server VM Corretto-21.0.10.7.1 (21.0.10+7-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)

Application Server: WildFly 34.0.1.Final

Additional context

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007fe234510bde, pid=280, tid=298

Problematic frame:

V [libjvm.so+0xfc8bde] Unsafe_GetByte+0x8e

Current thread (0x00005561c813a540): JavaThread "BatchSpanProcessor_WorkerThread-1" daemon [_thread_in_vm, id=298, stack(0x00007fe1c7ae8000,0x00007fe1c7be8000) (1024K)]

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xfc8bde] Unsafe_GetByte+0x8e
J 243 jdk.internal.misc.Unsafe.getByte(Ljava/lang/Object;J)B java.base@21.0.10 (0 bytes) @ 0x00007fe2230703f7 [0x00007fe223070320+0x00000000000000d7]
j sun.misc.Unsafe.getByte(Ljava/lang/Object;J)B+5 jdk.unsupported@21.0.10
j io.opentelemetry.exporter.internal.marshal.UnsafeStringEncoder.isLatin1(Ljava/lang/String;)Z+8
j io.opentelemetry.exporter.internal.marshal.AbstractStringEncoder.getUtf8Size(Ljava/lang/String;)I+2
j io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil.getUtf8Size(Ljava/lang/String;Lio/opentelemetry/exporter/internal/marshal/MarshalerContext;)I+12
J 59686 c1 io.opentelemetry.exporter.internal.otlp.StringAnyValueStatelessMarshaler.getBinarySerializedSize(Ljava/lang/Object;Lio/opentelemetry/exporter/internal/marshal/MarshalerContext;)I (10 bytes) @ 0x00007fe21c840884 [0x00007fe21c8406e0+0x00000000000001a4]
J 49171 c2 io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil.sizeRepeatedMessageWithContext(Lio/opentelemetry/exporter/internal/marshal/ProtoFieldInfo;Ljava/util/List;Lio/opentelemetry/exporter/internal/marshal/StatelessMarshaler;Lio/opentelemetry/exporter/internal/marshal/MarshalerContext;)I (94 bytes) @ 0x00007fe224e1e2a4 [0x00007fe224e1dec0+0x00000000000003e4]
J 61360 c1 io.opentelemetry.exporter.internal.otlp.AttributeArrayAnyValueStatelessMarshaler.getBinarySerializedSize(Lio/opentelemetry/javaagent/shaded/io/opentelemetry/api/common/AttributeType;Ljava/util/List;Lio/opentelemetry/exporter/internal/marshal/MarshalerContext;)I (98 bytes) @ 0x00007fe21d977124 [0x00007fe21d976fa0+0x0000000000000184]
J 61359 c1 io.opentelemetry.exporter.internal.otlp.AttributeArrayAnyValueStatelessMarshaler.getBinarySerializedSize(Ljava/lang/Object;Ljava/lang/Object;Lio/opentelemetry/exporter/internal/marshal/MarshalerContext;)I (14 bytes) @ 0x00007fe21d03e624 [0x00007fe21d03e540+0x00000000000000e4]
J 39678 c1 io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil.sizeMessageWithContext(Lio/opentelemetry/exporter/internal/marshal/ProtoFieldInfo;Ljava/lang/Object;Ljava/lang/Object;Lio/opentelemetry/exporter/internal/marshal/StatelessMarshaler2;Lio/opentelemetry/exporter/internal/marshal/MarshalerContext;)I (46 bytes) @ 0x00007fe21c0e3514 [0x00007fe21c0e3380+0x0000000000000194]
J 60909 c2 io.opentelemetry.exporter.internal.otlp.AttributeKeyValueStatelessMarshaler$ValueStatelessMarshaler.getBinarySerializedSize(Lio/opentelemetry/javaagent/shaded/io/opentelemetry/api/common/AttributeKey;Ljava/lang/Object;Lio/opentelemetry/exporter/internal/marshal/MarshalerContext;)I (155 bytes) @ 0x00007fe22538149c [0x00007fe225381220+0x000000000000027c]
...
J 53197 c2 io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.exportCurrentBatch()V (194 bytes) @ 0x00007fe2237a8234 [0x00007fe2237a7c40+0x00000000000005f4]
j io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.run()V+68
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5 java.base@21.0.10
j java.lang.Thread.run()V+19 java.base@21.0.10

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[laurit]

Additionally, attempting to work around the issue by disabling Unsafe optimizations via system properties (-Dotel.javaagent.experimental.disable-unsafe=true and -Dio.opentelemetry.exporter.internal.marshal.UnsafeStringEncoder.enabled=false) fails.

-Dotel.javaagent.experimental.disable-unsafe=true is intended for something different, it isn't supposed to work for this.
-Dio.opentelemetry.exporter.internal.marshal.UnsafeStringEncoder.enabled=false as far as I can tell this flag does not exist.

What is the address it is accessing when it crashes? e.g siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x0000000000000010
Is the address close to zero like in the sample above? If so there probably is an attribute with null value which is unusual since as far as I remember the default attributes implementation doesn't record null values. Just out of curiosity by any chance do you know where this null value comes from?

[Raushan-Sapiens]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000010 Because String.value or String.coder offsets in JDK 21 are right around 16 (0x10), this perfectly aligns with Unsafe.getByte(obj, offset) being called where obj is null.

As for where the null value is coming from, that is the tricky part. This is running a massive, highly complex platform on WildFly 34. While the default attribute implementation might not record nulls directly, my suspicion is that one of the auto-instrumentation modules is scooping up a null from the application context.

A few theories on where it might be originating:

MDC / Log Context: We heavily utilize logging context. If a key is initialized in the MDC but the value is explicitly set to null, the logging instrumentation might be picking it up and passing it to the span.

HTTP Headers / Parameters: A Jakarta EE / Servlet instrumentation might be capturing a specific header or query parameter that is present but evaluates to a null string.

JDBC / SQL: A captured database bind parameter that is explicitly null.

Given that Unsafe_GetByte instantly kills the JVM instead of throwing a catchable NullPointerException, it's incredibly difficult to trace exactly which span or attribute key is the culprit.

Is there a recommended way to defensively patch UnsafeStringEncoder.isLatin1 in the agent to handle a null java.lang.String reference gracefully, or is the fix to trace back up the stack to AttributeKeyValueStatelessMarshaler and ensure nulls are stripped before they reach the encoder?

[laurit]

A few theories on where it might be originating:

These shouldn't create null values since they use the default attributes implementation. Just curious where the null attribute comes from since we have had this code for a while and you are the first one to trigger the issue.

Given that Unsafe_GetByte instantly kills the JVM instead of throwing a catchable NullPointerException, it's incredibly difficult to trace exactly which span or attribute key is the culprit.

If you can reproduce it at will you could set a conditional breakpoint.

Is there a recommended way to defensively patch UnsafeStringEncoder.isLatin1 in the agent to handle a null java.lang.String reference gracefully, or is the fix to trace back up the stack to AttributeKeyValueStatelessMarshaler and ensure nulls are stripped before they reach the encoder?

I'll create a PR to address this. Meanwhile you can probably work around this by setting otel.java.exporter.memory_mode= immutable_data

[laurit]

Hopefully fixed with open-telemetry/opentelemetry-java#8312 Fix should be available in 2.28.0 that is due in about a month.