Skip to content

Commit bf241bf

Browse files
trasklaurit
trask
and
laurit
authored
Split runtime telemetry JFR config from experimental metrics (#18110)
Co-authored-by: Lauri Tulmin <ltulmin@splunk.com>
1 parent 64d55bf commit bf241bf

18 files changed

Lines changed: 243 additions & 97 deletions

File tree

‎instrumentation/runtime-telemetry/README.md‎

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,20 @@
11
# Settings for the Runtime Telemetry instrumentation
22

3-
| System property | Type | Default | Description |
4-
| ------------------------------------------------------------------------------------- | ------- | ------- | --------------------------------------------------------------------------------------- |
5-
| `otel.instrumentation.runtime-telemetry.emit-experimental-metrics` | Boolean | `false` | Enable the capture of experimental metrics. |
6-
| `otel.instrumentation.runtime-telemetry.experimental.prefer-jfr` | Boolean | `false` | Prefer JFR over JMX for metrics where both collection methods are available (Java 17+). |
7-
| `otel.instrumentation.runtime-telemetry.experimental.package-emitter.enabled` | Boolean | `false` | Enable creating events for JAR libraries used by the application. |
8-
| `otel.instrumentation.runtime-telemetry.experimental.package-emitter.jars-per-second` | Integer | 10 | The number of JAR files processed per second. |
3+
| System property | Type | Default | Description |
4+
| ------------------------------------------------------------------------------------- | ------- | ------- | -------------------------------------------------------------------------------- |
5+
| `otel.instrumentation.runtime-telemetry.emit-experimental-metrics` | Boolean | `false` | Enable the capture of experimental JMX-based metrics. |
6+
| `otel.instrumentation.runtime-telemetry.emit-experimental-jfr-metrics` | Boolean | `false` | Enable the capture of experimental JFR-based metrics on Java 17+. |
7+
| `otel.instrumentation.runtime-telemetry.experimental.prefer-jfr` | Boolean | `false` | Prefer JFR over JMX for metrics available from both sources, on Java 17+. |
8+
| `otel.instrumentation.runtime-telemetry.experimental.package-emitter.enabled` | Boolean | `false` | Enable creating events for JAR libraries used by the application. |
9+
| `otel.instrumentation.runtime-telemetry.experimental.package-emitter.jars-per-second` | Integer | 10 | The number of JAR files processed per second. |
910

1011
## Deprecated Properties (to be removed in 3.0)
1112

12-
| System property | Type | Default | Description |
13-
| ------------------------------------------------------------------------ | ------- | ------- | --------------------------------------------------------------------------------- |
14-
| `otel.instrumentation.runtime-telemetry.capture-gc-cause` | Boolean | `false` | Enable the capture of the jvm.gc.cause attribute. Will always be captured in 3.0. |
15-
| `otel.instrumentation.runtime-telemetry.emit-experimental-telemetry` | Boolean | `false` | Use `emit-experimental-metrics` instead. |
16-
| `otel.instrumentation.runtime-telemetry.package-emitter.enabled` | Boolean | `false` | Use `experimental.package-emitter.enabled` instead. |
17-
| `otel.instrumentation.runtime-telemetry.package-emitter.jars-per-second` | Integer | 10 | Use `experimental.package-emitter.jars-per-second` instead. |
18-
| `otel.instrumentation.runtime-telemetry-java17.enabled` | Boolean | `false` | Deprecated. Use `emit-experimental-metrics` for experimental JFR features. |
19-
| `otel.instrumentation.runtime-telemetry-java17.enable-all` | Boolean | `false` | Deprecated. Use `emit-experimental-metrics` and `experimental.prefer-jfr`. |
13+
| System property | Type | Default | Description |
14+
| ------------------------------------------------------------------------ | ------- | ------- | ------------------------------------------------------------------------------------------------------------ |
15+
| `otel.instrumentation.runtime-telemetry.capture-gc-cause` | Boolean | `false` | Enable the capture of the jvm.gc.cause attribute. Will always be captured in 3.0. |
16+
| `otel.instrumentation.runtime-telemetry.emit-experimental-telemetry` | Boolean | `false` | Use `emit-experimental-metrics` instead. |
17+
| `otel.instrumentation.runtime-telemetry.package-emitter.enabled` | Boolean | `false` | Use `experimental.package-emitter.enabled` instead. |
18+
| `otel.instrumentation.runtime-telemetry.package-emitter.jars-per-second` | Integer | 10 | Use `experimental.package-emitter.jars-per-second` instead. |
19+
| `otel.instrumentation.runtime-telemetry-java17.enabled` | Boolean | `false` | Deprecated. Use `emit-experimental-jfr-metrics` instead. |
20+
| `otel.instrumentation.runtime-telemetry-java17.enable-all` | Boolean | `false` | Deprecated. Use `experimental.prefer-jfr` and `emit-experimental-jfr-metrics` instead. |

‎instrumentation/runtime-telemetry/library/README.md‎

Lines changed: 48 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,28 @@ meter_provider:
8787

8888
## Metrics
8989

90+
### Configuration
91+
92+
The set of emitted metrics depends on three independent knobs. In autoconfigured
93+
environments, these map to system properties; programmatically, use the corresponding
94+
builder methods.
95+
96+
- `otel.instrumentation.runtime-telemetry.emit-experimental-metrics=true`
97+
(`emitExperimentalMetrics()`): enables additional JMX-based metrics that are not yet
98+
stable in the semantic conventions.
99+
- `otel.instrumentation.runtime-telemetry.emit-experimental-jfr-metrics=true` (Java 17+):
100+
enables additional JFR-based metrics that are not yet stable in the semantic conventions.
101+
- `otel.instrumentation.runtime-telemetry.experimental.prefer-jfr=true` (Java 17+): sources
102+
metrics from JFR instead of JMX wherever a JFR equivalent exists (see
103+
[JFR-based (Overlap with JMX)](#jfr-based-overlap-with-jmx) below). The corresponding
104+
JMX metrics are suppressed.
105+
106+
> **Warning**: JFR events might not be available for all JVMs or with a GraalVM native
107+
> image, therefore limiting the produced metrics. The original implementation was done
108+
> for Hotspot. OpenJ9 currently (Nov. 2025) only has the VM-level JFR implementation. So
109+
> events emitted at the Java level (ie. in jdk.jfr) will not be present. Meaning,
110+
> jdk.SocketRead, jdk.SocketWrite won't work.
111+
90112
### Stable Metrics (enabled by default)
91113

92114
These metrics are collected via JMX on all Java versions:
@@ -107,9 +129,7 @@ These metrics are collected via JMX on all Java versions:
107129

108130
### Experimental Metrics
109131

110-
These metrics are enabled with `emitExperimentalMetrics()`:
111-
112-
**JMX-based (all Java versions):**
132+
#### JMX-based (all Java versions)
113133

114134
| Metric | Description |
115135
| -------- | ----------- |
@@ -119,12 +139,7 @@ These metrics are enabled with `emitExperimentalMetrics()`:
119139
| `jvm.memory.init` | Measure of initial memory requested |
120140
| `jvm.system.cpu.utilization` | System-wide CPU utilization |
121141

122-
**JFR-based (Java 17+ only):**
123-
124-
> **Warning**: JFR events might not be available for all JVMs or with a GraalVM native image,
125-
> therefore limiting the produced metrics. The original implementation was done for Hotspot. OpenJ9
126-
> currently (Nov. 2025) only has the VM-level JFR implementation. So events emitted at the Java
127-
> level (ie. in jdk.jfr) will not be present. Meaning, jdk.SocketRead, jdk.SocketWrite won't work.
142+
#### JFR-based (Java 17+ only)
128143

129144
| Metric | Description |
130145
| -------- | ----------- |
@@ -134,6 +149,30 @@ These metrics are enabled with `emitExperimentalMetrics()`:
134149
| `jvm.network.io` | Network I/O bytes |
135150
| `jvm.network.time` | Network I/O time |
136151

152+
#### JFR-based (Overlap with JMX)
153+
154+
When `experimental.prefer-jfr=true`, the following metrics are sourced from JFR instead of JMX
155+
(the JMX-based registration is suppressed to avoid duplicates):
156+
157+
| Metric |
158+
| -------- |
159+
| `jvm.buffer.count` |
160+
| `jvm.buffer.memory.limit` |
161+
| `jvm.buffer.memory.used` |
162+
| `jvm.class.count` |
163+
| `jvm.class.loaded` |
164+
| `jvm.class.unloaded` |
165+
| `jvm.cpu.count` |
166+
| `jvm.cpu.recent_utilization` |
167+
| `jvm.gc.duration` |
168+
| `jvm.memory.committed` |
169+
| `jvm.memory.init` |
170+
| `jvm.memory.limit` |
171+
| `jvm.memory.used` |
172+
| `jvm.memory.used_after_last_gc` |
173+
| `jvm.system.cpu.utilization` |
174+
| `jvm.thread.count` |
175+
137176
## Garbage Collector Dependent Metrics
138177

139178
The attributes reported on the memory metrics (`jvm.memory.*`) and gc metrics (`jvm.gc.*`) are dependent on the garbage collector used by the application, since each garbage collector organizes memory pools differently and has different strategies for reclaiming memory during garbage collection.

‎instrumentation/runtime-telemetry/library/src/main/java/io/opentelemetry/instrumentation/runtimetelemetry/RuntimeTelemetryBuilder.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,9 @@ public final class RuntimeTelemetryBuilder {
3737

3838
static {
3939
Experimental.internalSetEmitExperimentalMetrics(
40+
(builder, emit) -> builder.emitExperimentalMetrics = emit);
41+
Experimental.internalSetEmitExperimentalJfrMetrics(
4042
(builder, emit) -> {
41-
builder.emitExperimentalMetrics = emit;
4243
if (emit) {
4344
builder.jfrConfig.enableExperimentalFeatures();
4445
}

‎instrumentation/runtime-telemetry/library/src/main/java/io/opentelemetry/instrumentation/runtimetelemetry/internal/Cpu.java‎

Lines changed: 1 addition & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@
1010
import io.opentelemetry.api.metrics.Meter;
1111
import java.util.ArrayList;
1212
import java.util.List;
13-
import java.util.function.IntSupplier;
1413
import java.util.function.Supplier;
1514
import javax.annotation.Nullable;
1615

@@ -33,16 +32,12 @@ public class Cpu {
3332
/** Register observers for java runtime CPU metrics. */
3433
public static List<AutoCloseable> registerObservers(Meter meter) {
3534
return INSTANCE.registerObservers(
36-
meter,
37-
Runtime.getRuntime()::availableProcessors,
38-
CpuMethods.processCpuTime(),
39-
CpuMethods.processCpuUtilization());
35+
meter, CpuMethods.processCpuTime(), CpuMethods.processCpuUtilization());
4036
}
4137

4238
// Visible for testing
4339
List<AutoCloseable> registerObservers(
4440
Meter meter,
45-
IntSupplier availableProcessors,
4641
@Nullable Supplier<Long> processCpuTime,
4742
@Nullable Supplier<Double> processCpuUtilization) {
4843
List<AutoCloseable> observables = new ArrayList<>();
@@ -62,14 +57,6 @@ List<AutoCloseable> registerObservers(
6257
}
6358
}));
6459
}
65-
observables.add(
66-
meter
67-
.upDownCounterBuilder("jvm.cpu.count")
68-
.setDescription("Number of processors available to the Java virtual machine.")
69-
.setUnit("{cpu}")
70-
.buildWithCallback(
71-
observableMeasurement ->
72-
observableMeasurement.record(availableProcessors.getAsInt())));
7360
if (processCpuUtilization != null) {
7461
observables.add(
7562
meter

‎instrumentation/runtime-telemetry/library/src/main/java/io/opentelemetry/instrumentation/runtimetelemetry/internal/CpuCount.java‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
/*
2+
* Copyright The OpenTelemetry Authors
3+
* SPDX-License-Identifier: Apache-2.0
4+
*/
5+
6+
package io.opentelemetry.instrumentation.runtimetelemetry.internal;
7+
8+
import static java.util.Collections.singletonList;
9+
10+
import io.opentelemetry.api.metrics.Meter;
11+
import java.util.List;
12+
import java.util.function.IntSupplier;
13+
14+
/**
15+
* Registers measurements that generate the JVM CPU count metric.
16+
*
17+
* <p>This class is internal and is hence not for public use. Its APIs are unstable and can change
18+
* at any time.
19+
*/
20+
public class CpuCount {
21+
22+
// Visible for testing
23+
static final CpuCount INSTANCE = new CpuCount();
24+
25+
/** Register observers for the JVM CPU count metric. */
26+
public static List<AutoCloseable> registerObservers(Meter meter) {
27+
return INSTANCE.registerObservers(meter, Runtime.getRuntime()::availableProcessors);
28+
}
29+
30+
// Visible for testing
31+
List<AutoCloseable> registerObservers(Meter meter, IntSupplier availableProcessors) {
32+
AutoCloseable observable =
33+
meter
34+
.upDownCounterBuilder("jvm.cpu.count")
35+
.setDescription("Number of processors available to the Java virtual machine.")
36+
.setUnit("{cpu}")
37+
.buildWithCallback(
38+
observableMeasurement ->
39+
observableMeasurement.record(availableProcessors.getAsInt()));
40+
return singletonList(observable);
41+
}
42+
43+
protected CpuCount() {}
44+
}

‎instrumentation/runtime-telemetry/library/src/main/java/io/opentelemetry/instrumentation/runtimetelemetry/internal/Experimental.java‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,17 +19,21 @@ public final class Experimental {
1919
@Nullable
2020
private static volatile BiConsumer<RuntimeTelemetryBuilder, Boolean> setEmitExperimentalMetrics;
2121

22+
@Nullable
23+
private static volatile BiConsumer<RuntimeTelemetryBuilder, Boolean>
24+
setEmitExperimentalJfrMetrics;
25+
2226
@Nullable
2327
private static volatile BiConsumer<RuntimeTelemetryBuilder, Boolean> setPreferJfrMetrics;
2428

2529
/**
26-
* Sets whether experimental metrics should be emitted. Experimental metrics are those not marked
27-
* as stable in the <a
30+
* Sets whether experimental JMX-based metrics should be emitted. Experimental metrics are those
31+
* not marked as stable in the <a
2832
* href="https://github.com/open-telemetry/semantic-conventions/blob/main/docs/runtime/jvm-metrics.md">semantic
2933
* conventions</a>.
3034
*
3135
* @param builder the runtime telemetry builder
32-
* @param emitExperimentalMetrics {@code true} to emit experimental metrics
36+
* @param emitExperimentalMetrics {@code true} to emit experimental JMX metrics
3337
*/
3438
public static void setEmitExperimentalMetrics(
3539
RuntimeTelemetryBuilder builder, boolean emitExperimentalMetrics) {
@@ -43,6 +47,27 @@ public static void internalSetEmitExperimentalMetrics(
4347
Experimental.setEmitExperimentalMetrics = setEmitExperimentalMetrics;
4448
}
4549

50+
/**
51+
* Sets whether experimental JFR-based metrics should be emitted (Java 17+). Experimental metrics
52+
* are those not marked as stable in the <a
53+
* href="https://github.com/open-telemetry/semantic-conventions/blob/main/docs/runtime/jvm-metrics.md">semantic
54+
* conventions</a>.
55+
*
56+
* @param builder the runtime telemetry builder
57+
* @param emitExperimentalJfrMetrics {@code true} to emit experimental JFR metrics
58+
*/
59+
public static void setEmitExperimentalJfrMetrics(
60+
RuntimeTelemetryBuilder builder, boolean emitExperimentalJfrMetrics) {
61+
if (setEmitExperimentalJfrMetrics != null) {
62+
setEmitExperimentalJfrMetrics.accept(builder, emitExperimentalJfrMetrics);
63+
}
64+
}
65+
66+
public static void internalSetEmitExperimentalJfrMetrics(
67+
BiConsumer<RuntimeTelemetryBuilder, Boolean> setEmitExperimentalJfrMetrics) {
68+
Experimental.setEmitExperimentalJfrMetrics = setEmitExperimentalJfrMetrics;
69+
}
70+
4671
/**
4772
* Sets whether to prefer JFR over JMX for metrics where both collection methods are available.
4873
* When set to {@code true}, metrics available from both sources will be collected using JFR. When

‎instrumentation/runtime-telemetry/library/src/main/java/io/opentelemetry/instrumentation/runtimetelemetry/internal/Internal.java‎

Lines changed: 21 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -284,7 +284,7 @@ private static void configureJava17EnableAll(
284284
RuntimeTelemetryBuilder builder, DeclarativeConfigProperties config) {
285285
logger.warning(
286286
"otel.instrumentation.runtime-telemetry-java17.enable-all is deprecated and will be"
287-
+ " removed in 3.0. Use otel.instrumentation.runtime-telemetry.emit-experimental-metrics"
287+
+ " removed in 3.0. Use otel.instrumentation.runtime-telemetry.emit-experimental-jfr-metrics"
288288
+ " and otel.instrumentation.runtime-telemetry.experimental.prefer-jfr instead.");
289289
// For backward compatibility: route JMX metrics to java8 scope, JFR metrics to java17 scope
290290
Internal.setJmxInstrumentationName(builder, "io.opentelemetry.runtime-telemetry-java8");
@@ -311,27 +311,33 @@ private static void configureJava17EnableAll(
311311
private static void configureJava17Enabled(RuntimeTelemetryBuilder builder) {
312312
logger.warning(
313313
"otel.instrumentation.runtime-telemetry-java17.enabled is deprecated and will be"
314-
+ " removed in 3.0. Use otel.instrumentation.runtime-telemetry.emit-experimental-metrics"
315-
+ " for experimental JFR features.");
316-
// Enable default JFR features: context switches, CPU count, locks, allocations, network I/O
317-
Internal.setEnableJfrFeature(builder, "CONTEXT_SWITCH_METRICS");
314+
+ " removed in 3.0. Use"
315+
+ " otel.instrumentation.runtime-telemetry.emit-experimental-jfr-metrics instead.");
316+
enableDefaultJfrFeatures(builder);
317+
// Preserve legacy behavior of the deprecated runtime-telemetry-java17 module, which enabled
318+
// CPU_COUNT_METRICS by default (emitted as jvm.cpu.limit via useLegacyJfrCpuCountMetric).
318319
Internal.setEnableJfrFeature(builder, "CPU_COUNT_METRICS");
319-
Internal.setEnableJfrFeature(builder, "LOCK_METRICS");
320-
Internal.setEnableJfrFeature(builder, "MEMORY_ALLOCATION_METRICS");
321-
Internal.setEnableJfrFeature(builder, "NETWORK_IO_METRICS");
322320
Internal.setUseLegacyJfrCpuCountMetric(builder, true);
323321
// For backward compatibility: OLD java17 module used java8's JMX factory, so JMX -> java8 scope
324322
Internal.setJmxInstrumentationName(builder, "io.opentelemetry.runtime-telemetry-java8");
325323
Internal.setJfrInstrumentationName(builder, "io.opentelemetry.runtime-telemetry-java17");
326324
}
327325

326+
private static void enableDefaultJfrFeatures(RuntimeTelemetryBuilder builder) {
327+
Internal.setEnableJfrFeature(builder, "CONTEXT_SWITCH_METRICS");
328+
Internal.setEnableJfrFeature(builder, "LOCK_METRICS");
329+
Internal.setEnableJfrFeature(builder, "MEMORY_ALLOCATION_METRICS");
330+
Internal.setEnableJfrFeature(builder, "NETWORK_IO_METRICS");
331+
}
332+
328333
private static void configureUnified(
329334
RuntimeTelemetryBuilder builder, DeclarativeConfigProperties config) {
330-
// Check if user is using new unified config options
331335
boolean emitExperimentalMetrics =
332336
config.getBoolean("emit_experimental_metrics/development", false);
333-
boolean preferJfr = config.getBoolean("prefer_jfr/development", false);
334-
boolean newConfig = emitExperimentalMetrics || preferJfr;
337+
boolean emitExperimentalJfrMetrics =
338+
config.getBoolean("emit_experimental_jfr_metrics/development", false);
339+
boolean preferJfrMetrics = config.getBoolean("prefer_jfr/development", false);
340+
boolean newConfig = emitExperimentalMetrics || emitExperimentalJfrMetrics || preferJfrMetrics;
335341

336342
if (newConfig) {
337343
// New unified config: Use new instrumentation name for both JMX and JFR
@@ -357,8 +363,10 @@ private static void configureUnified(
357363
Experimental.setEmitExperimentalMetrics(builder, true);
358364
}
359365

360-
// Apply prefer_jfr
361-
if (preferJfr) {
366+
if (emitExperimentalJfrMetrics) {
367+
Experimental.setEmitExperimentalJfrMetrics(builder, true);
368+
}
369+
if (preferJfrMetrics) {
362370
Experimental.setPreferJfrMetrics(builder, true);
363371
}
364372

‎instrumentation/runtime-telemetry/library/src/main/java/io/opentelemetry/instrumentation/runtimetelemetry/internal/JmxRuntimeMetricsFactory.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ public static List<AutoCloseable> buildObservables(
2525
if (!preferJfrMetrics) {
2626
observables.addAll(Classes.registerObservers(meter));
2727
observables.addAll(Cpu.registerObservers(meter));
28+
observables.addAll(CpuCount.registerObservers(meter));
2829
observables.addAll(GarbageCollector.registerObservers(meter, captureGcCause));
2930
observables.addAll(MemoryPools.registerObservers(meter));
3031
observables.addAll(Threads.registerObservers(meter));

‎instrumentation/runtime-telemetry/library/src/main/java17/io/opentelemetry/instrumentation/runtimetelemetry/internal/JfrFeature.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
*
1111
* <p>Features that overlap with stable JMX-based instrumentation are disabled by default to avoid
1212
* duplicate metrics. Experimental features (those not marked stable in the semantic conventions)
13-
* are also disabled by default and require {@code emit_experimental_metrics=true} to enable.
13+
* are also disabled by default and require explicit JFR opt-in to enable.
1414
*
1515
* <p>This class is internal and is hence not for public use. Its APIs are unstable and can change
1616
* at any time.

0 commit comments

Comments
 (0)