Distinguish stable from nonstable JMX target systems

[trask]

E.g. by prefixing the target system name with experimental_.

Or alternatively we could say that they are all unstable for now by adding experimental to the otel.jmx.target.system configuration property (while gracefully emitting a warning for usage of the current configuration property).

[trask]

@SylvainJuge @breedx-splk @robsunday @PeterF778 any ideas?

Stable means that we don't break default telemetry emitted from them without a major version bump. So, e.g., it's ok to say they're all stable if we want to abide by that restriction.

[SylvainJuge]

I think we can consider them as "experimental by default" as we never had any formal status nor stability defined. This is also why we allowed ourselves with some breaking changes with #12158 and related changes.

Using a dedicated config option with experimental_ or prefixing the value for otel.jmx.target.system means the stability is defined on a per-system level, which might prevent evolution in the future.

I think that if we want to have stability we need to define it on a per-metric level, that means investing in some tooling and registry generation/verification like with open-telemetry/opentelemetry-java-contrib#2362.

In summary I would recommend to:

• document all those metrics to be experimental for now, document the values of otel.jmx.target.system as experimental as well (in a similar way as db.system.name)
• invest into tooling/definitions to allow adding stability (default to experimental) on the metric level, this should also cover metric attributes.
• start the process of stabilizing those metrics and their definitions, the biggest challenge here remains to find the right people to provide feedback/approval on this as it depends on actual products outside of OpenTelemetry scope.
• once we have at least one stable metric for a given system, we can promote the corresponding value for otel.jmx.target.system as stable.

However, I understand that it could feel a bit "heavy" in terms of effort, so maybe there are simpler ways to achieve this. We try to avoid breaking changes, so I think it could be the right time to start doing things "the right way" for JMX metrics.

Also, do you think we should have a configuration option to filter-out experimental metrics like we have for JVM Runtime Telemetry ?

[trask]

Using a dedicated config option with experimental_ or prefixing the value for otel.jmx.target.system means the stability is defined on a per-system level, which might prevent evolution in the future.

we should probably then change the property name to include the word experimental to align with https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/VERSIONING.md#compatibility-requirements

(we can deprecate otel.jmx.target.system and drop it in 3.0)

Also, do you think we should have a configuration option to filter-out experimental metrics like we have for JVM Runtime Telemetry ?

I'm simplifying the runtime metrics in #16087, using broader configuration properties to opt-in to all experimental metrics, and relying on metric views to filter some out. This proposal may help out in the future too: open-telemetry/opentelemetry-specification#4809

[SylvainJuge]

From what I understand, the goals here would be to:

• only enable stable JMX metrics by default
• offer a configuration option to opt-in for non-stable metrics, it could be otel.jmx.emit-experimental-metrics (false by default).

In order to do this, we need to have for each metric an attribute to define its stability, which is where the work with registry/weaver would be relevant.

Deprecating otel.jmx.target.system and replacing it with another option with experimental in the name would be a breaking change. Even if that allows to make the experimental/non-stable nature of this config option explicit, I am not sure it is worth a guaranteed extra burden on users (this would also include users of jmx-scraper).

However, in practice we currently ask the users to set the value of otel.jmx.target.system from what they know about the target JVM system, for example a tomcat server with an embedded activemq server it would be tomcat,activemq.
Instead of doing this we can automatically register all the metrics that have matching JMX rules, without having to ask user for any configuration.

Thus, I think here we could maybe provide a better option:

• modify the metrics definitions to include a maturity/stability indicator, probably by integrating with otel weaver.
• disable the non-stable metrics by default (unfortunately, for now it would be most of them excepted for jvm runtime metrics, but they are used only through jmx-scraper) and provide an opt-in configuration option to enable them.
• automatically register metrics whenever their JMX rules match and remove the otel.jmx.target.system configuration that becomes obsolete.

As a side-note, for the configuration options, do you think we should keep things in the otel.jmx namespace or do you think it would be simpler/better to duplicate a bit with for example otel.instrumentation.jmx (to configure JMX in instrumentation ) and otel.jmx-scraper (to configure JMX in scraper), or as an alternative configure everything with otel.instrumentation.jmx even when used through jmx-scraper ?

[trask]

plan sounds good, I don't have strong preference other than that by the time 3.0 rolls out, anything under JMX metrics (in this repo) that we don't want to guarantee stability of until 4.0, is hidden behind a flag with the word "experimental" (/development in declarative configuration).

As a side-note, for the configuration options, do you think we should keep things in the otel.jmx namespace or do you think it would be simpler/better to duplicate a bit with for example otel.instrumentation.jmx (to configure JMX in instrumentation ) and otel.jmx-scraper (to configure JMX in scraper), or as an alternative configure everything with otel.instrumentation.jmx even when used through jmx-scraper ?

currently we have to map them since they don't have the .instrumentation segment. It's not a big problem though, just keep in mind the overall move to declarative configuration.

opentelemetry-java-instrumentation/declarative-config-bridge/src/main/java/io/opentelemetry/instrumentation/config/bridge/ConfigPropertiesBackedDeclarativeConfigProperties.java

Lines 94 to 97 in 5f6ec3d

	// jmx properties don't have an "instrumentation" segment
	SPECIAL_MAPPINGS.put("java.jmx.enabled", "otel.jmx.enabled");
	SPECIAL_MAPPINGS.put("java.jmx.config", "otel.jmx.config");
	SPECIAL_MAPPINGS.put("java.jmx.target.system", "otel.jmx.target.system");