Allow to unregister/stop/destroy instruments

[mateuszrzeszutek]

What are you trying to achieve?

I'm currently working on the micrometer->OTel bridge instrumentation in the javaagent. Micrometer offers the possibility to remove a meter from the MeterRegistry and stop emitting whatever metrics it used to collect. For example, suppose you use a database connection pool that's instrumented with metrics - when you close/destroy the whole pool you probably want to stop collecting any metrics associated to it (because it doesn't exist anymore). This is useful for both asynchronous instruments (since once they're registered there's no way to stop them) and synchronous instruments (you can just stop using them, but the metrics SDK will still send in the last recorded value).

Additional context.

Micrometer bridge PR: open-telemetry/opentelemetry-java-instrumentation#4919

CC @jsuereth

[tigrannajaryan]

(Following https://github.com/open-telemetry/opentelemetry-specification/blob/main/CONTRIBUTING.md#issue-triaging process).

@jsuereth I am reassigning this to you, I believe you know the context of this better.

[bogdandrutu]

@mateuszrzeszutek micrometer is an API + Some implementation(s). I am not sure why that should translate into calls to our API instead of offering an implementation of micrometer that implements the MetricProducer, see https://github.com/open-telemetry/opentelemetry-java/blob/44dccd6c33c7c4cb58e1cc63f0db02b3bfec1804/opencensus-shim/src/main/java/io/opentelemetry/opencensusshim/metrics/OpenCensusMetricProducer.java#L24

/cc @jsuereth

[anuraaga]

@bogdandrutu Micrometer was just one example but do you think apps should not have a way to stop reporting metrics? Dropwizard metrics in Java also has registry.remove and it seems idiomatic for a library that allows registering a stateful object to always allow unregistering it. While the whole MeterProvider could be closed, metric definitions are generally completely separate from the MeterProvider lifecycle and that probably doesn't solve that case.

[mateuszrzeszutek]

There are advantages to calling OTel API in micrometer bridge instrumentation:

• Micrometer instruments like Timer and DistributionSummary can be translated to native OTel histograms; micrometer does not have a "histogram" meter by itself, it basically exports each bucket as a separate gauge.
• Due to the API/SDK split in the javaagent using a MetricProducer inside an instrumentation would be very painful; the SDK classes are not directly accessible in instrumentation code, they're hidden away inside the agent classloader -- we would have to bridge MetricProducer and all related classes (and do lots of copying) to make that work at all.

Also, micrometer aside, the unregister/remove functionality is still very much needed. The database connection pool example from my first post here is still valid.

[anuraaga]

Just another note about Micrometer, I expect it to be used for a very long time by Spring and likely Spring users too as a result. For example, for tracing they will use Micrometer Tracing, not the OpenTelemetry tracing API. I think the reality of the Java ecosystem means we should consider Micrometer as a first class API for Java metrics, and this seems OK to me in practice. As such, I'd like Micrometer usage to benefit from features like exemplars / baggage which means going through our API (or well our SDK implementations directly maybe, but not MetricProducer which is too late to add cross-cutting concern features to the micrometer metrics).

[bogdandrutu]

@mateuszrzeszutek

Micrometer instruments like Timer and DistributionSummary can be translated to native OTel histograms; micrometer does not have a "histogram" meter by itself, it basically exports each bucket as a separate gauge.

It is just a representation difference, it is still a Histogram, I don't see the point.

Due to the API/SDK split in the javaagent using a MetricProducer inside an instrumentation would be very painful; the SDK classes are not directly accessible in instrumentation code, they're hidden away inside the agent classloader -- we would have to bridge MetricProducer and all related classes (and do lots of copying) to make that work at all.

The micrometer instrumentation is not really an instrumentation, is actually an SDK/Producer whatever you call it, since micrometer is an API + Impl to produce metrics. Not sure if any metrics library should be seen as "instrumentation", but rather as producers of telemetry.

Just another note about Micrometer, I expect it to be used for a very long time by Spring and likely Spring users too as a result. For example, for tracing they will use Micrometer Tracing, not the OpenTelemetry tracing API. I think the reality of the Java ecosystem means we should consider Micrometer as a first class API for Java metrics, and this seems OK to me in practice.

I am not saying to not have an integration (not instrumentation) with micrometer.

As such, I'd like Micrometer usage to benefit from features like exemplars / baggage which means going through our API (or well our SDK implementations directly maybe, but not MetricProducer which is too late to add cross-cutting concern features to the micrometer metrics).

It is very hard to do that since their API is not well designed for that (or was not designed with that in mind).

FYI: We don't have enough tracing APIs let's create another one :)))

[mateuszrzeszutek]

Micrometer instruments like Timer and DistributionSummary can be translated to native OTel histograms; micrometer does not have a "histogram" meter by itself, it basically exports each bucket as a separate gauge.

It is just a representation difference, it is still a Histogram, I don't see the point.

Well that's true, conceptually it's the same. It results in different MetricData being exported though.

As such, I'd like Micrometer usage to benefit from features like exemplars / baggage which means going through our API (or well our SDK implementations directly maybe, but not MetricProducer which is too late to add cross-cutting concern features to the micrometer metrics).

It is very hard to do that since their API is not well designed for that (or was not designed with that in mind).

Doesn't that sort of work by default? If you're using a Counter and call counter.add(1) it'll capture Context.current() even if you don't pass the context; so if you use a bridged micrometer synchronous instrument it should get the exemplar from the current context even though micrometer API does not explicitly support this.