feat: support legacy and new prometheus client stacks

[SavitarC]

What is this PR for?

Fix #16109

This PR adds compatibility for both the legacy and the new Prometheus client stacks in Dubbo metrics reporting, allowing users to run with either dependency set without changing Dubbo-side logic. By default, the new stack is preferred when both are present on the classpath.

The reporter now detects a supported Prometheus stack at runtime and continues to support Pushgateway workflows, including basic authentication.

What changes were made?

• Added stack-detection utilities for:
  • the legacy micrometer-registry-prometheus / simpleclient stack
  • the new micrometer-registry-prometheus (prometheusmetrics) / new Pushgateway stack
• Refactored PrometheusMetricsReporter to use an adapter abstraction:
  • LegacyPrometheusClientAdapter
  • NewPrometheusClientAdapter
• Added runtime adapter selection based on classpath availability
• Kept the existing Pushgateway scheduling flow, but routed push/create/auth operations through the selected adapter
• Added support for the builder-style Pushgateway API used by the new stack
• Updated dependencies, BOM configuration, and Spring Boot autoconfiguration wiring to include both the old and new Prometheus artifacts, including prometheus-metrics-exporter-pushgateway
• Extended tests to cover Pushgateway behavior and authentication behavior on the new adapter path

Why is this needed?

Different deployments currently depend on different generations of Prometheus and Micrometer libraries. This change makes the Dubbo metrics reporter compatible with both stacks and reduces upgrade friction for users migrating between them.

How to use it

Use the following dependencies together:

• dubbo-spring-boot-starter
• dubbo-metrics-prometheus
• dubbo-observability-spring-boot-starter
• spring-boot-starter-actuator

With this combination, the new Prometheus stack is preferred by default when both stacks are available.

Additional Notes

Spring Boot 3.3.x and 3.4.x registry conflict

In Spring Boot 3.3.x and 3.4.x, both of the following auto-configurations may coexist:

• PrometheusSimpleclientMetricsExportAutoConfiguration
• PrometheusMetricsExportAutoConfiguration

When both are active at the same time, registry conflicts may occur, causing Prometheus metrics to become invisible.

Solutions

There are two possible solutions:

1. Manually exclude one of the two Prometheus registry implementations from dubbo-metrics-prometheus:

   • micrometer-registry-prometheus-simpleclient
   • micrometer-registry-prometheus

   Only one of them should remain.

2. Enable the following configuration:

dubbo.metrics.use-global-registry=true

[codecov-commenter]

Codecov Report

❌ Patch coverage is 62.50000% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.82%. Comparing base (af9a355) to head (37a549a).

Files with missing lines	Patch %	Lines
.../metrics/prometheus/PrometheusMetricsReporter.java	72.41%	13 Missing and 3 partials ⚠️
...s/prometheus/PrometheusMetricsReporterFactory.java	0.00%	13 Missing ⚠️
...apache/dubbo/metrics/utils/MetricsSupportUtil.java	88.88%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                3.3   #16132      +/-   ##
============================================
+ Coverage     60.77%   60.82%   +0.04%     
- Complexity       15    11758   +11743     
============================================
  Files          1953     1953              
  Lines         89119    89182      +63     
  Branches      13445    13453       +8     
============================================
+ Hits          54165    54243      +78     
+ Misses        29380    29370      -10     
+ Partials       5574     5569       -5     

Flag	Coverage Δ
integration-tests-java21	32.12% <38.75%> (-0.05%)	⬇️
integration-tests-java8	32.23% <38.75%> (+0.02%)	⬆️
samples-tests-java21	32.16% <38.75%> (-0.04%)	⬇️
samples-tests-java8	29.81% <38.75%> (+0.11%)	⬆️
unit-tests-java11	59.02% <46.25%> (?)
unit-tests-java17	58.53% <46.25%> (+<0.01%)	⬆️
unit-tests-java21	58.56% <46.25%> (+0.03%)	⬆️
unit-tests-java25	58.47% <46.25%> (-0.01%)	⬇️
unit-tests-java8	59.06% <46.25%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[zrlw]

@SavitarC

Do you test https://github.com/SavitarC/dubbo-support-prometheusmetrics with applying this pr?
i tested it but got the same result:

Run curl -s http://127.0.0.1:18180/metrics | grep '^dubbo_' | sort — no output is returned.

[zrlw]

still has problems

[zrlw]

i tested https://github.com/apache/dubbo-samples/tree/master/4-governance/dubbo-samples-metrics-prometheus with applying this pr by bumping the versions,

        <!-- applying this pr to latest dubbo 3.3  --->
        <dubbo.version>3.3.7-SNAPSHOT</dubbo.version>

        <!-- depends on micrometer-registry-prometheus 1.15.8 instead of 1.9.7 -->
        <spring-boot.version>3.5.10</spring-boot.version>

then build the test projects, run the provider app and execute such command,

curl -s http://127.0.0.1:20888/metrics

but i got such response:

There is no application with data

[SavitarC]

@zrlw
Thanks for the verification.

I found the root cause. The dubbo-samples-metrics-prometheus-provider sample uses <dubbo:metrics protocol="prometheus"> directly, but the Prometheus metrics extension module was not present on the runtime classpath.

In this PR, I moved the required Prometheus client dependencies and the old/new Prometheus client compatibility logic into dubbo-metrics-prometheus:

• dubbo-metrics-prometheus now contains the adapter logic for both the legacy simpleclient-based stack and the new prometheus-metrics stack.
• It prefers the new Prometheus client stack when available.
• It also brings the required Prometheus / Micrometer dependencies for the Prometheus reporter and PushGateway support.

With Spring Boot 3.5.x, the old simpleclient-based Prometheus stack is no longer brought in through the Spring Boot / Micrometer dependency path. Therefore, the Dubbo Prometheus extension module, org.apache.dubbo:dubbo-metrics-prometheus, must be present on the classpath.

I have tested again with:

• Dubbo: 3.3.7-SNAPSHOT
• Spring Boot: 3.5.10

After adding org.apache.dubbo:dubbo-metrics-prometheus, both my demo project and the official dubbo-samples-metrics-prometheus-provider sample can output Dubbo metrics correctly.

For the XML-based dubbo-samples-metrics-prometheus sample, I think explicitly adding dubbo-metrics-prometheus is reasonable because it uses <dubbo:metrics protocol="prometheus"> directly.

However, I am not fully sure about the expected dependency convention for dubbo-observability-spring-boot-starter. If users depend on the observability starter, should the starter transitively bring in dubbo-metrics-prometheus, so that users do not need to add it manually?

If that is the expected behavior, I will update this PR to make dubbo-observability-spring-boot-starter depend on dubbo-metrics-prometheus directly.

[zrlw]

As for me, making dubbo-observability-spring-boot-starter depend on dubbo-metrics-prometheus directly might be better than adding org.apache.dubbo:dubbo-metrics-prometheus manually.

[zrlw]

But it seemed a little bit weird that dubbo-observability-spring-boot-starter depends on dubbo-metrics-prometheus, maybe we have to modify the pom.xml by replacing micrometer-registry-prometheus dependency with dubbo-metrics-prometheus dependency.

[zrlw]

this pr should be ok since apache/dubbo-samples#1291 has been merged.

[zrlw]

LGTM

[zrlw]

@SavitarC

ConsumerMetricsIT and ProviderMetricsIT at apache/dubbo-samples#1291 will fail due to unmatched metric names if running them with springboot 3.5.10 and this pr, e.g.,

14:24:57.330 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_application_info_total doesn't exist
14:24:57.331 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_configcenter_total doesn't exist
14:24:57.332 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_provider_qps_total doesn't exist
14:24:57.333 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_provider_requests_processing_total doesn't exist
14:24:57.334 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_registry_subscribe_num_total doesn't exist
14:24:57.335 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_registry_register_requests_succeed_total doesn't exist
14:24:57.336 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_registry_register_requests_total doesn't exist
14:24:57.337 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_registry_subscribe_num_succeed_total doesn't exist
14:24:57.337 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_registry_notify_requests_total doesn't exist
14:24:57.338 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_registry_register_requests_failed_total doesn't exist
14:24:57.339 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_registry_subscribe_num_failed_total doesn't exist
14:24:57.341 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_registry_register_requests_succeed_total doesn't exist
14:24:57.342 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_metadata_subscribe_num_failed_total doesn't exist
14:24:57.344 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_metadata_push_num_failed_total doesn't exist
14:24:57.346 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_metadata_subscribe_num_total doesn't exist
14:24:57.346 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_metadata_subscribe_num_succeed_total doesn't exist
14:24:57.347 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_metadata_push_num_succeed_total doesn't exist
14:24:57.348 |-ERROR [main] cs.prometheus.consumer.ConsumerMetricsIT:150 -| metric key:dubbo_metadata_push_num_total doesn't exist

refer to the reason that you said at #14964

With Spring Boot 3.3.13 (micrometer-registry-prometheus 1.13.15 + prometheus-metrics-model 1.2.1), Dubbo’s raw metric name dubbo.application.info.total is first normalized by io.prometheus.metrics.model.snapshots.PrometheusNaming#sanitizeMetricName, which strips reserved suffixes such as .info and .total. Then, the Prometheus text writer (PrometheusTextFormatWriter#writeCounter) appends _total only for Counter metrics. As a result, the final exposed name becomes dubbo_application_total.

By contrast, Dubbo’s raw metric name dubbo_provider_qps_total is normalized to dubbo_provider_qps, which is produced as a GaugeMetricSample in Dubbo (AggregateMetricsCollector) and registered as a Gauge via AbstractMetricsReporter#registerGaugeSample. Since Gauges do not get the _total suffix, the metric name remains dubbo_provider_qps.

Could you provide a compatible solution?

[zrlw]

By the way, there are many metric names that does not be normalized, e.g., dubbo_provider_requests_business_failed_total, dubbo_consumer_requests_failed_service_unavailable_total

[SavitarC]

@zrlw Thanks for the reminder.

I rechecked both the Dubbo metric registration path and the Micrometer / Prometheus naming logic.

In Dubbo, these metrics are registered with different sample types:

Dubbo raw metric name	Dubbo sample type	Micrometer meter type
dubbo.application.info.total	CounterMetricSample	FunctionCounter
dubbo.provider.qps.total	GaugeMetricSample	Gauge
dubbo.provider.requests.business.failed.total	CounterMetricSample	FunctionCounter
dubbo.consumer.requests.failed.service.unavailable.total	CounterMetricSample	FunctionCounter

For the legacy micrometer-registry-prometheus-simpleclient path, the exposed name is produced by Micrometer's io.micrometer.prometheus.PrometheusNamingConvention. It applies NamingConvention.snakeCase, appends _total for Counter meters if missing, and replaces unsupported characters with _. It does not strip reserved suffixes such as .info or .total.

For the new micrometer-registry-prometheus path, Micrometer delegates to the new Prometheus client's PrometheusNaming. The new client strips reserved suffixes such as _total, _info, .total, and .info from the base metric name first. Then the Prometheus text writer appends suffixes according to the metric type. Counter metrics get _total appended back, while Gauge metrics do not.

So the expected exposed names are:

Dubbo raw metric name	Legacy simpleclient exposed name	New prometheus-metrics exposed name
dubbo.application.info.total	dubbo_application_info_total	dubbo_application_total
dubbo.provider.qps.total	dubbo_provider_qps_total	dubbo_provider_qps
dubbo.provider.requests.business.failed.total	dubbo_provider_requests_business_failed_total	dubbo_provider_requests_business_failed_total
dubbo.consumer.requests.failed.service.unavailable.total	dubbo_consumer_requests_failed_service_unavailable_total	dubbo_consumer_requests_failed_service_unavailable_total

This also explains why some _total metrics still keep the _total suffix: for Counter metrics, the new client strips the reserved suffix from the base name but the text writer appends _total again. For Gauge metrics, _total is stripped and not appended back.

Therefore, I do not think we should apply a generic _total stripping rule, and we should not rename Dubbo's raw metric names globally in this PR, because that may break existing dashboards, PromQL queries, and alert rules.

A safer compatible solution is to update ConsumerMetricsIT and ProviderMetricsIT with a centralized metric-name resolver. The resolver will take the Dubbo raw metric name and its Dubbo sample type, then derive the possible exposed names for both naming paths:

• legacy micrometer-registry-prometheus-simpleclient
• new micrometer-registry-prometheus / prometheus-metrics

Then the tests can assert that one of the resolved names exists.

I will update the tests in this direction. This keeps Dubbo's raw metric names unchanged, avoids hard-coded aliases scattered across tests, and makes the samples compatible with both Prometheus client implementations.

[zrlw]

A safer compatible solution is to update ConsumerMetricsIT and ProviderMetricsIT with a centralized metric-name resolver. The resolver will take the Dubbo raw metric name and its Dubbo sample type, then derive the possible exposed names for both naming paths:

Solved by apache/dubbo-samples#1293

Good job, thanks!

[zrlw]

@SavitarC
Pls send a email to me if you are willing to participate in PR reviewing.