K8s OpenTelemetry Java Agent, missing metrics?

[ynnckvdv]

Hi,

We are migrating our Wildfly VM's to Kubernetes. We used to use Prometheus to scrape metrics but we want to switch to OpenTelemetry. We are currently using the OpenTelemetry Operator with the Instrumentation CR:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-instrumentation
  namespace: monitoring
spec:
  exporter:
    endpoint: http://alloy-receiver.kube-monitoring:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.27.0
    env:
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf
      - name: OTEL_JMX_TARGET_SYSTEM
        value: wildfly
      - name: OTEL_METRICS_EXPORTER
        value: otlp

This works, but we are missing some metrics that we hope to be able to get. For example, with Prometheus we get wildfly_undertow_active_sessions. Now with OTEL we getwildfly_active_sessions but this value stays empty. There should also be metrics for JDBC connections, especially one for max pool connections, but we do not seem to get them via OTEL. We are not sure what we are missing. If I understand correctly, by setting the OTEL_JMX_TARGET_SYSTEM to wildfly, we should already receive a lot of these metrics. I read about the JDBC metrics here (https://github.com/open-telemetry/semantic-conventions/blob/main/docs/db/database-metrics.md), but I am not sure if these include the max pool connections metric.

What do we need to set to get the JDBC (and undertow if it is indeed not the same) metric?

Sorry about the vague issue, I'm just not sure on how to continue. I also wasn't sure if I should set this here or in the opentelemetry-operator repo.

Thanks a lot!

[laurit]

If I understand correctly, by setting the OTEL_JMX_TARGET_SYSTEM to wildfly, we should already receive a lot of these metrics.

See https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/jmx-metrics/library/wildfly.md for the list of supported metrics.

I read about the JDBC metrics here (https://github.com/open-telemetry/semantic-conventions/blob/main/docs/db/database-metrics.md), but I am not sure if these include the max pool connections metric.

db.client.operation.duration is currently only emitted with otel.semconv-stability.opt-in=database, will be enabled by default in the next release
db.client.response.returned_rows is not implemented, a PR exists
The availability of connection pool metrics depends on which connection pool you use. See https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md

[ynnckvdv]

I have set otel.semconv-stability.opt-in=database and gained the db.client.operation.duration metrics, thanks.
We use wildfly/wildfly-runtime:latest-jdk21 image with the Galleon feature pack wildfly-datasources-galleon-pack. As far as I understand, this uses IronJacamar. This probably means that connection pool metrics are not supported.

I would like to ask what my options are? Do I need to set wildfly.statistics-enabled to true in my standalone.xml? I wanted to avoid editing it, but I will if there aren't other options.

Sorry for the delayed response, thanks for helping out!

[laurit]

Do I need to set wildfly.statistics-enabled to true in my standalone.xml

Consult wildfly documentation for that. I'd guess that the answer is going to be yes. If you don't wish to edit the configuration file then I believe that it is possible to also set these via system properties.

There should also be metrics for JDBC connections, especially one for max pool connections,

Have a look at https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/jmx-metrics/library/wildfly.md for collected wildfly jmx metrics. It does not include all of the connection pool metrics. If you need more you can provide your own yaml configuration, see

opentelemetry-java-instrumentation/instrumentation/jmx-metrics/library/src/main/resources/jmx/rules/wildfly.yaml

Line 81 in 6a3862f

- bean: jboss.as:subsystem=datasources,data-source=*,statistics=pool

for ideas.