|
1 | 1 | = Monitoring |
2 | | -:description: The HDFS cluster can be monitored with Prometheus from inside or outside the K8S cluster. |
| 2 | +:description: The HDFS cluster is automatically configured to export Prometheus metrics. |
3 | 3 |
|
4 | 4 | The cluster can be monitored with Prometheus from inside or outside the K8S cluster. |
5 | 5 |
|
6 | | -All services (with the exception of the Zookeeper daemon on the node names) run with the JMX exporter agent enabled and expose metrics on the `metrics` port. |
7 | | -This port is available from the container level up to the NodePort services. |
| 6 | +The managed HDFS stacklets are automatically configured to export Prometheus metrics. |
| 7 | +See xref:operators:monitoring.adoc[] for more details. |
8 | 8 |
|
9 | 9 | [IMPORTANT] |
10 | 10 | ==== |
11 | | -Starting with Stackable Data Platform 25.7, the built-in Prometheus metrics are also available at the `/prom` endpoint of all the UI services. |
| 11 | +Starting with Stackable Data Platform 25.7, the built-in Prometheus metrics are available at the `/prom` endpoint of all the UI services. |
12 | 12 | The JMX exporter metrics are now deprecated and will be removed in a future release. |
13 | 13 | ==== |
14 | 14 |
|
15 | | -The metrics endpoints are also used as liveliness probes by Kubernetes. |
| 15 | +This endpoint, in the case of the Namenode service, is reachable via the the `metrics` service: |
| 16 | +[source,shell] |
| 17 | +---- |
| 18 | +http://<hdfs-stacklet>-namenode-<rolegroup-name>-metrics:9870/prom |
| 19 | +---- |
16 | 20 |
|
17 | | -See xref:operators:monitoring.adoc[] for more details. |
| 21 | +== Authentication when using TLS |
| 22 | + |
| 23 | +HDFS exposes metrics through the same port as their web UI. Hence, when configuring HDFS with TLS the metrics are also secured by TLS, |
| 24 | +and the clients scraping the metrics endpoint need to authenticate against it. This could for example be accomplished by utilizing mTLS |
| 25 | +between Kubernetes Pods with the xref:home:secret-operator:index.adoc[Secret Operator]. |
| 26 | + |
| 27 | +When using the Prometheus `ServiceMonitor` for scraping, the `address` label needs relabeling to use the `headless` Service instead of the |
| 28 | +`metrics` Service. This is because per default Prometheus targets the Pod IPs as endpoints, but since the Pod IPs are not |
| 29 | +part of the certificate, the authentication will fail. Instead, the FQDN of the Pods, which can be added to the certificate, is used, but |
| 30 | +this FQDN is only available through the `headless` Service. |
| 31 | + |
| 32 | +A more detailed explanation can be found in the xref:home:nifi:usage_guide/monitoring.adoc[NiFi Operator Monitoring Docs] with a similar situation |
| 33 | +and an example of a Prometheus `ServiceMonitor` configured for TLS in the |
| 34 | +https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus-service-monitors.yaml[Monitoring Stack{external-link-icon}^]. |
0 commit comments