Skip to content

monitoring.adoc

Latest commit

 

History

History
109 lines (93 loc) · 6.2 KB

monitoring.adoc

File metadata and controls

109 lines (93 loc) · 6.2 KB
Raw

Monitoring

Configure metrics in NiFi 2.x.x

The Prometheus Reporting Task was removed in NiFi 2.x.x in NIFI-13507. Metrics are now always exposed and can be scraped using the NiFi metrics Service and the HTTP path /nifi-api/flow/metrics/prometheus.

For a deployed NiFi cluster called simple-nifi, containing a rolegroup called default, the metrics endpoint is reachable under:

https://simple-nifi-node-default-metrics.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
Note
 The above URL connects to one of the Pods, reachable through the specified Service, therefore scraping metrics produced by that Pod only. To scrape metrics from a particular Pod, the FQDN of the Pod and the headless Service need to be used. For example: https://simple-nifi-node-default-0.simple-nifi-node-default-headless.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
Important
 If NiFi is configured to do any user authentication, requests to the metrics endpoint must be authenticated and authorized.

Authentication with NiFi 2.x.x

To authenticate against the NiFi 2.x.x API, you can configure mTLS between NiFi and the client calling NiFi. For more information about authentication between Kubernetes Pods, check out the Secret Operator documentation.

The following example illustrates the configuration of a Prometheus scraper for NiFi, using the aforementioned method of configuring mTLS and utilizing the internally available tls SecretClass.

To generate a client certificate signed by the tls SecretClass CA trusted in NiFi, add the following volume and volumeMount to the Prometheus Pod.

Important
 If the Prometheus Operator is used to deploy Prometheus, there is currently a known bug, which prevents adding an additional Volume containing annotations on the volumeClaimTemplate. The bug is tracked in the Prometheus Operator Issues. The annotations are necessary to configure the behavior of the Secret Operator. As a current workaround, until the issue is resolved, one could deploy an additional Pod only responsible for creating a TLS certificate as a Secret, which then can be used by the ServiceMonitor. This workaround is illustrated in the monitoring Stack. 
---
prometheus: (1)
  prometheusSpec:
    volumes:
      - name: tls
        ephemeral:
          volumeClaimTemplate:
            metadata:
              annotations:
                secrets.stackable.tech/class: tls # (2)
                secrets.stackable.tech/scope: pod,service=prometheus-kube-prometheus-prometheus # (3)
            spec:
              storageClassName: secrets.stackable.tech
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: "1"
    volumeMounts:
      - name: tls
        mountPath: /stackable/tls # (4)

  1. This given configuration is set in the Prometheus resource for the Prometheus Operator

  2. The tls SecretClass created by the Secret Operator, storing its CA in a Kubernetes Secret. Any other SecretClass can be used as well

  3. The service=prometheus-kube-prometheus-prometheus scope is added to include the subjectAlternateName of the Prometheus Service in the generated TLS certificate. This particular Service name, used here, refers to the Prometheus Service deployed by the Prometheus Operator

  4. The path where the mTLS certificates are mounted inside the Prometheus Pod

If you want to use a ServiceMonitor you’d need to configure it as follows:

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: scrape-nifi-2
  labels:
    stackable.tech/vendor: Stackable
    release: prometheus
spec:
  endpoints:
  - path: /nifi-api/flow/metrics/prometheus
    port: https
    scheme: https
    tlsConfig: # (1)
      caFile: /stackable/tls/ca.crt
      certFile: /stackable/tls/tls.crt
      keyFile: /stackable/tls/tls.key
    relabelings: # (2)
      - sourceLabels:
          - __meta_kubernetes_pod_name
          - __meta_kubernetes_service_name
          - __meta_kubernetes_namespace
          - __meta_kubernetes_pod_container_port_number
        targetLabel: __address__
        replacement: ${1}.${2}-headless.${3}.svc.cluster.local:${4} # (3)
        regex: (.+);(.+?)(?:-metrics)?;(.+);(.+)
  selector:
    matchLabels:
      stackable.tech/vendor: Stackable
      prometheus.io/scrape: "true"
  namespaceSelector:
    any: true
  jobLabel: app.kubernetes.io/instance

  1. In the TLS configuration of the ServiceMonitor, specify the location of the cert and key files mounted into the Prometheus Pod

  2. Relabel address to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary to scrape NiFi, since it requires a DNS name to address the NiFi REST API

  3. Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the headless Service, which is why we use the headless Service instead of the metrics Service to scrape NiFi metrics

Note
 The SDP exposes a dedicated metrics Service since the Listener integration.

The described example is part of the Prometheus and ServiceMonitor manifests used in the monitoring stack of the demos repository.