Skip to content

monitoring.adoc

Latest commit

 

History

History
141 lines (116 loc) · 4.94 KB

monitoring.adoc

File metadata and controls

141 lines (116 loc) · 4.94 KB
Raw

Monitoring

In November 2024, Apache NiFi released a new major version 2.0.0.

The NiFi 2.0.0 release changed the way of exposing Prometheus metrics significantly. The following steps explain on how to expose Metrics in NiFi versions 1.x.x and 2.x.x.

Configure metrics in NiFi 1.x.x

For NiFi versions 1.x.x, the operator automatically configures NiFi to export Prometheus metrics. This is done by creating a Job that connects to NiFi and configures a Prometheus Reporting Task.

Important
 Network access from the Job to NiFi is required. If you are running a Kubernetes with restrictive NetworkPolicies, make sure to allow access from the Job to NiFi.

See operators:monitoring.adoc for more details.

Disabling create-reporting-task Job

It can be helpful to disable the Job, e.g. when you configOverride an authentication mechanism, which the Job currently cannot use to authenticate against NiFi.

To achieve this use the following configuration:

spec:
  clusterConfig:
    createReportingTaskJob:
      enabled: false

Configure metrics in NiFi 2.x.x

The Prometheus Reporting Task was removed in NiFi 2.x.x in NIFI-13507. Metrics are now always exposed and can be scraped using the NiFi Pod FQDN and the HTTP path /nifi-api/flow/metrics/prometheus.

For a deployed single node NiFi cluster called simple-nifi, containing a rolegroup called default, the metrics endpoint is reachable under:

https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
Important
 If NiFi is configured to do any user authentication, requests to the metric endpoint must be authenticated and authorized.

Authentication with NiFi 2.x.x

To authenticate, you can use a bearer token created by your NiFi instance e.g.

curl -X POST https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/access/token -d 'username=<user>&password=<password>' -k

where -k equals verify=false. The reply is your bearer token.

You then can use the bearer token to authenticate with Prometheus replacing basic_auth with

---
# basic_auth:
  # username: <user>
  # password: <password>
authorization:
  type: Bearer
  credentials: "<Bearer Token>"
tls_config:
  insecure_skip_verify: true
static_configs:
  - targets:
    - '<pod>.<statefulset>.svc.cluster.local:8443' (1)
metrics_path: '/nifi-api/flow/metrics/prometheus'
scheme: https

  1. Static targets only scrapes one pod.

or use it in a NiFi secret which should look like

---
apiVersion: v1
kind: Secret
metadata:
  name: nifi-authorization-secret
type: Opaque
stringData:
  nifi_token: "<Bearer_token>"

If you want to use a ServiceMonitor you’d need to configure it as follows:

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: scrape-nifi2
  labels:
    stackable.tech/vendor: Stackable
    release: prometheus
spec:
  endpoints:
  - port: https
    path: 'nifi-api/flow/metrics/prometheus'
    scheme: https
    interval: 5s
    tlsConfig:
      insecureSkipVerify: true
    authorization:
      credentials: (1)
        key: "nifi_token"
        name: "nifi-authorization-secret"
        optional: false
      type: "Bearer"
    relabelings: (2)
      - sourceLabels:
          - __meta_kubernetes_pod_name
          - __meta_kubernetes_service_name
          - __meta_kubernetes_namespace
          - __meta_kubernetes_pod_container_port_number
        targetLabel: __address__
        replacement: ${1}.${2}.${3}.svc.cluster.local:${4}
        regex: (.+);(.+?)(?:-metrics)?;(.+);(.+)
  selector:
    matchLabels:
      prometheus.io/scrape: "true"
  namespaceSelector:
    any: true
  jobLabel: app.kubernetes.io/instance

  1. Authorization via Bearer Token stored in a secret

  2. Relabel __address__ to be a FQDN rather then the IP-Address of target pod

Note
 As of Listener integration, SDP exposes a Service with -metrics thus we need to regex this suffix.

Known Limitations

NiFi only allows authentication with JWT on pod level. Therefore you will need one endpoint per NiFi pod and a valid bearer token for each. This is a consequence of NiFi moving their metrics endpoint behind a strong authentication mechanism.