Scrape Vault metrics with Prometheus

[jchristgit]

Via #573, we currently do not scrape any metrics from Vault. This prevents us from monitoring Vault's health except via generic Kubernetes monitoring.

Vault exposes a metrics endpoint, see https://developer.hashicorp.com/vault/api-docs/system/metrics. The endpoint requires authentication. As far as I can see, we cannot cleanly configure headers to send along in the Prometheus server configuration in a generic manner.

Action items

• enable the endpoint in an unauthenticated manner
  • Helm chart docs: https://developer.hashicorp.com/vault/docs/platform/k8s
  • Helm chart values: https://github.com/hashicorp/vault-helm/blob/main/values.yaml
• disable public access to the metrics endpoint via the ingress using an nginx server config snippet
  • Example: Restrict access to /admin location nginx/kubernetes-ingress#161 (comment)
• Expand our Prometheus configuration to scrape Vault for metrics
  • This can be done by adding labels to the Helm chart values above.
• Check if we can alert on Vault's CA certificate lifetime and if yes, implement it
  • This needs to be accompany this with a runbook that instructs how to renew.