Prometheus metrics for operator and key-manager

[dcmcand]

Problem

Today the only way to know what the pack is doing is to read pod logs. There are no metrics, no dashboards, no SLO targets, and no alerting hooks. For a production deployment this means:

• No visibility into reconcile latency, error rates, or queue depth on the operator.
• No visibility into key-manager request rates, error rates, or which model groups are hot.
• No way to tell internal server error rates apart by handler / by model.
• No way to detect a stuck reconciler or a key-manager pod that has lost its k8s watch.

Proposal

Operator

Use the controller-runtime metrics endpoint that's already wired up by kubebuilder. Expose:

• controller_runtime_reconcile_total{controller, result} (built-in)
• controller_runtime_reconcile_errors_total{controller} (built-in)
• controller_runtime_reconcile_time_seconds{controller} histogram (built-in)
• Custom: nebari_llmmodel_total{phase} gauge (one per LLMModel by phase: Pending / Ready / Failed)
• Custom: nebari_llmmodel_reconcile_failures_total{model_namespace, model_name, reason} counter

The controller-runtime metrics are served at :8080/metrics by default - just confirm the chart enables it and exposes it via a Service that a ServiceMonitor (or PodMonitor) can scrape.

Key-manager

Add a /metrics endpoint on the existing HTTP server. Expose:

• keymanager_http_requests_total{handler, method, status} counter
• keymanager_http_request_duration_seconds{handler, method} histogram
• keymanager_keys_total{model} gauge (live count from the k8s cache)
• keymanager_key_operations_total{operation, result} counter (operation in {create, list, revoke}; result in {success, error})
• keymanager_k8s_watch_errors_total{resource} counter (informer watch failures)

Use prometheus/client_golang and the promhttp.Handler(). Wire structured-log error counts up to a metric so the #57 logging fix has a metric companion.

Charts

Both subcharts should ship a ServiceMonitor template gated on serviceMonitor.enabled (off by default), pointing at the metrics endpoint. Document the assumed Prometheus Operator availability.

Out of scope

• Dashboards (Grafana JSON) - separate issue once metrics exist.
• Tracing (OTLP) - separate issue.
• SLOs / alerts - separate issue.

This issue is just "metrics exist and a Prometheus can scrape them".

[dcmcand]

From the PR #117 security assessment (R-05): alongside the general metrics tracked here, the API-key auth path needs a few security-relevant signals so a stale or failed reconcile is visible instead of silent:

• Last successful api-keys Secret to SecurityPolicy allow-list reconcile, per model, so a stuck allow-list is detectable.
• A count of duplicate client IDs across the pooled api-keys Secrets (should be zero; see Per-model API-key scoping broken when one user has keys for multiple models (colliding client IDs) #122).
• Per-model SecurityPolicy Accepted / Programmed conditions.
• Prolonged "freshly minted key returns 403" windows (allow-list not yet converged) and Secret list/watch failures.

Convergence is fail-closed today, but a stale state currently only surfaces in operator logs. Recording this here so the metrics work covers the auth path too, rather than opening a duplicate issue.