Skip to content

fix(observability): resolve kube-ovn dashboard datasource references#150

Open
gurinderu wants to merge 1 commit into
mainfrom
fix/kube-ovn-grafana-datasource
Open

fix(observability): resolve kube-ovn dashboard datasource references#150
gurinderu wants to merge 1 commit into
mainfrom
fix/kube-ovn-grafana-datasource

Conversation

@gurinderu

Copy link
Copy Markdown
Contributor

Problem

Every panel on the Kube-OVN Grafana dashboards (Controller / CNI / OVN / OVS / Pinger) shows Datasource ... was not found.

Root cause

The upstream kube-ovn dashboards (dist/monitoring/*-grafana.json, release-1.14) reference their datasource two ways, neither of which resolves in this cluster:

  1. A DS_PROMETHEUS dashboard input (${DS_PROMETHEUS} references in panels/targets).
  2. A datasource template variable hardcoded to the value prometheus.

This cluster's Prometheus-compatible datasource is VictoriaMetrics (uid victoria-metrics, isDefault: true); there is no datasource named/uid prometheus. The dashboards that work (dotdc k8s-views, node-exporter) leave the datasource variable empty, so Grafana falls back to the default — kube-ovn hardcodes prometheus, so it errors.

Fix (two parts)

  • dashboard.yml — add spec.datasources mapping DS_PROMETHEUS → VictoriaMetrics to all 5 GrafanaDashboard CRs. Resolves the ${DS_PROMETHEUS} references (the bulk of panels; 9/11 on CNI, 12+ on Pinger).
  • datasource-prometheus-alias.yml (new) — a GrafanaDatasource with uid/name prometheus pointing at the same vmsingle-stack backend, isDefault: false, allowCrossNamespaceImport: true. Satisfies the hardcoded $datasource template variable. Non-default, so it does not affect dashboards that resolve the default datasource.

Verification

  • kubectl kustomize flux/apps/kube-system/kube-ovn/observability/ builds clean.
  • Part 1 verified live on a mainnet cluster: after applying spec.datasources, unresolved ${DS_PROMETHEUS} references dropped to 0 and now point at VictoriaMetrics; all kube-ovn panel queries return data through the VictoriaMetrics proxy.
  • Part 2 (alias) was intentionally not applied directly to prod (GitOps); the mechanism mirrors the already-working dashboards — a valid datasource reference resolves.

Deploy note

Reaches the target environment via the per-env tag move (Flux tag > branch).

Upstream kube-ovn dashboards declare a DS_PROMETHEUS input and hardcode the
`datasource` template variable to "prometheus", neither of which matches this
cluster's VictoriaMetrics datasource, so every panel errored with
"Datasource ... was not found".

- Add spec.datasources DS_PROMETHEUS -> VictoriaMetrics to all 5
  GrafanaDashboard CRs (resolves the ${DS_PROMETHEUS} panel/target refs).
- Add a `prometheus`-uid GrafanaDatasource alias pointing at the same
  vmsingle backend (non-default) to satisfy the hardcoded $datasource
  template variable.
@gurinderu
gurinderu requested a review from ValeryAntopol June 13, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant