fix(observability): resolve kube-ovn dashboard datasource references#150
Open
gurinderu wants to merge 1 commit into
Open
fix(observability): resolve kube-ovn dashboard datasource references#150gurinderu wants to merge 1 commit into
gurinderu wants to merge 1 commit into
Conversation
Upstream kube-ovn dashboards declare a DS_PROMETHEUS input and hardcode the
`datasource` template variable to "prometheus", neither of which matches this
cluster's VictoriaMetrics datasource, so every panel errored with
"Datasource ... was not found".
- Add spec.datasources DS_PROMETHEUS -> VictoriaMetrics to all 5
GrafanaDashboard CRs (resolves the ${DS_PROMETHEUS} panel/target refs).
- Add a `prometheus`-uid GrafanaDatasource alias pointing at the same
vmsingle backend (non-default) to satisfy the hardcoded $datasource
template variable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every panel on the Kube-OVN Grafana dashboards (Controller / CNI / OVN / OVS / Pinger) shows
Datasource ... was not found.Root cause
The upstream kube-ovn dashboards (
dist/monitoring/*-grafana.json, release-1.14) reference their datasource two ways, neither of which resolves in this cluster:DS_PROMETHEUSdashboard input (${DS_PROMETHEUS}references in panels/targets).datasourcetemplate variable hardcoded to the valueprometheus.This cluster's Prometheus-compatible datasource is VictoriaMetrics (uid
victoria-metrics,isDefault: true); there is no datasource named/uidprometheus. The dashboards that work (dotdc k8s-views, node-exporter) leave the datasource variable empty, so Grafana falls back to the default — kube-ovn hardcodesprometheus, so it errors.Fix (two parts)
dashboard.yml— addspec.datasourcesmappingDS_PROMETHEUS → VictoriaMetricsto all 5GrafanaDashboardCRs. Resolves the${DS_PROMETHEUS}references (the bulk of panels; 9/11 on CNI, 12+ on Pinger).datasource-prometheus-alias.yml(new) — aGrafanaDatasourcewith uid/nameprometheuspointing at the samevmsingle-stackbackend,isDefault: false,allowCrossNamespaceImport: true. Satisfies the hardcoded$datasourcetemplate variable. Non-default, so it does not affect dashboards that resolve the default datasource.Verification
kubectl kustomize flux/apps/kube-system/kube-ovn/observability/builds clean.spec.datasources, unresolved${DS_PROMETHEUS}references dropped to 0 and now point at VictoriaMetrics; all kube-ovn panel queries return data through the VictoriaMetrics proxy.Deploy note
Reaches the target environment via the per-env tag move (Flux tag > branch).