Skip to content

Make reconcile error panels honor Grafana time range#63

Open
swills wants to merge 1 commit into
cloudnative-pg:mainfrom
swills:reconcile-errors-fix
Open

Make reconcile error panels honor Grafana time range#63
swills wants to merge 1 commit into
cloudnative-pg:mainfrom
swills:reconcile-errors-fix

Conversation

@swills

@swills swills commented Jul 2, 2026

Copy link
Copy Markdown

This PR updates the cluster dashboard reconcile error panels so they honor the Grafana-selected time range instead of using raw cumulative counters.

This closes #62 and is related to #48

The dashboard currently uses raw controller_runtime_reconcile_total{result="error"} counters for several reconcile error panels. Examples:

clamp_max(max(controller_runtime_reconcile_total{namespace=~"$operatorNamespace", result="error", controller="backup"}), 1)

and:

max(controller_runtime_reconcile_total{namespace=~"$operatorNamespace", result="error", controller="backup"})

Since controller_runtime_reconcile_total is a monotonically increasing counter, these queries effectively answer:
"Has this controller ever seen a reconcile error since the metrics process started?" This makes the panels stay red/noisy long after transient issues have cleared.

This PR wraps those counters with increase(...[$__range]). For example:

clamp_max(max(increase(controller_runtime_reconcile_total{namespace=~"$operatorNamespace", result="error", controller="backup"}[$__range])), 1)

and:

max(increase(controller_runtime_reconcile_total{namespace=~"$operatorNamespace", result="error", controller="backup"}[$__range]))

Using increase(...[$__range]) makes the panel reflect reconcile errors that occurred within the currently selected Grafana time range. That means the panel semantics become:

"Did any reconcile errors occur during the selected dashboard time range?"

instead of:

"Did any reconcile errors ever occur since the operator process started?"

This works well for interactive dashboard use because it naturally follows the Grafana time picker:

  • 5m -> last 5 minutes
  • 1h -> last hour
  • 24h -> last day

This PR updates:

  • the top overview reconcile error panel queries
  • the related reconcile error stat/timeseries panel queries elsewhere in the dashboard

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reconcile errors panel

1 participant