fix: WAL health panel false positive on idle databases#56
Open
christophebeling wants to merge 1 commit into
Open
fix: WAL health panel false positive on idle databases#56christophebeling wants to merge 1 commit into
christophebeling wants to merge 1 commit into
Conversation
christophebeling
force-pushed
the
fix/wal-health-idle-false-positive
branch
3 times, most recently
from
March 13, 2026 07:42
0d629ba to
64c4b7c
Compare
The WAL health stat panel (ID 612) shows "Unsynced" (red) on idle databases where no WAL activity is occurring, even though the archiver is functioning correctly. PostgreSQL's archive_timeout only forces WAL segment switches when there are records in the current segment. On idle databases (common in staging/dev), no new WAL segments are produced, causing seconds_since_last_archival to grow indefinitely and the panel to turn red. Fix: multiply the archival age by whether there are actually WAL files pending archival (ready > 0). When no WALs are pending, the result is 0 (Healthy) since the archiver has nothing to do. Fixes cloudnative-pg#55 Signed-off-by: Christoph Ebeling <c.ebeling@nexode.de>
christophebeling
force-pushed
the
fix/wal-health-idle-false-positive
branch
from
March 13, 2026 07:44
64c4b7c to
ec0d993
Compare
|
I have the same issue |
jrmclx
added a commit
to jrmclx/grafana-dashboards
that referenced
this pull request
May 20, 2026
fix: WAL health panel false positive on idle databases cloudnative-pg#56
jrmclx
added a commit
to jrmclx/grafana-dashboards
that referenced
this pull request
May 28, 2026
* Change Prometheus expression to use max function
Objective is to fix "First Recoverability Point" panel (id 237) returning "vector cannot contain metrics with the same labelset" error
The panel query was failing with the error "vector cannot contain metrics with the same labelset" when both metrics matched by __name__=~ share the same labelset, because Prometheus drops the __name__ label during vector evaluation, causing a collision.
The fix wraps the selector with max(), which is already used in the equivalent panel elsewhere in the same dashboard (panel id 364). This deduplicates the series and resolves the collision.
Before:
{__name__=~"cnpg_collector_first_recoverability_point|barman_cloud_cloudnative_pg_io_first_recoverability_point",namespace=~"$namespace",pod=~"$instances"}*1000 > 0
After:
max({__name__=~"cnpg_collector_first_recoverability_point|barman_cloud_cloudnative_pg_io_first_recoverability_point",namespace=~"$namespace",pod=~"$instances"})*1000 > 0
* Changed reconcile errors panel queries, added "cnpg-cloudnative-pg.+" to Operator Status regex that did not match operator pod.
* Added cnpg-cloudnative-pg.+ to Operator related regex queries
* Added fixed query as proposed in PR56
Fix: WAL health panel false positive on idle databases
cloudnative-pg#56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ready > 0guard so the panel only reports degraded status when there are actually WAL files pending archivalProblem
PostgreSQL's
archive_timeoutonly forces WAL segment switches when there are records in the current segment. On idle databases (common in staging/dev environments), no new WAL segments are produced, causingseconds_since_last_archivalto grow indefinitely. The panel's thresholds (>360s = Delayed, >900s = Unsynced) trigger false positives.Evidence from a staging environment with 3 CNPG clusters:
failed_count= 0,readyWALs = 0,rate(wal_bytes)= 0Fix
The query is multiplied by
cnpg_collector_pg_wal_archive_status{value="ready"} > bool 0:ready > 0(WALs pending): returns the actual archival age → thresholds work as beforeready == 0(nothing to archive): returns 0 → maps to "Healthy"Test plan
Fixes #55