K8SPG-374: handle standby lag detection errors by pooknull · Pull Request #1462 · percona/percona-postgresql-operator

pooknull · 2026-02-26T12:24:33Z

https://perconadev.atlassian.net/browse/K8SPG-374

DESCRIPTION

This PR improves standby lag detection by handling 2 errors that can occur when the source cluster is paused.

If the primary pod cannot be identified during lag detection, the operator sets the following condition on the PerconaPGCluster resource:

 			Type:    postgrescluster.ConditionStandbyLagging,
 			Status:  metav1.ConditionUnknown,
 			Reason:  "ErrorGettingLag",
 			Message: "Cannot find primary for replication lag calculation",

If the lag detection query returns no rows/NULL (for example, when pg_stat_wal_receiver is empty), the operator sets the following condition on the PerconaPGCluster resource:

 			Type:    postgrescluster.ConditionStandbyLagging,
 			Status:  metav1.ConditionUnknown,
 			Reason:  "ErrorGettingLag",
 			Message: "Invalid output from lag query. The WAL receiver is probably not active",

Additionally, this PR moves the log message "Requeuing standby cluster for lag check" from INFO to DEBUG.

CHECKLIST

Jira

Is the Jira ticket created and referenced properly?
Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

Is an E2E test/test case added for the new feature/change?
Are unit tests added where appropriate?

Config/Logging/Testability

Are all needed new/changed options added to default YAML files?
Are all needed new/changed options added to the Helm Chart?
Did we add proper logging messages for operator actions?
Did we ensure compatibility with the previous version or cluster upgrade process?
Does the change support oldest and newest supported PG version?
Does the change support oldest and newest supported Kubernetes version?

https://perconadev.atlassian.net/browse/K8SPG-374

Copilot

Pull request overview

This pull request enhances error handling for standby lag detection in PostgreSQL cluster replication. It introduces sentinel errors and graceful error handling for transient conditions that can occur during cluster initialization or when replication is not yet established.

Changes:

Added sentinel errors ErrPrimaryPodNotFound and ErrInvalidLagQueryOutput for better error classification
Enhanced error handling in reconcileStandbyLag to set condition status to Unknown for recoverable error scenarios
Added empty string validation before parsing lag values from database queries
Reduced logging verbosity for periodic requeue operations
Removed unused fmt import from pgbackup controller

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
percona/controller/pgcluster/standby.go	Implements improved error handling for standby lag detection with sentinel errors, graceful handling of transient conditions, and empty string validation for query outputs
percona/controller/pgbackup/controller.go	Removes unused `fmt` import (cleanup)

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

gkech · 2026-03-02T09:49:58Z


 	lagBytes, err := r.getStandbyLag(ctx, cr)
 	if err != nil {
+		if errors.Is(err, ErrPrimaryPodNotFound) && cr.Status.State != v2.AppStateReady {


is it possible to have app state ready while the primary cannot be found?

It refers to the primary pod of the main site, not the standby

mayankshah1607 · 2026-03-02T11:46:09Z

+		if errors.Is(err, ErrPrimaryPodNotFound) && cr.Status.State != v2.AppStateReady {
+			meta.SetStatusCondition(&cr.Status.Conditions, metav1.Condition{
+				Type:    postgrescluster.ConditionStandbyLagging,
+				Status:  metav1.ConditionUnknown,
+				Reason:  "PrimaryNotFound",
+				Message: "Cannot find primary for replication lag calculation",
+			})
+			return nil
+		}
+		if errors.Is(err, ErrInvalidLagQueryOutput) {
+			meta.SetStatusCondition(&cr.Status.Conditions, metav1.Condition{
+				Type:    postgrescluster.ConditionStandbyLagging,
+				Status:  metav1.ConditionUnknown,
+				Reason:  "InvalidLagQueryOutput",
+				Message: "Invalid output from lag query. The WAL receiver is probably not active",
+			})
+			return nil
+		}


I think we should unconditionally add metav1.ConditionUnknown for this condition if we have a non-nil error, regardless of what the error actually is..

We can have a standard reason ErrorGettingLag, and the message can be the error string.. WDYT?

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Copilot · 2026-03-03T09:36:57Z

+var (
+	ErrPrimaryPodNotFound    = errors.New("primary pod not found")
+	ErrInvalidLagQueryOutput = errors.New("invalid lag query output")
+)


Problem: ErrPrimaryPodNotFound / ErrInvalidLagQueryOutput are exported but (currently) only used within this file.
Why it matters: Exporting expands the package API surface and makes future refactors harder.
Fix: Make these errors unexported (e.g., errPrimaryPodNotFound, errInvalidLagQueryOutput) unless they are intended to be referenced from other packages.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

You can also share your feedback on Copilot code review. Take the survey.

JNKPercona · 2026-03-04T08:56:21Z

Test Name	Result	Time
backup-enable-disable	passed	00:11:15
builtin-extensions	passed	00:04:51
cert-manager-tls	passed	00:05:05
custom-envs	passed	00:21:41
custom-extensions	passed	00:14:56
custom-tls	passed	00:06:40
database-init-sql	passed	00:02:10
demand-backup	passed	00:26:55
demand-backup-offline-snapshot	passed	00:17:56
dynamic-configuration	passed	00:05:04
finalizers	passed	00:04:41
init-deploy	passed	00:02:56
huge-pages	passed	00:02:57
monitoring	passed	00:07:41
monitoring-pmm3	passed	00:08:19
one-pod	passed	00:05:44
operator-self-healing	passed	00:10:38
pg-tde	passed	00:08:46
pitr	passed	00:12:05
scaling	passed	00:05:09
scheduled-backup	passed	00:30:17
self-healing	passed	00:09:04
sidecars	passed	00:02:43
standby-pgbackrest	passed	00:15:23
standby-streaming	passed	00:09:34
start-from-backup	passed	00:14:14
tablespaces	passed	00:08:11
telemetry-transfer	passed	00:06:17
upgrade-consistency	passed	00:05:50
upgrade-minor	passed	00:06:43
users	passed	00:04:31

Summary	Value
Tests Run	31/31
Job Duration	01:41:49
Total Test Time	04:58:30

commit: b00d26a
image: perconalab/percona-postgresql-operator:PR-1462-b00d26a70

K8SPG-374: handle standby lag detection errors

4e93cf0

https://perconadev.atlassian.net/browse/K8SPG-374

Copilot AI review requested due to automatic review settings February 26, 2026 12:24

Copilot started reviewing on behalf of pooknull February 26, 2026 12:25 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

Comment thread percona/controller/pgcluster/standby.go

Comment thread percona/controller/pgcluster/standby.go Outdated

Comment thread percona/controller/pgcluster/standby.go

Comment thread percona/controller/pgcluster/standby.go Outdated

pooknull added 2 commits February 26, 2026 15:57

Merge branch 'main' into K8SPG-374-fix

4777d26

add unit-test

ab5d6a7

Copilot AI review requested due to automatic review settings February 26, 2026 14:11

pooknull marked this pull request as ready for review February 26, 2026 14:11

pooknull requested review from egegunes, gkech, hors, mayankshah1607, nmarukovich and oksana-grishchenko as code owners February 26, 2026 14:11

Copilot started reviewing on behalf of pooknull February 26, 2026 14:12 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

Merge branch 'main' into K8SPG-374-fix

ba26542

egegunes previously approved these changes Feb 27, 2026

View reviewed changes

gkech reviewed Mar 2, 2026

View reviewed changes

mayankshah1607 requested changes Mar 2, 2026

View reviewed changes

egegunes added this to the v2.9.0 milestone Mar 3, 2026

always add condition on error

74d7974

Copilot AI review requested due to automatic review settings March 3, 2026 09:31

pooknull dismissed egegunes’s stale review via 74d7974 March 3, 2026 09:31

Merge branch 'main' into K8SPG-374-fix

3efc054

Copilot started reviewing on behalf of pooknull March 3, 2026 09:31 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

gkech previously approved these changes Mar 3, 2026

View reviewed changes

Merge branch 'main' into K8SPG-374-fix

dfb1050

egegunes previously approved these changes Mar 4, 2026

View reviewed changes

small fixes

b00d26a

Copilot AI review requested due to automatic review settings March 4, 2026 07:10

pooknull dismissed stale reviews from egegunes and gkech via b00d26a March 4, 2026 07:10

pooknull requested a review from gkech March 4, 2026 07:11

Copilot started reviewing on behalf of pooknull March 4, 2026 07:11 View session

pooknull requested review from egegunes and mayankshah1607 March 4, 2026 07:11

Copilot AI reviewed Mar 4, 2026

View reviewed changes

egegunes approved these changes Mar 4, 2026

View reviewed changes

mayankshah1607 approved these changes Mar 4, 2026

View reviewed changes

gkech approved these changes Mar 4, 2026

View reviewed changes

Merge branch 'main' into K8SPG-374-fix

4ad6157

hors merged commit 909fcd1 into main Mar 4, 2026
11 checks passed

hors deleted the K8SPG-374-fix branch March 4, 2026 13:20

Conversation

pooknull commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DESCRIPTION

CHECKLIST

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

gkech Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mayankshah1607 Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

mayankshah1607 Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

pooknull Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

JNKPercona commented Mar 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pooknull commented Feb 26, 2026 •

edited

Loading

gkech Mar 2, 2026 •

edited

Loading