K8SPG-804: Fix internal.percona.com/delete-backup finalizer

[pooknull]

https://perconadev.atlassian.net/browse/K8SPG-804

DESCRIPTION

Problem:
On rare occasions, after executing the finalizer internal.percona.com/delete-backup, operator creates a second backup job for single pg-backup resource. After that, it is not possible to start new backups

Cause:
The internal.percona.com/keep-job finalizer tries to delete the postgres-operator.crunchydata.com/pgbackrest-backup annotation from the postgrescluster. Afterwards, it deletes the labels from the backup job so that they are not included in the repoResources.manualBackupJobs used in the reconcileManualBackup method.

If the job is included in reconcileManualBackup, it gets stuck at https://github.com/percona/percona-postgresql-operator/blob/02cb95c196a93ff03ce044[...]af1f1fe9f0505/internal/controller/postgrescluster/pgbackrest.go, because the finalizer internal.percona.com/keep-job is present in the backup job.
The reconcileManualBackup method creates a new backup job if there is no backup job with these labels and the postgres-operator.crunchydata.com/pgbackrest-backup annotation is specified.

However, on rare occasions, crunchy's reconcile process can have an outdated version of postgrescluster with this annotation specified when it shouldn't be, while having the latest state of the jobs. In this case, the method doesn't find any backup jobs with labels, but it does see the annotation and creates a new backup job which the operator doesn't control. It is expected that backup labels will be deleted from the backup job after each backup.

Solution:
Fetch the latest state of the postgrescluster cluster in the reconcileManualBackup method.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PG version?
• Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test Name	Result	Time
backup-enable-disable	passed	00:06:42
custom-extensions	passed	00:08:14
custom-tls	passed	00:05:16
demand-backup	passed	00:25:24
finalizers	passed	00:03:36
init-deploy	passed	00:02:59
monitoring	passed	00:07:33
monitoring-pmm3	passed	00:07:22
one-pod	passed	00:05:28
operator-self-healing	passed	00:07:43
pitr	passed	00:11:15
scaling	passed	00:04:43
scheduled-backup	passed	00:26:16
self-healing	passed	00:08:30
sidecars	passed	00:02:25
start-from-backup	passed	00:12:45
tablespaces	passed	00:07:18
telemetry-transfer	passed	00:03:11
upgrade-consistency	passed	00:05:21
upgrade-minor	passed	00:06:25
users	passed	00:03:34
We run 21 out of 21		02:52:10

commit: 6e5dec0
image: perconalab/percona-postgresql-operator:PR-1182-6e5dec01f