K8SPXC-1732 Change ordering of operations in backup scripts to begin listening for SST earlier

[NotAndD]

Change ordering of operations in backup scripts to begin listening for SST earlier

Problem:

K8SPXC-1732

When Istio in ambient mode is deployed in the cluster and Percona pods are inside its service mesh, backups fail as the SST transfer is sometimes randomly lost, leaving the backup pod stuck running forever, or always failing because the first socat receives the snapshot files instead of the sst file (and the second socat remains hanging forever).

I've opened an issue here on Github as I had some issues creating a Jira ticket (not sure why).

Cause:

Honestly, a bit unclear. Since connections are wrapped by ztunnel, I suspect something breaks an assumption made by the donor and somehow the sst transfer is marked as okay, even if it fails. This was not happening with older version of Percona (1.14.0) so I think it's something on garbd / donor side.

Solution:

I've refactored the backup scripts so that they work in a different way:

• socat listening starts before garbd, this way the pod is ready to await the sst.
• also, garbd is closed without errors when sst transfer is done, and then the snapshot is awaited by the main script in a separate way.
• to obtain this different ordering, there are small changes here and there.

I've tested my solution in some of our test clusters and it looks it is working (for s3 storage at least). But I am searching for inputs as I am unsure if this is the right approach, or whatever. I think this could be an improvement on the scripts, but unsure if I missed something obvious in changing them.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?
• Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PXC version?
• Does the change support oldest and newest supported Kubernetes version?

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ hors
❌ NotAndD
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[egegunes]

@NotAndD thank you for your contribution. We'll start working on next PXC operator release soon, we'll review and test your changes as part of release.

[NotAndD]

@egegunes thanks a lot! Please let me know if there's anything wrong, or even if you prefer a different solution.

Like I wrote at the beginning, I am mostly searching for guidance here since I wasn't able to fully understand the issue, so my changes to the scripts are somehow done without full knowledge and more based on practical tests in our test environments.

[gkech]

hello @NotAndD, given that the changes on the scripts are substantial, we believe that we should not only review this PR but also test it a little bit more thoroughly.

[JNKPercona]

Test Name	Result	Time
auto-tuning-8-0	passed	00:19:21
allocator-8-0	passed	00:12:45
backup-storage-tls-8-0	passed	00:22:19
cross-site-8-0	passed	00:35:39
custom-users-8-0	passed	00:13:09
demand-backup-cloud-8-0	passed	00:59:17
demand-backup-cloud-pxb-8-0	passed	00:52:47
demand-backup-encrypted-with-tls-8-0	passed	00:46:00
demand-backup-encrypted-with-tls-8-4	passed	00:46:36
demand-backup-encrypted-with-tls-pxb-8-0	passed	00:16:26
demand-backup-encrypted-with-tls-pxb-8-4	passed	00:18:09
demand-backup-8-0	passed	00:43:43
demand-backup-flow-control-8-0	passed	00:11:21
demand-backup-parallel-8-0	passed	00:09:09
demand-backup-without-passwords-8-0	passed	00:16:15
extra-pvc-8-0	passed	00:24:10
haproxy-5-7	passed	00:14:40
haproxy-8-0	passed	00:14:10
init-deploy-5-7	passed	00:16:34
init-deploy-8-0	passed	00:16:58
limits-8-0	passed	00:11:57
monitoring-2-0-8-0	passed	00:22:02
monitoring-pmm3-8-0	passed	00:17:51
one-pod-5-7	passed	00:13:52
one-pod-8-0	passed	00:13:51
pitr-8-0	passed	00:43:00
pitr-pxb-8-0	failure	00:38:19
pitr-gap-errors-8-0	passed	00:55:57
proxy-protocol-8-0	passed	00:09:46
proxy-switch-8-0	passed	00:13:24
proxysql-sidecar-res-limits-8-0	passed	00:08:42
proxysql-scheduler-8-0	passed	00:22:02
pvc-resize-5-7	passed	00:17:00
pvc-resize-8-0	passed	00:16:03
recreate-8-0	passed	00:17:38
restore-to-encrypted-cluster-8-0	passed	00:25:58
restore-to-encrypted-cluster-8-4	passed	00:27:05
restore-to-encrypted-cluster-pxb-8-0	passed	00:16:24
scaling-proxysql-8-0	passed	00:08:59
scaling-8-0	passed	00:11:04
scheduled-backup-5-7	passed	01:06:36
scheduled-backup-8-0	passed	01:04:25
security-context-8-0	passed	00:26:25
smart-update1-8-0	passed	00:33:16
smart-update2-8-0	passed	00:38:55
storage-8-0	passed	00:10:48
tls-issue-cert-manager-ref-8-0	passed	00:08:32
tls-issue-cert-manager-8-0	passed	00:10:01
tls-issue-self-8-0	passed	00:13:22
upgrade-consistency-8-0	passed	00:10:59
upgrade-haproxy-5-7	passed	00:23:59
upgrade-haproxy-8-0	passed	00:24:53
upgrade-proxysql-5-7	passed	00:13:52
upgrade-proxysql-8-0	passed	00:14:33
users-5-7	passed	00:28:22
users-8-0	failure	00:06:53
validation-hook-8-0	passed	00:02:11

Summary	Value
Tests Run	57/57
Job Duration	03:19:28
Total Test Time	21:58:49

commit: deef583
image: perconalab/percona-xtradb-cluster-operator:PR-2221-deef583d

[egegunes]

@NotAndD I was not able to test this as part of v1.19.0, I'm sorry. I'll test and review as part of v1.20.0.

[NotAndD]

@egegunes no worries! If we update to 1.19.0 in our test environments (not sure) will test again without this change and then with this change.

Also, if you have doubts or whatever on this change please let me know!

[egegunes]

There will be some improvements in backup scripts in v1.19.0. Also v1.19.0 is going to introduce a new design for backups (as tech preview) that doesn't need SST, see K8SPXC-1576. That might help you as well.

[NotAndD]

Ah, interesting! So if I got it right, the backup is now done with a go utility in a sidecar of the actual database Pods, instead of spawning a job which joins the galera cluster with the arbitrator?

Will try to test this out when it gets released then, if I find the time xD