Skip to content

Commit 1ec30a1

Browse files
mayankshah1607github-actions[bot]
mayankshah1607
and
github-actions[bot]
authored
K8SPG-771: Offline snapshots (#1418)
* wip: add fencing mechanism Signed-off-by: Mayank Shah <mayank.shah@percona.com> * rename to suspended-instances Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add BackupSnapshots feature gate Signed-off-by: Mayank Shah <mayank.shah@percona.com> * implement reconciler logic Signed-off-by: Mayank Shah <mayank.shah@percona.com> * implement offline executor Signed-off-by: Mayank Shah <mayank.shah@percona.com> * naming improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add to scheme Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix reconcile state Signed-off-by: Mayank Shah <mayank.shah@percona.com> * bug fixes and improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * naming improvements and fixes Signed-off-by: Mayank Shah <mayank.shah@percona.com> * ran make generate Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix PGBackup field validations Signed-off-by: Mayank Shah <mayank.shah@percona.com> * refactors and stability improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * update cr.yaml examples Signed-off-by: Mayank Shah <mayank.shah@percona.com> * organize imports Signed-off-by: Mayank Shah <mayank.shah@percona.com> * misspells Signed-off-by: Mayank Shah <mayank.shah@percona.com> * improve fencing logic Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add extra validation Signed-off-by: Mayank Shah <mayank.shah@percona.com> * improvements to suspended logic Signed-off-by: Mayank Shah <mayank.shah@percona.com> * finalizer renaming Signed-off-by: Mayank Shah <mayank.shah@percona.com> * remove enabled field Signed-off-by: Mayank Shah <mayank.shah@percona.com> * implement snapshot schedules Signed-off-by: Mayank Shah <mayank.shah@percona.com> * linting Signed-off-by: Mayank Shah <mayank.shah@percona.com> * implement in-place restore Signed-off-by: Mayank Shah <mayank.shah@percona.com> * update cr.yaml example Signed-off-by: Mayank Shah <mayank.shah@percona.com> * linting Signed-off-by: Mayank Shah <mayank.shah@percona.com> * no need to use configmap to track Signed-off-by: Mayank Shah <mayank.shah@percona.com> * more improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * typo Signed-off-by: Mayank Shah <mayank.shah@percona.com> * wip: PiTR Signed-off-by: Mayank Shah <mayank.shah@percona.com> * support snapshots in WALWatcher Signed-off-by: Mayank Shah <mayank.shah@percona.com> * bug fix Signed-off-by: Mayank Shah <mayank.shah@percona.com> * update comments Signed-off-by: Mayank Shah <mayank.shah@percona.com> * linting Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add PGO_FEATURE_GATES env variable to deploy Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add retry mechanism Signed-off-by: Mayank Shah <mayank.shah@percona.com> * implement checkpointing Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add restore_command wrapper for snapshots Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add logic for creating snapshot signal file Signed-off-by: Mayank Shah <mayank.shah@percona.com> * handle leader ep reconcile Signed-off-by: Mayank Shah <mayank.shah@percona.com> * more improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add e2e test Signed-off-by: Mayank Shah <mayank.shah@percona.com> * wip: fix pitr Signed-off-by: Mayank Shah <mayank.shah@percona.com> * in-place restore improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * update e2e test Signed-off-by: Mayank Shah <mayank.shah@percona.com> * update test runs Signed-off-by: Mayank Shah <mayank.shah@percona.com> * test fixes Signed-off-by: Mayank Shah <mayank.shah@percona.com> * Update build/postgres-operator/restore_command.sh Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * support for WAL and Tablespace volumes Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix tests Signed-off-by: Mayank Shah <mayank.shah@percona.com> * workaround improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * update examples Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix unit test Signed-off-by: Mayank Shah <mayank.shah@percona.com> * remove unused code Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add unit test shouldFailSnapshot Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add unit tests for backup helpers Signed-off-by: Mayank Shah <mayank.shah@percona.com> * linting Signed-off-by: Mayank Shah <mayank.shah@percona.com> * cleanup unused code Signed-off-by: Mayank Shah <mayank.shah@percona.com> * formatting Signed-off-by: Mayank Shah <mayank.shah@percona.com> * code cleanup Signed-off-by: Mayank Shah <mayank.shah@percona.com> * status improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add more unit tests Signed-off-by: Mayank Shah <mayank.shah@percona.com> * update e2e test assertions Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix inconsistencies & address copilot comments Signed-off-by: Mayank Shah <mayank.shah@percona.com> * remove sh prefix Signed-off-by: Mayank Shah <mayank.shah@percona.com> * checkpoint timeout default to 5m Signed-off-by: Mayank Shah <mayank.shah@percona.com> * linting Signed-off-by: Mayank Shah <mayank.shah@percona.com> * POSIX-compliant script Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix potential nil-ptr Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix error messag Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix invalid status update Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix retries to get latest objecgt Signed-off-by: Mayank Shah <mayank.shah@percona.com> * bugfix: restore reconcile can use incorrect volume spec Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add unit test Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix linting suggestions Signed-off-by: Mayank Shah <mayank.shah@percona.com> * prepare job should work when using separate wal volumes Signed-off-by: Mayank Shah <mayank.shah@percona.com> * update unit tests Signed-off-by: Mayank Shah <mayank.shah@percona.com> * spelling errors Signed-off-by: Mayank Shah <mayank.shah@percona.com> * allow configuring checkpoint timeout Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix restore wrapper Signed-off-by: Mayank Shah <mayank.shah@percona.com> * consistent field naming Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix retry loop Signed-off-by: Mayank Shah <mayank.shah@percona.com> * no need to check PITR while unsuspending Signed-off-by: Mayank Shah <mayank.shah@percona.com> * error message consistency Signed-off-by: Mayank Shah <mayank.shah@percona.com> * remove duplicate code Signed-off-by: Mayank Shah <mayank.shah@percona.com> * use fmt.Errorf Signed-off-by: Mayank Shah <mayank.shah@percona.com> * catch potential nil error Signed-off-by: Mayank Shah <mayank.shah@percona.com> * allow skipping checkpointing Signed-off-by: Mayank Shah <mayank.shah@percona.com> * address copilot comments Signed-off-by: Mayank Shah <mayank.shah@percona.com> * fix error message Signed-off-by: Mayank Shah <mayank.shah@percona.com> --------- Signed-off-by: Mayank Shah <mayank.shah@percona.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent f9b2aa7 commit 1ec30a1

80 files changed

Lines changed: 4815 additions & 193 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎build/crd/percona/generated/pgv2.percona.com_perconapgbackups.yaml‎

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,13 @@ spec:
7070
type: object
7171
spec:
7272
properties:
73+
method:
74+
default: pgbackrest
75+
description: Method with which to perform the backup
76+
enum:
77+
- pgbackrest
78+
- volumeSnapshot
79+
type: string
7380
options:
7481
description: |-
7582
Command line options to include when running the pgBackRest backup command.
@@ -80,14 +87,17 @@ spec:
8087
pgCluster:
8188
type: string
8289
repoName:
83-
description: The name of the pgBackRest repo to run the backup command
84-
against.
90+
description: |-
91+
The name of the pgBackRest repo to run the backup command against.
92+
This is required when method is 'pgbackrest'.
8593
pattern: ^repo[1-4]
8694
type: string
8795
required:
8896
- pgCluster
89-
- repoName
9097
type: object
98+
x-kubernetes-validations:
99+
- message: repoName is required when method is 'pgbackrest'
100+
rule: self.method == "volumeSnapshot" || has(self.repoName)
91101
status:
92102
properties:
93103
backupName:
@@ -391,6 +401,24 @@ spec:
391401
required:
392402
- name
393403
type: object
404+
snapshot:
405+
properties:
406+
dataVolumeSnapshotRef:
407+
description: Name of the VolumeSnapshot containing data volume
408+
contents.
409+
type: string
410+
tablespaceVolumeSnapshotRefs:
411+
additionalProperties:
412+
type: string
413+
description: |-
414+
Names of the VolumeSnapshots containing tablespace volume contents.
415+
Key is the name of the tablespace, value is the name of the VolumeSnapshot.
416+
type: object
417+
walVolumeSnapshotRef:
418+
description: Name of the VolumeSnapshot containing WAL volume
419+
contents.
420+
type: string
421+
type: object
394422
state:
395423
type: string
396424
storageType:

‎build/crd/percona/generated/pgv2.percona.com_perconapgclusters.yaml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7110,6 +7110,51 @@ spec:
71107110
trackLatestRestorableTime:
71117111
description: Enable tracking latest restorable time
71127112
type: boolean
7113+
volumeSnapshots:
7114+
description: VolumeSnapshots configuration
7115+
properties:
7116+
className:
7117+
description: Name of the VolumeSnapshotClass to use.
7118+
type: string
7119+
mode:
7120+
default: offline
7121+
description: Mode of the VolumeSnapshot.
7122+
enum:
7123+
- offline
7124+
type: string
7125+
offlineConfig:
7126+
description: |-
7127+
Configuration for offline snapshot operations.
7128+
Ignored if mode is not offline.
7129+
properties:
7130+
checkpoint:
7131+
description: Checkpoint configuration for offline snapshot
7132+
operations.
7133+
properties:
7134+
enabled:
7135+
default: true
7136+
description: If set, a checkpoint is requested.
7137+
type: boolean
7138+
timeoutSeconds:
7139+
default: 300
7140+
description: |-
7141+
Timeout for the checkpoint operation.
7142+
Ignored if checkpoint is not enabled.
7143+
format: int32
7144+
minimum: 30
7145+
type: integer
7146+
type: object
7147+
type: object
7148+
schedule:
7149+
description: |-
7150+
Defines the Cron schedule for a VolumeSnapshot.
7151+
Follows the standard Cron schedule syntax:
7152+
https://k8s.io/docs/concepts/workloads/controllers/cron-jobs/#cron-schedule-syntax
7153+
minLength: 6
7154+
type: string
7155+
required:
7156+
- className
7157+
type: object
71137158
type: object
71147159
x-kubernetes-validations:
71157160
- message: At least one repository must be configured when backups

‎build/crd/percona/generated/pgv2.percona.com_perconapgrestores.yaml‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,17 +67,33 @@ spec:
6767
pgCluster:
6868
description: The name of the PerconaPGCluster to perform restore.
6969
type: string
70+
x-kubernetes-validations:
71+
- message: pgCluster is an immutable field
72+
rule: self == oldSelf
7073
repoName:
7174
description: |-
7275
The name of the pgBackRest repo within the source PostgresCluster that contains the backups
7376
that should be utilized to perform a pgBackRest restore when initializing the data source
7477
for the new PostgresCluster.
7578
pattern: ^repo[1-4]
7679
type: string
80+
x-kubernetes-validations:
81+
- message: repoName is an immutable field
82+
rule: self == oldSelf
83+
volumeSnapshotBackupName:
84+
description: The name of the backup to perform in-place volume snapshot
85+
restores from.
86+
type: string
87+
x-kubernetes-validations:
88+
- message: volumeSnapshotBackupName is an immutable field
89+
rule: self == oldSelf
7790
required:
7891
- pgCluster
79-
- repoName
8092
type: object
93+
x-kubernetes-validations:
94+
- message: either repoName or volumeSnapshotBackupName must be set
95+
rule: ((has(self.repoName) && self.repoName != "") || (has(self.volumeSnapshotBackupName)
96+
&& self.volumeSnapshotBackupName != ""))
8197
status:
8298
properties:
8399
completed:

‎build/postgres-operator/Dockerfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ COPY build/postgres-operator/init-entrypoint.sh /usr/local/bin
6969
COPY build/postgres-operator/postgres-entrypoint.sh /usr/local/bin
7070
COPY build/postgres-operator/postgres-liveness-check.sh /usr/local/bin
7171
COPY build/postgres-operator/postgres-readiness-check.sh /usr/local/bin
72+
COPY build/postgres-operator/restore-command-wrapper.sh /usr/local/bin
7273
COPY hack/tools/queries /opt/crunchy/conf
7374

7475
RUN chgrp -R 0 /opt/crunchy/conf && chmod -R g=u opt/crunchy/conf

‎build/postgres-operator/init-entrypoint.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,4 @@ install -o "$(id -u)" -g "$(id -g)" -m 0755 -D "/usr/local/bin/postgres-entrypoi
1010
install -o "$(id -u)" -g "$(id -g)" -m 0755 -D "/usr/local/bin/postgres-liveness-check.sh" "${CRUNCHY_BINDIR}/bin/postgres-liveness-check.sh"
1111
install -o "$(id -u)" -g "$(id -g)" -m 0755 -D "/usr/local/bin/postgres-readiness-check.sh" "${CRUNCHY_BINDIR}/bin/postgres-readiness-check.sh"
1212
install -o "$(id -u)" -g "$(id -g)" -m 0755 -D "/usr/local/bin/relocate-extensions.sh" "${CRUNCHY_BINDIR}/bin/relocate-extensions.sh"
13+
install -o "$(id -u)" -g "$(id -g)" -m 0755 -D "/usr/local/bin/restore-command-wrapper.sh" "${CRUNCHY_BINDIR}/bin/restore-command-wrapper.sh"

‎build/postgres-operator/restore-command-wrapper.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/bin/sh
2+
set -e
3+
4+
# When this marker exists (e.g. after a snapshot restore), skip all WAL recovery by
5+
# exiting non-zero. Do not remove the file so every restore_command call is skipped.
6+
if [ -f "${PGDATA}/skip-wal-recovery" ]; then
7+
exit 1
8+
fi
9+
10+
exec "$@"

‎cmd/postgres-operator/main.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import (
1414
"time"
1515
"unicode"
1616

17+
volumesnapshotv1 "github.com/kubernetes-csi/external-snapshotter/client/v8/apis/volumesnapshot/v1"
1718
"github.com/pkg/errors"
1819
"go.opentelemetry.io/otel"
1920
uzap "go.uber.org/zap"
@@ -125,6 +126,8 @@ func main() {
125126
// Add Percona custom resource types to scheme
126127
assertNoError(v2.AddToScheme(mgr.GetScheme()))
127128

129+
assertNoError(volumesnapshotv1.AddToScheme(mgr.GetScheme()))
130+
128131
// add all PostgreSQL Operator controllers to the runtime manager
129132
err = addControllersToManager(ctx, mgr)
130133
assertNoError(err)

‎config/crd/bases/pgv2.percona.com_perconapgclusters.yaml‎

Lines changed: 93 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,13 @@ spec:
6969
type: object
7070
spec:
7171
properties:
72+
method:
73+
default: pgbackrest
74+
description: Method with which to perform the backup
75+
enum:
76+
- pgbackrest
77+
- volumeSnapshot
78+
type: string
7279
options:
7380
description: |-
7481
Command line options to include when running the pgBackRest backup command.
@@ -79,14 +86,17 @@ spec:
7986
pgCluster:
8087
type: string
8188
repoName:
82-
description: The name of the pgBackRest repo to run the backup command
83-
against.
89+
description: |-
90+
The name of the pgBackRest repo to run the backup command against.
91+
This is required when method is 'pgbackrest'.
8492
pattern: ^repo[1-4]
8593
type: string
8694
required:
8795
- pgCluster
88-
- repoName
8996
type: object
97+
x-kubernetes-validations:
98+
- message: repoName is required when method is 'pgbackrest'
99+
rule: self.method == "volumeSnapshot" || has(self.repoName)
90100
status:
91101
properties:
92102
backupName:
@@ -390,6 +400,24 @@ spec:
390400
required:
391401
- name
392402
type: object
403+
snapshot:
404+
properties:
405+
dataVolumeSnapshotRef:
406+
description: Name of the VolumeSnapshot containing data volume
407+
contents.
408+
type: string
409+
tablespaceVolumeSnapshotRefs:
410+
additionalProperties:
411+
type: string
412+
description: |-
413+
Names of the VolumeSnapshots containing tablespace volume contents.
414+
Key is the name of the tablespace, value is the name of the VolumeSnapshot.
415+
type: object
416+
walVolumeSnapshotRef:
417+
description: Name of the VolumeSnapshot containing WAL volume
418+
contents.
419+
type: string
420+
type: object
393421
state:
394422
type: string
395423
storageType:
@@ -7515,6 +7543,51 @@ spec:
75157543
trackLatestRestorableTime:
75167544
description: Enable tracking latest restorable time
75177545
type: boolean
7546+
volumeSnapshots:
7547+
description: VolumeSnapshots configuration
7548+
properties:
7549+
className:
7550+
description: Name of the VolumeSnapshotClass to use.
7551+
type: string
7552+
mode:
7553+
default: offline
7554+
description: Mode of the VolumeSnapshot.
7555+
enum:
7556+
- offline
7557+
type: string
7558+
offlineConfig:
7559+
description: |-
7560+
Configuration for offline snapshot operations.
7561+
Ignored if mode is not offline.
7562+
properties:
7563+
checkpoint:
7564+
description: Checkpoint configuration for offline snapshot
7565+
operations.
7566+
properties:
7567+
enabled:
7568+
default: true
7569+
description: If set, a checkpoint is requested.
7570+
type: boolean
7571+
timeoutSeconds:
7572+
default: 300
7573+
description: |-
7574+
Timeout for the checkpoint operation.
7575+
Ignored if checkpoint is not enabled.
7576+
format: int32
7577+
minimum: 30
7578+
type: integer
7579+
type: object
7580+
type: object
7581+
schedule:
7582+
description: |-
7583+
Defines the Cron schedule for a VolumeSnapshot.
7584+
Follows the standard Cron schedule syntax:
7585+
https://k8s.io/docs/concepts/workloads/controllers/cron-jobs/#cron-schedule-syntax
7586+
minLength: 6
7587+
type: string
7588+
required:
7589+
- className
7590+
type: object
75187591
type: object
75197592
x-kubernetes-validations:
75207593
- message: At least one repository must be configured when backups
@@ -21968,17 +22041,33 @@ spec:
2196822041
pgCluster:
2196922042
description: The name of the PerconaPGCluster to perform restore.
2197022043
type: string
22044+
x-kubernetes-validations:
22045+
- message: pgCluster is an immutable field
22046+
rule: self == oldSelf
2197122047
repoName:
2197222048
description: |-
2197322049
The name of the pgBackRest repo within the source PostgresCluster that contains the backups
2197422050
that should be utilized to perform a pgBackRest restore when initializing the data source
2197522051
for the new PostgresCluster.
2197622052
pattern: ^repo[1-4]
2197722053
type: string
22054+
x-kubernetes-validations:
22055+
- message: repoName is an immutable field
22056+
rule: self == oldSelf
22057+
volumeSnapshotBackupName:
22058+
description: The name of the backup to perform in-place volume snapshot
22059+
restores from.
22060+
type: string
22061+
x-kubernetes-validations:
22062+
- message: volumeSnapshotBackupName is an immutable field
22063+
rule: self == oldSelf
2197822064
required:
2197922065
- pgCluster
21980-
- repoName
2198122066
type: object
22067+
x-kubernetes-validations:
22068+
- message: either repoName or volumeSnapshotBackupName must be set
22069+
rule: ((has(self.repoName) && self.repoName != "") || (has(self.volumeSnapshotBackupName)
22070+
&& self.volumeSnapshotBackupName != ""))
2198222071
status:
2198322072
properties:
2198422073
completed:

‎config/manager/default/manager.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ spec:
3939
value: "1"
4040
- name: PPROF_BIND_ADDRESS
4141
value: "0"
42+
- name: PGO_FEATURE_GATES
43+
value: ""
4244
ports:
4345
- containerPort: 8080
4446
name: metrics

‎deploy/backup.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,6 @@ metadata:
55
spec:
66
pgCluster: cluster1
77
repoName: repo1
8+
# method: volumeSnapshot
89
# options:
910
# - --type=full

0 commit comments

Comments
 (0)