Skip to content

Commit 987ee32

Browse files
mattlordmattlord
authored
Merge pull request #758 from planetscale/keyspace_level_backups
Backups: Add support for cluster and keyspace level backup schedules
2 parents cf1e45d + 6066c57 commit 987ee32

24 files changed

Lines changed: 6217 additions & 3080 deletions

‎.buildkite/pipeline.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,27 @@ steps:
5656
retry:
5757
<<: *retry_policy_tests
5858

59+
- name: "Backup Schedule Cluster/Keyspace Scope Test"
60+
command:
61+
- apk add --no-progress --quiet g++ make bash gcompat curl mysql-client libc6-compat
62+
- wget -q https://golang.org/dl/$GO_VERSION_FILE
63+
- tar -C /usr/local -xzf $GO_VERSION_FILE
64+
- export PATH=$PATH:/usr/local/go/bin:/bin
65+
- rm $GO_VERSION_FILE
66+
- ln -s /lib/libc.so.6 /usr/lib/libresolv.so.2
67+
- make backup-schedule-keyspace-test
68+
concurrency: 1
69+
concurrency_group: 'vtop/backup-schedule-keyspace-test'
70+
timeout_in_minutes: 20
71+
plugins:
72+
- docker#v3.12.0:
73+
image: "docker:latest"
74+
propagate-environment: true
75+
volumes:
76+
- "/var/run/docker.sock:/var/run/docker.sock"
77+
retry:
78+
<<: *retry_policy_tests
79+
5980
- name: "Backup Schedule Test"
6081
command:
6182
- apk add --no-progress --quiet g++ make bash gcompat curl mysql-client libc6-compat

‎Makefile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,10 @@ backup-schedule-test: e2e-test-setup
7373
echo "Running Backup-Schedule test"
7474
test/endtoend/backup_schedule_test.sh
7575

76+
backup-schedule-keyspace-test: e2e-test-setup
77+
echo "Running Backup-Schedule Keyspace-wide test"
78+
test/endtoend/backup_schedule_keyspace_test.sh
79+
7680
vtorc-vtadmin-test: e2e-test-setup
7781
echo "Running VTOrc and VtAdmin test"
7882
test/endtoend/vtorc_vtadmin_test.sh

‎deploy/crds/planetscale.com_vitessbackupschedules.yaml‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,8 @@ spec:
4848
format: int32
4949
minimum: 0
5050
type: integer
51+
frequency:
52+
type: string
5153
image:
5254
type: string
5355
imagePullPolicy:
@@ -115,13 +117,17 @@ spec:
115117
type: string
116118
name:
117119
type: string
120+
scope:
121+
enum:
122+
- Shard
123+
- Keyspace
124+
- Cluster
125+
type: string
118126
shard:
119127
example: '-'
120128
type: string
121129
required:
122-
- keyspace
123130
- name
124-
- shard
125131
type: object
126132
minItems: 1
127133
type: array
@@ -135,7 +141,6 @@ spec:
135141
- cluster
136142
- name
137143
- resources
138-
- schedule
139144
- strategies
140145
type: object
141146
status:
@@ -160,6 +165,10 @@ spec:
160165
type: object
161166
x-kubernetes-map-type: atomic
162167
type: array
168+
generatedSchedules:
169+
additionalProperties:
170+
type: string
171+
type: object
163172
lastScheduledTime:
164173
format: date-time
165174
type: string
@@ -168,6 +177,11 @@ spec:
168177
format: date-time
169178
type: string
170179
type: object
180+
nextScheduledTimes:
181+
additionalProperties:
182+
format: date-time
183+
type: string
184+
type: object
171185
type: object
172186
type: object
173187
served: true

‎deploy/crds/planetscale.com_vitessclusters.yaml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,8 @@ spec:
178178
format: int32
179179
minimum: 0
180180
type: integer
181+
frequency:
182+
type: string
181183
jobTimeoutMinute:
182184
default: 10
183185
format: int32
@@ -241,13 +243,17 @@ spec:
241243
type: string
242244
name:
243245
type: string
246+
scope:
247+
enum:
248+
- Shard
249+
- Keyspace
250+
- Cluster
251+
type: string
244252
shard:
245253
example: '-'
246254
type: string
247255
required:
248-
- keyspace
249256
- name
250-
- shard
251257
type: object
252258
minItems: 1
253259
type: array
@@ -260,7 +266,6 @@ spec:
260266
required:
261267
- name
262268
- resources
263-
- schedule
264269
- strategies
265270
type: object
266271
type: array

‎docs/api.md‎

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -647,6 +647,15 @@ SecretSource
647647
</tr>
648648
</tbody>
649649
</table>
650+
<h3 id="planetscale.com/v2.BackupScope">BackupScope
651+
(<code>string</code> alias)</p></h3>
652+
<p>
653+
(<em>Appears on:</em>
654+
<a href="#planetscale.com/v2.VitessBackupScheduleStrategy">VitessBackupScheduleStrategy</a>)
655+
</p>
656+
<p>
657+
<p>BackupScope defines the scope at which a backup strategy operates.</p>
658+
</p>
650659
<h3 id="planetscale.com/v2.CephBackupLocation">CephBackupLocation
651660
</h3>
652661
<p>
@@ -2685,6 +2694,35 @@ map[string]*k8s.io/apimachinery/pkg/apis/meta/v1.Time
26852694
Note that these are not the times when the last execution started, only the scheduled times.</p>
26862695
</td>
26872696
</tr>
2697+
<tr>
2698+
<td>
2699+
<code>generatedSchedules</code><br>
2700+
<em>
2701+
map[string]string
2702+
</em>
2703+
</td>
2704+
<td>
2705+
<em>(Optional)</em>
2706+
<p>GeneratedSchedules maps expanded strategy names to their generated cron expressions.
2707+
This is populated when Frequency is used instead of Schedule, providing observability
2708+
into the deterministic per-shard cron schedules.</p>
2709+
</td>
2710+
</tr>
2711+
<tr>
2712+
<td>
2713+
<code>nextScheduledTimes</code><br>
2714+
<em>
2715+
<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#*k8s.io/apimachinery/pkg/apis/meta/v1.time--">
2716+
map[string]*k8s.io/apimachinery/pkg/apis/meta/v1.Time
2717+
</a>
2718+
</em>
2719+
</td>
2720+
<td>
2721+
<em>(Optional)</em>
2722+
<p>NextScheduledTimes maps expanded strategy names to the next scheduled execution time.
2723+
This is populated for both cron-based and frequency-based schedules.</p>
2724+
</td>
2725+
</tr>
26882726
</tbody>
26892727
</table>
26902728
<h3 id="planetscale.com/v2.VitessBackupScheduleStrategy">VitessBackupScheduleStrategy
@@ -2719,13 +2757,30 @@ string
27192757
</tr>
27202758
<tr>
27212759
<td>
2760+
<code>scope</code><br>
2761+
<em>
2762+
<a href="#planetscale.com/v2.BackupScope">
2763+
BackupScope
2764+
</a>
2765+
</em>
2766+
</td>
2767+
<td>
2768+
<em>(Optional)</em>
2769+
<p>Scope defines whether this strategy targets a single Shard, all shards in a Keyspace,
2770+
or all shards in the Cluster. Defaults to &ldquo;Shard&rdquo; for backward compatibility.</p>
2771+
</td>
2772+
</tr>
2773+
<tr>
2774+
<td>
27222775
<code>keyspace</code><br>
27232776
<em>
27242777
string
27252778
</em>
27262779
</td>
27272780
<td>
2728-
<p>Keyspace defines the keyspace on which we want to take the backup.</p>
2781+
<em>(Optional)</em>
2782+
<p>Keyspace defines the keyspace on which we want to take the backup.
2783+
Required for Shard and Keyspace scopes.</p>
27292784
</td>
27302785
</tr>
27312786
<tr>
@@ -2736,7 +2791,9 @@ string
27362791
</em>
27372792
</td>
27382793
<td>
2739-
<p>Shard defines the shard on which we want to take a backup.</p>
2794+
<em>(Optional)</em>
2795+
<p>Shard defines the shard on which we want to take a backup.
2796+
Required only for Shard scope.</p>
27402797
</td>
27412798
</tr>
27422799
<tr>
@@ -2792,7 +2849,26 @@ string
27922849
</em>
27932850
</td>
27942851
<td>
2795-
<p>The schedule in Cron format, see <a href="https://en.wikipedia.org/wiki/Cron">https://en.wikipedia.org/wiki/Cron</a>.</p>
2852+
<em>(Optional)</em>
2853+
<p>The schedule in Cron format, see <a href="https://en.wikipedia.org/wiki/Cron">https://en.wikipedia.org/wiki/Cron</a>.
2854+
Mutually exclusive with Frequency.</p>
2855+
</td>
2856+
</tr>
2857+
<tr>
2858+
<td>
2859+
<code>frequency</code><br>
2860+
<em>
2861+
string
2862+
</em>
2863+
</td>
2864+
<td>
2865+
<em>(Optional)</em>
2866+
<p>Frequency is a Go duration string that defines how often backups should run.
2867+
Since schedules are executed via cron, only frequencies that can be represented exactly
2868+
in cron are supported. Examples include 1m, 5m, 15m, 30m, 1h, 2h, 4h, 6h, 12h, and 24h.
2869+
When set, the controller generates deterministic per-shard cron schedules staggered
2870+
across the interval to avoid bandwidth spikes.
2871+
Mutually exclusive with Schedule.</p>
27962872
</td>
27972873
</tr>
27982874
<tr>

‎docs/api/index.html‎

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -649,6 +649,15 @@ <h3 id="planetscale.com/v2.AzblobBackupLocation">AzblobBackupLocation
649649
</tr>
650650
</tbody>
651651
</table>
652+
<h3 id="planetscale.com/v2.BackupScope">BackupScope
653+
(<code>string</code> alias)</p></h3>
654+
<p>
655+
(<em>Appears on:</em>
656+
<a href="#planetscale.com/v2.VitessBackupScheduleStrategy">VitessBackupScheduleStrategy</a>)
657+
</p>
658+
<p>
659+
<p>BackupScope defines the scope at which a backup strategy operates.</p>
660+
</p>
652661
<h3 id="planetscale.com/v2.CephBackupLocation">CephBackupLocation
653662
</h3>
654663
<p>
@@ -2687,6 +2696,35 @@ <h3 id="planetscale.com/v2.VitessBackupScheduleStatus">VitessBackupScheduleStatu
26872696
Note that these are not the times when the last execution started, only the scheduled times.</p>
26882697
</td>
26892698
</tr>
2699+
<tr>
2700+
<td>
2701+
<code>generatedSchedules</code><br>
2702+
<em>
2703+
map[string]string
2704+
</em>
2705+
</td>
2706+
<td>
2707+
<em>(Optional)</em>
2708+
<p>GeneratedSchedules maps expanded strategy names to their generated cron expressions.
2709+
This is populated when Frequency is used instead of Schedule, providing observability
2710+
into the deterministic per-shard cron schedules.</p>
2711+
</td>
2712+
</tr>
2713+
<tr>
2714+
<td>
2715+
<code>nextScheduledTimes</code><br>
2716+
<em>
2717+
<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#*k8s.io/apimachinery/pkg/apis/meta/v1.time--">
2718+
map[string]*k8s.io/apimachinery/pkg/apis/meta/v1.Time
2719+
</a>
2720+
</em>
2721+
</td>
2722+
<td>
2723+
<em>(Optional)</em>
2724+
<p>NextScheduledTimes maps expanded strategy names to the next scheduled execution time.
2725+
This is populated for both cron-based and frequency-based schedules.</p>
2726+
</td>
2727+
</tr>
26902728
</tbody>
26912729
</table>
26922730
<h3 id="planetscale.com/v2.VitessBackupScheduleStrategy">VitessBackupScheduleStrategy
@@ -2721,13 +2759,30 @@ <h3 id="planetscale.com/v2.VitessBackupScheduleStrategy">VitessBackupScheduleStr
27212759
</tr>
27222760
<tr>
27232761
<td>
2762+
<code>scope</code><br>
2763+
<em>
2764+
<a href="#planetscale.com/v2.BackupScope">
2765+
BackupScope
2766+
</a>
2767+
</em>
2768+
</td>
2769+
<td>
2770+
<em>(Optional)</em>
2771+
<p>Scope defines whether this strategy targets a single Shard, all shards in a Keyspace,
2772+
or all shards in the Cluster. Defaults to &ldquo;Shard&rdquo; for backward compatibility.</p>
2773+
</td>
2774+
</tr>
2775+
<tr>
2776+
<td>
27242777
<code>keyspace</code><br>
27252778
<em>
27262779
string
27272780
</em>
27282781
</td>
27292782
<td>
2730-
<p>Keyspace defines the keyspace on which we want to take the backup.</p>
2783+
<em>(Optional)</em>
2784+
<p>Keyspace defines the keyspace on which we want to take the backup.
2785+
Required for Shard and Keyspace scopes.</p>
27312786
</td>
27322787
</tr>
27332788
<tr>
@@ -2738,7 +2793,9 @@ <h3 id="planetscale.com/v2.VitessBackupScheduleStrategy">VitessBackupScheduleStr
27382793
</em>
27392794
</td>
27402795
<td>
2741-
<p>Shard defines the shard on which we want to take a backup.</p>
2796+
<em>(Optional)</em>
2797+
<p>Shard defines the shard on which we want to take a backup.
2798+
Required only for Shard scope.</p>
27422799
</td>
27432800
</tr>
27442801
<tr>
@@ -2794,7 +2851,26 @@ <h3 id="planetscale.com/v2.VitessBackupScheduleTemplate">VitessBackupScheduleTem
27942851
</em>
27952852
</td>
27962853
<td>
2797-
<p>The schedule in Cron format, see <a href="https://en.wikipedia.org/wiki/Cron">https://en.wikipedia.org/wiki/Cron</a>.</p>
2854+
<em>(Optional)</em>
2855+
<p>The schedule in Cron format, see <a href="https://en.wikipedia.org/wiki/Cron">https://en.wikipedia.org/wiki/Cron</a>.
2856+
Mutually exclusive with Frequency.</p>
2857+
</td>
2858+
</tr>
2859+
<tr>
2860+
<td>
2861+
<code>frequency</code><br>
2862+
<em>
2863+
string
2864+
</em>
2865+
</td>
2866+
<td>
2867+
<em>(Optional)</em>
2868+
<p>Frequency is a Go duration string that defines how often backups should run.
2869+
Since schedules are executed via cron, only frequencies that can be represented exactly
2870+
in cron are supported. Examples include 1m, 5m, 15m, 30m, 1h, 2h, 4h, 6h, 12h, and 24h.
2871+
When set, the controller generates deterministic per-shard cron schedules staggered
2872+
across the interval to avoid bandwidth spikes.
2873+
Mutually exclusive with Schedule.</p>
27982874
</td>
27992875
</tr>
28002876
<tr>

0 commit comments

Comments
 (0)