fix: scope S3/minio cleanup to the current job's backup files#478
Open
alwynpan wants to merge 1 commit into
Open
fix: scope S3/minio cleanup to the current job's backup files#478alwynpan wants to merge 1 commit into
alwynpan wants to merge 1 commit into
Conversation
S3 cleanup deleted every object older than the retention window with no filename filter, risking deletion of other jobs' backups sharing the bucket/path. Scope deletion to this job's filename base, matching the filesystem and blobxfer branches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cleanup_old_data()deletes every object older than the retention window unders3://${bucket}/${s3_path}/with no filename filter:When multiple backup jobs (different databases/hosts) share one bucket + path prefix, or unrelated objects live under it, one job's cleanup deletes the other jobs' backups and any unrelated files older than its own
cleanup_time— cross-job data loss and incorrect retention.The
file/filesystemandblobxferbranches already scope deletion to the job's own files via-iname "${backup_job_filename_base}*"(and"${backup_job_global_base}*"for postgres globals). The S3/minio branch had no equivalent.Fix
Apply the same job-scoping to the S3/minio branch: only remove an object when its name matches this job's filename base (case-insensitive prefix), guarded against an empty base so it can't fall back to matching everything. Non-matching objects are logged at debug level instead of deleted.
This mirrors the matching behaviour of the existing filesystem/blobxfer branches, so retention semantics stay consistent across all three backup locations.
Test plan
bash -n install/assets/functions/10-db-backup— syntax OKValidation
Tested locally against a real S3 backend: a MinIO container with the actual (unmodified)
cleanup_old_data()function exercised over the liveaws s3 ls/rmAPI. The function was run inside a Linux container (amazon/aws-cli) so the GNUdate/awsenvironment matches production, and the function bodies were extracted from both the pre-fix commit and this fix to confirm the real code paths were exercised. Bucket seeded with two jobs' backups (pgsql_db1_myhost_*,pgsql_db2_myhost_*), a postgres globals file (pgsql_globals_myhost_*), and an unrelated object; retention set so every object is age-eligible, isolating the new prefix filter.pgsql_db1_myhostpgsql_db1_myhost, globals offpgsql_db1_myhost_*(2)db2, globals, unrelatedpgsql_db1_myhost, globals onpgsql_db1_myhost_*+ globalsdb2, unrelated