Skip to content

feat: Add BLOBXFER_REMOTE_DOWNLOAD flag #475

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eoehen wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: Add BLOBXFER_REMOTE_DOWNLOAD flag #475

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
47d68ed
fix: Fix download blobxfer when BLOBXFER_REMOTE_PATH has folders
eoehen Apr 28, 2026
a3fbbad
feat: Add BLOBXFER_REMOTE_DOWNLOAD flag
eoehen Apr 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,10 +362,12 @@ If `DEFAULT_BACKUP_LOCATION` = `blobxfer` then the following options are used:.
| -------------------------------------- | ------------------------------------------------------------------- | ------------------- | ------- |
| `DEFAULT_BLOBXFER_STORAGE_ACCOUNT` | Microsoft Azure Cloud storage account name. | | x |
| `DEFAULT_BLOBXFER_STORAGE_ACCOUNT_KEY` | Microsoft Azure Cloud storage account key. | | x |
| `DEFAULT_BLOBXFER_REMOTE_DOWNLOAD` | Download remote path before upload to keep local/remote in sync | `TRUE` | x |
| `DEFAULT_BLOBXFER_REMOTE_PATH` | Remote Azure path | `/docker-db-backup` | x |
| `DEFAULT_BLOBXFER_MODE` | Azure Storage mode e.g. `auto`, `file`, `append`, `block` or `page` | `auto` | x |

- When `DEFAULT_BLOBXFER_MODE` is set to auto it will use blob containers by default. If the `DEFAULT_BLOBXFER_REMOTE_PATH` path does not exist a blob container with that name will be created.
- Set `DEFAULT_BLOBXFER_REMOTE_DOWNLOAD=FALSE` to skip downloading remote files before upload. Keeping this set to `TRUE` is recommended when you use cleanup sync behavior.

> This service uploads files from backup targed directory `DEFAULT_FILESYSTEM_PATH`.
> If the a cleanup configuration in `DEFAULT_CLEANUP_TIME` is defined, the remote directory on Azure storage will also be cleaned automatically.
Expand Down Expand Up @@ -645,10 +647,12 @@ If `DB01_BACKUP_LOCATION` = `blobxfer` then the following options are used:.
| -------------------------------------- | ------------------------------------------------------------------- | ------------------- | ------- |
| `DB01_BLOBXFER_STORAGE_ACCOUNT` | Microsoft Azure Cloud storage account name. | | x |
| `DB01_BLOBXFER_STORAGE_ACCOUNT_KEY` | Microsoft Azure Cloud storage account key. | | x |
| `DB01_BLOBXFER_REMOTE_DOWNLOAD` | Download remote path before upload to keep local/remote in sync | `TRUE` | x |
| `DB01_BLOBXFER_REMOTE_PATH` | Remote Azure path | `/docker-db-backup` | x |
| `DB01_BLOBXFER_REMOTE_MODE` | Azure Storage mode e.g. `auto`, `file`, `append`, `block` or `page` | `auto` | x |

- When `DEFAULT_BLOBXFER_MODE` is set to auto it will use blob containers by default. If the `DEFAULT_BLOBXFER_REMOTE_PATH` path does not exist a blob container with that name will be created.
- Set `DB01_BLOBXFER_REMOTE_DOWNLOAD=FALSE` to skip downloading remote files before upload. Keeping this set to `TRUE` is recommended when you use cleanup sync behavior.

> This service uploads files from backup directory `DB01_BACKUP_FILESYSTEM_PATH`.
> If the a cleanup configuration in `DB01_CLEANUP_TIME` is defined, the remote directory on Azure storage will also be cleaned automatically.
Expand Down
9 changes: 6 additions & 3 deletions examples/mssql-blobxfer/compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ services:
# image: tiredofit/db-backup
image: tiredofit/db-backup
links:
- example-mssql-s3-db
- example-mssql-s3-db
volumes:
- ./backups:/backup
- ./tmp/backups:/tmp/backups # shared tmp backup directory
#- ./post-script.sh:/assets/custom-scripts/post-script.sh
#- ./post-script.sh:/assets/custom-scripts/post-script.sh
environment:
- TIMEZONE=America/Vancouver
- CONTAINER_ENABLE_MONITORING=FALSE
Expand All @@ -58,7 +58,10 @@ services:
- DB01_BLOBXFER_STORAGE_ACCOUNT={TODO Add Storage Name}
# Add here azure storage account key
- DB01_BLOBXFER_STORAGE_ACCOUNT_KEY={TODO Add Key}
- DB01_BLOBXFER_REMOTE_PATH=docker-db-backup
# Keep TRUE to sync remote/local state before upload
- DB01_BLOBXFER_REMOTE_DOWNLOAD=TRUE
# Optional nested path, e.g. test/weekly (leading/trailing slashes are not required)
- DB01_BLOBXFER_REMOTE_PATH=backup01/docker-db-backup
restart: always
networks:
example-mssql-blobxfer-net:
Expand Down
1 change: 1 addition & 0 deletions install/assets/defaults/10-db-backup
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ DBBACKUP_GROUP=${DBBACKUP_GROUP:-"${DBBACKUP_USER}"} # Must go after DBBACKUP_US
DEFAULT_BACKUP_BEGIN=${DEFAULT_BACKUP_BEGIN:-+0}
DEFAULT_BACKUP_INTERVAL=${DEFAULT_BACKUP_INTERVAL:-1440}
DEFAULT_BACKUP_LOCATION=${DEFAULT_BACKUP_LOCATION:-"FILESYSTEM"}
DEFAULT_BLOBXFER_REMOTE_DOWNLOAD=${DEFAULT_BLOBXFER_REMOTE_DOWNLOAD:-"TRUE"}
DEFAULT_BLOBXFER_REMOTE_PATH=${DEFAULT_BLOBXFER_REMOTE_PATH:-"/docker-db-backup"}
DEFAULT_BLOBXFER_MODE=${DEFAULT_BLOBXFER_MODE:-"auto"}
DEFAULT_CHECKSUM=${DEFAULT_CHECKSUM:-"MD5"}
Expand Down
26 changes: 24 additions & 2 deletions install/assets/functions/10-db-backup
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ bootstrap_variables() {
DEFAULT_S3_CERT_CA_FILE \
DEFAULT_BLOBXFER_STORAGE_ACCOUNT \
DEFAULT_BLOBXFER_STORAGE_ACCOUNT_KEY \
DEFAULT_BLOBXFER_REMOTE_DOWNLOAD \
DEFAULT_BLOBXFER_REMOTE_PATH \
DEFAULT_BLOBXFER_MODE \
DB"${backup_instance_number}"_AUTH \
Expand Down Expand Up @@ -93,10 +94,12 @@ bootstrap_variables() {
DB"${backup_instance_number}"_S3_CERT_CA_FILE \
DB"${backup_instance_number}"_BLOBXFER_STORAGE_ACCOUNT \
DB"${backup_instance_number}"_BLOBXFER_STORAGE_ACCOUNT_KEY \
DB"${backup_instance_number}"_BLOBXFER_REMOTE_DOWNLOAD \
DB"${backup_instance_number}"_BLOBXFER_REMOTE_PATH \
DB"${backup_instance_number}"_BLOBXFER_MODE \
BLOBXFER_STORAGE_ACCOUNT \
BLOBXFER_STORAGE_ACCOUNT_KEY \
BLOBXFER_REMOTE_DOWNLOAD \
DB_HOST \
DB_NAME \
DB_PORT \
Expand Down Expand Up @@ -206,6 +209,7 @@ bootstrap_variables() {
transform_backup_instance_variable "${backup_instance_number}" BLOBXFER_REMOTE_PATH backup_job_blobxfer_remote_path
transform_backup_instance_variable "${backup_instance_number}" BLOBXFER_STORAGE_ACCOUNT backup_job_blobxfer_storage_account
transform_backup_instance_variable "${backup_instance_number}" BLOBXFER_STORAGE_ACCOUNT_KEY backup_job_blobxfer_storage_account_key
transform_backup_instance_variable "${backup_instance_number}" BLOBXFER_REMOTE_DOWNLOAD backup_job_blobxfer_remote_download
transform_backup_instance_variable "${backup_instance_number}" BLOBXFER_MODE backup_job_blobxfer_mode
transform_backup_instance_variable "${backup_instance_number}" CHECKSUM backup_job_checksum
transform_backup_instance_variable "${backup_instance_number}" CLEANUP_TIME backup_job_cleanup_time
Expand Down Expand Up @@ -1109,6 +1113,8 @@ cleanup_old_data() {

if [ -z "${backup_job_blobxfer_storage_account}" ] || [ -z "${backup_job_blobxfer_storage_account_key}" ]; then
write_log warn "Variable _BLOBXFER_STORAGE_ACCOUNT or _BLOBXFER_STORAGE_ACCOUNT_KEY is not set. Skipping blobxfer functions"
elif var_false "${backup_job_blobxfer_remote_download}"; then
write_log info "Skipping remote cleanup sync because _BLOBXFER_REMOTE_DOWNLOAD is disabled"
else
write_log info "Syncing changes via blobxfer"
silent run_as_user blobxfer upload --no-overwrite --mode ${backup_job_blobxfer_mode} --remote-path ${backup_job_blobxfer_remote_path} --storage-account ${backup_job_blobxfer_storage_account} --storage-account-key ${backup_job_blobxfer_storage_account_key} --local-path ${backup_job_filesystem_path} --delete --delete-only
Expand Down Expand Up @@ -1674,8 +1680,24 @@ EOF
if [ -z "${backup_job_blobxfer_storage_account}" ] || [ -z "${backup_job_blobxfer_storage_account_key}" ]; then
write_log warn "Variable _BLOBXFER_STORAGE_ACCOUNT or _BLOBXFER_STORAGE_ACCOUNT_KEY is not set. Skipping blobxfer functions"
else
write_log info "Synchronize local storage from blob container with blobxfer"
${play_fair} blobxfer download --no-overwrite --mode ${backup_job_blobxfer_mode} --remote-path ${backup_job_blobxfer_remote_path} --storage-account ${backup_job_blobxfer_storage_account} --storage-account-key ${backup_job_blobxfer_storage_account_key} --local-path ${backup_job_filesystem_path} --restore-file-lmt --delete
if var_true "${backup_job_blobxfer_remote_download}"; then
write_log info "Synchronize local storage from blob container with blobxfer"
${play_fair} blobxfer download --no-overwrite --mode ${backup_job_blobxfer_mode} --remote-path ${backup_job_blobxfer_remote_path} --storage-account ${backup_job_blobxfer_storage_account} --storage-account-key ${backup_job_blobxfer_storage_account_key} --local-path ${backup_job_filesystem_path} --restore-file-lmt --delete

# Avoid recursively re-uploading the downloaded remote path when a nested remote directory is configured.
blobxfer_remote_path_clean=$(echo "${backup_job_blobxfer_remote_path}" | sed -e 's#^/*##' -e 's#/*$##')
case "${blobxfer_remote_path_clean}" in
*/*)
blobxfer_local_mirror_path="${backup_job_filesystem_path}/${blobxfer_remote_path_clean}"
if [ -d "${blobxfer_local_mirror_path}" ]; then
write_log debug "Removing local blobxfer mirror path ${blobxfer_local_mirror_path} to prevent recursive nesting"
run_as_user rm -rf "${blobxfer_local_mirror_path}"
fi
;;
esac
else
write_log info "Skipping blobxfer remote download because _BLOBXFER_REMOTE_DOWNLOAD is disabled"
fi

write_log info "Moving backup to external storage with blobxfer"
mkdir -p "${backup_job_filesystem_path}"
Expand Down