Skip to content

Commit 28e527f

Browse files
committed
[#1895] Added fresh DB download support for Acquia.
1 parent 109a0ad commit 28e527f

6 files changed

Lines changed: 177 additions & 14 deletions

File tree

.ahoy.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,10 +120,10 @@ commands:
120120

121121
#;< !PROVISION_TYPE_PROFILE
122122
download-db:
123-
usage: Download database. Run with "--no-cache" option to force fresh database backup.
123+
usage: Download database. Run with "--fresh" option to force fresh database backup.
124124
aliases: [fetch-db]
125125
cmd: |
126-
case " $* " in *" --no-cache "*) export VORTEX_DB_DOWNLOAD_NO_CACHE=1;; esac
126+
case " $* " in *" --fresh "*) export VORTEX_DB_DOWNLOAD_FRESH=1;; esac
127127
./scripts/vortex/download-db.sh
128128
#;> !PROVISION_TYPE_PROFILE
129129

.vortex/docs/content/workflows/development.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ To fetch the database with the latest data from the production environment,
4747
use the `ahoy fetch-db` command, which will download the latest database
4848
dump from the production environment into a `.data` directory. If there was a
4949
download attempt on the same day (locally or from CI), it will download the
50-
cached database dump. Use `ahoy fetch-db --no-cache` to create a new database
50+
cached database dump. Use `ahoy fetch-db --fresh` to create a new database
5151
dump regardless of the cache.
5252

5353
:::note

.vortex/docs/content/workflows/variables.mdx

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -586,6 +586,22 @@ Default value: `./.data`
586586

587587
Defined or used in: `.env`, `scripts/vortex/download-db-acquia.sh`, `scripts/vortex/download-db-container-registry.sh`, `scripts/vortex/download-db-ftp.sh`, `scripts/vortex/download-db-lagoon.sh`, `scripts/vortex/download-db-url.sh`, `scripts/vortex/download-db.sh`, `scripts/vortex/provision.sh`
588588

589+
### `VORTEX_DB_DOWNLOAD_ACQUIA_BACKUP_MAX_WAIT`
590+
591+
Maximum time in seconds to wait for backup completion.
592+
593+
Default value: `600`
594+
595+
Defined or used in: `scripts/vortex/download-db-acquia.sh`
596+
597+
### `VORTEX_DB_DOWNLOAD_ACQUIA_BACKUP_WAIT_INTERVAL`
598+
599+
Interval in seconds to wait between backup status checks.
600+
601+
Default value: `10`
602+
603+
Defined or used in: `scripts/vortex/download-db-acquia.sh`
604+
589605
### `VORTEX_DB_DOWNLOAD_ACQUIA_DB_NAME`
590606

591607
Acquia database name to download the database from.
@@ -612,6 +628,14 @@ Default value: `1`
612628

613629
Defined or used in: `.env.local.example`, `scripts/vortex/download-db.sh`
614630

631+
### `VORTEX_DB_DOWNLOAD_FRESH`
632+
633+
Flag to download a fresh copy of the database by triggering a new backup.
634+
635+
Default value: `UNDEFINED`
636+
637+
Defined or used in: `scripts/vortex/download-db-acquia.sh`, `scripts/vortex/download-db-lagoon.sh`
638+
615639
### `VORTEX_DB_DOWNLOAD_FTP_FILE`
616640

617641
Database dump FTP file name.
@@ -702,14 +726,6 @@ Default value: `${LAGOON_PROJECT}-${VORTEX_DB_DOWNLOAD_ENVIRONMENT}`
702726

703727
Defined or used in: `scripts/vortex/download-db-lagoon.sh`
704728

705-
### `VORTEX_DB_DOWNLOAD_NO_CACHE`
706-
707-
Flag to download a fresh copy of the database.
708-
709-
Default value: `UNDEFINED`
710-
711-
Defined or used in: `scripts/vortex/download-db-lagoon.sh`
712-
713729
### `VORTEX_DB_DOWNLOAD_PROCEED`
714730

715731
Proceed with download.

.vortex/tests/bats/unit/download-db-acquia.bats

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -445,3 +445,84 @@ load ../_helper.bash
445445

446446
popd >/dev/null
447447
}
448+
449+
@test "download-db-acquia: Create fresh backup when requested" {
450+
pushd "${LOCAL_REPO_DIR}" >/dev/null || exit 1
451+
452+
# Clean up any existing test files
453+
rm -rf .data
454+
mkdir -p .data
455+
456+
# Create .env.local with the fresh flag
457+
echo "VORTEX_DB_DOWNLOAD_FRESH=1" >.env.local
458+
459+
declare -a STEPS=(
460+
"[INFO] Started database dump download from Acquia."
461+
462+
# Authentication
463+
"[TASK] Retrieving authentication token."
464+
'@curl -s -L https://accounts.acquia.com/api/auth/oauth/token --data-urlencode client_id=test-key --data-urlencode client_secret=test-secret --data-urlencode grant_type=client_credentials # {"access_token":"test-token","expires_in":3600}'
465+
466+
# Application UUID
467+
"[TASK] Retrieving testapp application UUID."
468+
'@curl -s -L -H Accept: application/json, version=2 -H Authorization: Bearer test-token https://cloud.acquia.com/api/applications?filter=name%3Dtestapp # {"_embedded":{"items":[{"uuid":"app-uuid-123","name":"testapp"}]}}'
469+
470+
# Environment ID
471+
"[TASK] Retrieving prod environment ID."
472+
'@curl -s -L -H Accept: application/json, version=2 -H Authorization: Bearer test-token https://cloud.acquia.com/api/applications/app-uuid-123/environments?filter=name%3Dprod # {"_embedded":{"items":[{"id":"env-id-456","name":"prod"}]}}'
473+
474+
# Create backup
475+
"[TASK] Creating new database backup for testdb."
476+
'@curl -s -L -X POST -H Accept: application/json, version=2 -H Authorization: Bearer test-token https://cloud.acquia.com/api/environments/env-id-456/databases/testdb/backups # {"_links":{"notification":{"href":"https://cloud.acquia.com/api/notifications/notification-uuid-123"}}}'
477+
478+
# Wait for backup - mock status checks
479+
"[TASK] Waiting for backup to complete."
480+
'@sleep 10 # 0'
481+
'@curl -s -L -H Accept: application/json, version=2 -H Authorization: Bearer test-token https://cloud.acquia.com/api/notifications/notification-uuid-123 # {"status":"in-progress"}'
482+
" Backup in progress (10s elapsed)..."
483+
'@sleep 10 # 0'
484+
'@curl -s -L -H Accept: application/json, version=2 -H Authorization: Bearer test-token https://cloud.acquia.com/api/notifications/notification-uuid-123 # {"status":"completed"}'
485+
"[ OK ] Backup completed successfully."
486+
" Fresh backup will be downloaded."
487+
488+
# Continue with normal download flow
489+
"[TASK] Discovering latest backup ID for DB testdb."
490+
'@curl --progress-bar -L -H Accept: application/json, version=2 -H Authorization: Bearer test-token https://cloud.acquia.com/api/environments/env-id-456/databases/testdb/backups?sort=created # {"_embedded":{"items":[{"id":"backup-id-new-123","completed":"2024-01-02T00:00:00+00:00"}]}}'
491+
492+
# Rest of download steps...
493+
"[TASK] Discovering backup URL."
494+
'@curl --progress-bar -L -H Accept: application/json, version=2 -H Authorization: Bearer test-token https://cloud.acquia.com/api/environments/env-id-456/databases/testdb/backups/backup-id-new-123/actions/download # {"url":"https://backup.example.com/db-fresh.sql.gz"}'
495+
496+
"[TASK] Downloading DB dump into file .data/testdb_backup_backup-id-new-123.sql.gz."
497+
'@curl --progress-bar -L -H Accept: application/json, version=2 -H Authorization: Bearer test-token https://backup.example.com/db-fresh.sql.gz -o .data/testdb_backup_backup-id-new-123.sql.gz # 0 # # echo "CREATE TABLE fresh (id INT);" | gzip > .data/testdb_backup_backup-id-new-123.sql.gz'
498+
499+
"[TASK] Expanding DB file .data/testdb_backup_backup-id-new-123.sql.gz into .data/testdb_backup_backup-id-new-123.sql."
500+
"@gunzip -t .data/testdb_backup_backup-id-new-123.sql.gz # 0"
501+
"@gunzip -c .data/testdb_backup_backup-id-new-123.sql.gz # 0 # CREATE TABLE fresh (id INT);"
502+
503+
'[TASK] Renaming file ".data/testdb_backup_backup-id-new-123.sql" to ".data/db.sql".'
504+
'@mv .data/testdb_backup_backup-id-new-123.sql .data/db.sql # 0 # # echo "CREATE TABLE fresh (id INT);" > .data/db.sql'
505+
506+
"[ OK ] Finished database dump download from Acquia."
507+
)
508+
509+
export VORTEX_ACQUIA_KEY="test-key"
510+
export VORTEX_ACQUIA_SECRET="test-secret"
511+
export VORTEX_ACQUIA_APP_NAME="testapp"
512+
export VORTEX_DB_DOWNLOAD_ENVIRONMENT="prod"
513+
export VORTEX_DB_DOWNLOAD_ACQUIA_DB_NAME="testdb"
514+
export VORTEX_DB_DIR=".data"
515+
export VORTEX_DB_FILE="db.sql"
516+
export VORTEX_DB_DOWNLOAD_FRESH="1"
517+
518+
mocks="$(run_steps "setup")"
519+
run scripts/vortex/download-db-acquia.sh
520+
run_steps "assert" "${mocks}"
521+
522+
assert_success
523+
assert_file_exists ".data/db.sql"
524+
assert_file_contains ".data/db.sql" "CREATE TABLE fresh"
525+
526+
rm -rf .data
527+
popd >/dev/null
528+
}

scripts/vortex/download-db-acquia.sh

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,15 @@ VORTEX_DB_DIR="${VORTEX_DB_DIR:-./.data}"
4646
# Database dump file name.
4747
VORTEX_DB_FILE="${VORTEX_DB_FILE:-db.sql}"
4848

49+
# Flag to download a fresh copy of the database by triggering a new backup.
50+
VORTEX_DB_DOWNLOAD_FRESH="${VORTEX_DB_DOWNLOAD_FRESH:-}"
51+
52+
# Interval in seconds to wait between backup status checks.
53+
VORTEX_DB_DOWNLOAD_ACQUIA_BACKUP_WAIT_INTERVAL="${VORTEX_DB_DOWNLOAD_ACQUIA_BACKUP_WAIT_INTERVAL:-10}"
54+
55+
# Maximum time in seconds to wait for backup completion.
56+
VORTEX_DB_DOWNLOAD_ACQUIA_BACKUP_MAX_WAIT="${VORTEX_DB_DOWNLOAD_ACQUIA_BACKUP_MAX_WAIT:-600}"
57+
4958
#-------------------------------------------------------------------------------
5059

5160
# @formatter:off
@@ -130,6 +139,63 @@ env_id=$(echo "${envs_json}" | extract_json_value "_embedded" | extract_json_val
130139
[ "${VORTEX_DEBUG-}" = "1" ] && note "Extracted environment ID: ${env_id}"
131140
[ -z "${env_id}" ] && fail "Unable to retrieve an environment ID for '${VORTEX_DB_DOWNLOAD_ENVIRONMENT}'. API response: ${envs_json}" && exit 1
132141

142+
# If fresh backup requested, create a new backup and wait for it to complete.
143+
if [ "$VORTEX_DB_DOWNLOAD_FRESH" == "1" ]; then
144+
task "Creating new database backup for ${VORTEX_DB_DOWNLOAD_ACQUIA_DB_NAME}."
145+
146+
# Trigger backup creation
147+
create_backup_json=$(curl -s -L -X POST -H 'Accept: application/json, version=2' -H "Authorization: Bearer ${token}" "https://cloud.acquia.com/api/environments/${env_id}/databases/${VORTEX_DB_DOWNLOAD_ACQUIA_DB_NAME}/backups")
148+
[ "${VORTEX_DEBUG-}" = "1" ] && note "Create backup API response: ${create_backup_json}"
149+
150+
# Check for errors
151+
if echo "${create_backup_json}" | grep -q '"error"'; then
152+
fail "Failed to create backup for database '${VORTEX_DB_DOWNLOAD_ACQUIA_DB_NAME}'. API response: ${create_backup_json}"
153+
exit 1
154+
fi
155+
156+
# Extract notification URL for status checking
157+
notification_url=$(echo "${create_backup_json}" | extract_json_value "_links" | extract_json_value "notification" | extract_json_value "href")
158+
[ "${VORTEX_DEBUG-}" = "1" ] && note "Notification URL: ${notification_url}"
159+
160+
if [ -z "${notification_url}" ]; then
161+
fail "Unable to get notification URL for backup creation. API response: ${create_backup_json}"
162+
exit 1
163+
fi
164+
165+
task "Waiting for backup to complete."
166+
max_wait="${VORTEX_DB_DOWNLOAD_ACQUIA_BACKUP_MAX_WAIT}"
167+
wait_interval="${VORTEX_DB_DOWNLOAD_ACQUIA_BACKUP_WAIT_INTERVAL}"
168+
elapsed=0
169+
170+
while [ ${elapsed} -lt "${max_wait}" ]; do
171+
sleep "${wait_interval}"
172+
elapsed=$((elapsed + wait_interval))
173+
174+
# Check backup status
175+
status_json=$(curl -s -L -H 'Accept: application/json, version=2' -H "Authorization: Bearer ${token}" "${notification_url}")
176+
[ "${VORTEX_DEBUG-}" = "1" ] && note "Status check (${elapsed}s): ${status_json}"
177+
178+
status=$(echo "${status_json}" | extract_json_value "status")
179+
180+
if [ "${status}" = "completed" ]; then
181+
pass "Backup completed successfully."
182+
break
183+
elif [ "${status}" = "failed" ]; then
184+
fail "Backup creation failed. API response: ${status_json}"
185+
exit 1
186+
fi
187+
188+
note "Backup in progress (${elapsed}s elapsed)..."
189+
done
190+
191+
if [ ${elapsed} -ge "${max_wait}" ]; then
192+
fail "Backup creation timed out after ${max_wait} seconds."
193+
exit 1
194+
fi
195+
196+
note "Fresh backup will be downloaded."
197+
fi
198+
133199
task "Discovering latest backup ID for DB ${VORTEX_DB_DOWNLOAD_ACQUIA_DB_NAME}."
134200
backups_json=$(curl --progress-bar -L -H 'Accept: application/json, version=2' -H "Authorization: Bearer ${token}" "https://cloud.acquia.com/api/environments/${env_id}/databases/${VORTEX_DB_DOWNLOAD_ACQUIA_DB_NAME}/backups?sort=created")
135201
[ "${VORTEX_DEBUG-}" = "1" ] && note "Backups API response: ${backups_json}"

scripts/vortex/download-db-lagoon.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ set -eu
2323
[ "${VORTEX_DEBUG-}" = "1" ] && set -x
2424

2525
# Flag to download a fresh copy of the database.
26-
VORTEX_DB_DOWNLOAD_NO_CACHE="${VORTEX_DB_DOWNLOAD_NO_CACHE:-}"
26+
VORTEX_DB_DOWNLOAD_FRESH="${VORTEX_DB_DOWNLOAD_FRESH:-}"
2727

2828
# Lagoon project name.
2929
LAGOON_PROJECT="${LAGOON_PROJECT:?Missing required environment variable LAGOON_PROJECT.}"
@@ -100,7 +100,7 @@ if [ ! -d "${VORTEX_DB_DIR}" ]; then
100100
mkdir -p "${VORTEX_DB_DIR}"
101101
fi
102102

103-
if [ "$VORTEX_DB_DOWNLOAD_NO_CACHE" == "1" ]; then
103+
if [ "$VORTEX_DB_DOWNLOAD_FRESH" == "1" ]; then
104104
note "Database dump refresh requested. Will create a new dump."
105105
fi
106106

@@ -115,7 +115,7 @@ task "Discovering or creating a database dump on Lagoon."
115115
ssh \
116116
"${ssh_opts[@]}" \
117117
"${VORTEX_DB_DOWNLOAD_LAGOON_SSH_USER}@${VORTEX_DB_DOWNLOAD_LAGOON_SSH_HOST}" service=cli container=cli \
118-
"if [ ! -f \"${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_DIR}/${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_FILE}\" ] || [ \"${VORTEX_DB_DOWNLOAD_NO_CACHE}\" == \"1\" ] ; then \
118+
"if [ ! -f \"${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_DIR}/${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_FILE}\" ] || [ \"${VORTEX_DB_DOWNLOAD_FRESH}\" == \"1\" ] ; then \
119119
[ -n \"${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_FILE_CLEANUP}\" ] && rm -f \"${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_DIR}\"\/${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_FILE_CLEANUP} && echo \"Removed previously created DB dumps.\"; \
120120
echo \" > Creating a database dump ${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_DIR}/${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_FILE}.\"; \
121121
/app/vendor/bin/drush --root=./${WEBROOT} sql:dump --structure-tables-key=common --structure-tables-list=ban,event_log_track,flood,login_security_track,purge_queue,queue,webform_submission,webform_submission_data,webform_submission_log,watchdog,cache* --extra-dump='--disable-ssl --no-tablespaces' > \"${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_DIR}/${VORTEX_DB_DOWNLOAD_LAGOON_REMOTE_FILE}\"; \

0 commit comments

Comments
 (0)