Skip to content

Commit 28be00a

Browse files
anth-volkclaude
anth-volk
and
claude
committed
Automate Artifact Registry image cleanup + purge dead repos
The image repos accumulated ~628 GiB that nothing prunes (~$45/mo storage). Ongoing (in the deploy pipeline): - cleanup_app_engine_versions.sh: when it deletes an App Engine version, also delete that version's gae-flexible Flex image, so the repo stays in lockstep with the retained versions. Version-aware because a blind keep-N would delete a kept stopped-prod version's image (staging/prod images interleave). - cleanup_cloud_run_images.sh + a cleanup-cloud-run-images job: keep the 15 most recent Cloud Run images, delete older. Sorts in-script by createTime since gcloud's Artifact Registry --sort-by proved unreliable. One-time: - purge_stale_artifact_registry.sh (interactive): after a safety pre-check, delete the dead legacy repos (us.gcr.io ~540 GiB, gcr.io, cloud-run-source- deploy), prune gae-flexible orphans, and prune Cloud Run to 15. Tests extended with a stubbed gcloud artifacts. Cuts AR storage to a few $/mo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 6c661ae commit 28be00a

6 files changed

Lines changed: 317 additions & 1 deletion

File tree

‎.github/scripts/cleanup_app_engine_versions.sh‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,11 @@
1818
# staging, and legacy timestamp-named versions), to stay under App Engine's
1919
# 210-versions-per-service limit. Prod is kept deeper because only prod
2020
# versions are meaningful rollback targets.
21+
# 3. DELETEs the Artifact Registry image for each deleted version so the
22+
# `gae-flexible` repo stays in lockstep with the retained versions (App
23+
# Engine Flex builds one image per version — `<service>.<version-id>` — and
24+
# nothing else prunes them, so the repo grows without bound). Best-effort
25+
# per image; set CLEANUP_APP_ENGINE_IMAGES=0 to skip.
2126
#
2227
# Stopping (not deleting) preserves rollback: a stopped version can be brought
2328
# back with `gcloud app versions start <v>` then `gcloud app services set-traffic
@@ -37,6 +42,13 @@ KEEP_STOPPED_PROD="${KEEP_STOPPED_PROD:-10}"
3742
KEEP_STOPPED_STAGING="${KEEP_STOPPED_STAGING:-3}"
3843
DRY_RUN="${DRY_RUN:-0}"
3944

45+
# Delete each deleted version's App Engine Flex image from Artifact Registry so
46+
# the image repo does not grow one image per version forever.
47+
CLEANUP_APP_ENGINE_IMAGES="${CLEANUP_APP_ENGINE_IMAGES:-1}"
48+
AR_LOCATION="${AR_LOCATION:-us-central1}"
49+
AR_IMAGE_PROJECT="${AR_IMAGE_PROJECT:-${APP_ENGINE_PROJECT:-policyengine-api}}"
50+
AR_IMAGE_REPO="${AR_IMAGE_REPO:-gae-flexible}"
51+
4052
common_args=("--service=${APP_ENGINE_SERVICE}")
4153
if [[ -n "${APP_ENGINE_PROJECT:-}" ]]; then
4254
common_args+=("--project=${APP_ENGINE_PROJECT}")
@@ -156,3 +168,23 @@ if [[ "${#to_delete[@]}" -gt 0 ]]; then
156168
gcloud app versions delete "${to_delete[@]}" "${common_args[@]}" --quiet
157169
fi
158170
fi
171+
172+
# Delete each deleted version's App Engine Flex image so the image repo stays in
173+
# lockstep with the retained versions. Best-effort per image: a version may
174+
# predate this repo (older images live in the legacy us.gcr.io) or its image may
175+
# already be gone, so individual misses are ignored.
176+
if [[ "${CLEANUP_APP_ENGINE_IMAGES}" == "1" && "${#to_delete[@]}" -gt 0 ]]; then
177+
image_repo="${AR_LOCATION}-docker.pkg.dev/${AR_IMAGE_PROJECT}/${AR_IMAGE_REPO}"
178+
img_done=0
179+
for v in "${to_delete[@]}"; do
180+
[[ -z "${v}" ]] && continue
181+
image="${image_repo}/${APP_ENGINE_SERVICE}.${v}"
182+
if [[ "${DRY_RUN}" == "1" ]]; then
183+
echo "[dry-run] would delete image ${image}"
184+
elif gcloud artifacts docker images delete "${image}" --delete-tags --quiet \
185+
>/dev/null 2>&1; then
186+
img_done=$((img_done + 1))
187+
fi
188+
done
189+
[[ "${DRY_RUN}" == "1" ]] || echo "Deleted ${img_done}/${#to_delete[@]} gae-flexible image(s)."
190+
fi

‎.github/scripts/cleanup_cloud_run_images.sh‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#!/usr/bin/env bash
2+
3+
# Keep only the most recent Cloud Run images in Artifact Registry.
4+
#
5+
# The Cloud Run deploy pushes a new image on every release and nothing prunes
6+
# them, so the repo grows without bound. This keeps the newest KEEP image
7+
# versions (by createTime) and deletes the rest.
8+
#
9+
# Note: deleting an image that an older Cloud Run *revision* still references
10+
# stops that revision from starting again. Cloud Run here is the non-primary
11+
# migration candidate, so losing rollback to revisions older than KEEP deploys
12+
# is acceptable; KEEP=15 leaves a generous window.
13+
#
14+
# Images are sorted in-script (by createTime) rather than trusting gcloud's
15+
# Artifact Registry `--sort-by`, which has proven unreliable. Set DRY_RUN=1 to
16+
# print the plan without changing anything. Written for bash 3.2.
17+
18+
set -euo pipefail
19+
20+
KEEP="${KEEP:-15}"
21+
DRY_RUN="${DRY_RUN:-0}"
22+
AR_LOCATION="${AR_LOCATION:-us-central1}"
23+
AR_PROJECT="${AR_PROJECT:-${CLOUD_RUN_PROJECT:-policyengine-api}}"
24+
AR_REPO="${AR_REPO:-${CLOUD_RUN_ARTIFACT_REPOSITORY:-policyengine-api}}"
25+
26+
image_repo="${AR_LOCATION}-docker.pkg.dev/${AR_PROJECT}/${AR_REPO}"
27+
28+
# Deletable refs (<image>@<digest>), newest first — sorted here by createTime
29+
# (ISO timestamps sort chronologically), one row per image version.
30+
refs=()
31+
while IFS='|' read -r _ctime pkg ver; do
32+
[[ -n "${ver}" ]] && refs+=("${pkg}@${ver}")
33+
done < <(
34+
gcloud artifacts docker images list "${image_repo}" \
35+
--format="value[separator='|'](createTime,package,version)" \
36+
| sort -r
37+
)
38+
39+
total="${#refs[@]}"
40+
to_delete=()
41+
if [[ "${total}" -gt "${KEEP}" ]]; then
42+
to_delete=("${refs[@]:${KEEP}}")
43+
fi
44+
45+
echo "Repo: ${image_repo}"
46+
echo "Images: ${total} (keeping newest ${KEEP})"
47+
echo "Deleting: ${#to_delete[@]}"
48+
49+
deleted=0
50+
for ref in "${to_delete[@]:-}"; do
51+
[[ -z "${ref}" ]] && continue
52+
if [[ "${DRY_RUN}" == "1" ]]; then
53+
echo "[dry-run] would delete ${ref}"
54+
elif gcloud artifacts docker images delete "${ref}" --delete-tags --quiet \
55+
>/dev/null 2>&1; then
56+
deleted=$((deleted + 1))
57+
fi
58+
done
59+
[[ "${DRY_RUN}" == "1" ]] || echo "Deleted ${deleted}/${#to_delete[@]} image(s)."

‎.github/scripts/purge_stale_artifact_registry.sh‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
#!/usr/bin/env bash
2+
3+
# One-time purge of stale Artifact Registry storage in policyengine-api.
4+
#
5+
# The image repos accumulated ~628 GiB that is never cleaned up. This removes
6+
# the dead legacy repos and prunes the live repos to their retention windows.
7+
# INTERACTIVE: prints what it will do and asks before each destructive step.
8+
# Safe to re-run (idempotent).
9+
#
10+
# 1. Pre-check: abort if any App Engine version's image lives in a repo we
11+
# are about to delete.
12+
# 2. Delete dead repos wholesale: us.gcr.io, gcr.io, cloud-run-source-deploy.
13+
# 3. Prune gae-flexible: delete images whose App Engine version no longer
14+
# exists (orphans left by already-deleted versions).
15+
# 4. Prune the Cloud Run repo to the newest 15 (cleanup_cloud_run_images.sh).
16+
#
17+
# Usage: PROJECT=policyengine-api bash .github/scripts/purge_stale_artifact_registry.sh
18+
#
19+
# Written for bash 3.2.
20+
21+
set -euo pipefail
22+
23+
PROJECT="${PROJECT:-policyengine-api}"
24+
LOCATION="${LOCATION:-us-central1}"
25+
SERVICE="${SERVICE:-default}"
26+
GAE_REPO="gae-flexible"
27+
DEAD_REPOS=("us.gcr.io" "gcr.io" "cloud-run-source-deploy")
28+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
29+
30+
confirm() {
31+
local reply
32+
read -r -p "$1 (y/N) " reply
33+
[[ "${reply}" =~ ^[Yy]$ ]]
34+
}
35+
36+
echo "=== Purge stale Artifact Registry storage in ${PROJECT} ==="
37+
echo ""
38+
39+
# --- 1. Pre-check: dead repos are not referenced by any live version ---------
40+
echo "[1/4] Checking no App Engine version references the repos to be deleted..."
41+
version_images="$(gcloud app versions list --project="${PROJECT}" --service="${SERVICE}" \
42+
--format="value(deployment.container.image)" 2>/dev/null || true)"
43+
for repo in "${DEAD_REPOS[@]}"; do
44+
if printf '%s\n' "${version_images}" | grep -q "/${repo}/"; then
45+
echo "ABORT: an App Engine version still references '${repo}'. Not safe." >&2
46+
exit 1
47+
fi
48+
done
49+
echo " OK — no live version references ${DEAD_REPOS[*]}."
50+
echo ""
51+
52+
# --- 2. Delete the dead repos wholesale --------------------------------------
53+
for repo in "${DEAD_REPOS[@]}"; do
54+
if ! gcloud artifacts repositories describe "${repo}" --location="${LOCATION}" \
55+
--project="${PROJECT}" >/dev/null 2>&1; then
56+
echo "[2/4] Repo '${repo}' not found (already gone) — skipping."
57+
continue
58+
fi
59+
size="$(gcloud artifacts repositories describe "${repo}" --location="${LOCATION}" \
60+
--project="${PROJECT}" --format="value(sizeBytes.size())" 2>/dev/null || echo '?')"
61+
echo "[2/4] Dead repo '${repo}' (${size})."
62+
if confirm " Delete repository '${repo}' entirely?"; then
63+
gcloud artifacts repositories delete "${repo}" --location="${LOCATION}" \
64+
--project="${PROJECT}" --quiet
65+
echo " deleted ${repo}."
66+
else
67+
echo " skipped ${repo}."
68+
fi
69+
done
70+
echo ""
71+
72+
# --- 3. Prune gae-flexible orphans (images whose version is gone) ------------
73+
echo "[3/4] Pruning ${GAE_REPO} images whose App Engine version no longer exists..."
74+
existing_versions="$(gcloud app versions list --project="${PROJECT}" --service="${SERVICE}" \
75+
--format="value(id)" 2>/dev/null || true)"
76+
orphans=()
77+
while IFS= read -r pkg; do
78+
[[ -z "${pkg}" ]] && continue
79+
base="${pkg##*/}" # "<service>.<version-id>"
80+
[[ "${base}" == "${SERVICE}."* ]] || continue
81+
vid="${base#"${SERVICE}."}"
82+
printf '%s\n' "${existing_versions}" | grep -qxF "${vid}" || orphans+=("${pkg}")
83+
done < <(
84+
gcloud artifacts docker images list \
85+
"${LOCATION}-docker.pkg.dev/${PROJECT}/${GAE_REPO}" \
86+
--format="value(package)" 2>/dev/null | sort -u
87+
)
88+
echo " ${#orphans[@]} orphaned image(s) (version deleted)."
89+
if [[ "${#orphans[@]}" -gt 0 ]] && confirm " Delete ${#orphans[@]} orphaned ${GAE_REPO} image(s)?"; then
90+
for pkg in "${orphans[@]}"; do
91+
if gcloud artifacts docker images delete "${pkg}" --delete-tags --quiet >/dev/null 2>&1; then
92+
echo " deleted ${pkg##*/}"
93+
else
94+
echo " skip ${pkg##*/}"
95+
fi
96+
done
97+
fi
98+
echo ""
99+
100+
# --- 4. Prune the Cloud Run repo to the newest 15 ----------------------------
101+
echo "[4/4] Pruning the Cloud Run image repo to the newest 15..."
102+
if confirm " Run cleanup_cloud_run_images.sh (keep 15)?"; then
103+
CLOUD_RUN_PROJECT="${PROJECT}" bash "${SCRIPT_DIR}/cleanup_cloud_run_images.sh"
104+
fi
105+
echo ""
106+
echo "Done."

‎.github/workflows/push.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -599,6 +599,33 @@ jobs:
599599
- name: Wait for Cloud Run production service health
600600
run: bash .github/scripts/health_check.sh "${{ steps.cloud_run_service_url.outputs.url }}/readiness-check"
601601

602+
cleanup-cloud-run-images:
603+
name: Clean up old Cloud Run images
604+
runs-on: ubuntu-latest
605+
needs: deploy-cloud-run-candidate
606+
if: |
607+
(github.repository == 'PolicyEngine/policyengine-api')
608+
&& (github.event.head_commit.message == 'Update PolicyEngine API')
609+
environment: production
610+
permissions:
611+
contents: read
612+
id-token: write
613+
steps:
614+
- name: Checkout repo
615+
uses: actions/checkout@v4
616+
- name: GCP authentication
617+
uses: "google-github-actions/auth@v2"
618+
with:
619+
workload_identity_provider: "${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}"
620+
service_account: "${{ secrets.GCP_DEPLOY_SERVICE_ACCOUNT }}"
621+
- name: Set up GCloud
622+
uses: "google-github-actions/setup-gcloud@v2"
623+
# The Cloud Run deploy pushes a new image every release and nothing prunes
624+
# them, so the repo grows without bound. Keep the 15 most recent image
625+
# versions (default in the script) and delete older ones.
626+
- name: Prune old Cloud Run images
627+
run: bash .github/scripts/cleanup_cloud_run_images.sh
628+
602629
docker:
603630
name: Docker
604631
runs-on: ubuntu-latest

‎changelog.d/artifact-registry-image-cleanup.changed.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Prune Artifact Registry images automatically so the image repos stop growing without bound. After each deploy the pipeline keeps the 15 most recent Cloud Run images and deletes each removed App Engine version's Flex (`gae-flexible`) image. Adds a one-time `purge_stale_artifact_registry.sh` to delete the dead legacy image repos (`us.gcr.io`, `gcr.io`, `cloud-run-source-deploy`) and orphaned `gae-flexible` images. This cuts Artifact Registry storage (~628 GiB, ~$45/mo) to a few dollars.

‎tests/unit/test_app_engine_cleanup_scripts.py‎

Lines changed: 92 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
REPO = Path(__file__).resolve().parents[2]
1919
CLEANUP_SCRIPT = ".github/scripts/cleanup_app_engine_versions.sh"
2020
STOP_SCRIPT = ".github/scripts/stop_app_engine_version.sh"
21+
CR_IMAGES_SCRIPT = ".github/scripts/cleanup_cloud_run_images.sh"
2122

2223
# A stub `gcloud` that answers `app versions list` from MOCK_* env vars (each a
2324
# newline-joined, newest-first id list) and appends any stop/delete calls to
@@ -28,6 +29,8 @@
2829
case "$a" in --filter=*) filter="${a#--filter=}";; esac
2930
done
3031
case "$*" in
32+
*"artifacts docker images list"*) [ -n "${MOCK_AR_IMAGES:-}" ] && printf '%s\\n' "$MOCK_AR_IMAGES";;
33+
*"artifacts docker images delete"*) echo "IMG_DELETE $*" >> "$GCLOUD_CALLS";;
3134
*"versions list"*)
3235
case "$filter" in
3336
*"traffic_split>0"*) [ -n "${MOCK_TRAFFIC:-}" ] && printf '%s\\n' "$MOCK_TRAFFIC";;
@@ -48,6 +51,7 @@ def _run(
4851
serving: list[str] | None = None,
4952
traffic: list[str] | None = None,
5053
stopped: list[str] | None = None,
54+
ar_images: list[str] | None = None,
5155
extra_env: dict[str, str] | None = None,
5256
) -> tuple[subprocess.CompletedProcess[str], Path]:
5357
stub_dir = tmp_path / "bin"
@@ -68,6 +72,7 @@ def _run(
6872
"MOCK_SERVING": "\n".join(serving or []),
6973
"MOCK_TRAFFIC": "\n".join(traffic or []),
7074
"MOCK_STOPPED": "\n".join(stopped or []),
75+
"MOCK_AR_IMAGES": "\n".join(ar_images or []),
7176
}
7277
env.update(extra_env or {})
7378

@@ -94,13 +99,99 @@ def _list_after(label: str, stdout: str) -> list[str]:
9499

95100

96101
def test_cleanup_scripts_are_shell_syntax_valid():
97-
for script in (CLEANUP_SCRIPT, STOP_SCRIPT):
102+
for script in (CLEANUP_SCRIPT, STOP_SCRIPT, CR_IMAGES_SCRIPT):
98103
result = subprocess.run(
99104
["bash", "-n", script], cwd=REPO, capture_output=True, text=True
100105
)
101106
assert result.returncode == 0, f"{script}: {result.stderr}"
102107

103108

109+
# --- cloud run image cleanup (keep 15) -----------------------------------
110+
111+
_CR_PKG = (
112+
"us-central1-docker.pkg.dev/policyengine-api/policyengine-api/policyengine-api"
113+
)
114+
115+
116+
def _ar_rows(digests: list[str], package: str = _CR_PKG) -> list[str]:
117+
"""`createTime|package|version` rows, oldest first (day 01 = oldest). The
118+
script sorts by createTime descending itself, so order here is irrelevant."""
119+
return [
120+
f"2026-01-{i + 1:02d}T00:00:00|{package}|sha256:{d}"
121+
for i, d in enumerate(digests)
122+
]
123+
124+
125+
def test_cloud_run_images_keeps_newest_15(tmp_path):
126+
digests = [f"d{i:02d}" for i in range(18)] # d00 oldest ... d17 newest
127+
result, calls = _run(CR_IMAGES_SCRIPT, tmp_path, ar_images=_ar_rows(digests))
128+
assert result.returncode == 0, result.stderr
129+
assert "Images: 18 (keeping newest 15)" in result.stdout
130+
assert "Deleting: 3" in result.stdout
131+
# The 3 oldest digests are the ones deleted; the newest is kept.
132+
for old in ("sha256:d00", "sha256:d01", "sha256:d02"):
133+
assert f"would delete {_CR_PKG}@{old}" in result.stdout
134+
assert "sha256:d17" not in result.stdout.split("Deleting:")[1]
135+
assert calls.read_text() == "" # DRY_RUN performs no mutations
136+
137+
138+
def test_cloud_run_images_no_delete_when_within_keep(tmp_path):
139+
result, _ = _run(
140+
CR_IMAGES_SCRIPT, tmp_path, ar_images=_ar_rows([f"d{i}" for i in range(10)])
141+
)
142+
assert result.returncode == 0, result.stderr
143+
assert "Deleting: 0" in result.stdout
144+
assert "would delete" not in result.stdout
145+
146+
147+
def test_cloud_run_images_respects_keep_override(tmp_path):
148+
result, _ = _run(
149+
CR_IMAGES_SCRIPT,
150+
tmp_path,
151+
ar_images=_ar_rows([f"d{i}" for i in range(10)]),
152+
extra_env={"KEEP": "5"},
153+
)
154+
assert "keeping newest 5" in result.stdout
155+
assert "Deleting: 5" in result.stdout
156+
157+
158+
# --- app engine cleanup also prunes each deleted version's image ----------
159+
160+
161+
def test_app_engine_cleanup_deletes_gae_flexible_image_per_deleted_version(tmp_path):
162+
result, _ = _run(
163+
CLEANUP_SCRIPT,
164+
tmp_path,
165+
serving=["prod-110", "prod-108"],
166+
traffic=["prod-110"],
167+
stopped=[f"prod-{n}" for n in range(106, 84, -2)] # 11 prod -> 1 deleted
168+
+ ["staging-90", "staging-88", "staging-86", "staging-84"], # 1 deleted
169+
)
170+
assert result.returncode == 0, result.stderr
171+
deleted = _list_after("Deleting", result.stdout)
172+
assert deleted, "expected some versions to be deleted"
173+
# AR_IMAGE_PROJECT defaults to APP_ENGINE_PROJECT, which _run sets.
174+
repo = "us-central1-docker.pkg.dev/test-project/gae-flexible/default."
175+
# Every deleted version has a matching gae-flexible image deletion line.
176+
for v in deleted:
177+
assert f"would delete image {repo}{v}" in result.stdout
178+
# No image deletion for a kept (live/warm) version.
179+
assert f"{repo}prod-110" not in result.stdout
180+
181+
182+
def test_app_engine_cleanup_image_pruning_can_be_disabled(tmp_path):
183+
result, _ = _run(
184+
CLEANUP_SCRIPT,
185+
tmp_path,
186+
serving=["prod-110", "prod-108"],
187+
traffic=["prod-110"],
188+
stopped=[f"prod-{n}" for n in range(106, 84, -2)],
189+
extra_env={"CLEANUP_APP_ENGINE_IMAGES": "0"},
190+
)
191+
assert result.returncode == 0, result.stderr
192+
assert "would delete image" not in result.stdout
193+
194+
104195
# --- cleanup: stop selection ---------------------------------------------
105196

106197

0 commit comments

Comments
 (0)