Skip to content

Commit e523f8a

Browse files
abtreeceabtreece
committed
feat: add EOL package archival and pruning tooling
Add incremental APT/YUM archive migration scripts that move EOL distro packages to archive buckets, and pruning scripts that remove EOL Ruby version packages from still-supported distro repos. Scripts auto-detect EOL content by comparing repo state against config.yml and support --dry-run. Migration scripts are incremental — safe to re-run when future distros reach EOL. Includes dev-handbook runbook and updated architecture docs.
1 parent 190dcf4 commit e523f8a

9 files changed

Lines changed: 1738 additions & 14 deletions

‎dev-handbook/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040

4141
* [Adding support for a new distribution](add-new-distro.md)
4242
* [Adding support for a new Ruby version](add-new-ruby-version.md)
43+
* [Archiving EOL packages](archiving-eol-packages.md)
4344

4445
## Organizational (for team members)
4546

‎dev-handbook/apt-yum-repo-infra.drawio.svg‎

Lines changed: 2 additions & 2 deletions
Loading

‎dev-handbook/apt-yum-repo.md‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -48,11 +48,15 @@ Our design was made with the following requirements in mind, and addresses the f
4848

4949
We self-host our repositories: we store them in Google Cloud Storage buckets. [Google Cloud Storage is strongly consistent.](https://cloud.google.com/storage/docs/consistency)
5050

51-
There are three buckets:
51+
There are five buckets:
5252

53-
- `fullstaq-ruby-server-edition-apt-repo` stores the production APT repository.
54-
- `fullstaq-ruby-server-edition-yum-repo` stores the production YUM repository.
55-
- `fullstaq-ruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs.
53+
- `fsruby-server-edition-apt-repo` stores the production APT repository.
54+
- `fsruby-server-edition-yum-repo` stores the production YUM repository.
55+
- `fsruby-server-edition-apt-repo-archive` stores packages for EOL distributions (APT).
56+
- `fsruby-server-edition-yum-repo-archive` stores packages for EOL distributions (YUM).
57+
- `fsruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs.
58+
59+
The archive buckets are only updated during [EOL migration](archiving-eol-packages.md) — CI never writes to them. Each migration creates a new version that merges newly-archived distros with the existing archive contents. They are served at `apt-archive.fullstaqruby.org` and `yum-archive.fullstaqruby.org`.
5660

5761
We don't let users use the production bucket URLs directly. Instead, we let users use `https://apt.fullstaqruby.org` and `https://yum.fullstaqruby.org`. These domains redirect to the appropriate bucket URLs. We do this so that we avoid strongly coupling users with Google Cloud Storage. If in the future we want to move off Google Cloud, we can do so without breaking users' URLs.
5862

@@ -144,7 +148,7 @@ A downside of this versioning approach is that each version consumes a lot of sp
144148

145149
### CI bucket
146150

147-
During CI runs, we create temporary repositories in `gs://fullstaq-ruby-server-edition-ci-artifacts/$CI_RUN_NUMBER/{apt,yum}-repo`. These directories look as follows:
151+
During CI runs, we create temporary repositories in `gs://fsruby-server-edition-ci-artifacts/$CI_RUN_NUMBER/{apt,yum}-repo`. These directories look as follows:
148152

149153
~~~
150154
/$CI_RUN_NUMBER/{apt,yum}-repo/
@@ -162,7 +166,7 @@ These directories are very similar to the production buckets' contents. But ther
162166

163167
The CI tests run directly against this bucket URL instead of going through `https://{apt,yum}.fullstaqruby.org`.
164168

165-
We create a temporary repository by copying over `gs://fullstaq-ruby-server-edition-{apt,yum}-repo/versions/$LATEST_VERSION`. This way we achieve production data parity during testing.
169+
We create a temporary repository by copying over `gs://fsruby-server-edition-{apt,yum}-repo/versions/$LATEST_VERSION`. This way we achieve production data parity during testing.
166170

167171
## Locking
168172

@@ -179,10 +183,10 @@ The lock's critical section is quite large, and covers:
179183

180184
The locks are located in the following URLs:
181185

182-
* `gs://fullstaq-ruby-server-edition-apt-repo/locks/apt`
183-
* `gs://fullstaq-ruby-server-edition-yum-repo/locks/yum`
186+
* `gs://fsruby-server-edition-apt-repo/locks/apt`
187+
* `gs://fsruby-server-edition-yum-repo/locks/yum`
184188

185-
When publishing to a testing repository (in `fullstaq-ruby-server-edition-ci-artifacts`) we don't perform any locking, because each CI run is guaranteed to write to its own temporary repository.
189+
When publishing to a testing repository (in `fsruby-server-edition-ci-artifacts`) we don't perform any locking, because each CI run is guaranteed to write to its own temporary repository.
186190

187191
## Backups
188192

‎dev-handbook/archiving-eol-packages.md‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
# Archiving EOL packages
2+
3+
This document describes how to archive packages for end-of-life (EOL) distributions and prune EOL Ruby versions from the repositories. This is a routine maintenance task that frees CI disk space and keeps the repository lean.
4+
5+
## Background
6+
7+
The CI publish step downloads the full Aptly state archive (`state.tar.zst`) from Google Cloud Storage on every run. This archive grows with every distribution and Ruby version ever published. When distributions or Ruby versions reach EOL, their packages remain in the state archive indefinitely, consuming disk space on GitHub Actions runners.
8+
9+
To address this, we maintain **archive repositories** alongside the main repositories:
10+
11+
| Repository | Bucket | Domain | Purpose |
12+
|------------|--------|--------|---------|
13+
| APT (main) | `fsruby-server-edition-apt-repo` | `apt.fullstaqruby.org` | Current, supported packages |
14+
| APT (archive) | `fsruby-server-edition-apt-repo-archive` | `apt-archive.fullstaqruby.org` | Frozen packages for EOL distributions |
15+
| YUM (main) | `fsruby-server-edition-yum-repo` | `yum.fullstaqruby.org` | Current, supported packages |
16+
| YUM (archive) | `fsruby-server-edition-yum-repo-archive` | `yum-archive.fullstaqruby.org` | Frozen packages for EOL distributions |
17+
18+
Archive repositories are static — CI never writes to them. They use the same versioned bucket structure as the main repos. Each migration creates a new version that merges newly-archived distros with the existing archive contents, so the archive grows incrementally over time.
19+
20+
This pattern follows the precedent set by [PostgreSQL](https://apt-archive.postgresql.org/) (`apt-archive.postgresql.org`) and [HashiCorp](https://www.hashicorp.com/en/blog/announcing-the-linux-package-archive-site) (`archive.releases.hashicorp.com`).
21+
22+
## Two types of cleanup
23+
24+
There are two independent axes of cleanup, each with its own script:
25+
26+
### 1. Distro archival — moving entire EOL distribution repos
27+
28+
When a Linux distribution reaches EOL, we stop building packages for it and move its existing packages to the archive. Users on EOL distributions can still install packages by pointing at the archive repo.
29+
30+
**Scripts:**
31+
* `internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb`
32+
* `internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb`
33+
34+
### 2. Package pruning — removing EOL Ruby version packages
35+
36+
When a Ruby version reaches EOL, we stop building it (by removing it from `config.yml`), but its packages persist inside every distro's repository. Pruning removes these stale packages from the still-supported distro repos to reduce state size.
37+
38+
**Scripts:**
39+
* `internal-scripts/ci-cd/archive/prune-apt-packages.rb`
40+
* `internal-scripts/ci-cd/archive/prune-yum-packages.rb`
41+
42+
## Removing an EOL distribution
43+
44+
### Step 1: Remove from the build system
45+
46+
1. Edit `config.yml` and remove the distribution from the `distributions` list (or add it to an exclusion).
47+
2. Delete the `environments/<distro>/` directory.
48+
3. Regenerate CI/CD workflows:
49+
50+
~~~bash
51+
./internal-scripts/generate-ci-cd-yaml.rb
52+
~~~
53+
54+
4. Commit and merge these changes.
55+
56+
### Step 2: Migrate packages to the archive
57+
58+
**Prerequisites:**
59+
* `gcloud` CLI authenticated with write access to the GCS buckets
60+
* `az` CLI authenticated with access to the `fsruby2infraowners` Key Vault (for the GPG signing key)
61+
* `aptly`, `zstd`, and `gpg` installed locally
62+
* Docker running (for `createrepo_c` in YUM migration)
63+
64+
**Dry run first** to verify which distros will be archived:
65+
66+
~~~bash
67+
PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \
68+
ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \
69+
./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run
70+
~~~
71+
72+
The script auto-detects EOL distros by comparing `aptly repo list` output against the distributions defined in `config.yml`. You can also specify distros explicitly:
73+
74+
~~~bash
75+
./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run --distros centos-8,debian-9
76+
~~~
77+
78+
**Execute the migration** (removes `--dry-run`):
79+
80+
~~~bash
81+
PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \
82+
ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \
83+
./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb
84+
~~~
85+
86+
**Repeat for YUM:**
87+
88+
~~~bash
89+
PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \
90+
ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \
91+
./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb --dry-run
92+
93+
PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \
94+
ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \
95+
./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb
96+
~~~
97+
98+
### Step 3: Restart the web server
99+
100+
After migration, restart the web server so Caddy picks up the new version numbers:
101+
102+
~~~bash
103+
curl -X POST https://apt.fullstaqruby.org/admin/restart_web_server \
104+
-H "Authorization: Bearer $ID_TOKEN"
105+
~~~
106+
107+
Or restart the Caddy service directly via Ansible/SSH.
108+
109+
### Step 4: Verify
110+
111+
~~~bash
112+
# Archive should list the archived distros
113+
curl -s https://apt-archive.fullstaqruby.org/dists/
114+
115+
# Main repo should only contain supported distros
116+
curl -s https://apt.fullstaqruby.org/dists/
117+
118+
# Verify state archive size decreased
119+
gsutil ls -l gs://fsruby-server-edition-apt-repo/versions/*/state.tar.zst | tail -5
120+
~~~
121+
122+
## Pruning EOL Ruby versions
123+
124+
After removing a Ruby version from `config.yml`, its packages persist in the Aptly state. Run the pruning scripts to remove them.
125+
126+
**Dry run:**
127+
128+
~~~bash
129+
PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \
130+
./internal-scripts/ci-cd/archive/prune-apt-packages.rb --dry-run
131+
~~~
132+
133+
The script compares packages in the Aptly state against `minor_version_packages` in `config.yml` and identifies any `fullstaq-ruby-X.Y*` packages where `X.Y` is not an active minor version.
134+
135+
**Execute:**
136+
137+
~~~bash
138+
PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \
139+
./internal-scripts/ci-cd/archive/prune-apt-packages.rb
140+
~~~
141+
142+
**Repeat for YUM:**
143+
144+
~~~bash
145+
PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \
146+
./internal-scripts/ci-cd/archive/prune-yum-packages.rb --dry-run
147+
148+
PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \
149+
./internal-scripts/ci-cd/archive/prune-yum-packages.rb
150+
~~~
151+
152+
Restart the web server after pruning (same as above).
153+
154+
## Execution order
155+
156+
When performing both distro archival and package pruning in the same session, always run distro archival **first**. This ensures the archive captures the full historical packages for EOL distros before any pruning happens.
157+
158+
1. `migrate-apt-to-archive.rb`
159+
2. `prune-apt-packages.rb`
160+
3. `migrate-yum-to-archive.rb`
161+
4. `prune-yum-packages.rb`
162+
5. Restart web server
163+
164+
## Rollback
165+
166+
The versioned bucket structure makes rollback straightforward. Each migration creates a new version — the old version is never modified.
167+
168+
**Revert the main APT repo to a previous version:**
169+
170+
~~~bash
171+
# Find the pre-migration version number
172+
gsutil cat gs://fsruby-server-edition-apt-repo/versions/latest_version.txt
173+
174+
# Point back to the old version
175+
echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo/versions/latest_version.txt
176+
~~~
177+
178+
**Revert the archive to a previous version:**
179+
180+
~~~bash
181+
gsutil cat gs://fsruby-server-edition-apt-repo-archive/versions/latest_version.txt
182+
183+
echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo-archive/versions/latest_version.txt
184+
~~~
185+
186+
**Delete all archive contents** (if no users depend on it yet):
187+
188+
~~~bash
189+
gsutil -m rm -r gs://fsruby-server-edition-apt-repo-archive/versions/
190+
~~~
191+
192+
## How the migration scripts work
193+
194+
### APT migration (`migrate-apt-to-archive.rb`)
195+
196+
1. Downloads the current Aptly state archive from the main bucket.
197+
2. Identifies EOL distros (published in Aptly but not in `config.yml`).
198+
3. Fetches the existing archive state (if any) so new distros are merged into it.
199+
4. Creates or extends the archive Aptly instance with EOL distro data and package pool.
200+
5. Publishes all archive distros (existing + newly archived).
201+
6. Drops the EOL distro repos from the main Aptly database.
202+
7. Runs `aptly db cleanup` to compact the database and reclaim pool space.
203+
8. Re-publishes remaining distros in the main repo.
204+
9. Uploads the merged archive as a new archive version (N+1).
205+
10. Uploads the trimmed state as a new version of the main repo.
206+
207+
### YUM migration (`migrate-yum-to-archive.rb`)
208+
209+
1. Downloads the current YUM repo from the main bucket via `gsutil rsync`.
210+
2. Identifies EOL distro directories.
211+
3. Fetches the existing archive repo (if any) so new distros are merged into it.
212+
4. Copies EOL distro directories into the local archive copy.
213+
5. Uploads the merged archive as a new archive version (N+1).
214+
6. Removes EOL distro directories from the main local copy.
215+
7. Uploads the trimmed repo as a new version of the main bucket.
216+
217+
### APT pruning (`prune-apt-packages.rb`)
218+
219+
1. Downloads the Aptly state.
220+
2. Scans all packages across all distro repos, matching `fullstaq-ruby-X.Y*` against active minor versions.
221+
3. Removes EOL Ruby packages using `aptly repo remove`.
222+
4. Compacts, re-publishes, and uploads.
223+
224+
### YUM pruning (`prune-yum-packages.rb`)
225+
226+
1. Downloads the YUM repo.
227+
2. Deletes RPM files matching EOL Ruby versions from the filesystem.
228+
3. Regenerates `repodata/` with `createrepo_c` and re-signs.
229+
4. Uploads as a new version.

‎dev-handbook/ci-cd-resumption.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,12 @@ In order to aleviate this problem, we implement the ability to re-run only faile
88

99
Resumption support works by checking, for each CI job, whether the artifact that that job should produce, already exists. If so, then that job can be skipped.
1010

11-
When you re-run a CI run, Github Actions wipes all previous state (including artifacts). Therefore, we store artifacts primarily in a Google Cloud Storage bucket ([fullstaq-ruby-server-edition-ci-artifacts](https://storage.googleapis.com/fullstaq-ruby-server-edition-ci-artifacts), part of the [infrastructure](https://github.com/fullstaq-labs/fullstaq-ruby-infra)), which isn't wiped before a re-run.
11+
When you re-run a CI run, Github Actions wipes all previous state (including artifacts). Therefore, we store artifacts primarily in a Google Cloud Storage bucket ([fsruby-server-edition-ci-artifacts](https://storage.googleapis.com/fsruby-server-edition-ci-artifacts), part of the [infrastructure](https://github.com/fullstaq-labs/fullstaq-ruby-infra)), which isn't wiped before a re-run.
1212

1313
Here's an example artifact URL:
1414

1515
~~~
16-
gs://fullstaq-ruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst
16+
gs://fsruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst
1717
~~~
1818

1919
Artifacts are stored on a per-CI-run basis. Thus, they always contains the CI run's number. Note that the CI run number does not change even for re-runs.
@@ -23,7 +23,7 @@ At the beginning of a CI run, a job named `determine_necessary_jobs` checks whic
2323
~~~
2424
##### Determine whether Rbenv DEB needs to be built #####
2525
--> Run ./.github/actions/check-artifact-exists
26-
Checking gs://fullstaq-ruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst
26+
Checking gs://fsruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst
2727
Artifact exists
2828
~~~
2929

0 commit comments

Comments
 (0)