Skip to content

Commit 7670d64

Browse files
sfreudenthalerSteve Freudenthalerclaude
authored
feat: Release Tracks — floating Docker tags (latest/standard/trailing) promotion engine (#36160) (#36161)
Resolves #36160 · Epic #35693 ### Proposed Changes * Add a small Python promotion engine at `cicd/evergreen-tracks/` (managed with `uv`) that advances three **floating Docker tags** — `latest` / `standard` / `trailing` — across the linear GA CalVer stream by release **age** (newest GA, ~14d, ~28d; thresholds configurable). * State is **registry-only** via marker tags: `<version>_tainted` (forward-only block — a bad release can't propagate to more conservative tracks) and `<track>_hold` (sticky manual freeze). No separate datastore; the audit trail is the Actions run logs. * Tags are re-pointed **by digest** with `docker buildx imagetools create` (no layer re-push). Age is read from the **CalVer date in the version**, not build/publish date, so emergency backports of older releases can't be swept into a future promotion. * Two workflows: `cicd_evergreen-tracks-promote.yml` (daily cron + dispatch) and `cicd_evergreen-tracks-admin.yml` (manual taint/hold). * Document Release Tracks (usage + the "why") in the root `README.md`. ### Checklist - [x] Tests — 60 unit tests (pure planner, calver, markers, registry parser, CLI); run with `cd cicd/evergreen-tracks && uv run pytest`. - [ ] Translations — N/A (CI tooling, no UI strings). - [x] Security Implications Contemplated — see notes below. ### Additional Info **Tag control:** `latest` is moved on-demand by the release pipeline (`promote-latest` job in `cicd_6-release.yml`, `--tracks latest`) the moment a GA's images publish, for `dotcms/dotcms` and `dotcms/dotcms-dev` — the old `deploy-docker latest: true` path is unwired (`latest: false`). The daily cron ages `standard`/`trailing` forward and **always applies** (no separate enable gate). To pause promotion, disable the scheduled workflow or `hold` the track; to block a bad release, `taint` it. All registry mutations (release-driven latest, cron, admin) serialize under one concurrency group `evergreen-tracks-registry`. **Credential scope (important):** promotion needs only **write**, but `untaint` and `release-hold` call the Hub delete API — the `DOCKER_USERNAME`/`DOCKER_TOKEN` used here **must have Read/Write/Delete** scope, or those two admin actions fail. (Verified both ways: write-only 403s on delete; RWD succeeds.) **Validation:** the full lifecycle (promote-by-age, taint→skip, hold→freeze, release-hold→resume, untaint→restore, teardown) was exercised end-to-end against the `dotcms/dotcms-test` sandbox and verified by digest. The live smoke can't run in `core-workflow-test` CI (it intentionally carries no live Docker secrets), so it should run here in `core` CI / on dispatch. **Security notes:** free-text workflow inputs are passed via `env:` (not interpolated into `run:`) to avoid expression injection; no tokens are logged; least-privilege `permissions: contents: read` on the promote workflow. **Out of scope (per epic):** LTS-line tracks, Java-variant track tags, the Cloud control-plane UI, and update *cadence* changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Steve Freudenthaler <stevefreudenthaler@36:c1:bd:05:e8:20.home> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 98f8d66 commit 7670d64

21 files changed

Lines changed: 1787 additions & 1 deletion

‎.github/workflows/cicd_6-release.yml‎

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,11 @@ jobs:
139139
tag-identifier: ${{ needs.release-prepare.outputs.release_version }}
140140
omit-environment-prefix: true
141141
artifact-run-id: ${{ github.run_id }}
142-
latest: ${{ needs.release-prepare.outputs.is_latest == 'true' && github.ref_name == 'main' && github.event.inputs.update_latest == 'true' }}
142+
# `latest` is no longer moved here. evergreen-tracks is the single controller
143+
# of the floating latest/standard/trailing tags; the promote-latest job below
144+
# repoints `latest` on-demand once the release images are published (and honors
145+
# the `update_latest` opt-out for back-patches of older release lines).
146+
latest: false
143147
deploy-cli: true
144148
deploy-dev-image: true
145149
publish-npm-cli: false
@@ -154,6 +158,50 @@ jobs:
154158
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
155159
DEV_REQUEST_TOKEN: ${{ secrets.DEV_REQUEST_TOKEN }}
156160

161+
# Promote latest - move the floating `latest` Docker tag to this GA release.
162+
#
163+
# evergreen-tracks (cicd/evergreen-tracks) is the single controller of the
164+
# latest/standard/trailing tags. Its daily cron ages standard/trailing, but
165+
# `latest` must move the instant a GA ships — so the release pipeline invokes
166+
# the same engine on-demand, scoped to the latest track only (--tracks latest).
167+
# This replaces the old deploy-docker `latest: true` path (now `latest: false`).
168+
#
169+
# Applies for any real latest release from main on dotcms/core.
170+
promote-latest:
171+
name: Promote latest tag
172+
# Default success() gate: only runs when release-prepare AND deployment both
173+
# succeeded, so the release images are guaranteed to exist before we repoint.
174+
needs: [ release-prepare, deployment ]
175+
if: >-
176+
needs.release-prepare.outputs.is_latest == 'true'
177+
&& github.ref_name == 'main'
178+
&& github.repository == 'dotcms/core'
179+
&& github.event.inputs.update_latest == 'true'
180+
runs-on: ubuntu-${{ vars.UBUNTU_RUNNER_VERSION || '24.04' }}
181+
permissions:
182+
contents: read
183+
# Share the registry-mutation lock with the promote/admin workflows so this
184+
# on-demand promote can never race the daily cron or an admin run.
185+
concurrency:
186+
group: evergreen-tracks-registry
187+
cancel-in-progress: false
188+
steps:
189+
- uses: actions/checkout@v4
190+
- uses: astral-sh/setup-uv@v5
191+
- uses: docker/setup-buildx-action@v3
192+
- name: Docker login
193+
uses: docker/login-action@v3
194+
with:
195+
username: ${{ secrets.DOCKER_USERNAME }}
196+
password: ${{ secrets.DOCKER_TOKEN }}
197+
- name: Move latest to ${{ needs.release-prepare.outputs.release_version }}
198+
working-directory: cicd/evergreen-tracks
199+
run: |
200+
for repo in dotcms/dotcms dotcms/dotcms-dev; do
201+
echo "--- promoting latest on ${repo} ---"
202+
uv run evergreen-tracks promote --repo "${repo}" --tracks latest --apply
203+
done
204+
157205
# Release - release-specific operations (Artifactory, Javadocs, Plugins, SBOM, Labels)
158206
# Waits for deployment to complete to safely update labels only if both succeed
159207
release:

‎.github/workflows/cicd_evergreen-tracks-admin.yml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
name: evergreen-tracks-admin
2+
on:
3+
workflow_dispatch:
4+
inputs:
5+
repo:
6+
description: 'Docker repo to operate on'
7+
required: true
8+
default: 'dotcms/dotcms'
9+
action:
10+
description: 'taint | untaint | hold | release-hold'
11+
required: true
12+
type: choice
13+
options: [taint, untaint, hold, release-hold]
14+
version:
15+
description: 'GA version (for taint/untaint/hold)'
16+
required: false
17+
default: ''
18+
track:
19+
description: 'Track (for hold/release-hold): latest|standard|trailing'
20+
required: false
21+
default: ''
22+
force:
23+
description: 'Allow holding onto a tainted version'
24+
required: false
25+
default: 'false'
26+
apply:
27+
description: 'Actually mutate tags (false = dry-run)'
28+
required: true
29+
default: 'false'
30+
31+
# Share the registry-mutation lock with the promote workflows so an admin
32+
# taint/hold can never overlap a promote run on the same tags.
33+
concurrency:
34+
group: evergreen-tracks-registry
35+
cancel-in-progress: false
36+
37+
jobs:
38+
admin:
39+
runs-on: ubuntu-${{ vars.UBUNTU_RUNNER_VERSION || '24.04' }}
40+
steps:
41+
- uses: actions/checkout@v4
42+
- uses: astral-sh/setup-uv@v5
43+
- uses: docker/setup-buildx-action@v3
44+
- name: Docker login
45+
uses: docker/login-action@v3
46+
with:
47+
username: ${{ secrets.DOCKER_USERNAME }}
48+
password: ${{ secrets.DOCKER_TOKEN }}
49+
- name: Run admin action
50+
working-directory: cicd/evergreen-tracks
51+
env:
52+
# Scope Docker Hub creds to this step only (the CLI's untaint/release-hold
53+
# delete calls read them); other steps/actions don't need them.
54+
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
55+
DOCKER_TOKEN: ${{ secrets.DOCKER_TOKEN }}
56+
REPO: ${{ github.event.inputs.repo }}
57+
ACTION: ${{ github.event.inputs.action }}
58+
VERSION: ${{ github.event.inputs.version }}
59+
TRACK: ${{ github.event.inputs.track }}
60+
FORCE: ${{ github.event.inputs.force }}
61+
APPLY: ${{ github.event.inputs.apply }}
62+
run: |
63+
EXTRA=""
64+
if [ "$FORCE" = "true" ]; then EXTRA="$EXTRA --force"; fi
65+
if [ "$APPLY" = "true" ]; then EXTRA="$EXTRA --apply"; fi
66+
uv run evergreen-tracks admin \
67+
--repo "$REPO" \
68+
--action "$ACTION" \
69+
--version "$VERSION" \
70+
--track "$TRACK" \
71+
$EXTRA

‎.github/workflows/cicd_evergreen-tracks-promote.yml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
name: evergreen-tracks-promote
2+
on:
3+
schedule:
4+
- cron: '17 6 * * *' # daily at 06:17 UTC
5+
workflow_dispatch:
6+
inputs:
7+
repo:
8+
description: 'Docker repo to operate on'
9+
required: true
10+
default: 'dotcms/dotcms'
11+
apply:
12+
description: 'Actually move tags (false = dry-run)'
13+
required: true
14+
default: 'false'
15+
standard_days:
16+
description: 'Min age (days) for the standard track'
17+
required: false
18+
default: '14'
19+
trailing_days:
20+
description: 'Min age (days) for the trailing track'
21+
required: false
22+
default: '28'
23+
24+
permissions:
25+
contents: read
26+
27+
# Serialize every registry mutation. The daily cron, a manual promote/admin
28+
# dispatch, and the on-demand latest-promote in the release pipeline all read
29+
# live registry state and then apply tag moves, so two concurrent runs could act
30+
# on stale state and overwrite a hold/taint or move a track based on outdated
31+
# data. A single static group (not keyed by workflow/ref) makes them serialize
32+
# across all three workflows. For a tag-promotion engine queueing is safer than
33+
# cancelling, so cancel-in-progress is false: concurrent runs wait their turn
34+
# rather than aborting mid-promote and leaving tags half-moved.
35+
concurrency:
36+
group: evergreen-tracks-registry
37+
cancel-in-progress: false
38+
39+
jobs:
40+
promote:
41+
runs-on: ubuntu-${{ vars.UBUNTU_RUNNER_VERSION || '24.04' }}
42+
steps:
43+
- uses: actions/checkout@v4
44+
- uses: astral-sh/setup-uv@v5
45+
- uses: docker/setup-buildx-action@v3
46+
- name: Docker login
47+
uses: docker/login-action@v3
48+
with:
49+
username: ${{ secrets.DOCKER_USERNAME }}
50+
password: ${{ secrets.DOCKER_TOKEN }}
51+
- name: Promote tracks
52+
working-directory: cicd/evergreen-tracks
53+
env:
54+
# Scheduled runs default to dotcms/dotcms; dispatch can override.
55+
REPO: ${{ github.event.inputs.repo || 'dotcms/dotcms' }}
56+
STANDARD_DAYS: ${{ github.event.inputs.standard_days || '14' }}
57+
TRAILING_DAYS: ${{ github.event.inputs.trailing_days || '28' }}
58+
EVENT_NAME: ${{ github.event_name }}
59+
DISPATCH_APPLY: ${{ github.event.inputs.apply }}
60+
run: |
61+
# Scheduled (cron) runs always apply — this is the engine that ages
62+
# standard/trailing forward. To pause promotion, disable this workflow
63+
# in the Actions tab or `hold` the affected track; there is no separate
64+
# apply gate. Manual dispatch defaults to dry-run unless apply=true.
65+
APPLY=""
66+
if [ "$EVENT_NAME" = "schedule" ]; then
67+
APPLY="--apply"
68+
elif [ "$EVENT_NAME" = "workflow_dispatch" ] && [ "$DISPATCH_APPLY" = "true" ]; then
69+
APPLY="--apply"
70+
fi
71+
echo "event=$EVENT_NAME repo=$REPO apply=${APPLY:-<dry-run>}"
72+
uv run evergreen-tracks promote \
73+
--repo "$REPO" \
74+
--standard-days "$STANDARD_DAYS" \
75+
--trailing-days "$TRAILING_DAYS" \
76+
$APPLY

‎README.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,3 +53,43 @@ For a complete list of requirements, see [this page](http://www.dotcms.com/docs/
5353
| Forums/Listserv | [via Google Groups](https://groups.google.com/forum/#!forum/dotCMS) |
5454
| Twitter | @dotCMS |
5555
| Main Site | [dotCMS.com](https://www.dotcms.com/) |
56+
57+
## Release Tracks
58+
59+
dotCMS GA releases follow CalVer (`YY.0M.0D-NN`, e.g. `26.06.11-01`). On top of that linear
60+
release stream we publish three **floating Docker tags** — *release tracks* — so you can pick
61+
how fresh a release each environment receives:
62+
63+
| Tag | Age of release | Use for |
64+
| ------------------------- | ----------------- | ----------------------------------------- |
65+
| `dotcms/dotcms:latest` | newest GA (~days) | tracking the latest release |
66+
| `dotcms/dotcms:standard` | ~2 weeks old | a short soak before adopting a release |
67+
| `dotcms/dotcms:trailing` | ~4 weeks old | the most conservative posture |
68+
69+
Pin the track you want in your deployment manifest, e.g. `image: dotcms/dotcms:standard`, and
70+
you will roll forward automatically as releases age into that track. Pin an exact version
71+
(`dotcms/dotcms:26.06.11-01`) instead if you never want automatic movement.
72+
73+
One engine (under [`cicd/evergreen-tracks/`](cicd/evergreen-tracks/)) re-points every track tag
74+
by image digest, on two triggers: the release pipeline moves `latest` on-demand the moment a GA
75+
ships, and a daily scheduled job ages `standard`/`trailing` forward. Two design choices are worth
76+
understanding:
77+
78+
- **Age is measured from the CalVer date in the version string, not from when the image was
79+
built or published.** This protects emergency backports: if we cut a patch of an *older*
80+
release on short notice, it is built today but logically belongs to the older release line.
81+
Using the build/publish date would make that patch look brand new and let it jump onto the
82+
`standard`/`trailing` tracks ahead of releases that are genuinely older. Anchoring to the
83+
embedded CalVer date keeps every release in its true place on the timeline.
84+
- **A track never moves backward automatically, and a release can be "tainted".** If a bad
85+
release is found, it is tainted so it will not advance onto tracks it has not yet reached —
86+
a known-bad build can never propagate from `latest` down to `standard`/`trailing`. A track
87+
can also be **held** (frozen) at a specific version for incident response. Both controls live
88+
as marker tags in the registry; there is no separate datastore.
89+
90+
> **GitOps / Argo note:** because these tags float (the same tag is re-pointed to newer
91+
> digests over time), a tag reference alone will not trigger a redeploy. Use Argo CD Image
92+
> Updater or a periodic rollout refresh to pick up track movements.
93+
94+
The promotion engine and its scheduled/admin workflows live under
95+
[`cicd/evergreen-tracks/`](cicd/evergreen-tracks/).

‎cicd/evergreen-tracks/README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# evergreen-tracks
2+
3+
Advances floating Docker track tags (`latest`, `standard`, `trailing`) across the dotCMS
4+
GA CalVer release stream by release age. State lives entirely in registry tags (the track
5+
tags plus `<version>_tainted` / `<track>_hold` markers).
6+
7+
## Run locally (dry-run is the default)
8+
9+
uv run evergreen-tracks promote --repo dotcms/dotcms-test
10+
uv run evergreen-tracks admin --repo dotcms/dotcms-test --action taint --version 26.03.12-01
11+
12+
Pass `--apply` to actually move tags. Without it, the command prints the plan and exits.
13+
14+
## Test
15+
16+
uv run pytest

‎cicd/evergreen-tracks/pyproject.toml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
[project]
2+
name = "evergreen-tracks"
3+
version = "0.1.0"
4+
description = "Floating Docker track tags (latest/standard/trailing) for dotCMS releases"
5+
requires-python = ">=3.12"
6+
dependencies = [
7+
"requests>=2.32",
8+
]
9+
10+
[project.scripts]
11+
evergreen-tracks = "evergreen_tracks.cli:main"
12+
13+
[dependency-groups]
14+
dev = [
15+
"pytest>=8.0",
16+
"responses>=0.25",
17+
]
18+
19+
[build-system]
20+
requires = ["hatchling"]
21+
build-backend = "hatchling.build"
22+
23+
[tool.hatch.build.targets.wheel]
24+
packages = ["src/evergreen_tracks"]
25+
26+
[tool.pytest.ini_options]
27+
testpaths = ["tests"]

‎cicd/evergreen-tracks/src/evergreen_tracks/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
"""Evergreen release tracks: floating Docker tags advanced by release age."""
2+
3+
__all__ = []

‎cicd/evergreen-tracks/src/evergreen_tracks/calver.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
"""CalVer release parsing, age, and ordering. No I/O."""
2+
from __future__ import annotations
3+
4+
import datetime as dt
5+
import re
6+
from dataclasses import dataclass
7+
8+
# GA tags only: YY.0M.0D-NN (e.g. 26.06.11-01). Excludes SNAPSHOT, RC, _lts*, _javaNN, markers.
9+
GA_RE = re.compile(r"^(\d{2})\.(\d{2})\.(\d{2})-(\d{1,2})$")
10+
11+
12+
@dataclass(frozen=True)
13+
class Release:
14+
version: str # original tag, e.g. "26.06.11-01"
15+
date: dt.date # parsed from YY.0M.0D
16+
build: int # NN
17+
18+
@property
19+
def sort_key(self) -> tuple[dt.date, int]:
20+
return (self.date, self.build)
21+
22+
23+
def parse_release(tag: str) -> Release | None:
24+
"""Return a Release for a GA tag, or None for anything else."""
25+
m = GA_RE.match(tag)
26+
if not m:
27+
return None
28+
yy, mm, dd, nn = m.groups()
29+
try:
30+
date = dt.date(2000 + int(yy), int(mm), int(dd))
31+
except ValueError:
32+
return None
33+
return Release(version=tag, date=date, build=int(nn))
34+
35+
36+
def age_days(release: Release, today: dt.date) -> int:
37+
return (today - release.date).days
38+
39+
40+
def newest(releases: list[Release]) -> Release | None:
41+
"""Newest by (date, build); None if the list is empty."""
42+
return max(releases, key=lambda r: r.sort_key, default=None)

0 commit comments

Comments
 (0)