Skip to content

Commit 5513dfc

Browse files
sarahchen6sarahchen6
committed
Add workflows that create images PRs
1 parent 69469ea commit 5513dfc

5 files changed

Lines changed: 347 additions & 125 deletions

File tree

‎.github/workflows/create-test-mirror-pr.yml‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
name: Create test image mirror PR
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
pr_number:
7+
description: "PR number in dd-trace-java-docker-build (e.g. 123)"
8+
required: true
9+
10+
jobs:
11+
create-test-mirror-pr:
12+
runs-on: ubuntu-latest
13+
permissions:
14+
id-token: write # Required for OIDC token federation
15+
contents: read
16+
steps:
17+
- uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3
18+
id: octo-sts
19+
with:
20+
scope: DataDog/images
21+
policy: dd-trace-java-docker-build.update-mirror
22+
23+
- name: Checkout DataDog/images
24+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
25+
with:
26+
repository: DataDog/images
27+
token: ${{ steps.octo-sts.outputs.token }}
28+
29+
- name: Capture images HEAD SHA
30+
id: images-head
31+
run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
32+
33+
- name: Install crane
34+
run: |
35+
CRANE_VERSION="0.20.2"
36+
curl -fsSL "https://github.com/google/go-containerregistry/releases/download/v${CRANE_VERSION}/go-containerregistry_Linux_x86_64.tar.gz" -o crane.tar.gz
37+
tar -xzf crane.tar.gz crane
38+
sudo mv crane /usr/local/bin/crane
39+
rm crane.tar.gz
40+
41+
- name: Resolve digests and update mirror files
42+
id: update-mirror
43+
env:
44+
PR_NUMBER: ${{ github.event.inputs.pr_number }}
45+
run: |
46+
python3 - <<'PYEOF'
47+
import subprocess, re, os
48+
49+
SOURCE_REPO = "ghcr.io/datadog/dd-trace-java-docker-build"
50+
VARIANTS = [
51+
"base", "7", "8", "11", "17", "21", "25", "tip",
52+
"zulu8", "zulu11", "oracle8", "ibm8",
53+
"semeru8", "semeru11", "semeru17",
54+
"graalvm17", "graalvm21", "graalvm25",
55+
]
56+
57+
pr_number = os.environ["PR_NUMBER"]
58+
if not pr_number.isdigit():
59+
raise ValueError(f"PR_NUMBER must be numeric, got: {pr_number!r}")
60+
61+
prefix = f"{pr_number}_merge-"
62+
print(f"Resolving digests for prefix: {prefix!r}")
63+
64+
digests = {}
65+
for variant in VARIANTS:
66+
tag = f"{prefix}{variant}"
67+
result = subprocess.run(
68+
["crane", "digest", f"{SOURCE_REPO}:{tag}"],
69+
capture_output=True, text=True, check=True,
70+
)
71+
digest = result.stdout.strip()
72+
digests[variant] = digest
73+
print(f" {tag}: {digest}")
74+
75+
# Check whether entries already exist in mirror.yaml (use base as sentinel)
76+
with open("mirror.yaml", "r") as f:
77+
yaml_content = f.read()
78+
79+
entries_exist = f"{SOURCE_REPO}:{prefix}base" in yaml_content
80+
mode = "update" if entries_exist else "add"
81+
print(f"\nMode: {mode} ({'entries exist, updating digests only' if entries_exist else 'no entries found, adding new entries'})")
82+
83+
github_output = os.environ.get("GITHUB_OUTPUT", "")
84+
if github_output:
85+
with open(github_output, "a") as f:
86+
f.write(f"mode={mode}\n")
87+
88+
if mode == "add":
89+
yaml_entries = []
90+
for variant in VARIANTS:
91+
tag = f"{prefix}{variant}"
92+
source = f"{SOURCE_REPO}:{tag}"
93+
yaml_entries.append(
94+
f' - source: "{source}"\n'
95+
f' dest:\n'
96+
f' repo: "dd-trace-java-docker-build"\n'
97+
f' tag: "{tag}"\n'
98+
f' replication_target: ""\n'
99+
)
100+
with open("mirror.yaml", "a") as f:
101+
f.write("".join(yaml_entries))
102+
print(f"Appended {len(yaml_entries)} entries to mirror.yaml")
103+
104+
# Always update mirror.lock.yaml: replace digest in-place if entry exists, append if not
105+
with open("mirror.lock.yaml", "r") as f:
106+
lock_content = f.read()
107+
108+
for variant in VARIANTS:
109+
tag = f"{prefix}{variant}"
110+
source = f"{SOURCE_REPO}:{tag}"
111+
digest = digests[variant]
112+
pattern = rf"( - source: {re.escape(source)}\n digest: )sha256:[a-f0-9]+"
113+
if re.search(pattern, lock_content):
114+
lock_content = re.sub(pattern, rf"\g<1>{digest}", lock_content)
115+
print(f"Updated mirror.lock.yaml: {tag}")
116+
else:
117+
lock_content = lock_content.rstrip("\n") + "\n"
118+
lock_content += f" - source: {source}\n digest: {digest}\n"
119+
print(f"Appended to mirror.lock.yaml: {tag}")
120+
121+
with open("mirror.lock.yaml", "w") as f:
122+
f.write(lock_content)
123+
124+
PYEOF
125+
126+
- name: Define branch name
127+
id: define-branch
128+
run: echo "branch=ci/add-dd-trace-java-docker-build-test-images-pr${{ github.event.inputs.pr_number }}" >> "$GITHUB_OUTPUT"
129+
130+
- name: Commit changes
131+
id: create-commit
132+
env:
133+
PR_NUMBER: ${{ github.event.inputs.pr_number }}
134+
MODE: ${{ steps.update-mirror.outputs.mode }}
135+
run: |
136+
git config user.name "github-actions[bot]"
137+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
138+
git add mirror.yaml mirror.lock.yaml
139+
if [[ "$MODE" == "update" ]]; then
140+
git commit -m "chore: Update dd-trace-java-docker-build test image digests for PR #${PR_NUMBER}"
141+
else
142+
git commit -m "chore: Add dd-trace-java-docker-build test images for PR #${PR_NUMBER}"
143+
fi
144+
echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
145+
146+
- name: Push changes
147+
uses: DataDog/commit-headless@05d7b7ee023e2c7d01c47832d420c2503cd416f3 # action/v2.0.3
148+
with:
149+
token: "${{ steps.octo-sts.outputs.token }}"
150+
branch: "${{ steps.define-branch.outputs.branch }}"
151+
head-sha: "${{ steps.images-head.outputs.sha }}"
152+
create-branch: true
153+
command: push
154+
commits: "${{ steps.create-commit.outputs.commit }}"
155+
156+
- name: Create or identify pull request
157+
env:
158+
GH_TOKEN: ${{ steps.octo-sts.outputs.token }}
159+
PR_NUMBER: ${{ github.event.inputs.pr_number }}
160+
run: |
161+
BRANCH="${{ steps.define-branch.outputs.branch }}"
162+
EXISTING_PR=$(gh pr list --repo DataDog/images --head "$BRANCH" --json url -q '.[0].url' 2>/dev/null || true)
163+
if [[ -n "$EXISTING_PR" ]]; then
164+
echo "PR already exists: $EXISTING_PR"
165+
else
166+
gh pr create \
167+
--repo DataDog/images \
168+
--draft \
169+
--title "Add dd-trace-java-docker-build test images for PR #${PR_NUMBER}" \
170+
--base master \
171+
--head "$BRANCH" \
172+
--body "Adds mirror entries for \`${PR_NUMBER}_merge-*\` test images from DataDog/dd-trace-java-docker-build#${PR_NUMBER}. To use in dd-trace-java CI, set \`TESTER_IMAGE_VERSION_PREFIX: \"${PR_NUMBER}_merge-\"\` in \`.gitlab-ci.yml\`."
173+
fi

‎.github/workflows/docker-tag.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Tag new images version
1+
name: Tag new images version # triggers update-mirror-digests workflow
22
on:
33
schedule:
44
# Quarterly schedule, roughly aligned with JDK CPU

‎.github/workflows/update-mirror-digests.yml‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
name: Update mirror digests for ci-* images
2+
3+
on:
4+
workflow_run:
5+
workflows: ["Tag new images version"]
6+
types: [completed]
7+
workflow_dispatch:
8+
9+
jobs:
10+
update-mirror-digests:
11+
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
12+
runs-on: ubuntu-latest
13+
permissions:
14+
id-token: write # Required for OIDC token federation
15+
contents: read
16+
steps:
17+
- uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3
18+
id: octo-sts
19+
with:
20+
scope: DataDog/images
21+
policy: dd-trace-java-docker-build.update-mirror
22+
23+
- name: Checkout DataDog/images
24+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
25+
with:
26+
repository: DataDog/images
27+
token: ${{ steps.octo-sts.outputs.token }}
28+
29+
- name: Capture images HEAD SHA
30+
id: images-head
31+
run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
32+
33+
- name: Install crane
34+
run: |
35+
CRANE_VERSION="0.20.2"
36+
curl -fsSL "https://github.com/google/go-containerregistry/releases/download/v${CRANE_VERSION}/go-containerregistry_Linux_x86_64.tar.gz" -o crane.tar.gz
37+
tar -xzf crane.tar.gz crane
38+
sudo mv crane /usr/local/bin/crane
39+
rm crane.tar.gz
40+
41+
- name: Get baseline digest for ci-base image
42+
id: baseline
43+
run: |
44+
BASELINE=$(awk '/source:.*dd-trace-java-docker-build:ci-base/{found=1; next} found && /digest:/{print $2; exit}' mirror.lock.yaml || true)
45+
echo "digest=${BASELINE}" >> "$GITHUB_OUTPUT"
46+
echo "Baseline ci-base digest: ${BASELINE:-<none found>}"
47+
48+
- name: Wait for new ci-base image to be published
49+
run: |
50+
BASELINE="${{ steps.baseline.outputs.digest }}"
51+
DEADLINE=$((SECONDS + 1800)) # 30 min timeout
52+
echo "Waiting for ci-base digest to differ from: ${BASELINE:-<none>}"
53+
while [[ $SECONDS -lt $DEADLINE ]]; do
54+
CURRENT=$(crane digest ghcr.io/datadog/dd-trace-java-docker-build:ci-base 2>/dev/null || true)
55+
if [[ -n "$CURRENT" && "$CURRENT" != "$BASELINE" ]]; then
56+
echo "New ci-base digest detected: $CURRENT"
57+
exit 0
58+
fi
59+
echo "No change yet (current: ${CURRENT:-unavailable}), retrying in 60s..."
60+
sleep 60
61+
done
62+
echo "::error::Timeout after 30 minutes: ci-base digest did not change"
63+
exit 1
64+
65+
- name: Resolve digests and update mirror files
66+
run: |
67+
python3 - <<'PYEOF'
68+
import subprocess, re
69+
70+
SOURCE_REPO = "ghcr.io/datadog/dd-trace-java-docker-build"
71+
VARIANTS = [
72+
"base", "7", "8", "11", "17", "21", "25", "tip",
73+
"zulu8", "zulu11", "oracle8", "ibm8",
74+
"semeru8", "semeru11", "semeru17",
75+
"graalvm17", "graalvm21", "graalvm25",
76+
]
77+
78+
# Verify all ci-* entries are already present in both files before proceeding
79+
with open("mirror.yaml", "r") as f:
80+
yaml_content = f.read()
81+
with open("mirror.lock.yaml", "r") as f:
82+
lock_content = f.read()
83+
84+
missing_yaml = [v for v in VARIANTS if f"{SOURCE_REPO}:ci-{v}" not in yaml_content]
85+
missing_lock = [v for v in VARIANTS if f"{SOURCE_REPO}:ci-{v}" not in lock_content]
86+
if missing_yaml or missing_lock:
87+
if missing_yaml:
88+
print(f"::error::ci-* entries missing from mirror.yaml: {missing_yaml}")
89+
if missing_lock:
90+
print(f"::error::ci-* entries missing from mirror.lock.yaml: {missing_lock}")
91+
print("Bootstrap the ci-* entries manually before running this workflow.")
92+
raise SystemExit(1)
93+
94+
print("Resolving digests for ci-* variants...")
95+
digests = {}
96+
for variant in VARIANTS:
97+
tag = f"ci-{variant}"
98+
result = subprocess.run(
99+
["crane", "digest", f"{SOURCE_REPO}:{tag}"],
100+
capture_output=True, text=True, check=True,
101+
)
102+
digest = result.stdout.strip()
103+
digests[variant] = digest
104+
print(f" {tag}: {digest}")
105+
106+
# Update existing digest entries in mirror.lock.yaml in-place
107+
for variant in VARIANTS:
108+
tag = f"ci-{variant}"
109+
source = f"{SOURCE_REPO}:{tag}"
110+
digest = digests[variant]
111+
pattern = rf"( - source: {re.escape(source)}\n digest: )sha256:[a-f0-9]+"
112+
lock_content = re.sub(pattern, rf"\g<1>{digest}", lock_content)
113+
print(f"Updated mirror.lock.yaml: {tag}")
114+
115+
with open("mirror.lock.yaml", "w") as f:
116+
f.write(lock_content)
117+
118+
PYEOF
119+
120+
- name: Check for changes
121+
id: check-changes
122+
run: |
123+
if [[ -z "$(git status -s)" ]]; then
124+
echo "No changes to commit."
125+
echo "commit_changes=false" >> "$GITHUB_OUTPUT"
126+
else
127+
echo "commit_changes=true" >> "$GITHUB_OUTPUT"
128+
fi
129+
130+
- name: Define branch name
131+
if: steps.check-changes.outputs.commit_changes == 'true'
132+
id: define-branch
133+
run: echo "branch=ci/update-dd-trace-java-docker-build-ci-digests-$(date +'%Y%m%d')" >> "$GITHUB_OUTPUT"
134+
135+
- name: Commit changes
136+
if: steps.check-changes.outputs.commit_changes == 'true'
137+
id: create-commit
138+
run: |
139+
git config user.name "github-actions[bot]"
140+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
141+
git add mirror.yaml mirror.lock.yaml
142+
git commit -m "chore: Update dd-trace-java-docker-build ci-* image digests"
143+
echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
144+
145+
- name: Push changes
146+
if: steps.check-changes.outputs.commit_changes == 'true'
147+
uses: DataDog/commit-headless@05d7b7ee023e2c7d01c47832d420c2503cd416f3 # action/v2.0.3
148+
with:
149+
token: "${{ steps.octo-sts.outputs.token }}"
150+
branch: "${{ steps.define-branch.outputs.branch }}"
151+
head-sha: "${{ steps.images-head.outputs.sha }}"
152+
create-branch: true
153+
command: push
154+
commits: "${{ steps.create-commit.outputs.commit }}"
155+
156+
- name: Create pull request
157+
if: steps.check-changes.outputs.commit_changes == 'true'
158+
env:
159+
GH_TOKEN: ${{ steps.octo-sts.outputs.token }}
160+
run: |
161+
gh pr create \
162+
--repo DataDog/images \
163+
--draft \
164+
--title "Update dd-trace-java-docker-build ci-* image digests" \
165+
--base master \
166+
--head "${{ steps.define-branch.outputs.branch }}" \
167+
--body "Automated digest update for \`dd-trace-java-docker-build\` \`ci-*\` images after tagging."

‎README.md‎

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Image variants are available on a per JDK basis:
1111
- The `zulu8`, `zulu11`, `oracle8`, `ibm8`, `semeru8`, `semeru11`, `semeru17`, `graalvm17`, `graalvm21`, and `graalvm25` variants all contain the base JDKs in addition to the specific JDK from their name.
1212
- The `latest` variant contains the base JDKs and all of the specific JDKs above.
1313

14-
Images are tagged via the [Tag new images version](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/docker-tag.yml) workflow. This workflow tags the latest images built from the specified branch with a `ci-` prefix. It runs quarterly on `master` but can also be triggered manually as needed.
14+
Images are tagged via the [Tag new images version](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/docker-tag.yml) workflow. This workflow tags the latest images built from the specified branch with a `ci-` prefix and triggers the [Update mirror digests for ci-* images](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/update-mirror-digests.yml) workflow which automatically creates a PR that updates the pinned `ci-*` image digests in `DataDog/images`. It runs quarterly on `master` but can also be triggered manually as needed.
1515

1616
## Development
1717

@@ -34,17 +34,8 @@ Images are built per PR for ease in testing. These test images are prefixed with
3434
To test these images in `dd-trace-java` CI:
3535

3636
1. Open a PR in [DataDog/dd-trace-java-docker-build](https://github.com/DataDog/dd-trace-java-docker-build) with the changes you want to test. Let's say these changes are made in PR #123 ([example](https://github.com/DataDog/dd-trace-java-docker-build/pull/123)).
37-
2. Run the [generate-test-image-yaml](https://github.com/DataDog/dd-trace-java-docker-build/tree/master/scripts/generate-test-image-yaml.sh) script to generate the YAML snippets that you will need in the following steps:
38-
```bash
39-
./scripts/generate-test-image-yaml.sh 123
40-
```
41-
Note that the test images have `renovate` enabled. This means that any changes to your upstream image (i.e. `DataDog/dd-trace-java-docker-build` PR images) will be auto-updated in `DataDog/images`. The update is done via a bot ([example](https://github.com/DataDog/images/pull/8756)).
42-
43-
3. Open a PR in [DataDog/images](https://github.com/DataDog/images) that adds the generated snippets to the corresponding files. This PR should be automatically merged with the `dd-prapprover` bot. If not, then manually merge the PR. The images should be generated a few minutes after the PR lands on `master`. One way to confirm this is by pulling a test image locally, e.g. by running:
44-
```bash
45-
docker pull registry.ddbuild.io/images/mirror/datadog/dd-trace-java-docker-build:123_merge-base
46-
```
47-
4. Open a PR in [DataDog/dd-trace-java](https://github.com/DataDog/dd-trace-java) that updates the `BUILDER_IMAGE_VERSION_PREFIX` variable according to the output from step 2. Here, you can check your test images with `DataDog/dd-trace-java` CI.
48-
5. For each following change made to your original PR #123, ensure the test image (i.e. prefixed with `123_merge-`) SHAs in `DataDog/images` are updated. This should be done via the bot mentioned in step 2 but can also be updated manually.
49-
6. When the test images look good and `DataDog/dd-trace-java` CI is green, merge your `DataDog/dd-trace-java-docker-build` PR #123, close the test `DataDog/dd-trace-java` PR, and **revert the `DataDog/images` PR**.
50-
7. Finally, run the [Tag new images version](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/docker-tag.yml) workflow and confirm that the corresponding image (i.e. `ci-` prefix) SHAs in `DataDog/images` are updated.
37+
2. Run the [Create test image mirror PR](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/create-test-mirror-pr.yml) workflow with `pr_number=123`. This automatically opens a PR in [DataDog/images](https://github.com/DataDog/images) that adds mirror entries for the `123_merge-*` test images. The PR should be automatically merged by the `dd-prapprover` bot.
38+
3. Open a PR in [DataDog/dd-trace-java](https://github.com/DataDog/dd-trace-java) that sets `BUILDER_IMAGE_VERSION_PREFIX: "123_merge-"` in `.gitlab-ci.yml`. Here, you can check your test images with `DataDog/dd-trace-java` CI.
39+
4. Every time you want to test changes made in PR #123, ensure the test image SHAs in `DataDog/images` are updated. This should be done by running the [Create test image mirror PR](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/create-test-mirror-pr.yml) workflow with `pr_number=123`.
40+
5. When the test images look good and `DataDog/dd-trace-java` CI is green, merge your `DataDog/dd-trace-java-docker-build` PR #123, close the test `DataDog/dd-trace-java` PR, and **remove the test images from the `DataDog/images` repo**.
41+
6. Finally, run the [Tag new images version](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/docker-tag.yml) workflow. The [Update mirror digests for ci-* images](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/update-mirror-digests.yml) workflow will automatically open a PR in `DataDog/images`, updating the pinned `ci-*` digests.

0 commit comments

Comments
 (0)