Skip to content

Commit 8e3e105

Browse files
sarahchen6sarahchen6
committed
Re-organize scripts
1 parent d11eb1a commit 8e3e105

File tree

6 files changed

+193
-175
lines changed

6 files changed

+193
-175
lines changed

‎.github/workflows/create-test-mirror-pr.yml‎

Lines changed: 42 additions & 99 deletions
Original file line numberDiff line numberDiff line change
@@ -13,22 +13,30 @@ jobs:
1313
permissions:
1414
id-token: write # Required for OIDC token federation
1515
contents: read
16+
pull-requests: write
1617
steps:
1718
- uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3
1819
id: octo-sts
1920
with:
2021
scope: DataDog/images
2122
policy: dd-trace-java-docker-build.update-mirror
2223

24+
- name: Checkout DataDog/dd-trace-java-docker-build
25+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
26+
with:
27+
path: dd-trace-java-docker-build
28+
2329
- name: Checkout DataDog/images
2430
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
2531
with:
2632
repository: DataDog/images
2733
token: ${{ steps.octo-sts.outputs.token }}
34+
path: images
2835

2936
- name: Capture images HEAD SHA
3037
id: images-head
3138
run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
39+
working-directory: images
3240

3341
- name: Install crane
3442
run: |
@@ -38,90 +46,12 @@ jobs:
3846
sudo mv crane /usr/local/bin/crane
3947
rm crane.tar.gz
4048
41-
- name: Resolve digests and update mirror files
49+
- name: Resolve digests and add new or update existing digests in mirror files
4250
id: update-mirror
4351
env:
4452
PR_NUMBER: ${{ github.event.inputs.pr_number }}
45-
run: |
46-
python3 - <<'PYEOF'
47-
import subprocess, re, os
48-
49-
SOURCE_REPO = "ghcr.io/datadog/dd-trace-java-docker-build"
50-
VARIANTS = [
51-
"base", "7", "8", "11", "17", "21", "25", "tip",
52-
"zulu8", "zulu11", "oracle8", "ibm8",
53-
"semeru8", "semeru11", "semeru17",
54-
"graalvm17", "graalvm21", "graalvm25",
55-
]
56-
57-
pr_number = os.environ["PR_NUMBER"]
58-
if not pr_number.isdigit():
59-
raise ValueError(f"PR_NUMBER must be numeric, got: {pr_number!r}")
60-
61-
prefix = f"{pr_number}_merge-"
62-
print(f"Resolving digests for prefix: {prefix!r}")
63-
64-
digests = {}
65-
for variant in VARIANTS:
66-
tag = f"{prefix}{variant}"
67-
result = subprocess.run(
68-
["crane", "digest", f"{SOURCE_REPO}:{tag}"],
69-
capture_output=True, text=True, check=True,
70-
)
71-
digest = result.stdout.strip()
72-
digests[variant] = digest
73-
print(f" {tag}: {digest}")
74-
75-
# Check whether entries already exist in mirror.yaml (use base as sentinel)
76-
with open("mirror.yaml", "r") as f:
77-
yaml_content = f.read()
78-
79-
entries_exist = f"{SOURCE_REPO}:{prefix}base" in yaml_content
80-
mode = "update" if entries_exist else "add"
81-
print(f"\nMode: {mode} ({'entries exist, updating digests only' if entries_exist else 'no entries found, adding new entries'})")
82-
83-
github_output = os.environ.get("GITHUB_OUTPUT", "")
84-
if github_output:
85-
with open(github_output, "a") as f:
86-
f.write(f"mode={mode}\n")
87-
88-
if mode == "add":
89-
yaml_entries = []
90-
for variant in VARIANTS:
91-
tag = f"{prefix}{variant}"
92-
source = f"{SOURCE_REPO}:{tag}"
93-
yaml_entries.append(
94-
f' - source: "{source}"\n'
95-
f' dest:\n'
96-
f' repo: "dd-trace-java-docker-build"\n'
97-
f' tag: "{tag}"\n'
98-
f' replication_target: ""\n'
99-
)
100-
with open("mirror.yaml", "a") as f:
101-
f.write("".join(yaml_entries))
102-
print(f"Appended {len(yaml_entries)} entries to mirror.yaml")
103-
104-
# Always update mirror.lock.yaml: replace digest in-place if entry exists, append if not
105-
with open("mirror.lock.yaml", "r") as f:
106-
lock_content = f.read()
107-
108-
for variant in VARIANTS:
109-
tag = f"{prefix}{variant}"
110-
source = f"{SOURCE_REPO}:{tag}"
111-
digest = digests[variant]
112-
pattern = rf"( - source: {re.escape(source)}\n digest: )sha256:[a-f0-9]+"
113-
if re.search(pattern, lock_content):
114-
lock_content = re.sub(pattern, rf"\g<1>{digest}", lock_content)
115-
print(f"Updated mirror.lock.yaml: {tag}")
116-
else:
117-
lock_content = lock_content.rstrip("\n") + "\n"
118-
lock_content += f" - source: {source}\n digest: {digest}\n"
119-
print(f"Appended to mirror.lock.yaml: {tag}")
120-
121-
with open("mirror.lock.yaml", "w") as f:
122-
f.write(lock_content)
123-
124-
PYEOF
53+
run: bash "${GITHUB_WORKSPACE}/dd-trace-java-docker-build/scripts/create-test-mirror-entries.sh"
54+
working-directory: images
12555

12656
- name: Define branch name
12757
id: define-branch
@@ -136,14 +66,18 @@ jobs:
13666
git config user.name "github-actions[bot]"
13767
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
13868
git add mirror.yaml mirror.lock.yaml
139-
if [[ "$MODE" == "update" ]]; then
140-
git commit -m "chore: Update dd-trace-java-docker-build test image digests for PR #${PR_NUMBER}"
141-
else
142-
git commit -m "chore: Add dd-trace-java-docker-build test images for PR #${PR_NUMBER}"
69+
if git diff --cached --quiet; then
70+
echo "No changes detected in mirror files; skipping commit."
71+
echo "has_changes=false" >> "$GITHUB_OUTPUT"
72+
exit 0
14373
fi
74+
git commit -m "chore: Update dd-trace-java-docker-build test image digests for PR #${PR_NUMBER}"
75+
echo "has_changes=true" >> "$GITHUB_OUTPUT"
14476
echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
77+
working-directory: images
14578

14679
- name: Push changes
80+
if: ${{ steps.create-commit.outputs.has_changes == 'true' }}
14781
uses: DataDog/commit-headless@05d7b7ee023e2c7d01c47832d420c2503cd416f3 # action/v2.0.3
14882
with:
14983
token: "${{ steps.octo-sts.outputs.token }}"
@@ -152,22 +86,31 @@ jobs:
15286
create-branch: true
15387
command: push
15488
commits: "${{ steps.create-commit.outputs.commit }}"
89+
working-directory: images
15590

156-
- name: Create or identify pull request
91+
- name: Create pull request
92+
id: images-pr
93+
if: ${{ steps.create-commit.outputs.has_changes == 'true' }}
15794
env:
15895
GH_TOKEN: ${{ steps.octo-sts.outputs.token }}
15996
PR_NUMBER: ${{ github.event.inputs.pr_number }}
16097
run: |
161-
BRANCH="${{ steps.define-branch.outputs.branch }}"
162-
EXISTING_PR=$(gh pr list --repo DataDog/images --head "$BRANCH" --json url -q '.[0].url' 2>/dev/null || true)
163-
if [[ -n "$EXISTING_PR" ]]; then
164-
echo "PR already exists: $EXISTING_PR"
165-
else
166-
gh pr create \
167-
--repo DataDog/images \
168-
--draft \
169-
--title "Add dd-trace-java-docker-build test images for PR #${PR_NUMBER}" \
170-
--base master \
171-
--head "$BRANCH" \
172-
--body "Adds mirror entries for \`${PR_NUMBER}_merge-*\` test images from DataDog/dd-trace-java-docker-build#${PR_NUMBER}. To use in dd-trace-java CI, set \`TESTER_IMAGE_VERSION_PREFIX: \"${PR_NUMBER}_merge-\"\` in \`.gitlab-ci.yml\`."
173-
fi
98+
PR_URL=$(gh pr create \
99+
--repo DataDog/images \
100+
--draft \
101+
--title "Update dd-trace-java-docker-build test images for PR #${PR_NUMBER}" \
102+
--base master \
103+
--head "${{ steps.define-branch.outputs.branch }}" \
104+
--body "Adds/updates mirror entries for \`${PR_NUMBER}_merge-*\` test images from DataDog/dd-trace-java-docker-build#${PR_NUMBER}. These images should be removed after testing.")
105+
echo "pr_url=${PR_URL}" >> "$GITHUB_OUTPUT"
106+
107+
- name: Comment on source PR with mirror cleanup reminder
108+
if: ${{ steps.update-mirror.outputs.mode == 'add' && steps.create-commit.outputs.has_changes == 'true' }}
109+
env:
110+
GH_TOKEN: ${{ github.token }}
111+
PR_NUMBER: ${{ github.event.inputs.pr_number }}
112+
IMAGES_PR_URL: ${{ steps.images-pr.outputs.pr_url }}
113+
run: |
114+
gh pr comment "${PR_NUMBER}" \
115+
--repo DataDog/dd-trace-java-docker-build \
116+
--body "Mirrored test images for \`${PR_NUMBER}_merge-*\` were added in ${IMAGES_PR_URL}. When you've finished validating the image, please remove the mirrored test images."

‎.github/workflows/update-mirror-digests.yml‎

Lines changed: 25 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -20,15 +20,22 @@ jobs:
2020
scope: DataDog/images
2121
policy: dd-trace-java-docker-build.update-mirror
2222

23+
- name: Checkout DataDog/dd-trace-java-docker-build
24+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
25+
with:
26+
path: dd-trace-java-docker-build
27+
2328
- name: Checkout DataDog/images
2429
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
2530
with:
2631
repository: DataDog/images
2732
token: ${{ steps.octo-sts.outputs.token }}
33+
path: images
2834

2935
- name: Capture images HEAD SHA
3036
id: images-head
3137
run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
38+
working-directory: images
3239

3340
- name: Install crane
3441
run: |
@@ -38,17 +45,17 @@ jobs:
3845
sudo mv crane /usr/local/bin/crane
3946
rm crane.tar.gz
4047
41-
- name: Get baseline digest for ci-base image
48+
- name: Get baseline digest for ci-base image # base variant used to check freshness
4249
id: baseline
4350
run: |
44-
BASELINE=$(awk '/source:.*dd-trace-java-docker-build:ci-base/{found=1; next} found && /digest:/{print $2; exit}' mirror.lock.yaml || true)
51+
BASELINE=$(awk '/source:.*dd-trace-java-docker-build:ci-base/{found=1; next} found && /digest:/{print $2; exit}' images/mirror.lock.yaml || true)
4552
echo "digest=${BASELINE}" >> "$GITHUB_OUTPUT"
4653
echo "Baseline ci-base digest: ${BASELINE:-<none found>}"
4754
4855
- name: Wait for new ci-base image to be published
4956
run: |
5057
BASELINE="${{ steps.baseline.outputs.digest }}"
51-
DEADLINE=$((SECONDS + 1800)) # 30 min timeout
58+
DEADLINE=$((SECONDS + 1800))
5259
echo "Waiting for ci-base digest to differ from: ${BASELINE:-<none>}"
5360
while [[ $SECONDS -lt $DEADLINE ]]; do
5461
CURRENT=$(crane digest ghcr.io/datadog/dd-trace-java-docker-build:ci-base 2>/dev/null || true)
@@ -59,91 +66,35 @@ jobs:
5966
echo "No change yet (current: ${CURRENT:-unavailable}), retrying in 60s..."
6067
sleep 60
6168
done
62-
echo "::error::Timeout after 30 minutes: ci-base digest did not change"
69+
echo "::error::Timeout after 30 minutes: ci-base digest did not change from existing mirror"
6370
exit 1
6471
65-
- name: Resolve digests and update mirror files
66-
run: |
67-
python3 - <<'PYEOF'
68-
import subprocess, re
69-
70-
SOURCE_REPO = "ghcr.io/datadog/dd-trace-java-docker-build"
71-
VARIANTS = [
72-
"base", "7", "8", "11", "17", "21", "25", "tip",
73-
"zulu8", "zulu11", "oracle8", "ibm8",
74-
"semeru8", "semeru11", "semeru17",
75-
"graalvm17", "graalvm21", "graalvm25",
76-
]
77-
78-
# Verify all ci-* entries are already present in both files before proceeding
79-
with open("mirror.yaml", "r") as f:
80-
yaml_content = f.read()
81-
with open("mirror.lock.yaml", "r") as f:
82-
lock_content = f.read()
83-
84-
missing_yaml = [v for v in VARIANTS if f"{SOURCE_REPO}:ci-{v}" not in yaml_content]
85-
missing_lock = [v for v in VARIANTS if f"{SOURCE_REPO}:ci-{v}" not in lock_content]
86-
if missing_yaml or missing_lock:
87-
if missing_yaml:
88-
print(f"::error::ci-* entries missing from mirror.yaml: {missing_yaml}")
89-
if missing_lock:
90-
print(f"::error::ci-* entries missing from mirror.lock.yaml: {missing_lock}")
91-
print("Bootstrap the ci-* entries manually before running this workflow.")
92-
raise SystemExit(1)
93-
94-
print("Resolving digests for ci-* variants...")
95-
digests = {}
96-
for variant in VARIANTS:
97-
tag = f"ci-{variant}"
98-
result = subprocess.run(
99-
["crane", "digest", f"{SOURCE_REPO}:{tag}"],
100-
capture_output=True, text=True, check=True,
101-
)
102-
digest = result.stdout.strip()
103-
digests[variant] = digest
104-
print(f" {tag}: {digest}")
105-
106-
# Update existing digest entries in mirror.lock.yaml in-place
107-
for variant in VARIANTS:
108-
tag = f"ci-{variant}"
109-
source = f"{SOURCE_REPO}:{tag}"
110-
digest = digests[variant]
111-
pattern = rf"( - source: {re.escape(source)}\n digest: )sha256:[a-f0-9]+"
112-
lock_content = re.sub(pattern, rf"\g<1>{digest}", lock_content)
113-
print(f"Updated mirror.lock.yaml: {tag}")
114-
115-
with open("mirror.lock.yaml", "w") as f:
116-
f.write(lock_content)
117-
118-
PYEOF
119-
120-
- name: Check for changes
121-
id: check-changes
122-
run: |
123-
if [[ -z "$(git status -s)" ]]; then
124-
echo "No changes to commit."
125-
echo "commit_changes=false" >> "$GITHUB_OUTPUT"
126-
else
127-
echo "commit_changes=true" >> "$GITHUB_OUTPUT"
128-
fi
72+
- name: Resolve digests and update mirror.lock.yaml files
73+
run: bash "${GITHUB_WORKSPACE}/dd-trace-java-docker-build/scripts/update-ci-image-digests.sh"
74+
working-directory: images
12975

13076
- name: Define branch name
131-
if: steps.check-changes.outputs.commit_changes == 'true'
13277
id: define-branch
13378
run: echo "branch=ci/update-dd-trace-java-docker-build-ci-digests-$(date +'%Y%m%d')" >> "$GITHUB_OUTPUT"
13479

13580
- name: Commit changes
136-
if: steps.check-changes.outputs.commit_changes == 'true'
13781
id: create-commit
13882
run: |
13983
git config user.name "github-actions[bot]"
14084
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
141-
git add mirror.yaml mirror.lock.yaml
85+
git add mirror.lock.yaml
86+
if git diff --cached --quiet; then
87+
echo "No changes detected in mirror files; skipping commit."
88+
echo "has_changes=false" >> "$GITHUB_OUTPUT"
89+
exit 0
90+
fi
14291
git commit -m "chore: Update dd-trace-java-docker-build ci-* image digests"
92+
echo "has_changes=true" >> "$GITHUB_OUTPUT"
14393
echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
94+
working-directory: images
14495

14596
- name: Push changes
146-
if: steps.check-changes.outputs.commit_changes == 'true'
97+
if: ${{ steps.create-commit.outputs.has_changes == 'true' }}
14798
uses: DataDog/commit-headless@05d7b7ee023e2c7d01c47832d420c2503cd416f3 # action/v2.0.3
14899
with:
149100
token: "${{ steps.octo-sts.outputs.token }}"
@@ -152,9 +103,10 @@ jobs:
152103
create-branch: true
153104
command: push
154105
commits: "${{ steps.create-commit.outputs.commit }}"
106+
working-directory: images
155107

156108
- name: Create pull request
157-
if: steps.check-changes.outputs.commit_changes == 'true'
109+
if: ${{ steps.create-commit.outputs.has_changes == 'true' }}
158110
env:
159111
GH_TOKEN: ${{ steps.octo-sts.outputs.token }}
160112
run: |

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,8 @@ Images are built per PR for ease in testing. These test images are prefixed with
3434
To test these images in `dd-trace-java` CI:
3535

3636
1. Open a PR in [DataDog/dd-trace-java-docker-build](https://github.com/DataDog/dd-trace-java-docker-build) with the changes you want to test. Let's say these changes are made in PR #123 ([example](https://github.com/DataDog/dd-trace-java-docker-build/pull/123)).
37-
2. Run the [Create test image mirror PR](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/create-test-mirror-pr.yml) workflow with `pr_number=123`. This automatically opens a PR in [DataDog/images](https://github.com/DataDog/images) that adds mirror entries for the `123_merge-*` test images. The PR should be automatically merged by the `dd-prapprover` bot.
37+
2. Run the [Create test image mirror PR](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/create-test-mirror-pr.yml) workflow with `PR_NUMBER=123`. This automatically opens a PR in [DataDog/images](https://github.com/DataDog/images) that adds mirror entries for the `123_merge-*` test images. Merge the PR if not done automatically by the `dd-prapprover` bot.
3838
3. Open a PR in [DataDog/dd-trace-java](https://github.com/DataDog/dd-trace-java) that sets `BUILDER_IMAGE_VERSION_PREFIX: "123_merge-"` in `.gitlab-ci.yml`. Here, you can check your test images with `DataDog/dd-trace-java` CI.
39-
4. Every time you want to test changes made in PR #123, ensure the test image SHAs in `DataDog/images` are updated. This should be done by running the [Create test image mirror PR](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/create-test-mirror-pr.yml) workflow with `pr_number=123`.
39+
4. Every time you want to test changes made in PR #123, ensure the test image SHAs in `DataDog/images` are updated. This is done by running the [Create test image mirror PR](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/create-test-mirror-pr.yml) workflow each time with `PR_NUMBER=123`.
4040
5. When the test images look good and `DataDog/dd-trace-java` CI is green, merge your `DataDog/dd-trace-java-docker-build` PR #123, close the test `DataDog/dd-trace-java` PR, and **remove the test images from the `DataDog/images` repo**.
41-
6. Finally, run the [Tag new images version](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/docker-tag.yml) workflow. The [Update mirror digests for ci-* images](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/update-mirror-digests.yml) workflow will automatically open a PR in `DataDog/images`, updating the pinned `ci-*` digests.
41+
6. Finally, run the [Tag new images version](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/docker-tag.yml) workflow. The [Update mirror digests for ci-* images](https://github.com/DataDog/dd-trace-java-docker-build/actions/workflows/update-mirror-digests.yml) workflow will automatically open a PR in `DataDog/images`, updating the pinned `ci-*` digests. `dd-trace-java` CI should automatically pick up these updated images a few minutes after the PR is merged.

0 commit comments

Comments
 (0)