-
Notifications
You must be signed in to change notification settings - Fork 4
Add image mirroring support #139
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
1e6af16
Add image mirroring support
sarahchen6 69469ea
Tag images with ci- prefix instead
sarahchen6 5513dfc
Add workflows that create images PRs
sarahchen6 0805c51
Separate out ci- tagging logic
sarahchen6 d11eb1a
Merge branch 'master' into sarahchen6/mirror-images
sarahchen6 8e3e105
Re-organize scripts
sarahchen6 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,173 @@ | ||
| name: Create test image mirror PR | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| pr_number: | ||
| description: "PR number in dd-trace-java-docker-build (e.g. 123)" | ||
| required: true | ||
|
|
||
| jobs: | ||
| create-test-mirror-pr: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| id-token: write # Required for OIDC token federation | ||
| contents: read | ||
| steps: | ||
| - uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3 | ||
| id: octo-sts | ||
| with: | ||
| scope: DataDog/images | ||
| policy: dd-trace-java-docker-build.update-mirror | ||
|
|
||
| - name: Checkout DataDog/images | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| repository: DataDog/images | ||
| token: ${{ steps.octo-sts.outputs.token }} | ||
|
|
||
| - name: Capture images HEAD SHA | ||
| id: images-head | ||
| run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Install crane | ||
| run: | | ||
| CRANE_VERSION="0.20.2" | ||
| curl -fsSL "https://github.com/google/go-containerregistry/releases/download/v${CRANE_VERSION}/go-containerregistry_Linux_x86_64.tar.gz" -o crane.tar.gz | ||
| tar -xzf crane.tar.gz crane | ||
| sudo mv crane /usr/local/bin/crane | ||
| rm crane.tar.gz | ||
|
|
||
| - name: Resolve digests and update mirror files | ||
| id: update-mirror | ||
| env: | ||
| PR_NUMBER: ${{ github.event.inputs.pr_number }} | ||
| run: | | ||
| python3 - <<'PYEOF' | ||
| import subprocess, re, os | ||
|
|
||
| SOURCE_REPO = "ghcr.io/datadog/dd-trace-java-docker-build" | ||
| VARIANTS = [ | ||
| "base", "7", "8", "11", "17", "21", "25", "tip", | ||
| "zulu8", "zulu11", "oracle8", "ibm8", | ||
| "semeru8", "semeru11", "semeru17", | ||
| "graalvm17", "graalvm21", "graalvm25", | ||
| ] | ||
|
|
||
| pr_number = os.environ["PR_NUMBER"] | ||
| if not pr_number.isdigit(): | ||
| raise ValueError(f"PR_NUMBER must be numeric, got: {pr_number!r}") | ||
|
|
||
| prefix = f"{pr_number}_merge-" | ||
| print(f"Resolving digests for prefix: {prefix!r}") | ||
|
|
||
| digests = {} | ||
| for variant in VARIANTS: | ||
| tag = f"{prefix}{variant}" | ||
| result = subprocess.run( | ||
| ["crane", "digest", f"{SOURCE_REPO}:{tag}"], | ||
| capture_output=True, text=True, check=True, | ||
| ) | ||
| digest = result.stdout.strip() | ||
| digests[variant] = digest | ||
| print(f" {tag}: {digest}") | ||
|
|
||
| # Check whether entries already exist in mirror.yaml (use base as sentinel) | ||
| with open("mirror.yaml", "r") as f: | ||
| yaml_content = f.read() | ||
|
|
||
| entries_exist = f"{SOURCE_REPO}:{prefix}base" in yaml_content | ||
| mode = "update" if entries_exist else "add" | ||
| print(f"\nMode: {mode} ({'entries exist, updating digests only' if entries_exist else 'no entries found, adding new entries'})") | ||
|
|
||
| github_output = os.environ.get("GITHUB_OUTPUT", "") | ||
| if github_output: | ||
| with open(github_output, "a") as f: | ||
| f.write(f"mode={mode}\n") | ||
|
|
||
| if mode == "add": | ||
| yaml_entries = [] | ||
| for variant in VARIANTS: | ||
| tag = f"{prefix}{variant}" | ||
| source = f"{SOURCE_REPO}:{tag}" | ||
| yaml_entries.append( | ||
| f' - source: "{source}"\n' | ||
| f' dest:\n' | ||
| f' repo: "dd-trace-java-docker-build"\n' | ||
| f' tag: "{tag}"\n' | ||
| f' replication_target: ""\n' | ||
| ) | ||
| with open("mirror.yaml", "a") as f: | ||
| f.write("".join(yaml_entries)) | ||
| print(f"Appended {len(yaml_entries)} entries to mirror.yaml") | ||
|
|
||
| # Always update mirror.lock.yaml: replace digest in-place if entry exists, append if not | ||
| with open("mirror.lock.yaml", "r") as f: | ||
| lock_content = f.read() | ||
|
|
||
| for variant in VARIANTS: | ||
| tag = f"{prefix}{variant}" | ||
| source = f"{SOURCE_REPO}:{tag}" | ||
| digest = digests[variant] | ||
| pattern = rf"( - source: {re.escape(source)}\n digest: )sha256:[a-f0-9]+" | ||
| if re.search(pattern, lock_content): | ||
| lock_content = re.sub(pattern, rf"\g<1>{digest}", lock_content) | ||
| print(f"Updated mirror.lock.yaml: {tag}") | ||
| else: | ||
| lock_content = lock_content.rstrip("\n") + "\n" | ||
| lock_content += f" - source: {source}\n digest: {digest}\n" | ||
| print(f"Appended to mirror.lock.yaml: {tag}") | ||
|
|
||
| with open("mirror.lock.yaml", "w") as f: | ||
| f.write(lock_content) | ||
|
|
||
| PYEOF | ||
|
|
||
| - name: Define branch name | ||
| id: define-branch | ||
| run: echo "branch=ci/add-dd-trace-java-docker-build-test-images-pr${{ github.event.inputs.pr_number }}" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Commit changes | ||
| id: create-commit | ||
| env: | ||
| PR_NUMBER: ${{ github.event.inputs.pr_number }} | ||
| MODE: ${{ steps.update-mirror.outputs.mode }} | ||
| run: | | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
| git add mirror.yaml mirror.lock.yaml | ||
| if [[ "$MODE" == "update" ]]; then | ||
| git commit -m "chore: Update dd-trace-java-docker-build test image digests for PR #${PR_NUMBER}" | ||
| else | ||
| git commit -m "chore: Add dd-trace-java-docker-build test images for PR #${PR_NUMBER}" | ||
| fi | ||
| echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Push changes | ||
| uses: DataDog/commit-headless@05d7b7ee023e2c7d01c47832d420c2503cd416f3 # action/v2.0.3 | ||
| with: | ||
| token: "${{ steps.octo-sts.outputs.token }}" | ||
| branch: "${{ steps.define-branch.outputs.branch }}" | ||
| head-sha: "${{ steps.images-head.outputs.sha }}" | ||
| create-branch: true | ||
| command: push | ||
| commits: "${{ steps.create-commit.outputs.commit }}" | ||
|
|
||
| - name: Create or identify pull request | ||
| env: | ||
| GH_TOKEN: ${{ steps.octo-sts.outputs.token }} | ||
| PR_NUMBER: ${{ github.event.inputs.pr_number }} | ||
| run: | | ||
| BRANCH="${{ steps.define-branch.outputs.branch }}" | ||
| EXISTING_PR=$(gh pr list --repo DataDog/images --head "$BRANCH" --json url -q '.[0].url' 2>/dev/null || true) | ||
| if [[ -n "$EXISTING_PR" ]]; then | ||
| echo "PR already exists: $EXISTING_PR" | ||
| else | ||
| gh pr create \ | ||
| --repo DataDog/images \ | ||
| --draft \ | ||
| --title "Add dd-trace-java-docker-build test images for PR #${PR_NUMBER}" \ | ||
| --base master \ | ||
| --head "$BRANCH" \ | ||
| --body "Adds mirror entries for \`${PR_NUMBER}_merge-*\` test images from DataDog/dd-trace-java-docker-build#${PR_NUMBER}. To use in dd-trace-java CI, set \`TESTER_IMAGE_VERSION_PREFIX: \"${PR_NUMBER}_merge-\"\` in \`.gitlab-ci.yml\`." | ||
| fi | ||
|
sarahchen6 marked this conversation as resolved.
Outdated
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,167 @@ | ||
| name: Update mirror digests for ci-* images | ||
|
|
||
| on: | ||
| workflow_run: | ||
| workflows: ["Tag new images version"] | ||
| types: [completed] | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| update-mirror-digests: | ||
| if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| id-token: write # Required for OIDC token federation | ||
| contents: read | ||
| steps: | ||
| - uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3 | ||
| id: octo-sts | ||
| with: | ||
| scope: DataDog/images | ||
| policy: dd-trace-java-docker-build.update-mirror | ||
|
|
||
| - name: Checkout DataDog/images | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| repository: DataDog/images | ||
| token: ${{ steps.octo-sts.outputs.token }} | ||
|
|
||
| - name: Capture images HEAD SHA | ||
| id: images-head | ||
| run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Install crane | ||
| run: | | ||
| CRANE_VERSION="0.20.2" | ||
| curl -fsSL "https://github.com/google/go-containerregistry/releases/download/v${CRANE_VERSION}/go-containerregistry_Linux_x86_64.tar.gz" -o crane.tar.gz | ||
| tar -xzf crane.tar.gz crane | ||
| sudo mv crane /usr/local/bin/crane | ||
| rm crane.tar.gz | ||
|
|
||
| - name: Get baseline digest for ci-base image | ||
| id: baseline | ||
| run: | | ||
| BASELINE=$(awk '/source:.*dd-trace-java-docker-build:ci-base/{found=1; next} found && /digest:/{print $2; exit}' mirror.lock.yaml || true) | ||
| echo "digest=${BASELINE}" >> "$GITHUB_OUTPUT" | ||
| echo "Baseline ci-base digest: ${BASELINE:-<none found>}" | ||
|
|
||
| - name: Wait for new ci-base image to be published | ||
| run: | | ||
| BASELINE="${{ steps.baseline.outputs.digest }}" | ||
| DEADLINE=$((SECONDS + 1800)) # 30 min timeout | ||
| echo "Waiting for ci-base digest to differ from: ${BASELINE:-<none>}" | ||
| while [[ $SECONDS -lt $DEADLINE ]]; do | ||
| CURRENT=$(crane digest ghcr.io/datadog/dd-trace-java-docker-build:ci-base 2>/dev/null || true) | ||
| if [[ -n "$CURRENT" && "$CURRENT" != "$BASELINE" ]]; then | ||
| echo "New ci-base digest detected: $CURRENT" | ||
| exit 0 | ||
| fi | ||
| echo "No change yet (current: ${CURRENT:-unavailable}), retrying in 60s..." | ||
| sleep 60 | ||
| done | ||
| echo "::error::Timeout after 30 minutes: ci-base digest did not change" | ||
| exit 1 | ||
|
|
||
| - name: Resolve digests and update mirror files | ||
| run: | | ||
| python3 - <<'PYEOF' | ||
| import subprocess, re | ||
|
|
||
| SOURCE_REPO = "ghcr.io/datadog/dd-trace-java-docker-build" | ||
| VARIANTS = [ | ||
| "base", "7", "8", "11", "17", "21", "25", "tip", | ||
| "zulu8", "zulu11", "oracle8", "ibm8", | ||
| "semeru8", "semeru11", "semeru17", | ||
| "graalvm17", "graalvm21", "graalvm25", | ||
| ] | ||
|
|
||
| # Verify all ci-* entries are already present in both files before proceeding | ||
| with open("mirror.yaml", "r") as f: | ||
| yaml_content = f.read() | ||
| with open("mirror.lock.yaml", "r") as f: | ||
| lock_content = f.read() | ||
|
|
||
| missing_yaml = [v for v in VARIANTS if f"{SOURCE_REPO}:ci-{v}" not in yaml_content] | ||
| missing_lock = [v for v in VARIANTS if f"{SOURCE_REPO}:ci-{v}" not in lock_content] | ||
| if missing_yaml or missing_lock: | ||
| if missing_yaml: | ||
| print(f"::error::ci-* entries missing from mirror.yaml: {missing_yaml}") | ||
| if missing_lock: | ||
| print(f"::error::ci-* entries missing from mirror.lock.yaml: {missing_lock}") | ||
| print("Bootstrap the ci-* entries manually before running this workflow.") | ||
| raise SystemExit(1) | ||
|
|
||
| print("Resolving digests for ci-* variants...") | ||
| digests = {} | ||
| for variant in VARIANTS: | ||
| tag = f"ci-{variant}" | ||
| result = subprocess.run( | ||
| ["crane", "digest", f"{SOURCE_REPO}:{tag}"], | ||
| capture_output=True, text=True, check=True, | ||
| ) | ||
| digest = result.stdout.strip() | ||
| digests[variant] = digest | ||
| print(f" {tag}: {digest}") | ||
|
|
||
| # Update existing digest entries in mirror.lock.yaml in-place | ||
| for variant in VARIANTS: | ||
| tag = f"ci-{variant}" | ||
| source = f"{SOURCE_REPO}:{tag}" | ||
| digest = digests[variant] | ||
| pattern = rf"( - source: {re.escape(source)}\n digest: )sha256:[a-f0-9]+" | ||
| lock_content = re.sub(pattern, rf"\g<1>{digest}", lock_content) | ||
| print(f"Updated mirror.lock.yaml: {tag}") | ||
|
|
||
| with open("mirror.lock.yaml", "w") as f: | ||
| f.write(lock_content) | ||
|
|
||
| PYEOF | ||
|
|
||
| - name: Check for changes | ||
| id: check-changes | ||
| run: | | ||
| if [[ -z "$(git status -s)" ]]; then | ||
| echo "No changes to commit." | ||
| echo "commit_changes=false" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "commit_changes=true" >> "$GITHUB_OUTPUT" | ||
| fi | ||
|
|
||
| - name: Define branch name | ||
| if: steps.check-changes.outputs.commit_changes == 'true' | ||
| id: define-branch | ||
| run: echo "branch=ci/update-dd-trace-java-docker-build-ci-digests-$(date +'%Y%m%d')" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Commit changes | ||
| if: steps.check-changes.outputs.commit_changes == 'true' | ||
| id: create-commit | ||
| run: | | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
| git add mirror.yaml mirror.lock.yaml | ||
| git commit -m "chore: Update dd-trace-java-docker-build ci-* image digests" | ||
| echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Push changes | ||
| if: steps.check-changes.outputs.commit_changes == 'true' | ||
| uses: DataDog/commit-headless@05d7b7ee023e2c7d01c47832d420c2503cd416f3 # action/v2.0.3 | ||
| with: | ||
| token: "${{ steps.octo-sts.outputs.token }}" | ||
| branch: "${{ steps.define-branch.outputs.branch }}" | ||
| head-sha: "${{ steps.images-head.outputs.sha }}" | ||
| create-branch: true | ||
| command: push | ||
| commits: "${{ steps.create-commit.outputs.commit }}" | ||
|
|
||
| - name: Create pull request | ||
| if: steps.check-changes.outputs.commit_changes == 'true' | ||
| env: | ||
| GH_TOKEN: ${{ steps.octo-sts.outputs.token }} | ||
| run: | | ||
| gh pr create \ | ||
| --repo DataDog/images \ | ||
| --draft \ | ||
| --title "Update dd-trace-java-docker-build ci-* image digests" \ | ||
| --base master \ | ||
| --head "${{ steps.define-branch.outputs.branch }}" \ | ||
| --body "Automated digest update for \`dd-trace-java-docker-build\` \`ci-*\` images after tagging." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❔ question: About this step, what motivated you to switch from bash to python as you had a working version in bash previously?
🎯 suggestion: It might be easier to run a script that is versioned in the repository rather than adding it inline into yml? 🤷 And as they are similar scripts in the update-mirror-digests workflow, you can reuse it there too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No good reason - I started with trying to bring the bash over, but found it easier to format and test out python with Claude... I think I ended up over-engineering with python though, so will switch back to bash!
Hmm that could make the workflows more readable too 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, think about what could be the simplest bricks to be reused by the workflows.
Ideally, if we could run them outside the workflow (when needed or to test them) like you did for the pin system test script, that would be ideal.
About python or bash, whatever. Just pick one that work well for the task and that will be easy to maintain by the team :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In 8e3e105, I split the logic into 3 scripts:
The
create-test-mirror-pr.ymlworkflow callscreate-test-mirror-entries.shwhich either adds test image entries or updates the existing test image digests inDataDog/images.The
update-mirror-digests.ymlworkflow callsupdate-ci-image-digests.shwhich updates the existingci-*image digests.Both workflows source
get-image-digests.shwhich usescraneto get the latest digests for all CI variants inghcr.io, given the prefixci-orPRNUM_merge-.The scripts were successful locally!