-
Notifications
You must be signed in to change notification settings - Fork 4
Add image mirroring support #139
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
1e6af16
Add image mirroring support
sarahchen6 69469ea
Tag images with ci- prefix instead
sarahchen6 5513dfc
Add workflows that create images PRs
sarahchen6 0805c51
Separate out ci- tagging logic
sarahchen6 d11eb1a
Merge branch 'master' into sarahchen6/mirror-images
sarahchen6 8e3e105
Re-organize scripts
sarahchen6 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| name: Create test image mirror PR | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| pr_number: | ||
| description: "PR number in dd-trace-java-docker-build (e.g. 123)" | ||
| required: true | ||
|
|
||
| jobs: | ||
| create-test-mirror-pr: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| id-token: write # Required for OIDC token federation | ||
| contents: read | ||
| pull-requests: write | ||
| steps: | ||
| - uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3 | ||
| id: octo-sts | ||
| with: | ||
| scope: DataDog/images | ||
| policy: dd-trace-java-docker-build.update-mirror | ||
|
|
||
| - name: Checkout DataDog/dd-trace-java-docker-build | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| path: dd-trace-java-docker-build | ||
|
|
||
| - name: Checkout DataDog/images | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| repository: DataDog/images | ||
| token: ${{ steps.octo-sts.outputs.token }} | ||
| path: images | ||
|
|
||
| - name: Capture images HEAD SHA | ||
| id: images-head | ||
| run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" | ||
| working-directory: images | ||
|
|
||
| - name: Install crane | ||
| run: | | ||
| CRANE_VERSION="0.20.2" | ||
| curl -fsSL "https://github.com/google/go-containerregistry/releases/download/v${CRANE_VERSION}/go-containerregistry_Linux_x86_64.tar.gz" -o crane.tar.gz | ||
| tar -xzf crane.tar.gz crane | ||
| sudo mv crane /usr/local/bin/crane | ||
| rm crane.tar.gz | ||
|
|
||
| - name: Resolve digests and add new or update existing digests in mirror files | ||
| id: update-mirror | ||
| env: | ||
| PR_NUMBER: ${{ github.event.inputs.pr_number }} | ||
| run: bash "${GITHUB_WORKSPACE}/dd-trace-java-docker-build/scripts/create-test-mirror-entries.sh" | ||
| working-directory: images | ||
|
|
||
| - name: Define branch name | ||
| id: define-branch | ||
| run: echo "branch=ci/add-dd-trace-java-docker-build-test-images-pr${{ github.event.inputs.pr_number }}" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Commit changes | ||
| id: create-commit | ||
| env: | ||
| PR_NUMBER: ${{ github.event.inputs.pr_number }} | ||
| MODE: ${{ steps.update-mirror.outputs.mode }} | ||
| run: | | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
| git add mirror.yaml mirror.lock.yaml | ||
| if git diff --cached --quiet; then | ||
| echo "No changes detected in mirror files; skipping commit." | ||
| echo "has_changes=false" >> "$GITHUB_OUTPUT" | ||
| exit 0 | ||
| fi | ||
| git commit -m "chore: Update dd-trace-java-docker-build test image digests for PR #${PR_NUMBER}" | ||
| echo "has_changes=true" >> "$GITHUB_OUTPUT" | ||
| echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" | ||
| working-directory: images | ||
|
|
||
| - name: Push changes | ||
| if: ${{ steps.create-commit.outputs.has_changes == 'true' }} | ||
| uses: DataDog/commit-headless@05d7b7ee023e2c7d01c47832d420c2503cd416f3 # action/v2.0.3 | ||
| with: | ||
| token: "${{ steps.octo-sts.outputs.token }}" | ||
| branch: "${{ steps.define-branch.outputs.branch }}" | ||
| head-sha: "${{ steps.images-head.outputs.sha }}" | ||
| create-branch: true | ||
| command: push | ||
| commits: "${{ steps.create-commit.outputs.commit }}" | ||
| working-directory: images | ||
|
|
||
| - name: Create pull request | ||
| id: images-pr | ||
| if: ${{ steps.create-commit.outputs.has_changes == 'true' }} | ||
| env: | ||
| GH_TOKEN: ${{ steps.octo-sts.outputs.token }} | ||
| PR_NUMBER: ${{ github.event.inputs.pr_number }} | ||
| run: | | ||
| PR_URL=$(gh pr create \ | ||
| --repo DataDog/images \ | ||
| --draft \ | ||
| --title "Update dd-trace-java-docker-build test images for PR #${PR_NUMBER}" \ | ||
| --base master \ | ||
| --head "${{ steps.define-branch.outputs.branch }}" \ | ||
| --body "Adds/updates mirror entries for \`${PR_NUMBER}_merge-*\` test images from DataDog/dd-trace-java-docker-build#${PR_NUMBER}. These images should be removed after testing.") | ||
| echo "pr_url=${PR_URL}" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Comment on source PR with mirror cleanup reminder | ||
| if: ${{ steps.update-mirror.outputs.mode == 'add' && steps.create-commit.outputs.has_changes == 'true' }} | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| PR_NUMBER: ${{ github.event.inputs.pr_number }} | ||
| IMAGES_PR_URL: ${{ steps.images-pr.outputs.pr_url }} | ||
| run: | | ||
| gh pr comment "${PR_NUMBER}" \ | ||
| --repo DataDog/dd-trace-java-docker-build \ | ||
| --body "Mirrored test images for \`${PR_NUMBER}_merge-*\` were added in ${IMAGES_PR_URL}. When you've finished validating the image, please remove the mirrored test images." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| name: Update mirror digests for ci-* images | ||
|
|
||
| on: | ||
| workflow_run: | ||
| workflows: ["Tag new images version"] | ||
| types: [completed] | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| update-mirror-digests: | ||
| if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| id-token: write # Required for OIDC token federation | ||
| contents: read | ||
| steps: | ||
| - uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3 | ||
| id: octo-sts | ||
| with: | ||
| scope: DataDog/images | ||
| policy: dd-trace-java-docker-build.update-mirror | ||
|
|
||
| - name: Checkout DataDog/dd-trace-java-docker-build | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| path: dd-trace-java-docker-build | ||
|
|
||
| - name: Checkout DataDog/images | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| repository: DataDog/images | ||
| token: ${{ steps.octo-sts.outputs.token }} | ||
| path: images | ||
|
|
||
| - name: Capture images HEAD SHA | ||
| id: images-head | ||
| run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" | ||
| working-directory: images | ||
|
|
||
| - name: Install crane | ||
| run: | | ||
| CRANE_VERSION="0.20.2" | ||
| curl -fsSL "https://github.com/google/go-containerregistry/releases/download/v${CRANE_VERSION}/go-containerregistry_Linux_x86_64.tar.gz" -o crane.tar.gz | ||
| tar -xzf crane.tar.gz crane | ||
| sudo mv crane /usr/local/bin/crane | ||
| rm crane.tar.gz | ||
|
|
||
| - name: Get baseline digest for ci-base image # base variant used to check freshness | ||
| id: baseline | ||
| run: | | ||
| BASELINE=$(awk '/source:.*dd-trace-java-docker-build:ci-base/{found=1; next} found && /digest:/{print $2; exit}' images/mirror.lock.yaml || true) | ||
| echo "digest=${BASELINE}" >> "$GITHUB_OUTPUT" | ||
| echo "Baseline ci-base digest: ${BASELINE:-<none found>}" | ||
|
|
||
| - name: Wait for new ci-base image to be published | ||
| run: | | ||
| BASELINE="${{ steps.baseline.outputs.digest }}" | ||
| DEADLINE=$((SECONDS + 1800)) | ||
| echo "Waiting for ci-base digest to differ from: ${BASELINE:-<none>}" | ||
| while [[ $SECONDS -lt $DEADLINE ]]; do | ||
| CURRENT=$(crane digest ghcr.io/datadog/dd-trace-java-docker-build:ci-base 2>/dev/null || true) | ||
| if [[ -n "$CURRENT" && "$CURRENT" != "$BASELINE" ]]; then | ||
| echo "New ci-base digest detected: $CURRENT" | ||
| exit 0 | ||
| fi | ||
| echo "No change yet (current: ${CURRENT:-unavailable}), retrying in 60s..." | ||
| sleep 60 | ||
| done | ||
| echo "::error::Timeout after 30 minutes: ci-base digest did not change from existing mirror" | ||
| exit 1 | ||
|
|
||
| - name: Resolve digests and update mirror.lock.yaml files | ||
| run: bash "${GITHUB_WORKSPACE}/dd-trace-java-docker-build/scripts/update-ci-image-digests.sh" | ||
| working-directory: images | ||
|
|
||
| - name: Define branch name | ||
| id: define-branch | ||
| run: echo "branch=ci/update-dd-trace-java-docker-build-ci-digests-$(date +'%Y%m%d')" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Commit changes | ||
| id: create-commit | ||
| run: | | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
| git add mirror.lock.yaml | ||
| if git diff --cached --quiet; then | ||
| echo "No changes detected in mirror files; skipping commit." | ||
| echo "has_changes=false" >> "$GITHUB_OUTPUT" | ||
| exit 0 | ||
| fi | ||
| git commit -m "chore: Update dd-trace-java-docker-build ci-* image digests" | ||
| echo "has_changes=true" >> "$GITHUB_OUTPUT" | ||
| echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" | ||
| working-directory: images | ||
|
|
||
| - name: Push changes | ||
| if: ${{ steps.create-commit.outputs.has_changes == 'true' }} | ||
| uses: DataDog/commit-headless@05d7b7ee023e2c7d01c47832d420c2503cd416f3 # action/v2.0.3 | ||
| with: | ||
| token: "${{ steps.octo-sts.outputs.token }}" | ||
| branch: "${{ steps.define-branch.outputs.branch }}" | ||
| head-sha: "${{ steps.images-head.outputs.sha }}" | ||
| create-branch: true | ||
| command: push | ||
| commits: "${{ steps.create-commit.outputs.commit }}" | ||
| working-directory: images | ||
|
|
||
| - name: Create pull request | ||
| if: ${{ steps.create-commit.outputs.has_changes == 'true' }} | ||
| env: | ||
| GH_TOKEN: ${{ steps.octo-sts.outputs.token }} | ||
| run: | | ||
| gh pr create \ | ||
| --repo DataDog/images \ | ||
| --draft \ | ||
| --title "Update dd-trace-java-docker-build ci-* image digests" \ | ||
| --base master \ | ||
| --head "${{ steps.define-branch.outputs.branch }}" \ | ||
| --body "Automated digest update for \`dd-trace-java-docker-build\` \`ci-*\` images after tagging." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| #!/usr/bin/env bash | ||
| # create-test-mirror-entries.sh — add or update {PR_NUMBER}_merge-* test image entries | ||
| # in mirror.yaml and mirror.lock.yaml files in the DataDog/images repo. | ||
| # | ||
| # This script is called in the create-test-mirror-pr Github workflow. | ||
| # It must be run from the root of DataDog/images and requires crane to be installed. | ||
| # | ||
| # Required env var: | ||
| # PR_NUMBER — pull request number in dd-trace-java-docker-build (numeric) | ||
| # | ||
| # Outputs (when GITHUB_OUTPUT is set): | ||
| # mode=add|update, indicating whether the test images were added or their digests were updated | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
|
|
||
| if ! [[ "${PR_NUMBER}" =~ ^[0-9]+$ ]]; then | ||
| echo "::error::PR_NUMBER must be numeric (got: '${PR_NUMBER}')" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| readonly PREFIX="${PR_NUMBER}_merge-" | ||
|
|
||
| # Check if entries already exist in mirror.yaml (use base variant as tester) | ||
| if grep -qF "ghcr.io/datadog/dd-trace-java-docker-build:${PREFIX}base" mirror.yaml; then | ||
| MODE="update" | ||
| echo "Entries for '${PREFIX}' already exist — updating digests only" | ||
| else | ||
| MODE="add" | ||
| echo "No entries found for '${PREFIX}' — adding new entries" | ||
| fi | ||
|
|
||
| if [[ -n "${GITHUB_OUTPUT:-}" ]]; then | ||
| echo "mode=${MODE}" >> "${GITHUB_OUTPUT}" | ||
| fi | ||
|
|
||
| # shellcheck source=scripts/get-image-digests.sh | ||
| source "${SCRIPT_DIR}/get-image-digests.sh" | ||
|
|
||
| if [[ "$MODE" == "add" ]]; then | ||
| for variant in "${CI_VARIANTS[@]}"; do | ||
| tag="${PREFIX}${variant}" | ||
| printf ' - source: "%s:%s"\n dest:\n repo: "dd-trace-java-docker-build"\n tag: "%s"\n replication_target: ""\n' \ | ||
| "ghcr.io/datadog/dd-trace-java-docker-build" "${tag}" "${tag}" >> mirror.yaml | ||
| done | ||
| echo "Appended ${#CI_VARIANTS[@]} entries to mirror.yaml" | ||
|
|
||
| for variant in "${CI_VARIANTS[@]}"; do | ||
| tag="${PREFIX}${variant}" | ||
| printf ' - source: %s:%s\n digest: %s\n' \ | ||
| "ghcr.io/datadog/dd-trace-java-docker-build" "${tag}" "${DIGESTS[$variant]}" >> mirror.lock.yaml | ||
| done | ||
| echo "Appended ${#CI_VARIANTS[@]} entries to mirror.lock.yaml" | ||
| else | ||
| for variant in "${CI_VARIANTS[@]}"; do | ||
| tag="${PREFIX}${variant}" | ||
| update_digest "${tag}" "${DIGESTS[$variant]}" mirror.lock.yaml | ||
| echo "Updated mirror.lock.yaml: ${tag}" | ||
| done | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| #!/usr/bin/env bash | ||
| # get-image-digests.sh — source this script to populate CI_VARIANTS and DIGESTS. | ||
| # | ||
| # Required env vars: | ||
| # PREFIX — tag prefix (e.g. "ci-" or "138_merge-") | ||
| # | ||
| # After sourcing, callers have access to: | ||
| # CI_VARIANTS — indexed array of variant names | ||
| # DIGESTS — associative array mapping image variant to its latest digest (sha256:...) | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| readonly CI_VARIANTS=(base 7 8 11 17 21 25 tip zulu8 zulu11 oracle8 ibm8 semeru8 semeru11 semeru17 graalvm17 graalvm21 graalvm25) | ||
|
|
||
| # update_digest TAG DIGEST FILE | ||
| # Finds the line "source: ...:TAG" and updates the digest on the following line. | ||
| update_digest() { | ||
| local tag="$1" digest="$2" file="$3" | ||
| awk -v tag="${tag}" -v digest="${digest}" ' | ||
| $0 ~ ("source:.*:" tag "$") { found=1 } | ||
| found && /digest:/ { sub(/digest: sha256:[a-f0-9]*/, "digest: " digest); found=0 } | ||
| { print } | ||
| ' "${file}" > "${file}.tmp" && mv "${file}.tmp" "${file}" | ||
| } | ||
|
|
||
| echo "Resolving digests for ${#CI_VARIANTS[@]} variants (prefix: '${PREFIX}')..." >&2 | ||
| declare -A DIGESTS | ||
| for variant in "${CI_VARIANTS[@]}"; do | ||
| tag="${PREFIX}${variant}" | ||
| echo -n " ${tag} ... " >&2 | ||
| DIGESTS[$variant]="$(crane digest "ghcr.io/datadog/dd-trace-java-docker-build:${tag}")" | ||
| echo "${DIGESTS[$variant]}" >&2 | ||
| done |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.