Skip to content

Commit 2b35f9b

Browse files
committed
Add image sync configuration and workflow
Includes: - `images.yaml` for declaring images to mirror - `scripts/sync.sh` for mirroring images using `skopeo` - GitHub Actions workflow for automated syncing into GHCR
1 parent 933413a commit 2b35f9b

4 files changed

Lines changed: 285 additions & 0 deletions

File tree

.github/workflows/sync.yml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: Sync images to GHCR
2+
3+
on:
4+
# Manual trigger, with an optional dry run.
5+
workflow_dispatch:
6+
inputs:
7+
dry_run:
8+
description: "Print actions without pushing"
9+
type: boolean
10+
default: false
11+
# Re-sync whenever the image list changes.
12+
push:
13+
branches: [main]
14+
paths:
15+
- images.yaml
16+
- scripts/sync.sh
17+
- .github/workflows/sync.yml
18+
# Keep mirrored tags fresh (daily at 03:00 UTC).
19+
schedule:
20+
- cron: "0 3 * * *"
21+
22+
# Allow only one sync at a time; let an in-progress run finish.
23+
concurrency:
24+
group: sync-images
25+
cancel-in-progress: false
26+
27+
jobs:
28+
sync:
29+
runs-on: ubuntu-latest
30+
permissions:
31+
contents: read
32+
packages: write
33+
steps:
34+
- name: Checkout
35+
uses: actions/checkout@v4
36+
37+
- name: Install skopeo
38+
run: |
39+
sudo apt-get update
40+
sudo apt-get install -y skopeo
41+
42+
- name: Sync images
43+
env:
44+
GHCR_NAMESPACE: ghcr.io/openprojectx
45+
GHCR_USERNAME: ${{ github.actor }}
46+
GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
47+
DRY_RUN: ${{ inputs.dry_run || 'false' }}
48+
run: bash scripts/sync.sh

README.md

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# image-sync
2+
3+
Mirror container images from public registries (Docker Hub, quay.io,
4+
registry.k8s.io, …) into [GHCR](https://ghcr.io) under
5+
**`ghcr.io/openprojectx`** using [skopeo](https://github.com/containers/skopeo).
6+
7+
Useful when you want a stable, rate-limit-free, geographically closer copy of
8+
upstream images that your clusters and CI can pull from a single namespace.
9+
10+
## How it works
11+
12+
1. [`images.yaml`](images.yaml) declares the images to mirror.
13+
2. [`scripts/sync.sh`](scripts/sync.sh) reads that file and runs
14+
`skopeo copy` for each entry into `ghcr.io/openprojectx`.
15+
3. The [`Sync images to GHCR`](.github/workflows/sync.yml) workflow runs the
16+
script on a daily schedule, on every change to the image list, and on
17+
manual dispatch. It authenticates to GHCR with the built-in `GITHUB_TOKEN`.
18+
19+
The destination encodes the **source registry** as a leading namespace, so
20+
images with the same name from different registries never collide:
21+
22+
```
23+
docker.io/library/nginx:1.27 -> ghcr.io/openprojectx/dockerhub/library/nginx:1.27
24+
registry.k8s.io/pause:3.10 -> ghcr.io/openprojectx/k8s/pause:3.10
25+
quay.io/skopeo/stable:latest -> ghcr.io/openprojectx/quay/skopeo/stable:latest
26+
```
27+
28+
Registry aliases: `docker.io``dockerhub`, `registry.k8s.io``k8s`,
29+
`quay.io``quay`, `gcr.io``gcr`, `ghcr.io``ghcr`,
30+
`mcr.microsoft.com``mcr`, `public.ecr.aws``ecr`. Unknown hosts use the
31+
host name verbatim.
32+
33+
## Adding an image
34+
35+
Edit [`images.yaml`](images.yaml):
36+
37+
```yaml
38+
images:
39+
- source: docker.io/library/redis:7.4 # -> ghcr.io/openprojectx/dockerhub/library/redis:7.4
40+
- source: registry.k8s.io/coredns/coredns:v1.11.1
41+
# -> ghcr.io/openprojectx/k8s/coredns/coredns:v1.11.1
42+
- source: quay.io/prometheus/prometheus:v2.53.0
43+
target: monitoring/prometheus:v2.53.0 # explicit destination overrides the default
44+
- source: docker.io/library/nginx:1.27
45+
all: false # copy only the runner's arch
46+
```
47+
48+
- `source` — full upstream reference (`registry/repo:tag` or `…@sha256:…`).
49+
- `target` *(optional)* — explicit destination `repo[:tag]` under
50+
`ghcr.io/openprojectx`. **Omit it** to use the default convention
51+
(`<registry-alias>/<upstream/repo/path>:<tag>`); supply it only when you
52+
want a custom path.
53+
- `all` *(optional, default `true`)* — mirror every architecture in the
54+
manifest list (`skopeo copy --all`).
55+
56+
Commit to `main` and the workflow re-syncs automatically.
57+
58+
## Running locally
59+
60+
Requires `skopeo` and `yq` (v4, mikefarah).
61+
62+
```bash
63+
# Preview without pushing
64+
DRY_RUN=true bash scripts/sync.sh
65+
66+
# Real sync (must be logged in to ghcr.io with packages:write)
67+
echo "$GITHUB_TOKEN" | skopeo login ghcr.io -u <your-username> --password-stdin
68+
bash scripts/sync.sh
69+
```
70+
71+
Environment variables: `GHCR_NAMESPACE` (default `ghcr.io/openprojectx`),
72+
`IMAGES_FILE` (default `images.yaml`), `DRY_RUN`, and `GHCR_USERNAME` /
73+
`GHCR_TOKEN` for in-script login.
74+
75+
## Notes
76+
77+
- New packages are private by default. To make a mirror public, open the
78+
package in the org's **Packages** settings and change its visibility (or set
79+
it once via the API).
80+
- The `GITHUB_TOKEN` only has `packages: write` for the `OpenProjectX` org, so
81+
the workflow can push to `ghcr.io/openprojectx/*` out of the box.

images.yaml

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Images to mirror into ghcr.io/openprojectx
2+
#
3+
# Each entry copies an image from a public registry into GHCR. The source
4+
# registry is encoded as a leading namespace so that images with the same
5+
# name from different registries never collide.
6+
#
7+
# source: full reference of the upstream image (registry/repo:tag)
8+
# target: (optional) repo[:tag] under ghcr.io/openprojectx. Omit it to use
9+
# the default naming convention (recommended):
10+
# <registry-alias>/<upstream/repo/path>:<tag>
11+
# all: (optional, default true) copy every architecture in the
12+
# manifest list instead of just the runner's platform.
13+
#
14+
# Registry aliases: docker.io -> dockerhub, registry.k8s.io -> k8s,
15+
# quay.io -> quay, gcr.io -> gcr, ghcr.io -> ghcr, mcr.microsoft.com -> mcr,
16+
# public.ecr.aws -> ecr. Unknown hosts use the host name verbatim.
17+
#
18+
# Default-naming examples:
19+
# docker.io/library/nginx:1.27 -> ghcr.io/openprojectx/dockerhub/library/nginx:1.27
20+
# registry.k8s.io/pause:3.10 -> ghcr.io/openprojectx/k8s/pause:3.10
21+
# quay.io/skopeo/stable:latest -> ghcr.io/openprojectx/quay/skopeo/stable:latest
22+
23+
images:
24+
# - source: docker.io/library/alpine:3.20
25+
# - source: docker.io/library/busybox:1.36
26+
# - source: registry.k8s.io/pause:3.10
27+
# - source: quay.io/skopeo/stable:latest
28+
- source: localstack/localstack:4

scripts/sync.sh

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Mirror the images declared in images.yaml into ghcr.io/openprojectx
4+
# using skopeo.
5+
#
6+
# Environment variables:
7+
# GHCR_NAMESPACE Destination namespace (default: ghcr.io/openprojectx)
8+
# IMAGES_FILE Path to the image list (default: images.yaml)
9+
# DRY_RUN If "true", print actions instead of running skopeo
10+
# GHCR_USERNAME Username for skopeo login (optional; login skipped if unset)
11+
# GHCR_TOKEN Token/password for login (optional; login skipped if unset)
12+
#
13+
set -euo pipefail
14+
15+
GHCR_NAMESPACE="${GHCR_NAMESPACE:-ghcr.io/openprojectx}"
16+
IMAGES_FILE="${IMAGES_FILE:-images.yaml}"
17+
DRY_RUN="${DRY_RUN:-false}"
18+
19+
log() { printf '\033[0;34m==>\033[0m %s\n' "$*"; }
20+
warn() { printf '\033[0;33mwarn:\033[0m %s\n' "$*" >&2; }
21+
die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; }
22+
23+
command -v skopeo >/dev/null 2>&1 || die "skopeo is not installed"
24+
command -v yq >/dev/null 2>&1 || die "yq is not installed"
25+
[[ -f "$IMAGES_FILE" ]] || die "images file not found: $IMAGES_FILE"
26+
27+
# Optionally authenticate to GHCR so we can push.
28+
if [[ -n "${GHCR_USERNAME:-}" && -n "${GHCR_TOKEN:-}" ]]; then
29+
log "Logging in to ghcr.io as ${GHCR_USERNAME}"
30+
echo "$GHCR_TOKEN" | skopeo login ghcr.io -u "$GHCR_USERNAME" --password-stdin
31+
else
32+
warn "GHCR_USERNAME/GHCR_TOKEN not set; assuming skopeo is already authenticated"
33+
fi
34+
35+
# Map a source registry host to a short, friendly namespace so that images
36+
# with the same name from different registries don't collide in GHCR.
37+
# Unknown hosts fall back to the host itself (with any port colon sanitized).
38+
alias_for() {
39+
case "$1" in
40+
docker.io|index.docker.io|registry-1.docker.io) echo "dockerhub" ;;
41+
registry.k8s.io|k8s.gcr.io) echo "k8s" ;;
42+
quay.io) echo "quay" ;;
43+
gcr.io) echo "gcr" ;;
44+
ghcr.io) echo "ghcr" ;;
45+
mcr.microsoft.com) echo "mcr" ;;
46+
public.ecr.aws) echo "ecr" ;;
47+
*) echo "${1//:/_}" ;;
48+
esac
49+
}
50+
51+
# Derive a default target ("alias/repo:tag") from a source reference when the
52+
# config does not specify one explicitly. The source registry is encoded as a
53+
# leading namespace and the full upstream repo path is preserved, e.g.
54+
# docker.io/library/busybox:1.36 -> dockerhub/library/busybox:1.36
55+
# registry.k8s.io/pause:3.10 -> k8s/pause:3.10
56+
default_target() {
57+
local source="$1" host rest first repo tag lastseg
58+
59+
# Split off the registry host. A leading component is a host only if it
60+
# looks like one (contains a dot or port colon, or is localhost); otherwise
61+
# the reference is a Docker Hub shorthand.
62+
if [[ "$source" == */* ]]; then
63+
first="${source%%/*}"
64+
if [[ "$first" == *.* || "$first" == *:* || "$first" == "localhost" ]]; then
65+
host="$first"
66+
rest="${source#*/}"
67+
else
68+
host="docker.io"
69+
rest="$source"
70+
fi
71+
else
72+
host="docker.io"
73+
rest="library/$source" # e.g. "nginx" -> "library/nginx"
74+
fi
75+
76+
# Separate the tag/digest from the repo path (the tag is the part after the
77+
# last colon, but only when that colon follows the last path separator).
78+
if [[ "$rest" == *@* ]]; then
79+
repo="${rest%@*}"
80+
tag="latest" # digest pulls have no tag; override 'target' to set one
81+
else
82+
lastseg="${rest##*/}"
83+
if [[ "$lastseg" == *:* ]]; then
84+
tag="${lastseg##*:}"
85+
repo="${rest%:*}"
86+
else
87+
repo="$rest"
88+
tag="latest"
89+
fi
90+
fi
91+
92+
printf '%s/%s:%s' "$(alias_for "$host")" "$repo" "$tag"
93+
}
94+
95+
count="$(yq '.images | length' "$IMAGES_FILE")"
96+
[[ "$count" =~ ^[0-9]+$ && "$count" -gt 0 ]] || die "no images found in $IMAGES_FILE"
97+
log "Found $count image(s) to mirror into $GHCR_NAMESPACE"
98+
99+
failures=0
100+
for i in $(seq 0 $((count - 1))); do
101+
source="$(yq -r ".images[$i].source" "$IMAGES_FILE")"
102+
target="$(yq -r ".images[$i].target // \"\"" "$IMAGES_FILE")"
103+
all="$(yq -r ".images[$i].all // \"true\"" "$IMAGES_FILE")"
104+
105+
[[ -n "$source" && "$source" != "null" ]] || die "images[$i] is missing 'source'"
106+
[[ -n "$target" ]] || target="$(default_target "$source")"
107+
108+
dest="${GHCR_NAMESPACE}/${target}"
109+
110+
copy_args=(--retry-times 3)
111+
[[ "$all" == "true" ]] && copy_args+=(--all)
112+
113+
log "[$((i + 1))/$count] $source -> $dest"
114+
if [[ "$DRY_RUN" == "true" ]]; then
115+
echo " DRY_RUN: skopeo copy ${copy_args[*]} docker://$source docker://$dest"
116+
continue
117+
fi
118+
119+
if ! skopeo copy "${copy_args[@]}" "docker://$source" "docker://$dest"; then
120+
warn "failed to copy $source"
121+
failures=$((failures + 1))
122+
fi
123+
done
124+
125+
if [[ "$failures" -gt 0 ]]; then
126+
die "$failures image(s) failed to sync"
127+
fi
128+
log "All images synced successfully"

0 commit comments

Comments
 (0)