Skip to content

Commit 45ebaf7

Browse files
friedrichgfriedrichg
authored
Have packages be built and test by CI for the release (only on master and tags) (#7351)
* Have packages be built by CI Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Fix package testing for modern base images and improve debugging - Upgrade deb test image from debian:10 to debian:12 - Migrate rpm test image from centos:8 (EOL) to rockylinux:9 - Fix systemd unit paths for Rocky Linux (/lib -> /usr/lib) - Add proper Cortex config and data directories in test containers - Increase readiness timeout and add diagnostic output on failure - Run containers in privileged mode for systemd compatibility Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Rename centos-systemd to rockylinux-systemd The base image was migrated from centos:8 to rockylinux:9 in the previous commit. Rename the directory and update all references in the Makefile and test-packages script to match. Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * create test-packages step Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Correctly build packages Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Fix build issues Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Build the binaries once and re-use them Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Prevent packages being rebuilt Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Split to build-dist the building step Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * let's just run the test Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * avoid using qemu for this Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Now each runner only builds the test image for its own architecture. Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Teting for cgroups v2 Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Fix binary permissions before building packages Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Split to its own file Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Implement loop Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> * Apply only to master Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> --------- Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>
1 parent d69156b commit 45ebaf7

8 files changed

Lines changed: 148 additions & 37 deletions

File tree

‎.github/workflows/distribution.yml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
name: distribution
2+
permissions: read-all
3+
on:
4+
push:
5+
branches: [master]
6+
tags:
7+
- v[0-9]+.[0-9]+.[0-9]+** # Tag filters not as strict due to different regex system on Github Actions
8+
9+
jobs:
10+
build-dist:
11+
runs-on: ubuntu-24.04
12+
container:
13+
image: quay.io/cortexproject/build-image:master-ee0b97cc37
14+
steps:
15+
- name: Checkout Repo
16+
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
17+
- name: Setup Git safe.directory
18+
run: |
19+
echo "this step is needed because when running in container, actions/checkout does not set safe.directory effectively."
20+
echo "See https://github.com/actions/runner/issues/2033. We should use --system instead of --global"
21+
git config --system --add safe.directory $GITHUB_WORKSPACE
22+
- name: Sym Link Expected Path to Workspace
23+
run: |
24+
mkdir -p /go/src/github.com/cortexproject/cortex
25+
ln -s $GITHUB_WORKSPACE/* /go/src/github.com/cortexproject/cortex
26+
- name: Build Dist Binaries
27+
run: |
28+
touch build-image/.uptodate
29+
make BUILD_IN_CONTAINER=false dist
30+
- name: Upload Dist Binaries
31+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
32+
with:
33+
name: Dist Binaries
34+
path: dist/
35+
36+
packages:
37+
needs: build-dist
38+
runs-on: ubuntu-24.04
39+
steps:
40+
- name: Checkout Repo
41+
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
42+
- name: Download Dist Binaries
43+
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
44+
with:
45+
name: Dist Binaries
46+
path: dist
47+
- name: Fix Permissions
48+
run: chmod +x dist/cortex-* dist/query-tee-*
49+
- name: Build Packages
50+
run: |
51+
touch dist/.uptodate
52+
make TTY='' ARCHS=amd64 packages
53+
- name: Upload Packages
54+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
55+
with:
56+
name: Packages
57+
path: |
58+
dist/*.deb
59+
dist/*.rpm
60+
dist/*-sha-256
61+
62+
test-packages:
63+
needs: packages
64+
runs-on: ${{ matrix.runner }}
65+
strategy:
66+
fail-fast: false
67+
matrix:
68+
include:
69+
- runner: ubuntu-24.04
70+
arch: amd64
71+
- runner: ubuntu-24.04-arm
72+
arch: arm64
73+
steps:
74+
- name: Checkout Repo
75+
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
76+
- name: Download Packages
77+
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
78+
with:
79+
name: Packages
80+
path: dist
81+
- name: Build Test Images
82+
run: make TTY='' ARCHS=${{ matrix.arch }} packaging/rpm/rockylinux-systemd/.uptodate packaging/deb/debian-systemd/.uptodate
83+
- name: Test Packages
84+
run: ./tools/packaging/test-packages quay.io/cortexproject/ $(cat VERSION) ${{ matrix.arch }}

‎Makefile‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -384,17 +384,19 @@ dist/$(UPTODATE)-packages: dist $(wildcard packaging/deb/**) $(wildcard packagin
384384

385385
endif
386386

387-
# Build both arm64 and amd64 images, so that we can test deb/rpm packages for both architectures.
388-
packaging/rpm/centos-systemd/$(UPTODATE): packaging/rpm/centos-systemd/Dockerfile
389-
$(SUDO) docker build --platform linux/amd64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):amd64 $(@D)/
390-
$(SUDO) docker build --platform linux/arm64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):arm64 $(@D)/
387+
# Build test images for the architectures specified by ARCHS.
388+
packaging/rpm/rockylinux-systemd/$(UPTODATE): packaging/rpm/rockylinux-systemd/Dockerfile
389+
@for arch in $(ARCHS); do \
390+
$(SUDO) docker build --platform linux/$$arch --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):$$arch $(@D)/ ; \
391+
done
391392
touch $@
392393

393394
packaging/deb/debian-systemd/$(UPTODATE): packaging/deb/debian-systemd/Dockerfile
394-
$(SUDO) docker build --platform linux/amd64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):amd64 $(@D)/
395-
$(SUDO) docker build --platform linux/arm64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):arm64 $(@D)/
395+
@for arch in $(ARCHS); do \
396+
$(SUDO) docker build --platform linux/$$arch --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):$$arch $(@D)/ ; \
397+
done
396398
touch $@
397399

398400
.PHONY: test-packages
399-
test-packages: packages packaging/rpm/centos-systemd/$(UPTODATE) packaging/deb/debian-systemd/$(UPTODATE)
400-
./tools/packaging/test-packages $(IMAGE_PREFIX) $(VERSION)
401+
test-packages: packages packaging/rpm/rockylinux-systemd/$(UPTODATE) packaging/deb/debian-systemd/$(UPTODATE)
402+
./tools/packaging/test-packages $(IMAGE_PREFIX) $(VERSION) $(ARCHS)

‎RELEASE.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -102,8 +102,8 @@ To publish a release candidate:
102102
1. Wait until CI pipeline succeeded (once a tag is created, the release process through GitHub actions will be triggered for this tag)
103103
1. Create a pre-release in GitHub
104104
- Write the release notes (including a copy-paste of the changelog)
105-
- Build binaries with `make dist` and attach them to the release
106-
- Build packages with `make packages`, test them with `make test-packages` and attach them to the release
105+
- Download binaries from the `Dist Binaries` artifact in the [distribution](https://github.com/cortexproject/cortex/actions/workflows/distribution.yml) GitHub Actions workflow for the tag and attach them to the release
106+
- Download packages from the `Packages` artifact in the [distribution](https://github.com/cortexproject/cortex/actions/workflows/distribution.yml) GitHub Actions workflow for the tag and attach them to the release
107107
1. [Sign the artifact and generate SBOM for the release](#sing-and-sbom)
108108

109109
### Publish a stable release
@@ -119,8 +119,8 @@ To publish a stable release:
119119
1. Wait until CI pipeline succeeded (once a tag is created, the release process through GitHub actions will be triggered for this tag)
120120
1. Create a release in GitHub
121121
- Write the release notes (including a copy-paste of the changelog)
122-
- Build binaries with `make dist` and attach them to the release
123-
- Build packages with `make packages`, test them with `make test-packages` and attach them to the release
122+
- Download binaries from the `Dist Binaries` artifact in the [distribution](https://github.com/cortexproject/cortex/actions/workflows/distribution.yml) GitHub Actions workflow for the tag and attach them to the release
123+
- Download packages from the `Packages` artifact in the [distribution](https://github.com/cortexproject/cortex/actions/workflows/distribution.yml) GitHub Actions workflow for the tag and attach them to the release
124124
1. [Sign the artifact and generate SBOM for the release](#sing-and-sbom)
125125
1. Merge the release branch `release-x.y` to `master`
126126
- Create `merge-release-X.Y-to-master` branch **from `release-X.Y` branch** locally

‎packaging/deb/debian-systemd/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM debian:10
1+
FROM debian:12
22
ENV container docker
33
ENV LC_ALL C
44
ENV DEBIAN_FRONTEND noninteractive

‎packaging/fpm/Dockerfile‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ FROM alpine:3.23
33
RUN apk add --no-cache \
44
ruby \
55
ruby-dev \
6-
ruby-etc \
76
gcc \
87
git \
98
libc-dev \

‎packaging/rpm/centos-systemd/Dockerfile‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎packaging/rpm/rockylinux-systemd/Dockerfile‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
FROM rockylinux:9
2+
ENV container docker
3+
RUN dnf -y install systemd && dnf clean all && \
4+
(cd /usr/lib/systemd/system/sysinit.target.wants/; for i in *; do [ "$i" = \
5+
systemd-tmpfiles-setup.service ] || rm -f "$i"; done); \
6+
rm -f /usr/lib/systemd/system/multi-user.target.wants/*; \
7+
rm -f /etc/systemd/system/*.wants/*; \
8+
rm -f /usr/lib/systemd/system/local-fs.target.wants/*; \
9+
rm -f /usr/lib/systemd/system/sockets.target.wants/*udev*; \
10+
rm -f /usr/lib/systemd/system/sockets.target.wants/*initctl*; \
11+
rm -f /usr/lib/systemd/system/basic.target.wants/*;
12+
13+
VOLUME [ "/sys/fs/cgroup"]
14+
CMD ["/usr/sbin/init"]

‎tools/packaging/test-packages‎

Lines changed: 35 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ set -euf -o pipefail
44

55
readonly IMAGE_PREFIX=$1
66
readonly VERSION=$2
7+
shift 2
8+
readonly ARCHS=("$@")
79
readonly DISABLE_CLEANUP=${DISABLE_CLEANUP:-0}
810

911
declare -a CONTAINERS=()
@@ -46,18 +48,42 @@ function test_with_systemd() {
4648

4749
echo "Testing $install_command on $image ($platform)"
4850

49-
container=$(docker run --platform="${platform}" --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:ro -itd -v "$(pwd)"/dist:/opt/cortex -p 9009 "${image}")
51+
container=$(docker run --platform="${platform}" --privileged --cgroupns=host --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:rw -td -v "$(pwd)"/dist:/opt/cortex -v "$(pwd)"/docs/configuration:/opt/config:ro -p 9009 "${image}")
5052
CONTAINERS+=("${container}")
5153

52-
port=$(docker inspect --format='{{(index (index .NetworkSettings.Ports "9009/tcp") 0).HostPort}}' "${container}")
54+
# Wait for container to be running (timeout 10 seconds)
55+
timeout=10
56+
while [ $timeout -gt 0 ]; do
57+
if docker inspect --format='{{.State.Running}}' "${container}" | grep -q true; then
58+
break
59+
fi
60+
sleep 1
61+
timeout=$((timeout - 1))
62+
done
5363

54-
docker exec -it "${container}" /bin/bash -c "${install_command}; systemctl start cortex.service; systemctl enable cortex.service"
64+
if [ $timeout -eq 0 ]; then
65+
echo "Container failed to start:"
66+
docker logs "${container}" || true
67+
error "Container ${container} is not running"
68+
fi
5569

56-
ready 10 1 "${port}" || error "Testing image: ${image} with command: '${install_command}' failed"
57-
}
70+
port=$(docker inspect --format='{{(index (index .NetworkSettings.Ports "9009/tcp") 0).HostPort}}' "${container}")
5871

59-
test_with_systemd "${IMAGE_PREFIX}"debian-systemd:amd64 linux/amd64 "dpkg -i /opt/cortex/cortex-${VERSION}_amd64.deb"
60-
test_with_systemd "${IMAGE_PREFIX}"debian-systemd:arm64 linux/arm64 "dpkg -i /opt/cortex/cortex-${VERSION}_arm64.deb"
72+
docker exec "${container}" /bin/bash -c "${install_command}; mkdir -p /tmp/cortex/tsdb /tmp/cortex/tsdb-sync /tmp/cortex/compactor /tmp/cortex/rules /tmp/cortex/alerts /var/lib/cortex/data/tsdb && chown -R cortex:cortex /tmp/cortex /var/lib/cortex; cp /opt/config/single-process-config-blocks-local.yaml /etc/cortex/single-process-config.yaml && chown root:cortex /etc/cortex/single-process-config.yaml; systemctl start cortex.service; systemctl enable cortex.service"
73+
74+
ready 30 2 "${port}" || {
75+
echo "--- curl localhost:${port}/ready ---"
76+
curl -s localhost:"${port}"/ready || true
77+
echo ""
78+
echo "--- systemctl status cortex.service ---"
79+
docker exec "${container}" systemctl status cortex.service || true
80+
echo "--- journalctl -u cortex.service ---"
81+
docker exec "${container}" journalctl -u cortex.service --no-pager -n 50 || true
82+
error "Testing image: ${image} with command: '${install_command}' failed"
83+
}
84+
}
6185

62-
test_with_systemd "${IMAGE_PREFIX}"centos-systemd:amd64 linux/amd64 "rpm -i /opt/cortex/cortex-${VERSION}_amd64.rpm"
63-
test_with_systemd "${IMAGE_PREFIX}"centos-systemd:arm64 linux/arm64 "rpm -i /opt/cortex/cortex-${VERSION}_arm64.rpm"
86+
for arch in "${ARCHS[@]}"; do
87+
test_with_systemd "${IMAGE_PREFIX}"debian-systemd:"${arch}" linux/"${arch}" "dpkg -i /opt/cortex/cortex-${VERSION}_${arch}.deb"
88+
test_with_systemd "${IMAGE_PREFIX}"rockylinux-systemd:"${arch}" linux/"${arch}" "rpm -i /opt/cortex/cortex-${VERSION}_${arch}.rpm"
89+
done

0 commit comments

Comments
 (0)