Skip to content

Commit 1ca2ce3

Browse files
authored
Test composefs (#81)
* Add composefs label and node image override to cluster test Use the Ginkgo Label("composefs") decorator so that the cluster creation test can be selected with --label-filter. Read BINK_NODE_IMAGE env var to allow overriding the default node image at runtime. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Alice Frosi <afrosi@redhat.com> * Add test-integration-composefs Makefile target Provide a dedicated target to run only composefs-labeled tests using ginkgo --label-filter, mirroring the existing test targets. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Alice Frosi <afrosi@redhat.com> * Add composefs integration test GHA workflow Run the composefs-labeled subset of integration tests against the composefs node image to validate the composefs backend on every push and PR. Sets BINK_NODE_IMAGE to the composefs disk image so the test uses it instead of the default. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Alice Frosi <afrosi@redhat.com> * Guard /etc/fstab sed in cloud-init for composefs Composefs-based images do not have /etc/fstab, causing cloud-init runcmd to fail and report error status. Traditional bootc images use a regular filesystem where /etc/fstab lists mount points. Composefs-based images use a content-addressed, read-only erofs filesystem overlay where root mounting is handled by ostree deployment metadata and the initramfs boot logic, not by /etc/fstab. Since there are no traditional block-device mounts to describe, the file is simply absent. Guard the sed command with a file-existence check so cloud-init succeeds on both composefs and non-composefs images. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Alice Frosi <afrosi@redhat.com> * Fix target-imgref override for composefs images The origin file path and format differ between ostree and composefs: - ostree: /ostree/deploy/default/deploy/<checksum>.<serial>.origin - composefs: /sysroot/state/deploy/<verity>/<verity>.origin - composefs uses spaces around '=' in key-value pairs Use find to locate the origin file in both paths, and fix the sed pattern to match both formats. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Alice Frosi <afrosi@redhat.com> * Treat cloud-init error status as terminal Previously WaitForCloudInit retried forever when cloud-init reported "error", but that is a terminal state — the status will never change to "done". Accept it with a warning instead of blocking for the full timeout. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Alice Frosi <afrosi@redhat.com> * Increase waitForAPI timeout from 2 to 5 minutes Signed-off-by: Alice Frosi <afrosi@redhat.com> --------- Signed-off-by: Alice Frosi <afrosi@redhat.com>
1 parent 3300f82 commit 1ca2ce3

6 files changed

Lines changed: 176 additions & 11 deletions

File tree

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
name: Integration Tests (composefs)
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
concurrency:
10+
group: integration-composefs-${{ github.head_ref || github.ref }}
11+
cancel-in-progress: true
12+
13+
env:
14+
BINK_IMAGES: >-
15+
ghcr.io/bootc-dev/bink/cluster:latest
16+
ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk-composefs
17+
ghcr.io/bootc-dev/bink/dns:latest
18+
EXTERNAL_IMAGES: >-
19+
quay.io/libpod/busybox:latest
20+
21+
jobs:
22+
integration-tests-composefs:
23+
runs-on: ubuntu-latest
24+
timeout-minutes: 120
25+
26+
steps:
27+
- name: Checkout code
28+
uses: actions/checkout@v4
29+
30+
- name: Configure kernel for nested containers
31+
run: |
32+
sudo aa-teardown 2>/dev/null || true
33+
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
34+
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
35+
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
36+
37+
- name: Enable KSM (Kernel Same-page Merging)
38+
run: |
39+
sudo sh -c 'echo 1 > /sys/kernel/mm/ksm/run'
40+
sudo sh -c 'echo 5000 > /sys/kernel/mm/ksm/pages_to_scan'
41+
cat /sys/kernel/mm/ksm/run
42+
43+
- name: Set up Go
44+
uses: actions/setup-go@v5
45+
with:
46+
go-version-file: go.mod
47+
cache: true
48+
49+
- name: Install system dependencies
50+
run: |
51+
sudo apt-get update
52+
sudo apt-get install -y \
53+
podman \
54+
libgpgme-dev \
55+
libbtrfs-dev \
56+
libdevmapper-dev \
57+
pkg-config
58+
59+
- name: Set up KVM
60+
run: |
61+
sudo chmod 666 /dev/kvm
62+
ls -la /dev/kvm
63+
64+
- name: Configure Podman
65+
run: |
66+
podman --version
67+
sudo mkdir -p /etc/containers
68+
echo '{"defaultAction":"SCMP_ACT_ALLOW"}' | sudo tee /etc/containers/seccomp.json
69+
printf '[containers]\napparmor_profile = "unconfined"\nseccomp_profile = "/etc/containers/seccomp.json"\n' | sudo tee /etc/containers/containers.conf
70+
grep -q '^root:' /etc/subuid || echo 'root:100000:65536' | sudo tee -a /etc/subuid
71+
grep -q '^root:' /etc/subgid || echo 'root:100000:65536' | sudo tee -a /etc/subgid
72+
sudo systemctl start podman.socket
73+
sudo podman info --format '{{.Store.GraphRoot}}'
74+
75+
- name: Build bink binary
76+
run: sudo make build-bink
77+
78+
- name: Verify prerequisites
79+
run: |
80+
test -f ./bink
81+
sudo podman images --format "table {{.Repository}}:{{.Tag}}\t{{.Size}}"
82+
df -h /
83+
free -h
84+
85+
- name: Get image digests
86+
id: digests
87+
run: |
88+
ALL_DIGESTS=""
89+
for img in $BINK_IMAGES $EXTERNAL_IMAGES; do
90+
digest=$(skopeo inspect --no-creds "docker://${img}" --format '{{.Digest}}')
91+
echo "${img}: ${digest}"
92+
ALL_DIGESTS="${ALL_DIGESTS}${digest}"
93+
done
94+
echo "hash=$(echo -n "${ALL_DIGESTS}" | sha256sum | cut -d' ' -f1)" >> "$GITHUB_OUTPUT"
95+
96+
- name: Restore cached images
97+
id: image-cache
98+
uses: actions/cache/restore@v4
99+
with:
100+
path: /tmp/podman-image-cache
101+
key: podman-images-v2-composefs-${{ steps.digests.outputs.hash }}
102+
103+
- name: Load cached images
104+
if: steps.image-cache.outputs.cache-hit == 'true'
105+
run: |
106+
for f in /tmp/podman-image-cache/*.tar; do
107+
sudo podman load -i "$f"
108+
done
109+
sudo podman images --format "table {{.Repository}}:{{.Tag}}\t{{.Size}}"
110+
111+
- name: Pre-pull container images
112+
if: steps.image-cache.outputs.cache-hit != 'true'
113+
run: |
114+
mkdir -p /tmp/podman-image-cache
115+
for img in $BINK_IMAGES $EXTERNAL_IMAGES; do
116+
sudo podman pull "$img"
117+
name=$(echo "$img" | sed 's|[/:]|_|g')
118+
sudo podman save -o "/tmp/podman-image-cache/${name}.tar" "$img"
119+
done
120+
sudo chown -R $(id -u):$(id -g) /tmp/podman-image-cache
121+
122+
- name: Save image cache
123+
if: steps.image-cache.outputs.cache-hit != 'true'
124+
uses: actions/cache/save@v4
125+
with:
126+
path: /tmp/podman-image-cache
127+
key: podman-images-v2-composefs-${{ steps.digests.outputs.hash }}
128+
129+
- name: Run composefs integration tests
130+
run: sudo make test-integration-composefs
131+
timeout-minutes: 90
132+
env:
133+
CONTAINER_HOST: unix:///run/podman/podman.sock
134+
BINK_NODE_IMAGE: ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk-composefs
135+
136+
- name: Collect logs
137+
if: failure()
138+
run: .github/collect-logs.sh
139+
140+
- name: Upload logs
141+
if: failure()
142+
uses: actions/upload-artifact@v4
143+
with:
144+
name: test-logs-composefs
145+
path: /tmp/bink-logs/
146+
147+
- name: Cleanup test clusters
148+
if: always()
149+
run: |
150+
sudo podman ps -a --filter "name=k8s-test-bink" --format '{{.Names}}' | \
151+
xargs -r sudo podman rm -f 2>/dev/null || true
152+
sudo podman volume prune -f 2>/dev/null || true

Makefile

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,11 @@ test-integration:
8888
@echo "=== Running Integration Tests ==="
8989
$(GINKGO) -v --procs=$(TEST_PROCS) $(GINKGO_FOCUS_FLAG) --fail-fast --randomize-all --randomize-suites test/integration/
9090

91+
test-integration-composefs:
92+
@test -f ./$(BINK_BINARY) || (echo "Error: bink binary not found. Run 'make build-bink' first" && exit 1)
93+
@echo "=== Running Composefs Integration Tests ==="
94+
$(GINKGO) -v --label-filter="composefs" --fail-fast test/integration/
95+
9196
test-integration-quick:
9297
@test -f ./$(BINK_BINARY) || (echo "Error: bink binary not found. Run 'make build-bink' first" && exit 1)
9398
@echo "=== Running Quick Integration Tests ==="

internal/cluster/cluster.go

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -112,14 +112,18 @@ func (c *Cluster) WaitForCloudInit(ctx context.Context, nodeName string, timeout
112112

113113
c.logger.Infof("cloud-init status: %s (attempt %d/%d)", status, i, maxRetries)
114114

115-
// Accept "done" (done with or without warnings is OK)
116-
if status == "done" {
115+
switch status {
116+
case "done":
117117
c.logger.Info("✓ cloud-init completed")
118118
return nil
119+
case "error":
120+
c.logger.Warn("cloud-init finished with errors (non-critical modules may have failed)")
121+
fullStatus, _ := sshClient.Exec(ctx, "cloud-init status --long")
122+
c.logger.Debugf("cloud-init full status:\n%s", fullStatus)
123+
return nil
119124
}
120125

121126
if i == maxRetries {
122-
// Get full status for debugging
123127
fullStatus, _ := sshClient.Exec(ctx, "cloud-init status --long")
124128
return fmt.Errorf("timeout waiting for cloud-init to complete on %s. Status: %s\nFull status:\n%s",
125129
nodeName, status, fullStatus)

internal/kube/client.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ func waitForAPI(ctx context.Context, config *rest.Config) error {
7171
return err
7272
}
7373

74-
deadline := time.After(2 * time.Minute)
74+
deadline := time.After(5 * time.Minute)
7575
ticker := time.NewTicker(2 * time.Second)
7676
defer ticker.Stop()
7777

internal/node/templates/user-data.yaml.tmpl

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ write_files:
6868

6969
runcmd:
7070
- swapoff -a
71-
- sed -i '/swap/d' /etc/fstab
71+
- "[ -f /etc/fstab ] && sed -i '/swap/d' /etc/fstab || true"
7272
- modprobe br_netfilter
7373
- echo 'br_netfilter' > /etc/modules-load.d/k8s-bridge.conf
7474
- sysctl -w net.ipv4.ip_forward=1
@@ -88,8 +88,8 @@ runcmd:
8888
{{- if .TargetImgRef}}
8989
- |
9090
mount -o remount,rw /sysroot
91-
DEPLOY_PATH=$(bootc status --json | jq -r '.status.booted.ostree.deploySerial // empty')
92-
CHECKSUM=$(bootc status --json | jq -r '.status.booted.ostree.checksum')
93-
ORIGIN="/ostree/deploy/default/deploy/${CHECKSUM}.${DEPLOY_PATH}.origin"
94-
sed -i 's|^container-image-reference=.*|container-image-reference=ostree-unverified-registry:{{.TargetImgRef}}|' "$ORIGIN"
91+
ORIGIN=$(find /ostree/deploy/default/deploy/ /sysroot/state/deploy/ -name "*.origin" -type f 2>/dev/null | head -1)
92+
if [ -n "$ORIGIN" ]; then
93+
sed -i '/^container-image-reference/s|=.*|= ostree-unverified-registry:{{.TargetImgRef}}|' "$ORIGIN"
94+
fi
9595
{{- end}}

test/integration/cluster_test.go

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,15 +33,19 @@ var _ = Describe("Cluster Lifecycle", func() {
3333
helpers.CleanupCluster(clusterName)
3434
})
3535

36-
It("should create and initialize a complete Kubernetes cluster", func() {
36+
It("should create and initialize a complete Kubernetes cluster", Label("composefs"), func() {
3737
customNodeName := "cp1"
3838
kubeconfigPath := fmt.Sprintf("../../kubeconfig-%s", clusterName)
3939
defer helpers.CleanupKubeconfig(kubeconfigPath)
4040

4141
targetImgRef := "registry.cluster.local:5000/node:latest"
4242

4343
By("Creating cluster with --expose, custom node name, memory ballooning, and target-imgref")
44-
cmd := helpers.BinkCmd("cluster", "start", "--cluster-name", clusterName, "--api-port", "0", "--memory", "1900", "--max-memory", "4096", "--node-name", customNodeName, "--expose", kubeconfigPath, "--target-imgref", targetImgRef)
44+
args := []string{"cluster", "start", "--cluster-name", clusterName, "--api-port", "0", "--memory", "1900", "--max-memory", "4096", "--node-name", customNodeName, "--expose", kubeconfigPath, "--target-imgref", targetImgRef}
45+
if nodeImage := os.Getenv("BINK_NODE_IMAGE"); nodeImage != "" {
46+
args = append(args, "--node-image", nodeImage)
47+
}
48+
cmd := helpers.BinkCmd(args...)
4549
session := helpers.RunCommand(cmd)
4650

4751
By("Verifying cluster creation command succeeded")

0 commit comments

Comments
 (0)