Skip to content

Commit 6c08a3f

Browse files
ValentaTomasValentaTomas
committed
refactor: simplify to "build everything, calver release per run"
- workflow_dispatch with no inputs; pick the branch in the GitHub UI. - Always build every entry in kernel_versions.txt for amd64 and arm64 in parallel (one runner per arch). - One release per run, tagged YYYY.MM.DD (with .N suffix on collision), with every binary uploaded as vmlinux-<version>-<arch>.bin (plus the legacy vmlinux-<version>.bin for amd64). - Same binaries pushed to GCS under kernels/vmlinux-<version>/... Drops the per-build content-hash version_name machinery and the validate.py helper that fed it.
1 parent 1e82d8d commit 6c08a3f

4 files changed

Lines changed: 73 additions & 381 deletions

File tree

‎.github/workflows/release.yml‎

Lines changed: 53 additions & 143 deletions
Original file line numberDiff line numberDiff line change
@@ -2,77 +2,39 @@ name: Manual Build & Release
22

33
on:
44
workflow_dispatch:
5-
inputs:
6-
kernel_versions:
7-
description: 'Comma-separated kernel versions to build (default: kernel_versions.txt)'
8-
required: false
9-
type: string
10-
default: ''
11-
build_amd64:
12-
description: 'Build for amd64 architecture'
13-
required: false
14-
type: boolean
15-
default: true
16-
build_arm64:
17-
description: 'Build for arm64 architecture'
18-
required: false
19-
type: boolean
20-
default: true
215

226
permissions:
237
contents: write
248
id-token: write
259

2610
jobs:
27-
validate:
28-
runs-on: ubuntu-24.04
29-
outputs:
30-
build_matrix: ${{ steps.validate.outputs.build_matrix }}
31-
versions: ${{ steps.validate.outputs.versions }}
32-
has_new_artifacts: ${{ steps.validate.outputs.has_new_artifacts }}
33-
steps:
34-
- uses: actions/checkout@v4
35-
36-
- uses: actions/setup-python@v5
37-
with:
38-
python-version: '3.11'
39-
40-
- id: validate
41-
env:
42-
GH_TOKEN: ${{ github.token }}
43-
run: |
44-
python3 scripts/validate.py \
45-
--kernel-versions "${{ inputs.kernel_versions }}" \
46-
--build-amd64 "${{ inputs.build_amd64 }}" \
47-
--build-arm64 "${{ inputs.build_arm64 }}"
48-
4911
build:
50-
needs: validate
51-
if: needs.validate.outputs.has_new_artifacts == 'true'
12+
name: Build kernels (${{ matrix.arch }})
5213
strategy:
5314
fail-fast: false
54-
matrix: ${{ fromJson(needs.validate.outputs.build_matrix) }}
15+
matrix:
16+
include:
17+
- arch: amd64
18+
target_arch: x86_64
19+
runner: ubuntu-24.04
20+
- arch: arm64
21+
target_arch: arm64
22+
runner: ubuntu-24.04-arm
5523
runs-on: ${{ matrix.runner }}
5624
steps:
5725
- uses: actions/checkout@v4
5826

59-
- name: Build kernel ${{ matrix.version_name }} (${{ matrix.arch }})
60-
env:
61-
VERSION_NAME: ${{ matrix.version_name }}
62-
run: |
63-
target_arch="${{ matrix.arch }}"
64-
[ "$target_arch" = "amd64" ] && target_arch="x86_64"
65-
sudo VERSION_NAME="$VERSION_NAME" ./build.sh "${{ matrix.kernel_version }}" "$target_arch"
27+
- name: Build kernels
28+
run: sudo TARGET_ARCH=${{ matrix.target_arch }} ./build.sh
6629

6730
- uses: actions/upload-artifact@v4
6831
with:
69-
name: ${{ matrix.version_name }}-${{ matrix.arch }}
32+
name: kernels-${{ matrix.arch }}
7033
path: ./builds
7134
retention-days: 7
7235

7336
publish:
74-
needs: [validate, build]
75-
if: needs.validate.outputs.has_new_artifacts == 'true'
37+
needs: build
7638
runs-on: ubuntu-24.04
7739
steps:
7840
- uses: actions/checkout@v4
@@ -84,108 +46,56 @@ jobs:
8446
path: ./builds
8547
merge-multiple: true
8648

87-
- name: Show build artifacts
88-
run: find ./builds -type f | head -50
49+
- name: Prepare release assets
50+
run: |
51+
set -euo pipefail
52+
mkdir -p release-assets
53+
for dir in ./builds/vmlinux-*/; do
54+
name=$(basename "$dir")
55+
[ -f "$dir/amd64/vmlinux.bin" ] && cp "$dir/amd64/vmlinux.bin" "release-assets/${name}-amd64.bin"
56+
[ -f "$dir/arm64/vmlinux.bin" ] && cp "$dir/arm64/vmlinux.bin" "release-assets/${name}-arm64.bin"
57+
# Legacy non-arch-suffixed asset (= amd64) for backwards compat.
58+
[ -f "$dir/vmlinux.bin" ] && cp "$dir/vmlinux.bin" "release-assets/${name}.bin"
59+
done
60+
ls -la release-assets/
8961
90-
- name: Create or update releases
62+
- name: Pick calver tag
63+
id: tag
9164
env:
9265
GH_TOKEN: ${{ github.token }}
93-
VERSIONS: ${{ needs.validate.outputs.versions }}
9466
run: |
9567
set -euo pipefail
96-
git config user.name "github-actions[bot]"
97-
git config user.email "github-actions[bot]@users.noreply.github.com"
98-
99-
echo "$VERSIONS" | jq -c '.[]' | while read -r entry; do
100-
version_name=$(echo "$entry" | jq -r '.version_name')
101-
kernel_version=$(echo "$entry" | jq -r '.kernel_version')
102-
103-
if ! gh release view "$version_name" >/dev/null 2>&1; then
104-
if ! git rev-parse "refs/tags/$version_name" >/dev/null 2>&1; then
105-
git tag "$version_name"
106-
git push origin "$version_name"
107-
fi
108-
gh release create "$version_name" \
109-
--title "Kernel $version_name" \
110-
--notes "Linux kernel $kernel_version built from configs at ${{ github.sha }}"
111-
fi
112-
113-
existing=$(gh release view "$version_name" --json assets -q '.assets[].name' || true)
114-
115-
for arch in amd64 arm64; do
116-
local_path="./builds/$version_name/$arch/vmlinux.bin"
117-
[ -f "$local_path" ] || continue
118-
119-
asset="vmlinux-${arch}.bin"
120-
if echo "$existing" | grep -qx "$asset"; then
121-
echo "Release $version_name: $asset exists, skipping"
122-
else
123-
tmp=$(mktemp -d)/$asset
124-
cp "$local_path" "$tmp"
125-
gh release upload "$version_name" "$tmp"
126-
rm -f "$tmp"
127-
fi
128-
129-
# Legacy non-arch-suffixed asset (amd64 only) for backwards compat.
130-
if [ "$arch" = "amd64" ] && ! echo "$existing" | grep -qx "vmlinux.bin"; then
131-
tmp=$(mktemp -d)/vmlinux.bin
132-
cp "$local_path" "$tmp"
133-
gh release upload "$version_name" "$tmp"
134-
rm -f "$tmp"
135-
fi
136-
done
68+
base="$(date -u +%Y.%m.%d)"
69+
tag="$base"
70+
n=1
71+
while gh release view "$tag" >/dev/null 2>&1; do
72+
n=$((n + 1))
73+
tag="${base}.${n}"
13774
done
75+
echo "tag=$tag" >> "$GITHUB_OUTPUT"
13876
139-
deploy:
140-
needs: [validate, publish]
141-
if: needs.validate.outputs.has_new_artifacts == 'true'
142-
runs-on: ubuntu-24.04
143-
steps:
144-
- uses: actions/checkout@v4
145-
146-
- uses: google-github-actions/auth@v2
147-
with:
148-
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
149-
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT_EMAIL }}
150-
151-
- uses: google-github-actions/setup-gcloud@v2
152-
153-
- name: Download release assets and upload to GCS
77+
- name: Create release
15478
env:
15579
GH_TOKEN: ${{ github.token }}
156-
GCP_BUCKET_NAME: ${{ vars.GCP_BUCKET_NAME }}
157-
VERSIONS: ${{ needs.validate.outputs.versions }}
15880
run: |
15981
set -euo pipefail
160-
echo "$VERSIONS" | jq -c '.[]' | while read -r entry; do
161-
version_name=$(echo "$entry" | jq -r '.version_name')
162-
163-
for arch in amd64 arm64; do
164-
asset="vmlinux-${arch}.bin"
165-
dl_dir="./dl/$version_name/$arch"
166-
mkdir -p "$dl_dir"
167-
if ! gh release download "$version_name" \
168-
--repo "${{ github.repository }}" \
169-
--pattern "$asset" \
170-
--output "$dl_dir/vmlinux.bin" 2>/dev/null; then
171-
continue
172-
fi
82+
git config user.name "github-actions[bot]"
83+
git config user.email "github-actions[bot]@users.noreply.github.com"
84+
git tag "${{ steps.tag.outputs.tag }}"
85+
git push origin "${{ steps.tag.outputs.tag }}"
86+
gh release create "${{ steps.tag.outputs.tag }}" \
87+
--title "Kernels ${{ steps.tag.outputs.tag }}" \
88+
--notes "Built from commit ${{ github.sha }} on ${{ github.ref_name }}" \
89+
./release-assets/*
17390
174-
gcs_path="gs://${GCP_BUCKET_NAME}/kernels/${version_name}/${arch}/vmlinux.bin"
175-
if gcloud storage ls "$gcs_path" >/dev/null 2>&1; then
176-
echo "GCS: $gcs_path exists, skipping"
177-
else
178-
gcloud storage cp "$dl_dir/vmlinux.bin" "$gcs_path"
179-
fi
91+
- uses: google-github-actions/auth@v2
92+
with:
93+
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
94+
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT_EMAIL }}
18095

181-
# Legacy non-arch path (amd64 only) for backwards compat.
182-
if [ "$arch" = "amd64" ]; then
183-
legacy_path="gs://${GCP_BUCKET_NAME}/kernels/${version_name}/vmlinux.bin"
184-
if gcloud storage ls "$legacy_path" >/dev/null 2>&1; then
185-
echo "GCS: $legacy_path exists, skipping"
186-
else
187-
gcloud storage cp "$dl_dir/vmlinux.bin" "$legacy_path"
188-
fi
189-
fi
190-
done
191-
done
96+
- uses: google-github-actions/upload-cloud-storage@v2
97+
with:
98+
path: ./builds
99+
destination: ${{ vars.GCP_BUCKET_NAME }}/kernels
100+
gzip: false
101+
parent: false

‎README.md‎

Lines changed: 12 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,6 @@
44

55
This project builds custom Linux kernels for Firecracker microVMs from the same kernel sources as the official Firecracker repo, using the configuration files (and optional patches) that live in this repo.
66

7-
Each kernel build is identified by a content hash of its inputs (configs + patches), so changing a flag or adding a patch produces a new, traceable artifact:
8-
9-
```
10-
vmlinux-<kernel_version>_<sha256[:7]>
11-
```
12-
137
## Prerequisites
148

159
- Linux environment (for building kernels)
@@ -29,26 +23,27 @@ vmlinux-<kernel_version>_<sha256[:7]>
2923
./build.sh 6.1.158 arm64
3024
```
3125

32-
Output: `builds/vmlinux-<version>_<hash>/<arch>/vmlinux.bin` where `<arch>` is `amd64` or `arm64` (Go/OCI convention). For x86_64 a legacy copy is also placed at `builds/vmlinux-<version>_<hash>/vmlinux.bin`.
26+
Output: `builds/vmlinux-<version>/<arch>/vmlinux.bin` where `<arch>` is `amd64` or `arm64` (Go/OCI convention). For x86_64 a legacy copy is also placed at `builds/vmlinux-<version>/vmlinux.bin`.
3327

3428
## Releasing
3529

36-
1. Run the **Manual Build & Release** workflow (Actions → Manual Build & Release → Run workflow).
37-
2. The workflow:
38-
- Computes a content hash for each kernel version from its configs and patches.
39-
- Skips arches whose artifact is already present in the matching GitHub release.
40-
- Builds the missing arches, creates/updates the `vmlinux-<version>_<hash>` release, uploads `vmlinux-amd64.bin` / `vmlinux-arm64.bin` (and a legacy `vmlinux.bin` for amd64), and pushes the same files to GCS under `gs://$GCP_BUCKET_NAME/kernels/<version_name>/`.
30+
1. Pick the branch and run the **Manual Build & Release** workflow (Actions → Manual Build & Release → Run workflow → branch). The workflow takes no inputs.
31+
2. Every kernel version in `kernel_versions.txt` is built for both `amd64` and `arm64` in parallel.
32+
3. A single GitHub release is created per run, tagged with calver `YYYY.MM.DD` (with a `.N` suffix for additional runs the same day). The release contains every binary for that commit:
4133

42-
### Workflow inputs
34+
```
35+
vmlinux-<version>-amd64.bin
36+
vmlinux-<version>-arm64.bin
37+
vmlinux-<version>.bin # legacy (= amd64) for backwards compat
38+
```
4339

44-
- `kernel_versions` (optional): comma-separated kernel versions. Defaults to all versions in `kernel_versions.txt`.
45-
- `build_amd64` / `build_arm64` (optional, default `true`): which architectures to build.
40+
4. The same binaries are uploaded to GCS at `gs://$GCP_BUCKET_NAME/kernels/vmlinux-<version>/<arch>/vmlinux.bin`.
4641

4742
## New kernel in E2B's infra
4843
_Note: these steps should give you a new kernel on your self-hosted E2B using https://github.com/e2b-dev/infra_
4944

50-
- Run the release workflow to publish the new kernel build.
51-
- Update `DefaultKernelVersion` in [packages/api/internal/cfg/model.go](https://github.com/e2b-dev/infra/blob/main/packages/api/internal/cfg/model.go) to the new `vmlinux-<version>_<hash>` name.
45+
- Run the release workflow on the branch with the new config/patch.
46+
- Update `DefaultKernelVersion` in [packages/api/internal/cfg/model.go](https://github.com/e2b-dev/infra/blob/main/packages/api/internal/cfg/model.go) if you changed the kernel version.
5247
- Build and deploy `api`.
5348

5449
## Architecture naming

0 commit comments

Comments
 (0)