feat: hash-based manual release workflow (#20)

* feat: hash-based manual release workflow

Mirror the manual build/release pipeline from e2b-dev/fc-versions, but
identify each kernel build by a content hash of its inputs (configs +
optional patches). Changing a flag or adding a patch yields a new
artifact named vmlinux-<version>_<sha256[:7]>.

- scripts/validate.py: resolves version names, computes hashes, builds
  the matrix and skips arches whose artifact is already in the GitHub
  release.
- build.sh: now accepts <kernel_version> [arch], computes the same
  version_name (or honors VERSION_NAME), supports patches/<version>/.
- .github/workflows/release.yml: workflow_dispatch with validate ->
  build -> publish (GH release) -> deploy (GCS), skipping work whose
  artifacts already exist.
- Drops the old build-on-every-push workflow; releases are explicit.

* fix: preserve version_name in artifact, force checkout

- upload-artifact: keep version_name in the artifact tree so the
  publish step's ./builds/<version_name>/<arch>/vmlinux.bin lookup
  resolves after merge-multiple download.
- build.sh: git checkout -f so a dirty tree from a prior iteration's
  apply_patches doesn't abort multi-version local builds.

* fix: hash only top-level *.patch files

apply_patches only processes patches/<v>/*.patch, so the hash inputs
must match. Otherwise unrelated/nested files in patches/<v>/ would
silently change the version_name without affecting the build.

* fix: align bash kernel_versions.txt parser with python

Strip inline comments and trim whitespace so build.sh and
scripts/validate.py produce identical version names for entries like
`6.1.158 # stable`.

* refactor: simplify to "build everything, calver release per run"

- workflow_dispatch with no inputs; pick the branch in the GitHub UI.
- Always build every entry in kernel_versions.txt for amd64 and arm64
  in parallel (one runner per arch).
- One release per run, tagged YYYY.MM.DD (with .N suffix on collision),
  with every binary uploaded as vmlinux-<version>-<arch>.bin (plus the
  legacy vmlinux-<version>.bin for amd64).
- Same binaries pushed to GCS under kernels/vmlinux-<version>/...

Drops the per-build content-hash version_name machinery and the
validate.py helper that fed it.

* fix: calver suffix sequence base, .1, .2 (was skipping .1)

* feat: also build on pull_request, gate publish on workflow_dispatch

Open PRs now run the build matrix and upload kernel binaries as
workflow artifacts so reviewers can grab and inspect them from the
PR's checks tab. Release/GCS publishing remains manual-only.

* feat: parallelize build per (kernel_version, arch)

Generate the build matrix dynamically from kernel_versions.txt so each
(version, arch) pair runs on its own runner. With N versions × 2 archs
that's 2N parallel jobs (e.g. 2 versions -> 4 runners).

build.sh already supports single-version mode, so no script changes;
the matrix job emits a JSON include list that the build job fans out
over.

* fix: scope id-token to publish, harden version handling, idempotent calver

- Drop workflow-level id-token: write; only the publish job (which
  needs OIDC for GCP auth) gets id-token: write. PR builds no longer
  receive a token they can exchange for cloud creds.
- Validate kernel versions against [0-9]+(\.[0-9]+)+ in the matrix job
  and pass matrix.version to build.sh via env vars instead of inline
  YAML expression interpolation, so a malicious kernel_versions.txt
  entry can't shell-inject into the runner.
- Calver tag picker now also checks local and remote git tags, not
  just GH releases, so retries after a partial publish no longer pick
  a tag that's already been pushed.

* fix: emit Build kernels (x86_64|arm64) checks for branch protection

The 'Main' ruleset on this repo requires status checks named exactly
'Build kernels (x86_64)' and 'Build kernels (arm64)' (left over from
the previous workflow). The new per-(version, arch) jobs use different
names, so those required checks never reported and PRs couldn't merge.

Add a tiny aggregator job whose name interpolates to those exact two
strings and that gates on the build matrix's combined result, so
parallelism stays per-(version, arch) but branch protection is
satisfied.

Author: ValentaTomas

.github/workflows/fc-kernels.yml

@@ -0,0 +1,156 @@
+name: Build & Release
+
+on:
+  workflow_dispatch:
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  matrix:
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+    outputs:
+      build_matrix: ${{ steps.gen.outputs.matrix }}
+    steps:
+      - uses: actions/checkout@v4
+      - id: gen
+        run: |
+          python3 - >> "$GITHUB_OUTPUT" <<'PY'
+          import json
+          import re
+          import sys
+          VERSION_RE = re.compile(r"^[0-9]+(?:\.[0-9]+)+$")
+          versions = []
+          for line in open("kernel_versions.txt").read().splitlines():
+              v = line.split("#", 1)[0].strip()
+              if not v:
+                  continue
+              if not VERSION_RE.match(v):
+                  sys.exit(f"::error::invalid kernel version in kernel_versions.txt: {v!r}")
+              versions.append(v)
+          include = []
+          for v in versions:
+              include.append({"version": v, "arch": "amd64", "target_arch": "x86_64", "runner": "ubuntu-24.04"})
+              include.append({"version": v, "arch": "arm64", "target_arch": "arm64", "runner": "ubuntu-24.04-arm"})
+          print(f"matrix={json.dumps({'include': include})}")
+          PY
+
+  build:
+    needs: matrix
+    name: Build ${{ matrix.version }} (${{ matrix.arch }})
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(needs.matrix.outputs.build_matrix) }}
+    runs-on: ${{ matrix.runner }}
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Build kernel
+        env:
+          KERNEL_VERSION: ${{ matrix.version }}
+          KERNEL_TARGET_ARCH: ${{ matrix.target_arch }}
+        run: sudo ./build.sh "$KERNEL_VERSION" "$KERNEL_TARGET_ARCH"
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: kernel-${{ matrix.version }}-${{ matrix.arch }}
+          path: ./builds
+          retention-days: 7
+
+  # Aggregator with the exact name required by the `Main` ruleset on this
+  # repo. Per-(version, arch) jobs above run in parallel; this single check
+  # gates the whole arch on their combined success.
+  build-status:
+    needs: build
+    if: always()
+    name: Build kernels (${{ matrix.target_arch }})
+    strategy:
+      matrix:
+        target_arch: [x86_64, arm64]
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+    steps:
+      - run: |
+          if [[ "${{ needs.build.result }}" != "success" ]]; then
+            echo "build matrix did not succeed (result=${{ needs.build.result }})"
+            exit 1
+          fi
+
+  publish:
+    needs: build
+    if: github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: write
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - uses: actions/download-artifact@v4
+        with:
+          path: ./builds
+          merge-multiple: true
+
+      - name: Prepare release assets
+        run: |
+          set -euo pipefail
+          mkdir -p release-assets
+          for dir in ./builds/vmlinux-*/; do
+            name=$(basename "$dir")
+            [ -f "$dir/amd64/vmlinux.bin" ] && cp "$dir/amd64/vmlinux.bin" "release-assets/${name}-amd64.bin"
+            [ -f "$dir/arm64/vmlinux.bin" ] && cp "$dir/arm64/vmlinux.bin" "release-assets/${name}-arm64.bin"
+            # Legacy non-arch-suffixed asset (= amd64) for backwards compat.
+            [ -f "$dir/vmlinux.bin" ] && cp "$dir/vmlinux.bin" "release-assets/${name}.bin"
+          done
+          ls -la release-assets/
+
+      - name: Pick calver tag
+        id: tag
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          set -euo pipefail
+          base="$(date -u +%Y.%m.%d)"
+          tag="$base"
+          n=0
+          while gh release view "$tag" >/dev/null 2>&1 \
+                || git rev-parse "refs/tags/$tag" >/dev/null 2>&1 \
+                || git ls-remote --exit-code --tags origin "refs/tags/$tag" >/dev/null 2>&1; do
+            n=$((n + 1))
+            tag="${base}.${n}"
+          done
+          echo "tag=$tag" >> "$GITHUB_OUTPUT"
+
+      - name: Create release
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          set -euo pipefail
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git tag "${{ steps.tag.outputs.tag }}"
+          git push origin "${{ steps.tag.outputs.tag }}"
+          gh release create "${{ steps.tag.outputs.tag }}" \
+            --title "Kernels ${{ steps.tag.outputs.tag }}" \
+            --notes "Built from commit ${{ github.sha }} on ${{ github.ref_name }}" \
+            ./release-assets/*
+
+      - uses: google-github-actions/auth@v2
+        with:
+          workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
+          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT_EMAIL }}
+
+      - uses: google-github-actions/upload-cloud-storage@v2
+        with:
+          path: ./builds
+          destination: ${{ vars.GCP_BUCKET_NAME }}/kernels
+          gzip: false
+          parent: false

.github/workflows/release.yml

@@ -2,40 +2,57 @@
 
 ## Overview
 
-This project automates the building of custom Linux kernels for Firecracker microVMs, using the same kernel sources as official Firecracker repo and custom configuration files. It supports building specific kernel versions and uploading the resulting binaries to a Google Cloud Storage (GCS) bucket.
+This project builds custom Linux kernels for Firecracker microVMs from the same kernel sources as the official Firecracker repo, using the configuration files (and optional patches) that live in this repo.
 
 ## Prerequisites
 
 - Linux environment (for building kernels)
 
-## Building Kernels
+## Building locally
 
 1. **Configure kernel versions:**
-   - Edit `kernel_versions.txt` to specify which kernel versions to build (one per line, e.g., `6.1.102`).
-   - Place the corresponding config file in `configs/` (e.g., `configs/6.1.102.config`).
+   - Edit `kernel_versions.txt` to specify which kernel versions to build (one per line, e.g. `6.1.158`).
+   - Place the corresponding config(s) in `configs/x86_64/<version>.config` and `configs/arm64/<version>.config`.
+   - (Optional) Drop `*.patch` files into `patches/<version>/` to apply on top of the upstream tree before build.
 
 2. **Build:**
    ```sh
-   make build
-   # or directly
-   ./build.sh
+   make build              # builds all versions in kernel_versions.txt for x86_64
+   make build-arm64        # same, for arm64
+   ./build.sh 6.1.158      # build a single version (x86_64)
+   ./build.sh 6.1.158 arm64
    ```
-   The built kernels will be placed in `builds/vmlinux-<version>/<arch>/vmlinux.bin` where `<arch>` is `amd64` or `arm64` (Go/OCI convention). For x86_64 backward compatibility, a legacy copy is also placed at `builds/vmlinux-<version>/vmlinux.bin`.
 
-## Development Workflow
-  - On every push, GitHub Actions will automatically build the kernels and save it as an artifact.
+   Output: `builds/vmlinux-<version>/<arch>/vmlinux.bin` where `<arch>` is `amd64` or `arm64` (Go/OCI convention). For x86_64 a legacy copy is also placed at `builds/vmlinux-<version>/vmlinux.bin`.
+
+## CI / Releasing
+
+The **Build & Release** workflow runs in two modes:
+
+- **On every pull request**: builds every (kernel version × arch) combination from `kernel_versions.txt` in parallel (one runner per pair) and uploads the binaries as workflow artifacts (downloadable from the PR's checks tab) so reviewers can inspect them. No release or GCS upload happens.
+- **Manually (workflow_dispatch)**: pick the branch in the GitHub UI and run. It does the same build as a PR and additionally creates a GitHub release tagged `YYYY.MM.DD` (with a `.N` suffix for additional runs the same day) containing every binary, and uploads them to GCS.
+
+Release asset naming for that commit:
+
+   ```
+   vmlinux-<version>-amd64.bin
+   vmlinux-<version>-arm64.bin
+   vmlinux-<version>.bin           # legacy (= amd64) for backwards compat
+   ```
+
+4. The same binaries are uploaded to GCS at `gs://$GCP_BUCKET_NAME/kernels/vmlinux-<version>/<arch>/vmlinux.bin`.
+
+## New kernel in E2B's infra
+_Note: these steps should give you a new kernel on your self-hosted E2B using https://github.com/e2b-dev/infra_
+
+- Run the release workflow on the branch with the new config/patch.
+- Update `DefaultKernelVersion` in [packages/api/internal/cfg/model.go](https://github.com/e2b-dev/infra/blob/main/packages/api/internal/cfg/model.go) if you changed the kernel version.
+- Build and deploy `api`.
 
 ## Architecture naming
 
 Output directories use Go's `runtime.GOARCH` convention (`amd64`, `arm64`) so they match the infra orchestrator's `TargetArch()` path resolution. The build-time variable `TARGET_ARCH` (`x86_64`, `arm64`) is only used internally for config paths and cross-compilation flags.
 
-## New Kernel in E2B's infra
-_Note: these steps should give you new kernel on your self-hosted E2B using https://github.com/e2b-dev/infra_
-
-  - Copy the kernel build in your project's object storage under `e2b-*-fc-kernels`
-  - In [packages/api/internal/cfg/model.go](https://github.com/e2b-dev/infra/blob/main/packages/api/internal/cfg/model.go) update `DefaultKernelVersion`
-  - Build and deploy `api`
-  
 ## License
 
-This project is licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details. 
+This project is licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details.