ci: add Dockerized GPU self-hosted runner with cron auto-restart

Adds ci/gpu-runner/: a GPU-capable, ephemeral GitHub Actions runner
packaged as a Docker image and kept alive by a cron watchdog.

- Dockerfile: nvidia/cuda 12.8 devel (ubuntu24.04) + actions runner +
  Rust toolchain + git-lfs/m4; carries no secrets. Rewrites only the
  private ceno-gpu-mock dep URL https->ssh (public deps stay on HTTPS).
- entrypoint.sh: mints a fresh registration token from a PAT on every
  start (survives restarts), runs an ephemeral runner.
- start-runner.sh: builds/(re)launches with --gpus all and persistent
  cargo-registry + CARGO_TARGET_DIR volumes for warm incremental builds.
- watchdog.sh: per-minute cron check; restarts on container stop or
  GPU-unreachable.
- README.md: host setup, secrets (host PAT + repo deploy key via
  ssh-agent), and cron registration.

The runner-registration PAT lives in ci/gpu-runner/runner.env, which is
gitignored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: kunxian-xia

.gitignore

@@ -15,4 +15,7 @@ docs/book
 # ceno serialized files
 *.bin
 *.json
-*.srs
+*.srs
+
+# GPU CI runner secrets (never commit the PAT)
+ci/gpu-runner/runner.env

ci/gpu-runner/Dockerfile

@@ -0,0 +1,78 @@
+# GPU-capable self-hosted GitHub Actions runner for the Ceno zkVM.
+#
+# Modeled on scroll-tech/ceno-reth-benchmark:ci/Dockerfile. This image is purely
+# the runner + build prerequisites — it carries NO secrets. The real GPU backend
+# is the PRIVATE scroll-tech/ceno-gpu repo, which Cargo.toml's active [patch]
+# pulls in as the local path ../ceno-gpu/cuda_hal (the public ceno-gpu-mock dep
+# is just the registry placeholder that the patch overrides). The workflow clones
+# ceno-gpu to ../ceno-gpu using a deploy key loaded into ssh-agent from GitHub
+# secrets; see ci/gpu-runner/README.md for the snippet.
+#
+# Build:
+#   docker build -t ceno-gpu-runner:latest -f ci/gpu-runner/Dockerfile .
+#
+# Match CUDA to your host driver (`nvidia-smi` top-right = max supported CUDA).
+FROM nvidia/cuda:12.8.0-devel-ubuntu24.04
+
+ARG RUNNER_VERSION="2.321.0"
+ARG RUNNER_SHA256="ba46ba7ce3a4d7236b16fbe44419fb453bc08f866b24f04d549ec89f1722a29e"
+ARG RUST_TOOLCHAIN="nightly-2025-11-20"
+ARG DEBIAN_FRONTEND=noninteractive
+
+# OS deps: build toolchain + Ceno CI needs (m4, git-lfs), runner needs (curl/jq),
+# and SSH/git so the workflow can clone the private ceno-gpu repo.
+# No nvidia-utils here: with `--gpus` the NVIDIA Container Toolkit injects the
+# host's nvidia-smi + driver libs at runtime, so pinning a driver userspace in
+# the image is redundant and risks mismatching the host driver.
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+        curl jq ca-certificates \
+        build-essential cmake pkg-config libssl-dev libffi-dev \
+        m4 git git-lfs openssh-client \
+        sudo tar gzip \
+    && git lfs install --system \
+    && rm -rf /var/lib/apt/lists/* \
+    && useradd -m -s /bin/bash docker \
+    && echo "docker ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
+
+# Install the actions runner (checksum-verified, like the reference).
+RUN mkdir -p /home/docker/actions-runner \
+    && cd /home/docker/actions-runner \
+    && curl -fsSL -o runner.tar.gz \
+        "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" \
+    && echo "${RUNNER_SHA256}  runner.tar.gz" | shasum -a 256 -c \
+    && tar xzf runner.tar.gz \
+    && rm runner.tar.gz \
+    && ./bin/installdependencies.sh \
+    && chown -R docker:docker /home/docker \
+    # Persistent build cache mount point. Creating it owned by `docker` here
+    # means the (initially empty) named volume mounted at /cache inherits that
+    # ownership on first use, so cargo can write CARGO_TARGET_DIR=/cache/target.
+    && mkdir -p /cache/target && chown -R docker:docker /cache
+
+# All cargo git deps are public (fetched over HTTPS, no auth). The only private
+# repo is scroll-tech/ceno-gpu, which the workflow clones to ../ceno-gpu over SSH
+# using a deploy key loaded into ssh-agent from GitHub secrets — no key or URL
+# rewrite is baked into the image. Pre-seed github.com in known_hosts so that
+# clone isn't blocked on host-key prompting.
+USER docker
+RUN git config --global --add safe.directory '*' \
+    && mkdir -p /home/docker/.ssh && chmod 700 /home/docker/.ssh \
+    && ssh-keyscan github.com >> /home/docker/.ssh/known_hosts
+
+# Rust toolchain pinned to rust-toolchain.toml + riscv guest target + cargo-make.
+ENV PATH="/home/docker/.cargo/bin:${PATH}"
+RUN curl --proto '=https' --tlsv1.2 -fsSL https://sh.rustup.rs \
+        | sh -s -- -y --default-toolchain "${RUST_TOOLCHAIN}" --profile minimal \
+    && rustup component add rust-src clippy rustfmt \
+    && rustup target add riscv32im-unknown-none-elf \
+    && cargo install cargo-make --locked
+
+WORKDIR /home/docker/actions-runner
+
+COPY --chown=docker:docker ci/gpu-runner/entrypoint.sh /usr/local/bin/runner-entrypoint.sh
+USER root
+RUN chmod +x /usr/local/bin/runner-entrypoint.sh
+USER docker
+
+ENTRYPOINT ["/usr/local/bin/runner-entrypoint.sh"]

ci/gpu-runner/README.md

@@ -0,0 +1,138 @@
+# Ceno GPU self-hosted CI runner (Docker + cron auto-restart)
+
+A GPU-capable, **ephemeral** GitHub Actions runner for this repo, packaged as a
+Docker image and kept alive by a cron watchdog.
+
+- **Ephemeral**: the runner exits cleanly after each job, so no state leaks
+  between CI runs. The watchdog immediately brings up a fresh one.
+- **Self-healing tokens**: the container mints a fresh registration token from
+  the GitHub API on every start, so restarts never fail on an expired token.
+- **Cron watchdog**: restarts the container if it stops *or* if the GPU becomes
+  unreachable inside it (your stated failure mode).
+
+## One-time host setup
+
+The host must have an NVIDIA driver, Docker, and the **NVIDIA Container Toolkit**
+(this is what makes `--gpus all` work):
+
+```sh
+# NVIDIA Container Toolkit (Ubuntu) — see NVIDIA docs for the current repo lines
+curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
+  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
+curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
+  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
+  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
+sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+
+# sanity check: should print your GPU(s)
+docker run --rm --gpus all nvidia/cuda:12.6.2-base-ubuntu24.04 nvidia-smi
+```
+
+Check the CUDA version your driver supports (top-right of `nvidia-smi`) and set
+`CUDA_VERSION` in the Dockerfile / build arg to match (must be ≤ that).
+
+## Private GPU backend: deploy key comes from GitHub secrets (not the image)
+
+All cargo *git* deps pinned in `Cargo.toml` are **public** (`ceno-gpu-mock`,
+`gkr-backend`, `ceno-patch`, `openvm`) and fetch over HTTPS with no auth. The
+real GPU backend, however, is the **private `scroll-tech/ceno-gpu`** repo:
+Cargo.toml's active `[patch]` redirects `ceno_gpu` away from the public
+`ceno-gpu-mock` placeholder to the **local path `../ceno-gpu/cuda_hal`**. So a
+`--features gpu` build needs `scroll-tech/ceno-gpu` cloned to `../ceno-gpu`
+(a sibling of the checkout).
+
+The runner **image carries no secrets**. The deploy key for `ceno-gpu` is stored
+as a repo secret and loaded into an ssh-agent *inside each job*, which then
+clones the repo to the path the patch expects.
+
+Setup:
+1. Add a **read-only deploy key** to the `scroll-tech/ceno-gpu` repo, and store
+   the private half as a repo secret, e.g. `CENO_GPU_DEPLOY_KEY`.
+2. In any GPU workflow, load it and clone `ceno-gpu` before the build:
+
+```yaml
+jobs:
+  gpu-job:
+    runs-on: [ self-hosted, Linux, X64, gpu ]
+    steps:
+      - uses: webfactory/ssh-agent@v0.9.0
+        with:
+          ssh-private-key: ${{ secrets.CENO_GPU_DEPLOY_KEY }}
+      - uses: actions/checkout@v4
+        with:
+          lfs: true
+      # the active [patch] expects ../ceno-gpu/cuda_hal to exist
+      - run: git clone git@github.com:scroll-tech/ceno-gpu.git ../ceno-gpu
+      - run: cargo build --release --features gpu
+```
+
+(`webfactory/ssh-agent` also adds github.com to known_hosts for the job.)
+
+## Configure & first run
+
+```sh
+cp ci/gpu-runner/runner.env.example ci/gpu-runner/runner.env
+$EDITOR ci/gpu-runner/runner.env     # paste PAT + repo URL
+
+ci/gpu-runner/start-runner.sh
+docker logs -f ceno-gpu-runner       # confirm it registered
+```
+
+To build manually:
+
+```sh
+docker build -t ceno-gpu-runner:latest -f ci/gpu-runner/Dockerfile .
+```
+
+In repo **Settings → Actions → Runners** you should now see a runner with the
+`gpu` label. Point GPU jobs at it:
+
+```yaml
+runs-on: [ self-hosted, Linux, X64, gpu ]
+```
+
+(Non-GPU jobs keep using `[ self-hosted, Linux, X64 ]` and won't be scheduled
+here because they don't request the `gpu` label.)
+
+## Enable the cron watchdog
+
+```sh
+crontab -e
+# add (use the ABSOLUTE path):
+* * * * * /home/zbx/blockchain/ceno/ci/gpu-runner/watchdog.sh >> /var/log/ceno-gpu-runner.log 2>&1
+```
+
+The watchdog checks every minute and restarts on stop or GPU-unreachable.
+
+## Files
+
+| file                  | purpose                                                        |
+|-----------------------|----------------------------------------------------------------|
+| `Dockerfile`          | CUDA-devel image + actions runner + Rust toolchain + git-lfs   |
+| `entrypoint.sh`       | mint token → configure ephemeral runner → run → de-register    |
+| `start-runner.sh`     | (re)launch the container with `--gpus all`; idempotent         |
+| `watchdog.sh`         | cron health check + restart                                    |
+| `runner.env.example`  | template for `runner.env` (PAT + repo URL; gitignored)         |
+
+## Notes & gotchas
+
+- **PAT scope**: repo-level runner needs `repo` (classic) or fine-grained
+  "Administration: Read and write" on this repo.
+- **Warm builds across ephemeral restarts**: two named volumes persist between
+  containers — `ceno-gpu-runner-cargo` (the cargo registry, so deps aren't
+  re-downloaded) and `ceno-gpu-runner-target` (mounted at `/cache/target`, with
+  `CARGO_TARGET_DIR` pointed at it so incremental compile artifacts survive).
+  This keeps recompiles fast even though each job runs in a fresh container.
+  Because of this, **GPU jobs can drop the `actions/cache` step for `target/`** —
+  the host volume is bigger (no ~10 GB cache limit) and faster (no network
+  restore) than GitHub's cache backend for a workspace this size. The single
+  ephemeral runner serves one job at a time, so there's no concurrent writer on
+  the shared target dir. (To reset: `docker volume rm ceno-gpu-runner-target`.)
+- **Alternative to cron**: `--restart unless-stopped` in `start-runner.sh`
+  restarts on crash instantly, but won't catch a hung container or a GPU that
+  silently went away — that's why the watchdog also probes `nvidia-smi`.
+- **Concurrency**: this is a single ephemeral runner = one job at a time. For N
+  concurrent GPU jobs, run N containers (`CONTAINER_NAME=ceno-gpu-runner-2 …`)
+  and one watchdog line each.

ci/gpu-runner/entrypoint.sh

@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Entrypoint for the Ceno GPU self-hosted runner.
+#
+# Differs from the ceno-reth-benchmark reference entrypoint in one way: instead
+# of taking a hard-coded RUNNER_TOKEN (which GitHub expires after ~1h and would
+# make an auto-restart fail), it mints a FRESH registration token from the
+# GitHub API on every start using a stored PAT. That is what lets the cron
+# watchdog restart this container indefinitely. It also runs an EPHEMERAL runner
+# (exits cleanly after one job) so no state leaks between CI runs.
+#
+# Required env:
+#   GITHUB_PAT   - classic PAT with `repo` scope, or fine-grained token with
+#                  "Administration: read/write" on the repo.
+#   REPO_URL     - e.g. https://github.com/scroll-tech/ceno
+# Optional env:
+#   RUNNER_NAME  - defaults to gpu-<short-hostname>
+#   RUNNER_LABELS- defaults to "self-hosted,Linux,X64,gpu"
+set -euo pipefail
+
+: "${GITHUB_PAT:?set GITHUB_PAT}"
+: "${REPO_URL:?set REPO_URL, e.g. https://github.com/scroll-tech/ceno}"
+
+RUNNER_NAME="${RUNNER_NAME:-gpu-$(hostname | cut -c1-12)}"
+RUNNER_LABELS="${RUNNER_LABELS:-self-hosted,Linux,X64,gpu}"
+RUNNER_DIR="/home/docker/actions-runner"
+cd "${RUNNER_DIR}"
+
+# REPO_URL -> owner/repo
+REPO_PATH="$(echo "${REPO_URL}" | sed -E 's#https?://[^/]+/##; s#\.git$##')"
+API="https://api.github.com/repos/${REPO_PATH}/actions/runners"
+
+echo "[entrypoint] requesting registration token for ${REPO_PATH} ..."
+REG_TOKEN="$(curl -fsSL -X POST \
+    -H "Authorization: Bearer ${GITHUB_PAT}" \
+    -H "Accept: application/vnd.github+json" \
+    -H "X-GitHub-Api-Version: 2022-11-28" \
+    "${API}/registration-token" | jq -r .token)"
+
+if [[ -z "${REG_TOKEN}" || "${REG_TOKEN}" == "null" ]]; then
+    echo "[entrypoint] ERROR: could not obtain registration token (check PAT scope/REPO_URL)" >&2
+    exit 1
+fi
+
+cleanup() {
+    echo "[entrypoint] de-registering runner ..."
+    ./config.sh remove --token "${REG_TOKEN}" || true
+}
+trap 'cleanup' EXIT INT TERM
+
+# Fail fast if the GPU isn't actually reachable inside the container — otherwise
+# an ephemeral runner would pick up a GPU job and fail it.
+if command -v nvidia-smi >/dev/null 2>&1; then
+    echo "[entrypoint] GPU check:"
+    nvidia-smi -L || { echo "[entrypoint] ERROR: nvidia-smi failed; is --gpus set + toolkit installed?" >&2; exit 1; }
+else
+    echo "[entrypoint] WARNING: nvidia-smi not found in container" >&2
+fi
+
+echo "[entrypoint] configuring ephemeral runner '${RUNNER_NAME}' (labels: ${RUNNER_LABELS}) ..."
+./config.sh \
+    --url "${REPO_URL}" \
+    --token "${REG_TOKEN}" \
+    --name "${RUNNER_NAME}" \
+    --labels "${RUNNER_LABELS}" \
+    --work _work \
+    --ephemeral \
+    --unattended \
+    --replace
+
+echo "[entrypoint] starting runner ..."
+exec ./run.sh

ci/gpu-runner/runner.env.example

@@ -0,0 +1,13 @@
+# Copy to `runner.env` and fill in. runner.env is gitignored — never commit the PAT.
+#
+# PAT needs (repo-level runner):
+#   - classic PAT: `repo` scope, OR
+#   - fine-grained PAT: repo selected + "Administration: Read and write"
+GITHUB_PAT=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+# The repo this runner serves.
+REPO_URL=https://github.com/scroll-tech/ceno
+
+# Optional overrides:
+# RUNNER_NAME=gpu-box-1
+# RUNNER_LABELS=self-hosted,Linux,X64,gpu

ci/gpu-runner/start-runner.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# Build (if needed) and (re)start the GPU runner container.
+# Idempotent: safe to call from the cron watchdog — it removes any stale
+# container of the same name first.
+set -euo pipefail
+
+cd "$(dirname "$0")/../.."          # repo root = docker build context
+
+RUNNER_DIR="ci/gpu-runner"
+CONTAINER_NAME="${CONTAINER_NAME:-ceno-gpu-runner}"
+IMAGE_NAME="${IMAGE_NAME:-ceno-gpu-runner:latest}"
+ENV_FILE="${ENV_FILE:-$(pwd)/${RUNNER_DIR}/runner.env}"
+
+if [[ ! -f "${ENV_FILE}" ]]; then
+    echo "ERROR: ${ENV_FILE} not found. Copy runner.env.example -> runner.env and fill it in." >&2
+    exit 1
+fi
+
+# Build the image if it's missing (first run / after a host reboot+prune).
+# No secrets at build time — the SSH deploy key is injected per-job from GitHub
+# secrets via ssh-agent in the workflow.
+if ! docker image inspect "${IMAGE_NAME}" >/dev/null 2>&1; then
+    echo "[start] building ${IMAGE_NAME} ..."
+    docker build -t "${IMAGE_NAME}" -f "${RUNNER_DIR}/Dockerfile" .
+fi
+
+# Drop any previous instance (exited ephemeral runner, crashed container, etc.).
+docker rm -f "${CONTAINER_NAME}" >/dev/null 2>&1 || true
+
+echo "[start] launching ${CONTAINER_NAME} ..."
+docker run -d \
+    --name "${CONTAINER_NAME}" \
+    --gpus all \
+    --env-file "${ENV_FILE}" \
+    --restart no \
+    -e CARGO_TARGET_DIR=/cache/target \
+    -v ceno-gpu-runner-cargo:/home/docker/.cargo/registry \
+    -v ceno-gpu-runner-target:/cache/target \
+    "${IMAGE_NAME}"
+
+echo "[start] done. logs: docker logs -f ${CONTAINER_NAME}"

ci/gpu-runner/watchdog.sh

@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+# Cron watchdog for the Ceno GPU runner.
+#
+# Run it every minute from cron. It restarts the container if:
+#   - the container is not in `running` state (crashed, or an ephemeral runner
+#     finished a job and exited), OR
+#   - nvidia-smi inside the container fails (GPU became unreachable).
+#
+# Crontab line (run `crontab -e` as the host user that owns docker access):
+#   * * * * * /ABS/PATH/ceno/ci/gpu-runner/watchdog.sh >> /var/log/ceno-gpu-runner.log 2>&1
+set -uo pipefail
+
+cd "$(dirname "$0")"
+CONTAINER_NAME="${CONTAINER_NAME:-ceno-gpu-runner}"
+TS() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }
+
+state="$(docker inspect -f '{{.State.Status}}' "${CONTAINER_NAME}" 2>/dev/null || echo "absent")"
+
+if [[ "${state}" != "running" ]]; then
+    echo "$(TS) watchdog: container state='${state}', restarting"
+    ./start-runner.sh
+    exit 0
+fi
+
+# Container is running — verify the GPU is still actually usable.
+if ! docker exec "${CONTAINER_NAME}" nvidia-smi -L >/dev/null 2>&1; then
+    echo "$(TS) watchdog: GPU unreachable inside container, restarting"
+    ./start-runner.sh
+    exit 0
+fi
+
+# Healthy — stay quiet (no log spam every minute).
+exit 0