ci: Dockerized GPU self-hosted runner with cron auto-restart#1348
Merged
Conversation
Adds ci/gpu-runner/: a GPU-capable, ephemeral GitHub Actions runner packaged as a Docker image and kept alive by a cron watchdog. - Dockerfile: nvidia/cuda 12.8 devel (ubuntu24.04) + actions runner + Rust toolchain + git-lfs/m4; carries no secrets. Rewrites only the private ceno-gpu-mock dep URL https->ssh (public deps stay on HTTPS). - entrypoint.sh: mints a fresh registration token from a PAT on every start (survives restarts), runs an ephemeral runner. - start-runner.sh: builds/(re)launches with --gpus all and persistent cargo-registry + CARGO_TARGET_DIR volumes for warm incremental builds. - watchdog.sh: per-minute cron check; restarts on container stop or GPU-unreachable. - README.md: host setup, secrets (host PAT + repo deploy key via ssh-agent), and cron registration. The runner-registration PAT lives in ci/gpu-runner/runner.env, which is gitignored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kunxian-xia
force-pushed
the
ci/gpu-runner
branch
from
d9ada3e to
f733720
Compare
Adds .github/workflows/gpu-integration.yml: proves an example (default keccak_syscall) end-to-end with the GPU prover (--features gpu) on the self-hosted `gpu` runner. - Trigger: workflow_dispatch (with an `example` input) or a PR labeled `gpu-ci` (opt-in; GPU runner time is scarce). - Loads CENO_GPU_DEPLOY_KEY into ssh-agent and clones the private ceno-gpu backend to ../ceno-gpu (the path Cargo.toml's active [patch] expects) before building. - Runs the example single-shard and multi-shard; no actions/cache since the runner persists target/ on a host volume via CARGO_TARGET_DIR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closed
The committed Cargo.toml ships the ceno_gpu -> ../ceno-gpu/cuda_hal [patch] commented out, so a default --features gpu build resolves the public, API-incompatible ceno-gpu-mock and fails to compile. After cloning the private ceno-gpu backend, uncomment the patch with sed and fail fast if it's still commented (so we can never silently build the mock again). Also document that the job is release-only by design: running debug + release would compile the workspace (and the CUDA cuda_hal crate) twice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace a machine-specific absolute path with /ABS/PATH placeholder. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cron runs the watchdog every minute; the first invocation triggers a long image build during which the container doesn't yet exist, so later ticks would each launch another start-runner.sh and race on docker build/run. Guard with a non-blocking flock so overlapping ticks exit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
If the crontab user can't write the redirect target, cron aborts the line before watchdog.sh runs, so it silently never executes and the error is only mailed to the user. Default the example to $HOME and document the /var/log + docker-group requirements. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cuda_hal depends on the public `sppark` submodule (ceno-gpu/sppark/rust); a plain clone omits it and cargo fails with "failed to read sppark/rust/Cargo.toml". Clone with --recurse-submodules. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The e2e binary defaults to LevelFilter::DEBUG when RUST_LOG is unset (ceno_zkvm/src/bin/e2e.rs), producing ~1400 debug lines per run. Pin RUST_LOG=info for the GPU steps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the proving time (run_e2e_proof span from `--profiling 1`) and feed it to benchmark-action/github-action-benchmark (customSmallerIsBetter). History is stored on gh-pages: master pushes append the baseline; PRs compare and fail on a >10% regression (alert-threshold 110%, fail-on-alert), commenting on the PR. Adds a push:master trigger to seed/update the baseline, a dedicated build step, and contents/pull-requests write perms for the action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1.21.0+ run on node24, which the runner image (actions runner v2.321.0) doesn't support, failing at "Set up job". v1.20.7 is the last node20 release. Bump RUNNER_VERSION to a node24-capable runner to unpin later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The profiling tree's root is ZKVM_create_proof (pure proof generation), not run_e2e_proof, so the previous grep found nothing and the step errored. Parse ZKVM_create_proof, and guard the grep with `|| true` so a no-match yields a clear error instead of being swallowed by set -e. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-action-benchmark runs `git switch gh-pages` in the checkout, which aborts while the "Activate patch" edits to Cargo.toml/Cargo.lock are uncommitted. Move the multi-shard run ahead of it (still needs the patch), then `git checkout -- Cargo.toml Cargo.lock` to clean the tree before the benchmark action. bench.json is untracked and survives. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A self-hosted GPU CI runner for Ceno, plus a workflow that runs GPU integration tests on it.
ci/gpu-runner/— the runnernvidia/cuda:12.8-devel-ubuntu24.04+ actions runner + pinned Rust toolchain. No secrets baked in.docker run --gpus all; persists cargo registry andCARGO_TARGET_DIRon volumes for warm rebuilds.flock-guarded against overlap..github/workflows/gpu-integration.yml— the testProves an example (default
keccak_syscall) with--features gpuon thegpu-labeled runner. Triggered manually (workflow_dispatch) or by agpu-ciPR label. Release-only by design (debug + release would compile the workspace twice). Steps: loadCENO_GPU_DEPLOY_KEY→ clone privateceno-gpu(--recurse-submodules) → activate its Cargo[patch]→ build + prove.Secrets
GITHUB_PAT(hostrunner.env, gitignored): registers the runner.CENO_GPU_DEPLOY_KEY(repo secret): read-only deploy key for the privateceno-gpubackend, loaded per-job via ssh-agent.No existing workflows are modified.
🤖 Generated with Claude Code