ci(release): runner-native release workflow + actionlint#710
Merged
Conversation
nekomoto911
changed the title
…tionlint Self-hosted runner is Ubuntu 24.04 (glibc 2.39); staging-mainnet nodes are Debian 11 (glibc 2.31). Binaries built directly on the runner require GLIBC_2.32+ symbols and fail to start on target with `GLIBC_2.34 not found`. Changes to .github/workflows/release.yml: - Build inside container image, default `rust:1.93.0-bullseye` (glibc 2.31), overridable via `vars.RELEASE_BUILD_IMAGE` for future OS bumps. - Assert `rustc --version` matches `rust-toolchain.toml` so image drift surfaces as a clear error instead of silently building with the wrong toolchain. - Install gh CLI inside the container (the rust:* base image lacks it). - Require the GitHub Release to exist before tag push (UI-driven release notes flow); CI now only uploads assets and fails fast if no release. - Drop workflow_dispatch and the auto-create-release fallback — single trigger source avoids the dispatch-vs-tag commit mismatch and keeps release notes human-authored. - Add a pull_request trigger scoped to .github/workflows/release.yml so changes to this file run a full container build as a dry-run; binaries are uploaded as a 7-day artifact for reviewer inspection. Real upload step is skipped on PR events. New .github/workflows/lint-workflows.yml: - Runs actionlint@1.7.7 on PRs touching any .github/workflows/** file.
- gh v2.62.0 was never published; download 404'd and broke the dry-run. Bump to v2.92.0 (current latest) and add --retry 3 for transient self-hosted-runner network blips. - actionlint's bundled shellcheck integration treats info-level issues (SC2086 unquoted vars, SC2053 unquoted RHS) as errors, which fails on pre-existing scripts in gate.yml / rust-ci.yml that are out of scope for this PR. Disable shellcheck via `-shellcheck=`; the workflow still validates yaml + expression syntax. Re-enable after a separate cleanup.
The gh CLI source repo is `cli/cli`, not `cli/gh` — the prior URL 404'd on every release tag, not just 2.62.0. Both previous failed dry-runs were hitting this same path bug, masked as "version doesn't exist". Verified `https://github.com/cli/cli/releases/download/v2.92.0/gh_2.92.0_linux_amd64.tar.gz` returns HTTP 200.
bindgen needs libclang at build time; the bare `rust:1.93.0-bullseye` image doesn't ship it, so the previous dry-run died at: thread 'main' panicked at bindgen-0.72.1/lib.rs:616 Unable to find libclang: ... Mirror the apt list from gravity_e2e/build_docker.sh (the known-working in-container build env), minus python/test packages. On bullseye the `clang` package does not pull the unversioned libclang.so symlink that clang-sys searches for, so libclang-dev is added explicitly (bookworm gets it transitively, which is why e2e didn't need to call it out).
Decision change: instead of containerizing the build to match production glibc, the runner OS itself will be aligned to production OS. Native build is simpler — no container plumbing, no apt-install on every run, target/ cache reuse between builds. - Drop the `container:` block and the `vars.RELEASE_BUILD_IMAGE` override. - Drop the in-container `Install build dependencies` step (apt-get clang/llvm/...). - Drop the in-container `Install gh CLI` curl-tarball step. - Drop the `cli/gh → cli/cli` URL fix (step no longer exists). - Add a `Show build environment` step that logs runner os-release / ldd / rustc / cargo so any future runner OS drift from production is visible in the run record. - Keep the `Assert rustc matches rust-toolchain.toml` check (rephrased for the native-runner context). - Keep PR dry-run trigger + artifact upload (still valuable for review). Invariant now: runner OS must equal production node OS. Currently staging-mainnet is Debian 11 / glibc 2.31, runner is Ubuntu 24.04 / glibc 2.39 — that gap must be closed on the runner side before this workflow can produce shippable binaries. Runner prerequisites (install once on the runner host): - rust toolchain per rust-toolchain.toml - gh CLI - clang llvm libclang-dev build-essential pkg-config libssl-dev libudev-dev
nekomoto911
force-pushed
the
release-workflow-container
branch
from
5155983 to
d75fc1f
Compare
Testnet releases follow the same publish flow as mainnet — human builds the release notes in GitHub UI, publishes, tag push triggers binary upload via this workflow. Trigger filter and the Resolve tag case-guard updated symmetrically so the validation message stays accurate.
nekomoto911
changed the title
4 tasks
nekomoto911
pushed a commit
that referenced
this pull request
May 14, 2026
## Summary The `Release` workflow in PR #710 builds `gravity_node` with `default = []` features, so the released v1.6.0 binary (commit `e8594e7b`) panics on every Staging Mainnet node whose identity is sourced from GCP Secret Manager: \`\`\` panicked at .../config/network_config.rs:281:25: load identity blob for peer_id from GCP secret projects/mainnet-storage/secrets/staging-vfn-1-identity/versions/latest: GCP Secret Manager support not compiled in. Rebuild aptos-config with \`--features gcp-secret-manager\` (or enable the forwarding feature on the downstream crate, e.g. \`gaptos/gcp-secret-manager\`). \`\`\` Observed live on `vfn-1` and `vfn-2` after the v1.6.0 rollout (2026-05-13 ~15:37 UTC): both had `NRestarts > 2700`, `ActiveState=activating / auto-restart`, every ~5s, with the panic above logged in `journalctl`. \`core-1..4 / rpc-1\` showed the same v1.6.0 binary on disk but had not yet started (separate \`/data\` / unit ordering issue — not in this PR's scope). ## Fix Split the build step. \`gravity_node\` is now built with: \`\`\`bash cargo build --profile quick-release -p gravity_node --bin gravity_node --features gcp-secret-manager \`\`\` \`gravity_cli\` is built in its own step (the feature is declared per-package on \`bin/gravity_node/Cargo.toml\`; cargo's workspace-level \`--features\` would not resolve it for the combined multi-bin build). ## Test plan - [x] PR-trigger dry-run (this PR) — confirm both \`Build gravity_node\` and \`Build gravity_cli\` steps succeed and the resulting binary is uploaded as \`release-dryrun-<sha>\` artifact. - [x] Local build on LA Private Mainnet build host (\`gravity-mainnet-node-la-0\`, Debian 11, rustc/cargo 1.93.0 matching \`rust-toolchain.toml\`) using the same cargo invocation: succeeded in **3m 15s**, sha256 \`921dd5bfc8ee11ba0360c5343f105b579d2e3aed17ef29b7ab02513cdf6d278d\`. (LA artifact dynamic-links \`libssl.so.1.1\` so it's only usable on Debian 11 / Private Mainnet, but this validates that the cargo invocation works and gravity_node compiles cleanly with the feature on.) - [ ] After merge: cut a new release tag (e.g. \`gravity-mainnet-v1.6.1\`) via the Releases UI — the workflow only attaches assets to a pre-existing release, so create the release with notes first, then publish to push the tag. Release notes can simply reference this PR. - [ ] Verify the new release binary on a Staging Mainnet VFN (e.g. \`vfn-1\`): the panic at \`network_config.rs:281\` should be gone, \`journalctl -u gravity-vfn-1.service\` should advance past the identity-load step. ## Why this slipped through v1.6.0 The first end-to-end use of the \`Release\` workflow in PR #710 was the v1.6.0 tag itself, which happened after the Packer image switch to Ubuntu 24.04. The dry-run did exercise build + upload-artifact, but did not exercise running the binary against a node that uses \`source = "gcp_secret"\` identities — that path only lives in the deployed Staging Mainnet / mainnet config, and isn't covered by SDK unit/integration tests. Consider adding a smoke job that runs \`gravity_node --help\` (or a tiny config-load test) under \`--features gcp-secret-manager\` so future regressions are caught at PR time, not at deploy time. ## References - v1.6.0 panic backtrace captured live on \`vfn-1\` 2026-05-14 ~00:21 UTC - Feature declaration: \`bin/gravity_node/Cargo.toml:13-21\` (\`gcp-secret-manager = ["gaptos/gcp-secret-manager"]\`), explicitly notes \`Off by default to keep the HTTP/TLS stack out of the default build graph; enable when cluster.toml configures any node with \`identity = { source = "gcp_secret", ... }\`.\` - Original release workflow: PR #710 (commit \`e8594e7b\`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
keanji-x
added a commit
that referenced
this pull request
May 14, 2026
) ## Summary Removes the optional GCP Secret Manager build mode in `bin/gravity_node` and reverts the workflow split that #713 forced as a band-aid. - **`bin/gravity_node/Cargo.toml`** — drop the local `gcp-secret-manager` forwarding feature. Instead pull the `gaptos` dep with that feature always enabled (`features = [\"gcp-secret-manager\"]`). `reqwest` + `base64` become a permanent part of every `gravity_node` build. - **`.github/workflows/release.yml`** — collapse the two-step `gravity_node` / `gravity_cli` build back to the single pre-#713 step. With the feature permanently on at the dep level, the explicit `--features` flag (and the `-p gravity_node` scoping it required) is redundant. No upstream `gravity-aptos` change: the feature flag on `aptos-config` is left in place; we just unconditionally enable it from this side. `cargo` pin is unchanged at `e9544c8`. ## Why The feature was a footgun. v1.6.0 shipped with `default = []` and the Staging Mainnet VFNs auto-restart-looped on the panic in `aptos-config/.../network_config.rs:281` because their identity source is `gcp_secret`. #713 patched the workflow but the underlying design — an optional build flag that every production deploy must remember to enable — is what should go. `reqwest` is already in the workspace via `gravity_cli`'s upload-side `secret_manager.rs`, so making it unconditional in `aptos-config` (via the gaptos feature being always on) does not introduce a new transitive dep at workspace scope. ## Test plan - [x] Local \`cargo check -p gravity_node --profile=quick-release\` (with `RUSTFLAGS=\"--cfg tokio_unstable\"`) passes — finished in 2m27s. Confirms `gaptos = { features = [\"gcp-secret-manager\"] }` resolves and pulls `reqwest` + `base64` into the build graph. - [ ] CI release workflow dry-run (PR-trigger) — the workflow attaches \`release-dryrun-<sha>\` artifact when triggered on a PR. Single \`Build gravity_node and gravity_cli\` step should succeed. - [ ] After merge: cut a new release tag (e.g. \`gravity-mainnet-v1.6.2\`), then verify on a Staging Mainnet VFN that the panic at \`network_config.rs:281\` is gone and \`journalctl -u gravity-vfn-1.service\` advances past the identity-load step. ## References - #713 — the workflow-level fix this supersedes - #710 — original release workflow that shipped without the feature - #688 / #695 — the cluster.toml-side wiring that originally introduced the optional build mode 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New
.github/workflows/release.ymlfor shippinggravity_node/gravity_clibinaries when agravity-mainnet-*orgravity-testnet-*tag is pushed. Workflow runs natively on the self-hosted Ubuntu 24.04 runner — no container.Decision: align production node OS with the runner OS, not the other way around. Runner is already Ubuntu 24.04 (glibc 2.39); the planned production-VM Packer image is being moved to
ubuntu-2404-lts-amd64to match. Native runner build keeps the workflow simple (no per-run apt-install,target/cache reuse) at the cost of requiring runner state stays in sync.The earlier containerized direction (
rust:1.93.0-bullseye) is preserved in git history (abf88dd1..af3e3711) and can be brought back withgit revert d75fc1f2if the OS migration slips.Changes
.github/workflows/release.yml[self-hosted, linux]— nocontainer:field.push: tags: ['gravity-mainnet-*', 'gravity-testnet-*']andpull_request: paths: ['.github/workflows/release.yml'].Show build environmentstep logs/etc/os-release,ldd --version,rustc --version,cargo --versionevery run, so runner OS drift surfaces in the run record.Assert rustc matches rust-toolchain.tomlso toolchain drift fails fast with a clear remediation message.release-dryrun-<sha>artifact for reviewer inspection; real GH-release upload is skipped on PR events.gh release upload --clobber. No auto-create.Resolve tagstep rejects any tag that doesn't matchgravity-{mainnet,testnet}-*..github/workflows/lint-workflows.ymlactionlint@1.7.7on PRs touching.github/workflows/**. Shellcheck integration disabled (-shellcheck=) — pre-existing SC2086 / SC2053 noise ingate.yml/rust-ci.ymlis out of scope; re-enable after a separate cleanup PR.Runner prerequisites (install once on the runner host)
rust-toolchain.toml(currently 1.93.0)ghCLI (viacli.github.comapt repo) — uses workflow-injectedGH_TOKEN, nogh auth loginneededclang llvm libclang-dev build-essential pkg-config libssl-dev libudev-devThese are installed on
gravity-grafana-20250806-070307.Test plan
Lint workflows / actionlintgreenRelease / Build and releasedry-run green (run 25787189549, attempt 2, 10m57s)objdump -T gravity_node | grep GLIBC_→ maxGLIBC_2.39(matches Ubuntu 24.04)sha256sum -cmatches workflow output./gravity_node --versionon runner shows non-emptybuild_commit_hashandbuild_time(read by shadow_rs at compile time from the checkout; PR path shows merge sha, tag-push path will show tag commit)gravity-mainnet-*tag until staging-mainnet VMs are running Ubuntu 24.04.Follow-ups (separate PRs)
ubuntu-os-cloud/ubuntu-2404-lts-amd64). Runner-native binaries depend onGLIBC_2.39; older OS targets will refuse to start.gate.yml/rust-ci.yml, then drop-shellcheck=fromlint-workflows.yml.Notes for reviewers
build-infocrate has a runtimestd::env::var(\"GIT_SHA\")override path (crates/build-info/src/lib.rs:84-101). It looks like it could be wired to GitHub Actions context to populatebuild_branch/build_commit_hashfrom the env, but the override is runtime-only — productionsystemdunits won't set those env vars, so it's a no-op for shipped binaries. Compile-timeshadow_rsalready fillsbuild_commit_hashandbuild_tagcorrectly on tag-push builds;build_branchis empty under detached-HEAD checkout (acceptable for tag releases). Fixingbuild_branchproperly is a crate change (option_env!+cargo:rerun-if-env-changed), tracked separately.03fcff29, since reset) before this was understood; the commit has been dropped via force-push.