diff --git a/.github/README.md b/.github/README.md new file mode 100644 index 00000000000..d5123f0e242 --- /dev/null +++ b/.github/README.md @@ -0,0 +1,186 @@ +# GitHub Actions Workflows + +This directory contains GitHub Actions workflows, along with composites, helper scripts, and Dockerfiles. The system is organized around a central **CI orchestrator** (`ci.yml`) that triggers reusable workflows and composites for testing, building, and releasing. + +--- + +## Permissions Model + +- **ci.yml**: `contents: read` (default) – minimal permissions for linting/checks +- **release-github.yml**: `contents: write`, `packages: write`, `id-token: write` – required for release creation and image push +- **Individual workflows**: Request specific permissions (principle of least privilege) + +--- + +## Triggers + +### CI Workflow (`ci.yml`) — The Orchestrator + +Triggered by: +- **Pull requests**: Opened, reopened, or when new commits are pushed +- **Merge queue**: `merge_group` event when PR is queued for merge +- **Manual dispatch** + +### Test Workflows + +In addition to being triggered as reusable workflows from `ci.yml`, each test workflow may be run manually via dispatch. +If the required caches do not exist, they will be created to run the workflow steps. + + +### Release Detection + +Releases are detected automatically by analyzing the **branch name** in the check-release step of `ci.yml`: +- **Node release**: Branch matches `release/X.Y.Z.A.B` (5-part version) → builds stacks-node + all tests +- **Signer-only release**: Branch matches `release/signer-X.Y.Z.A.B.C` (6-part version) → builds stacks-signer + all tests +- **No release**: Any other branch or PR → skips release jobs and epoch-tests + +Detection is handled by `.github/scripts/check_release.sh` and controlled by the `check-release` job outputs. + +--- + +### PR, Manual Dispatch, and Merge Queue Workflow +**Duration**: ~45-90 min + +**Trigger**: Pull Request (*Note*: `clippy`, `nix-check`, `proptest-extra` are not triggered via the orchestrator workflow but will run on a PR event) + +``` +┌────────────────┐ ┌─────────────────┐ ┌────────────────┐ ┌────────────────┐ ┌─────────────────┐ +│ clippy │ │ changelog-check │-▷ │ rustfmt │ -▷ │ check-release │ ------▷ │ constants-check │ +│ nix-check │ └─────────────────┘ └────────────────┘ └────────────────┘ │ cargo-hack │ +│ proptest-extra │ │ └─────────────────┘ +└────────────────┘ │ + │ ┌─────────────────┐ + ▽ │ stacks-core │ + ┌────────────────┐ │ bitcoin │ + │ create-cache │ ------▷ │ bitcoin-rpc │ + └────────────────┘ │ p2p-tests │ + └─────────────────┘ + │ + ▽ + ┌─────────────────┐ + │ codecov │ + └─────────────────┘ +``` + +### Release Branch Workflow +**Duration**: ~45-90 min + release pipeline + +**Triggers**: Manual dispatch on a release branch - `release/X.Y.Z.A.B` or `release/signer/X.Y.Z.A.B.C` +``` +┌─────────────-───┐ ┌────────────────┐ ┌────────────────┐ ┌─────────────────┐ +│ changelog-check │-▷ │ rustfmt │ -▷ │ check-release │ --------▷ │ constants-check │ +└─────────────-───┘ └────────────────┘ └────────────────┘ │ │ cargo-hack │ + │ │ └─────────────────┘ ┌─────────────────┐ + │ │ │ build-binaries │ + │ └────────────────────────▷ │ release-docker │ + │ │ github-release │ + │ ┌─────────────────┐ └─────────────────┘ + ▽ │ stacks-core │ + ┌────────────────┐ │ bitcoin │ ┌─────────────────┐ + │ create-cache │ ------▷ │ bitcoin-rpc │ -─▷ │ codecov │ + └────────────────┘ │ p2p-tests │ └─────────────────┘ + │ epoch-tests │ + └─────────────────┘ +``` + +--- + +## Concurrency & Cancellation + +```yaml +concurrency: + group: ci-${{ github.head_ref || github.ref || github.run_id }} + cancel-in-progress: true +``` + +**Behavior:** +- Each PR/branch has a single concurrency group +- When new commits are pushed to a PR, previous runs are **automatically cancelled** +- Release branches have their own concurrency group, independent of PR runs + +**Example:** +- Push to PR → run 1 starts +- Push again immediately → run 1 is cancelled, run 2 starts +- Multiple PRs run in parallel (different concurrency groups) + +--- + +## Independent Workflows + +These workflows run **outside** the main `ci.yml` orchestrator and are triggered by different events: + +### Branch Push, or Merge Queue +- **`nix-check.yml`** Validates Nix environment setup. +- **`clippy.yml`** – Runs additional Rust linting checks. +- **`proptest-extra-tests.yml`** – on-demand property tests (with configurable base branch and case count). + +### Scheduled +- **`docker-image.yml`** – Scheduled daily (5am UTC) build of Docker images from the `develop` branch. +- **`proptest-nightly-tests.yml`** – Scheduled daily (5am UTC) property-based fuzz testing on `develop`. +- **`lock-threads.yml`** – Scheduled daily (midnight UTC) to lock stale issues and PRs. + +### Manual Dispatch Only +- **`sbtc-tests.yml`** – Manual sBTC test suite. + +**Note:** `nix-check.yml` and `clippy.yml` run independently on branch/PR events and are **not orchestrated by `ci.yml`**. + +--- + +## Directory Structure + +### `.github/workflows/` +GitHub Actions workflow definitions (`.yml` files). Each workflow can be: +- **Called by ci.yml** – Reusable workflows triggered by the orchestrator +- **Standalone** – Run manually or on schedule + +Key workflows: +- `ci.yml` – Main orchestrator for PR/release CI +- `release-github.yml` – Release coordination (calls build and docker) +- `release-build.yml` – Builds binaries for multiple architectures +- `release-docker.yml` – Builds and publishes Docker images to GHCR +- `docker-image.yml` – Independent nightly Docker builds +- Test workflows – stacks-core-tests.yml, bitcoin-tests.yml, etc. + +### `.github/actions/` +Reusable composite actions that encapsulate multi-step workflows: +- `docker/` – Docker setup, QEMU, buildx, registry login +- `setup-rust-toolchain/` – Rust environment configuration +- `cache/` – Cache management (bitcoin, cargo, test archives) +- `testenv/` – Test environment setup and cache restore +- `run-tests/` – Test execution with nextest +- `install-tool/` – Tool installation (nextest, grcov, etc.) +- `codecov/` – Code coverage integration +- `release/` – Release automation actions + +### `.github/scripts/` +Shell scripts and utilities executed by workflows. Any non-trivial script step should be added here, and should be able to run locally: +- `check_release.sh` – Detects if branch is a release and sets version tags +- `build_binaries.sh` – Builds binaries for target platform (Rust cross-compilation) +- `draft_release.sh` – Creates draft GitHub release with artifacts +- `rustfmt.sh` – Enforces Rust code formatting +- `changelog.js` – Validates CHANGELOG.md updates +- `logging.sh` – Common logging functions + +### `.github/dockerfiles/` +Dockerfile definitions for building images: +- `debian/` – Debian-based images (glibc) +- `alpine/` – Alpine-based images (musl) + +--- + +## Release Process + +When `ci.yml` is triggered via dispatch from a **release branch** (e.g., `release/1.2.3.4.5`): + +1. **Branch detection** – `check-release` job identifies the version tags +2. **Validation** – rustfmt and changelog checks must pass +3. **Binary builds** – `release-build.yml` compiles binaries for: + - Linux x86_64 (native for musl and glibc + - Linux ARM64 (cross-compiled for musl and glibc) + - Windows x86_64 + - MacOS ARM64 (native) +4. **Docker images** – `release-docker.yml` builds Linux based images and pushes to `ghcr.io/stacks-network/*` +5. **Release creation** – Draft GitHub release created with binary archive assets +6. **Attestation** – Build provenance attached to artifacts for supply chain security + +--- diff --git a/.github/actions/run-tests/action.yml b/.github/actions/run-tests/action.yml index e3c144c7505..db0e325d432 100644 --- a/.github/actions/run-tests/action.yml +++ b/.github/actions/run-tests/action.yml @@ -25,7 +25,7 @@ inputs: retries: description: "Number of test retries" required: false - default: "0" + default: "2" runs: using: "composite" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3e344dbee1d..448beb81411 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -131,12 +131,9 @@ jobs: create-cache: name: Create Test Cache needs: - - rustfmt - - changelog-check - check-release if: >- !cancelled() && - needs.rustfmt.result == 'success' && (needs.check-release.outputs.is_node_release == 'true' || needs.check-release.outputs.is_signer_release == 'true' || github.event_name == 'workflow_dispatch' || @@ -148,16 +145,10 @@ jobs: stacks-core-tests: name: Stacks Core Tests needs: - - rustfmt - - changelog-check - create-cache - - check-release if: >- !cancelled() && - needs.rustfmt.result == 'success' && - (needs.check-release.outputs.is_node_release == 'true' || - needs.check-release.outputs.is_signer_release == 'true' || - github.event_name == 'workflow_dispatch' || + (github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request' || github.event_name == 'merge_group') uses: ./.github/workflows/stacks-core-tests.yml @@ -166,16 +157,10 @@ jobs: bitcoin-tests: name: Bitcoin Tests needs: - - rustfmt - - changelog-check - create-cache - - check-release if: >- !cancelled() && - needs.rustfmt.result == 'success' && - (needs.check-release.outputs.is_node_release == 'true' || - needs.check-release.outputs.is_signer_release == 'true' || - github.event_name == 'workflow_dispatch' || + (github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request' || github.event_name == 'merge_group') uses: ./.github/workflows/bitcoin-tests.yml @@ -184,15 +169,10 @@ jobs: bitcoin-rpc-tests: name: Bitcoin RPC Tests needs: - - rustfmt - create-cache - - check-release if: >- !cancelled() && - needs.rustfmt.result == 'success' && - (needs.check-release.outputs.is_node_release == 'true' || - needs.check-release.outputs.is_signer_release == 'true' || - github.event_name == 'workflow_dispatch' || + (github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request' || github.event_name == 'merge_group') uses: ./.github/workflows/bitcoin-rpc-tests.yml @@ -201,16 +181,10 @@ jobs: p2p-tests: name: P2P Tests needs: - - rustfmt - - changelog-check - create-cache - - check-release if: >- !cancelled() && - needs.rustfmt.result == 'success' && - (needs.check-release.outputs.is_node_release == 'true' || - needs.check-release.outputs.is_signer_release == 'true' || - github.event_name == 'workflow_dispatch' || + (github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request' || github.event_name == 'merge_group') uses: ./.github/workflows/p2p-tests.yml @@ -225,12 +199,9 @@ jobs: cargo-hack-check: name: Cargo Hack Check needs: - - rustfmt - - changelog-check - check-release if: >- !cancelled() && - needs.rustfmt.result == 'success' && (github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request' || github.event_name == 'merge_group') @@ -240,12 +211,9 @@ jobs: constants-check: name: Constants Check needs: - - rustfmt - - changelog-check - check-release if: >- !cancelled() && - needs.rustfmt.result == 'success' && (github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request' || github.event_name == 'merge_group') @@ -260,18 +228,16 @@ jobs: # - Creates Docker images and pushes to ghcr registry # - Creates a draft github release create-release: + name: Create Release + needs: + - changelog-check + - check-release if: >- !cancelled() && github.event.repository.visibility == 'public' && - needs.rustfmt.result == 'success' && needs.changelog-check.result == 'success' && (needs.check-release.outputs.is_node_release == 'true' || needs.check-release.outputs.is_signer_release == 'true') - name: Create Release - needs: - - rustfmt - - changelog-check - - check-release permissions: contents: write # required for github release id-token: write # required for attestation @@ -287,13 +253,11 @@ jobs: epoch-tests: name: Epoch Tests needs: - - rustfmt - - changelog-check - create-cache - check-release if: >- !cancelled() && - needs.rustfmt.result == 'success' && + github.event.repository.visibility == 'public' && (needs.check-release.outputs.is_node_release == 'true' || needs.check-release.outputs.is_signer_release == 'true') uses: ./.github/workflows/epoch-tests.yml @@ -304,6 +268,12 @@ jobs: # Runs when: # - always (unless workflow is cancelled OR any of bitcoin or stacks-core or p2p tests are skipped) trigger-code-coverage-report: + name: Merge & Upload Code Coverage Report + runs-on: ubuntu-latest + needs: + - stacks-core-tests + - bitcoin-tests + - p2p-tests if: >- always() && !cancelled() && @@ -311,12 +281,6 @@ jobs: !contains(needs.stacks-core-tests.result, 'skipped') && !contains(needs.bitcoin-tests.result, 'skipped') && !contains(needs.p2p-tests.result, 'skipped') - name: Merge & Upload Code Coverage Report - runs-on: ubuntu-latest - needs: - - stacks-core-tests - - bitcoin-tests - - p2p-tests env: REPORT_FILES_DIR: "code_coverage_files" REPORT_FILES_EXT: "info" @@ -338,7 +302,7 @@ jobs: # Check for at least 1 code coverage files - name: Check for at least 1 code coverage file run: | - file_count=$(find -type f -wholename "./${{ env.REPORT_FILES_DIR }}/*.${{ env.REPORT_FILES_EXT }}" | wc -l) + file_count=$(find "./${{ env.REPORT_FILES_DIR }}/*.${{ env.REPORT_FILES_EXT }}" -type f -wholename | wc -l) if [ "$file_count" -eq 0 ]; then echo "ERROR: no code coverage files found to merge. Verify that they were correctly generated and uploaded in prior CI steps" exit 1