Skip to content

Migrate pipeline from CircleCI to GitHub Actions #215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
phelma wants to merge 5 commits into from
Closed

Migrate pipeline from CircleCI to GitHub Actions #215

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ea9b1aa
migrate pipeline from circleci to github actions
phelma Jul 16, 2026
87e6fba
publish and verify a pre-release image on pull requests
phelma Jul 16, 2026
fd22322
debug: show dirty files before git-crypt unlock
phelma Jul 16, 2026
7fbdd6e
pin ruby 3.2.11 to match Gemfile.lock resolution
phelma Jul 16, 2026
03e8be8
re-lock nokogiri for x86_64-linux-gnu
phelma Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
176 changes: 0 additions & 176 deletions .circleci/config.yml
View file
Open in desktop

This file was deleted.

0 .circleci/gpg.private.enc → .github/gpg.private.enc
View file
Open in desktop
File renamed without changes.
151 changes: 151 additions & 0 deletions .github/workflows/main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
name: Main
on:
push:
branches:
- main

permissions:
contents: read

# Every run publishes an rc image and tags the repo; serialise so
# close-together merges cannot race on version tags. queue: max keeps every
# queued run (the default keeps only the newest pending), so a slow release
# approval delays later prereleases instead of cancelling them.
# 'queue: max' — GA 2026-05-07: https://github.blog/changelog/2026-05-07-github-actions-concurrency-groups-now-allow-larger-queues/
concurrency:
group: main
cancel-in-progress: false
queue: max
Comment on lines +15 to +18

@phelma phelma Jul 16, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 PLAN CONCERN (flagged independently by the correctness and safety lenses) — this block is mandated verbatim by plan §4.2, so it is a plan-level issue, not a defect of this diff. Not a blocker for this PR, but the single item most worth resolving before the fleet rollout.

concurrency is declared at workflow level, so the group is held for the whole run — including while release sits in waiting on its environment: release approval. A release is only approved when someone deliberately wants to ship, so the modal outcome of a push to main is that the run parks in waiting and holds group: main until the approval expires (default 30 days).

With cancel-in-progress: false and queue: max, every subsequent push to main queues behind it and never starts: no tests, no rc image, no tag — silently, with no failure signal. The comment's framing ("a slow release approval delays later prereleases instead of cancelling them") describes a bounded delay; the unbounded case looks like the normal one.

This is also a regression from CircleCI rather than parity: CircleCI doesn't serialise workflows, so its hold job blocked only its own workflow — pushes 2, 3, 4 each ran test + prerelease independently.

Safety lens adds the 3am version: a release proposed on Friday that nobody approves means main has no CI feedback by Monday, plus a queue of runs that will each publish an rc image and push a tag the moment someone approves or cancels.

Suggested fix (for the plan): scope serialisation to the jobs that actually race on version tags, so the approval wait can't hold it — drop the workflow-level block and put concurrency: {group: main-prerelease, cancel-in-progress: false} on prerelease. Alternatively split release into a workflow_dispatch workflow so the approval wait lives outside the group entirely.

Comment on lines +15 to +18

@phelma phelma Jul 16, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Plan concern (not a defect — this is the plan's exact YAML) | Flagged independently by: correctness, safety

The main group spans the whole workflow, including the approval-gated release job. Two distinct consequences follow, and the plan's D4 note ("nothing lands on main at all") doesn't quite cover either:

1. Queued prereleases can fail on a rejected push. With queue: max, a run whose release job awaits approval holds the group; subsequent pushes queue behind it. When a queued run starts, actions/checkout fetches at github.sha (the old commit), leaving local main behind origin/main — and prerelease deliberately has no git pull. version:bump[rc] then calls repo.push('origin', 'main', tags: true), which the git gem implements as two commands: git push origin main, then git push --tags origin. The first is a non-fast-forward and is rejected — so the job fails and the tag repo.add_tag just created is never pushed, orphaned in the workspace. In the normal case local main matches origin and the push is a harmless no-op, which is why D4's framing holds day-to-day; serialisation makes the divergent case the expected outcome whenever an approval is slow, rather than a rare race.

2. A pending approval freezes all main-branch CI. While release waits (GitHub holds approvals pending for up to 30 days), the group is occupied, so no check or test runs on main at all — not just no prereleases. queue: max retains every queued run, so the backlog grows. One forgotten approval silently stops main-branch feedback for the whole team; runs show as queued, not failed, so it's easy to miss for days. Recovery is fast once diagnosed.

Suggested plan amendment: scope serialisation to the jobs that actually race on version tags rather than the whole workflow — drop the workflow-level concurrency and put concurrency: { group: main-publish, cancel-in-progress: false, queue: max } on prerelease and release. check/test mutate no shared state and are safe to run in parallel across main pushes. This preserves the tag-race protection while keeping feedback flowing. Giving prerelease a git pull would separately address (1).


jobs:
skip-ci-check:
runs-on: ubuntu-latest
outputs:
should_skip_ci: ${{ steps.skip_ci_check.outputs.should_skip_ci }}
steps:
- id: skip_ci_check
env:
HEAD_COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
run: |
if [[ "$HEAD_COMMIT_MESSAGE" == *"[no ci]"* ]] \
|| [[ "$HEAD_COMMIT_MESSAGE" == *"[skip ci]"* ]] \
|| [[ "$HEAD_COMMIT_MESSAGE" == *"[ci skip]"* ]]; then
echo "should_skip_ci=true" >> "$GITHUB_OUTPUT"
else
echo "should_skip_ci=false" >> "$GITHUB_OUTPUT"
fi

check:
needs: [skip-ci-check]
if: needs.skip-ci-check.outputs.should_skip_ci != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install tools
uses: infrablocks/github-actions/asdf_install@v1
- name: Check
run: ./go test:code:check
- name: Notify Slack
if: ${{ !cancelled() }}
continue-on-error: true
run: ./go "slack:notify[${{ job.status }}]"
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

test:
needs: [skip-ci-check]
if: needs.skip-ci-check.outputs.should_skip_ci != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install tools
uses: infrablocks/github-actions/asdf_install@v1
- name: Ensure docker-compose is available
run: |
if ! command -v docker-compose >/dev/null 2>&1; then
sudo curl -fsSL \
"https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64" \
-o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
fi
Comment on lines +65 to +72
Comment on lines +65 to +72

@phelma phelma Jul 16, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Plan concern (not a defect — plan §4.1/§4.2 specify this exact step) | Flagged independently by: security, safety, standards, code quality

Four lenses converged. The guard is documented as deliberate; the concerns are about properties of the guard that the plan doesn't address:

Supply chain — releases/latest is a mutable pointer resolved at run time. The bytes are never pinned, checksummed or signed, yet are installed to /usr/local/bin with sudo and executed. On main.yaml the surrounding jobs hold ENCRYPTION_PASSPHRASE, the unlocked secrets tree, and Docker Hub push credentials — so a compromised upstream artefact could exfiltrate the git-crypt passphrase or tamper with the published infrablocks/prometheus-aws image, with no commit in this repo to correlate against.

Reproducibility — every other tool in this pipeline is pinned via .tool-versions + asdf_install@v1. This is the one CI dependency whose version is declared nowhere, so a breaking compose release changes behaviour fleet-wide simultaneously and bisects to nothing.

Duplication — the block is repeated verbatim in pr.yaml. The PR description justifies repeating scaffolding on the grounds that logic lives in the build system; this is imperative provisioning logic in CI YAML, so it's the one spot that cuts against that principle.

Suggested plan amendments (in preference order):

  1. Point dependencies:test:provision/destroy at the docker compose plugin already present on ubuntu-latest — the guard step and its download disappear entirely.
  2. Failing that, factor the guard into a composite action in infrablocks/github-actions alongside asdf_install, with a pinned version and a checksum:
env:
  COMPOSE_VERSION: v2.32.4
  COMPOSE_SHA256: <sha>
run: |
  if ! command -v docker-compose >/dev/null 2>&1; then
    curl -fsSL "https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-linux-x86_64" -o /tmp/docker-compose
    echo "${COMPOSE_SHA256}  /tmp/docker-compose" | sha256sum -c -
    sudo install -m 0755 /tmp/docker-compose /usr/local/bin/docker-compose
  fi

- name: Test
run: ./go test:integration
- name: Notify Slack
if: ${{ !cancelled() }}
continue-on-error: true
run: ./go "slack:notify[${{ job.status }}]"
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

prerelease:
needs: [check, test]
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install tools
uses: infrablocks/github-actions/asdf_install@v1

@phelma phelma Jul 16, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 PLAN CONCERN — §1/D6 explicitly accept tracking the moving v1 tag, so not a blocker. Recorded only because it composes with the other two items in the "trust boundary at release time" theme.

infrablocks/github-actions/asdf_install@v1 is a mutable major-version tag, and it runs in prerelease and release — the two jobs holding ENCRYPTION_PASSPHRASE, contents: write, and (post-unlock) the decrypted Docker Hub credentials.

A push to the v1 tag in the actions repo — malicious or accidental by anyone with write access there — silently executes new code in every InfraBlocks release job across the fleet. First-party ownership reduces likelihood but not blast radius; a first-party repo with broad write access is often the softer target.

Suggested follow-up: pin to a commit SHA (asdf_install@<sha> # v1.2.3) at least in the secret-bearing jobs, and protect the v1 tag in infrablocks/github-actions. Dependabot's github-actions ecosystem keeps SHA pins current, so the maintenance cost is near zero.

- name: Install secrets tools
run: sudo apt-get update && sudo apt-get install -y git-crypt gnupg
- name: Unlock git-crypt
run: ./go git_crypt:unlock_with_encrypted_gpg_key
env:
ENCRYPTION_PASSPHRASE: ${{ secrets.ENCRYPTION_PASSPHRASE }}
- name: Set CI git author
run: ./go repository:set_ci_author
- name: Prerelease
run: ./go "version:bump[rc]" && ./go image:publish
- name: Notify Slack of release hold
continue-on-error: true
run: ./go "slack:notify[success,on_hold]"
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
- name: Notify Slack
if: ${{ !cancelled() }}
continue-on-error: true
run: ./go "slack:notify[${{ job.status }}]"
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

release:
needs: [prerelease]
runs-on: ubuntu-latest
timeout-minutes: 30
environment: release

@phelma phelma Jul 16, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Plan concern (not a defect — plan §7 covers provisioning) | Lens: safety

The approval gate depends entirely on the release environment existing with its reviewer protection rule. GitHub auto-creates an environment on first reference if it doesn't exist — and the auto-created environment has no protection rules. The environment is only created out-of-band by ./go pipeline:prepare, a manual operator step nothing in the workflow verifies.

Impact: if pipeline:prepare hasn't run against a repo before its first push to main (or if the environment is later deleted), release runs unapproved and publishes a full release image plus a release tag with no human in the loop — and the workflow reports success, so there is no signal the gate was absent. That's a fail-open default on the pipeline's most consequential job, at exactly the moment in a fleet migration when the provisioning step is most likely to be missed on some repo.

The plan's §7 verification (gh api .../environments/release --jq '.protection_rules') catches this if the operator runs it. Worth considering whether the family plan should make the gate self-verifying rather than checklist-verified:

      - name: Assert release was approved
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          rules=$(gh api "repos/${{ github.repository }}/environments/release" \
            --jq '.protection_rules | length')
          [ "$rules" -gt 0 ] || { echo "release environment has no protection rules"; exit 1; }

For this PR specifically: please confirm §7 provisioning was run and that release shows a maintainers required reviewer before merging.

permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
ref: main
fetch-depth: 0
- name: Pull latest main
# Approval can land long after the run starts; release publishes main
# as of approval time, not the tested SHA — parity with the old
# release.sh, which also pulled. prerelease.sh never pulled, so the
# prerelease job deliberately has no pull.
run: git pull

@phelma phelma Jul 16, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 PLAN CONCERN (flagged by both security and safety) — documented as deliberate parity with the old release.sh, which also pulled. Not a blocker.

ref: main + git pull means the approver approves run N (built from SHA X), but what gets tagged and published is main as of approval time. The environment: release reviewer is the only human gate on an irreversible Docker Hub publish plus a pushed release tag — and what they're approving isn't pinned to anything they can inspect.

The security framing worth adding to the plan's note: this is a time-of-check/time-of-use gap in the only human authorisation control on the release path. Anyone who can land on main between approval request and approval — including via the unfiltered dependabot auto-merge, which merges without triggering a build — gets that code published under a release tag with a maintainer's approval attached. The audit trail then attributes the release to an approver who never saw the content.

The workflow-level concurrency: group: main does mitigate the common case today (later pushes queue behind the pending release), so this mainly bites on a direct push during the approval wait — but note it stops mitigating if the group is re-scoped to fix the stall issue flagged above. The two interact.

Suggested fix (post-migration): ref: ${{ github.sha }} with no git pull, so the approval gates the exact tested tree.

- name: Install tools
uses: infrablocks/github-actions/asdf_install@v1
- name: Install secrets tools
run: sudo apt-get update && sudo apt-get install -y git-crypt gnupg
- name: Unlock git-crypt
run: ./go git_crypt:unlock_with_encrypted_gpg_key
env:
ENCRYPTION_PASSPHRASE: ${{ secrets.ENCRYPTION_PASSPHRASE }}
- name: Set CI git author
run: ./go repository:set_ci_author
Comment on lines +128 to +143

@phelma phelma Jul 16, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Plan concern (not a defect — this step order is the plan's exact YAML) | Lens: correctness

git pull runs before ./go repository:set_ci_author. It fast-forwards in the common case, but if it needs to create a merge commit, git aborts with "Committer identity unknown / Please tell me who you are" — no user.name/user.email is configured yet.

This inverts the old CircleCI ordering, where configure_tools (including configure-git.sh, which set the identity) ran before release.sh performed its git pull. So it's a small, probably unintended parity break rather than a carried-over wart.

Impact: a rare but real failure mode where the release job dies on the pull with a confusing identity error — precisely when main has diverged, which is when the release matters most.

Suggested plan amendment: move Set CI git author above Pull latest main to restore the original ordering. Alternatively git pull --ff-only, so a diverging main fails loudly and unambiguously instead of attempting a merge it cannot author.

- name: Release
run: ./go version:release && ./go image:publish
- name: Notify Slack
if: ${{ !cancelled() }}
continue-on-error: true
run: ./go "slack:notify[${{ job.status }}]"
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
Loading
Loading