Skip to content

Fix and modernize release workflows, complete ghcr.io migration, and parameterize jobs for easy fork usage #2363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ddelnano merged 6 commits into from
Apr 27, 2026
Merged

Fix and modernize release workflows, complete ghcr.io migration, and parameterize jobs for easy fork usage #2363

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2acddef
Upgrade docker buildx to v0.33.0 to fix deprecated client version error
ddelnano Apr 27, 2026
ed4f6e8
Fix and modernize release build scripts
ddelnano Apr 27, 2026
4acc0e0
Parameterize image registry via IMAGE_REPO
ddelnano Apr 27, 2026
f12f974
Parameterize buildbot identity and GH_REPO
ddelnano Apr 27, 2026
04d5def
Drop GCS release path; push images and helm charts to GHCR
ddelnano Apr 27, 2026
9aba7f3
Gate CLI macOS signing on vars.ENABLE_MACOS_SIGNING
ddelnano Apr 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 42 additions & 21 deletions .github/workflows/cli_release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,15 @@ jobs:
name: Build Release
runs-on: oracle-16cpu-64gb-x86-64
needs: get-dev-image
permissions:
contents: read
packages: write
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
env:
ARTIFACT_UPLOAD_LOG: "artifact_uploads.json"
# When macOS signing is enabled, push-signed-artifacts owns the manifest update.
MANIFEST_UPDATES: ${{ vars.ENABLE_MACOS_SIGNING == 'true' && '' || 'manifest_updates.json' }}
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
Expand All @@ -42,43 +47,48 @@ jobs:
BUILDBOT_GPG_KEY_B64: ${{ secrets.BUILDBOT_GPG_KEY_B64 }}
run: |
echo "${BUILDBOT_GPG_KEY_B64}" | base64 --decode | gpg --no-tty --batch --import
- id: gcloud-creds
uses: ./.github/actions/gcloud_creds
- name: Login to GHCR
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
SERVICE_ACCOUNT_KEY: ${{ secrets.GH_RELEASE_SA_PEM_B64 }}
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ github.token }}
- name: Build & Push Artifacts
env:
REF: ${{ github.event.ref }}
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_NUMBER: ${{ github.run_attempt }}
JOB_NAME: ${{ github.job }}
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
GH_REPO: ${{ github.repository }}
IMAGE_REPO: ${{ vars.IMAGE_REPO || 'ghcr.io/pixie-io' }}
shell: bash
run: |
export TAG_NAME="${REF#*/tags/}"
mkdir -p "artifacts/"
export ARTIFACTS_DIR="$(realpath artifacts/)"
./ci/save_version_info.sh
./ci/cli_build_release.sh
# Despite the name, linux-artifacts also contains the unsigned darwin
# binaries (cli_darwin_{amd64,arm64}_unsigned). sign-release downloads
# this artifact to feed cli_merge_sign.sh.
- name: Upload Github Artifacts
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: linux-artifacts
path: artifacts/
- name: Update GCS Manifest
env:
ARTIFACT_MANIFEST_BUCKET: "pixie-dev-public"
# Use the old style versions file instead of the new updates for the gcs manifest.
MANIFEST_UPDATES: ""
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
run: ./ci/update_artifact_manifest.sh
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: artifact-upload-log
path: ${{ env.ARTIFACT_UPLOAD_LOG }}
- if: vars.ENABLE_MACOS_SIGNING != 'true'
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: manifest-updates
path: manifest_updates.json
sign-release:
name: Sign Release for MacOS
if: vars.ENABLE_MACOS_SIGNING == 'true'
runs-on: macos-latest
needs: build-release
steps:
Expand All @@ -87,6 +97,10 @@ jobs:
fetch-depth: 0
- name: Add pwd to git safe dir
run: git config --global --add safe.directory `pwd`
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: linux-artifacts
path: artifacts/
- name: Install gon
run: brew install Bearer/tap/gon
- name: Sign CLI release
Expand All @@ -101,7 +115,6 @@ jobs:
export CERT_PATH="pixie.cert"
echo -n "$CERT_BASE64" | base64 --decode -o "$CERT_PATH"
export TAG_NAME="${REF#*/tags/}"
mkdir -p "artifacts/"
export ARTIFACTS_DIR="$(pwd)/artifacts"
./ci/cli_merge_sign.sh
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
Expand All @@ -110,6 +123,7 @@ jobs:
path: artifacts/
push-signed-artifacts:
name: Push Signed Artifacts for MacOS
if: vars.ENABLE_MACOS_SIGNING == 'true'
runs-on: ubuntu-latest
needs: [get-dev-image, sign-release]
container:
Expand All @@ -131,19 +145,15 @@ jobs:
BUILDBOT_GPG_KEY_B64: ${{ secrets.BUILDBOT_GPG_KEY_B64 }}
run: |
echo "${BUILDBOT_GPG_KEY_B64}" | base64 --decode | gpg --no-tty --batch --import
- id: gcloud-creds
uses: ./.github/actions/gcloud_creds
with:
SERVICE_ACCOUNT_KEY: ${{ secrets.GH_RELEASE_SA_PEM_B64 }}
- name: Add pwd to git safe dir
run: |
git config --global --add safe.directory `pwd`
- name: Upload signed CLI
env:
REF: ${{ github.event.ref }}
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
ARTIFACT_UPLOAD_LOG: "artifact_uploads.json"
GH_REPO: ${{ github.repository }}
shell: bash
run: |
export TAG_NAME="${REF#*/tags/}"
Expand All @@ -161,7 +171,11 @@ jobs:
create-github-release:
name: Create Release on Github
runs-on: ubuntu-latest
needs: push-signed-artifacts
needs: [build-release, push-signed-artifacts]
if: |
always() &&
needs.build-release.result == 'success' &&
(needs.push-signed-artifacts.result == 'success' || needs.push-signed-artifacts.result == 'skipped')
permissions:
contents: write
steps:
Expand All @@ -186,8 +200,15 @@ jobs:
gh release create "${TAG_NAME}" "${prerelease[@]}" \
--title "CLI ${TAG_NAME#release/cli/}" \
--notes $'Pixie CLI Release:\n'"${changelog}"
gh release upload "${TAG_NAME}" linux-artifacts/* macos-artifacts/*
shopt -s nullglob
upload_paths=(linux-artifacts/*)
if [[ -d macos-artifacts ]]; then
upload_paths+=(macos-artifacts/*)
fi
gh release upload "${TAG_NAME}" "${upload_paths[@]}"
update-gh-artifacts-manifest:
if: |
always() && needs.create-github-release.result == 'success'
runs-on: oracle-8cpu-32gb-x86-64
needs: [get-dev-image, create-github-release]
container:
Expand Down Expand Up @@ -217,8 +238,8 @@ jobs:
env:
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
run: |
git config --global user.name 'pixie-io-buildbot'
git config --global user.email 'build@pixielabs.ai'
git config --global user.name "${{ vars.BUILDBOT_NAME || 'pixie-io-buildbot' }}"
git config --global user.email "${{ vars.BUILDBOT_EMAIL || 'build@pixielabs.ai' }}"
git config --global user.signingkey "${BUILDBOT_GPG_KEY_ID}"
git config --global commit.gpgsign true
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
Expand Down
19 changes: 12 additions & 7 deletions .github/workflows/cloud_release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ jobs:
name: Build Release
runs-on: oracle-16cpu-64gb-x86-64
needs: get-dev-image
permissions:
contents: read
packages: write
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
steps:
Expand All @@ -30,15 +33,17 @@ jobs:
with:
download_toplevel: 'true'
BB_API_KEY: ${{ secrets.BB_IO_API_KEY }}
- id: gcloud-creds
uses: ./.github/actions/gcloud_creds
with:
SERVICE_ACCOUNT_KEY: ${{ secrets.GH_RELEASE_SA_PEM_B64 }}
- name: Import GPG key
env:
BUILDBOT_GPG_KEY_B64: ${{ secrets.BUILDBOT_GPG_KEY_B64 }}
run: |
echo "${BUILDBOT_GPG_KEY_B64}" | base64 --decode | gpg --no-tty --batch --import
- name: Login to GHCR
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ github.token }}
- name: Build & Push Artifacts
env:
REF: ${{ github.event.ref }}
Expand All @@ -47,8 +52,9 @@ jobs:
GH_API_KEY: ${{ secrets.GITHUB_TOKEN }}
COSIGN_PASSWORD: ${{secrets.COSIGN_PASSWORD}}
COSIGN_PRIVATE_KEY: ${{secrets.COSIGN_PRIVATE_KEY}}
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
IMAGE_REPO: ${{ vars.IMAGE_REPO || 'ghcr.io/pixie-io' }}
GH_REPO: ${{ github.repository }}
shell: bash
run: |
export TAG_NAME="${REF#*/tags/}"
Expand Down Expand Up @@ -76,8 +82,7 @@ jobs:
env:
REF: ${{ github.event.ref }}
GH_TOKEN: ${{ secrets.BUILDBOT_GH_API_TOKEN }}
OWNER: pixie-io
REPO: pixie
GH_REPO: ${{ github.repository }}
shell: bash
run: |
export TAG_NAME="${REF#*/tags/}"
Expand Down
30 changes: 14 additions & 16 deletions .github/workflows/operator_release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ jobs:
name: Build Release
runs-on: oracle-16cpu-64gb-x86-64
needs: get-dev-image
permissions:
contents: read
packages: write
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
env:
Expand All @@ -33,25 +36,27 @@ jobs:
with:
download_toplevel: 'true'
BB_API_KEY: ${{ secrets.BB_IO_API_KEY }}
- id: gcloud-creds
uses: ./.github/actions/gcloud_creds
with:
SERVICE_ACCOUNT_KEY: ${{ secrets.GH_RELEASE_SA_PEM_B64 }}
- name: Import GPG key
env:
BUILDBOT_GPG_KEY_B64: ${{ secrets.BUILDBOT_GPG_KEY_B64 }}
run: |
echo "${BUILDBOT_GPG_KEY_B64}" | base64 --decode | gpg --no-tty --batch --import
- name: Login to GHCR
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ github.token }}
- name: Build & Push Artifacts
env:
REF: ${{ github.event.ref }}
BUILD_NUMBER: ${{ github.run_attempt }}
JOB_NAME: ${{ github.job }}
COSIGN_PASSWORD: ${{secrets.COSIGN_PASSWORD}}
COSIGN_PRIVATE_KEY: ${{secrets.COSIGN_PRIVATE_KEY}}
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
GH_REPO: ${{ github.repository }}
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
IMAGE_REPO: ${{ vars.IMAGE_REPO || 'ghcr.io/pixie-io' }}
shell: bash
run: |
export TAG_NAME="${REF#*/tags/}"
Expand All @@ -60,13 +65,6 @@ jobs:
mkdir -p "${ARTIFACTS_DIR}"
./ci/save_version_info.sh
./ci/operator_build_release.sh
- name: Update GCS Manifest
env:
ARTIFACT_MANIFEST_BUCKET: "pixie-dev-public"
# Use the old style versions file instead of the new updates for the gcs manifest.
MANIFEST_UPDATES: ""
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
run: ./ci/update_artifact_manifest.sh
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: manifest-updates
Expand Down Expand Up @@ -127,8 +125,8 @@ jobs:
env:
GIT_SSH_COMMAND: "ssh -i /tmp/ssh.key"
run: |
git config --global user.name 'pixie-io-buildbot'
git config --global user.email 'build@pixielabs.ai'
git config --global user.name "${{ vars.BUILDBOT_NAME || 'pixie-io-buildbot' }}"
git config --global user.email "${{ vars.BUILDBOT_EMAIL || 'build@pixielabs.ai' }}"
- name: Push Helm YAML to gh-pages
shell: bash
env:
Expand Down Expand Up @@ -171,8 +169,8 @@ jobs:
env:
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
run: |
git config --global user.name 'pixie-io-buildbot'
git config --global user.email 'build@pixielabs.ai'
git config --global user.name "${{ vars.BUILDBOT_NAME || 'pixie-io-buildbot' }}"
git config --global user.email "${{ vars.BUILDBOT_EMAIL || 'build@pixielabs.ai' }}"
git config --global user.signingkey "${BUILDBOT_GPG_KEY_ID}"
git config --global commit.gpgsign true
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
Expand Down
37 changes: 14 additions & 23 deletions .github/workflows/vizier_release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ jobs:
name: Build Release
runs-on: oracle-16cpu-64gb-x86-64
needs: get-dev-image
permissions:
contents: read
packages: write
container:
image: ${{ needs.get-dev-image.outputs.image-with-tag }}
env:
Expand All @@ -33,25 +36,27 @@ jobs:
with:
download_toplevel: 'true'
BB_API_KEY: ${{ secrets.BB_IO_API_KEY }}
- id: gcloud-creds
uses: ./.github/actions/gcloud_creds
with:
SERVICE_ACCOUNT_KEY: ${{ secrets.GH_RELEASE_SA_PEM_B64 }}
- name: Import GPG key
env:
BUILDBOT_GPG_KEY_B64: ${{ secrets.BUILDBOT_GPG_KEY_B64 }}
run: |
echo "${BUILDBOT_GPG_KEY_B64}" | base64 --decode | gpg --no-tty --batch --import
- name: Login to GHCR
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ github.token }}
- name: Build & Push Artifacts
env:
REF: ${{ github.event.ref }}
BUILD_NUMBER: ${{ github.run_attempt }}
JOB_NAME: ${{ github.job }}
COSIGN_PASSWORD: ${{secrets.COSIGN_PASSWORD}}
COSIGN_PRIVATE_KEY: ${{secrets.COSIGN_PRIVATE_KEY}}
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
GH_REPO: ${{ github.repository }}
IMAGE_REPO: ${{ vars.IMAGE_REPO || 'ghcr.io/pixie-io' }}
shell: bash
run: |
export TAG_NAME="${REF#*/tags/}"
Expand All @@ -60,20 +65,6 @@ jobs:
export INDEX_FILE="$(pwd)/index.yaml"
./ci/save_version_info.sh
./ci/vizier_build_release.sh
- name: Build & Export Docs
env:
PXL_DOCS_GCS_PATH: "gs://pixie-dev-public/pxl-docs.json"
run: |
docs="$(mktemp)"
bazel run //src/carnot/docstring:docstring -- --output_json "${docs}"
gsutil cp "${docs}" "${PXL_DOCS_GCS_PATH}"
- name: Update GCS Manifest
env:
ARTIFACT_MANIFEST_BUCKET: "pixie-dev-public"
# Use the old style versions file instead of the new updates for the gcs manifest.
MANIFEST_UPDATES: ""
GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.gcloud-creds.outputs.gcloud-creds }}
run: ./ci/update_artifact_manifest.sh
Copy link
Copy Markdown
Member Author

@ddelnano ddelnano Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs are updated through a different workflow now.

- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: manifest-updates
Expand Down Expand Up @@ -134,8 +125,8 @@ jobs:
env:
GIT_SSH_COMMAND: "ssh -i /tmp/ssh.key"
run: |
git config --global user.name 'pixie-io-buildbot'
git config --global user.email 'build@pixielabs.ai'
git config --global user.name "${{ vars.BUILDBOT_NAME || 'pixie-io-buildbot' }}"
git config --global user.email "${{ vars.BUILDBOT_EMAIL || 'build@pixielabs.ai' }}"
- name: Push Helm YAML to gh-pages
shell: bash
env:
Expand Down Expand Up @@ -178,8 +169,8 @@ jobs:
env:
BUILDBOT_GPG_KEY_ID: ${{ secrets.BUILDBOT_GPG_KEY_ID }}
run: |
git config --global user.name 'pixie-io-buildbot'
git config --global user.email 'build@pixielabs.ai'
git config --global user.name "${{ vars.BUILDBOT_NAME || 'pixie-io-buildbot' }}"
git config --global user.email "${{ vars.BUILDBOT_EMAIL || 'build@pixielabs.ai' }}"
git config --global user.signingkey "${BUILDBOT_GPG_KEY_ID}"
git config --global commit.gpgsign true
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
Expand Down
5 changes: 0 additions & 5 deletions ci/artifact_mirrors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,3 @@
- name: gh-releases
type: gh-releases
url_format: 'https://github.com/${gh_repo}/releases/download/release/${component}/v${version}/${artifact_name}'
- name: pixie-oss-gcs
type: gcs
bucket: pixie-dev-public
path_format: '${component}/${version}/${artifact_name}'
url_format: 'https://storage.googleapis.com/pixie-dev-public/${component}/${version}/${artifact_name}'
Loading
Loading