ci(release): use reproducible Docker build for Linux release binaries

[decofe]

Follow-up to #3804 — switches the tempo binary build for Linux targets from bare cargo build to the reproducible Docker build pipeline. Mac (aarch64-apple-darwin) remains non-reproducible.

Changes

Dockerfile.reproducible

• Parameterized with TARGET and JEMALLOC_ARCH build args (defaults to x86_64)
• aarch64 builds run natively on ARM runners via --platform linux/arm64

scripts/reproducible-build.sh

• Accepts TARGET env var, auto-derives DOCKER_PLATFORM and JEMALLOC_ARCH
• Passes TARGET and JEMALLOC_ARCH as --build-args to Docker

release.yml

• Adds reproducible: true/false flag to platform matrix entries
• For tempo on Linux: builds via Docker reproducible pipeline instead of bare cargo
• For tempo-bench, tempo-sidecar, and macOS: keeps bare cargo
• Emits .reproducible.sha256 sidecar alongside release artifacts for Linux tempo binaries
• Adds fetch-depth: 0 (needed by reproducible-build.sh for SOURCE_DATE_EPOCH)
• Sets up Docker Buildx conditionally for reproducible builds

reproducible-build.yml

• Adds workflow_call trigger with ref and target inputs
• Adds target choice to workflow_dispatch
• Picks runner dynamically based on target arch (depot-ubuntu-latest-arm-16 for aarch64)

Dockerfile.reproducible and scripts/reproducible-build.sh are already on the branch. The workflow file diffs need to be applied manually:

Patch for workflow files (click to expand)

diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 7811c971..efcd4e7f 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -99,15 +99,19 @@ jobs:
         platform:
           - os: depot-ubuntu-latest-16
             target: x86_64-unknown-linux-gnu
+            reproducible: true
           - os: depot-ubuntu-latest-arm-16
             target: aarch64-unknown-linux-gnu
+            reproducible: true
           - os: depot-macos-15
             target: aarch64-apple-darwin
+            reproducible: false
 
     steps:
       - name: Checkout repository
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
+          fetch-depth: 0
           persist-credentials: false
 
       - name: Install Rust toolchain
@@ -118,6 +122,10 @@ jobs:
       - name: Setup mold linker
         uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
 
+      - name: Set up Docker Buildx
+        if: matrix.platform.reproducible && matrix.binary.name == 'tempo'
+        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
+
       - name: Get build profile
         id: profile
         run: |
@@ -128,7 +136,28 @@ jobs:
           fi
         shell: bash
 
-      - name: Build binary
+      - name: Build binary (reproducible – Linux)
+        if: matrix.platform.reproducible && matrix.binary.name == 'tempo'
+        env:
+          VERSION: ${{ needs.get-version.outputs.version }}
+          TARGET: ${{ matrix.platform.target }}
+        # Retry to absorb snapshot.debian.org transient 503s.
+        run: |
+          set -euo pipefail
+          for attempt in 1 2 3; do
+            if ./scripts/reproducible-build.sh; then
+              break
+            fi
+            echo "reproducible-build.sh failed (attempt $attempt/3); retrying after backoff"
+            sleep $((attempt * 30))
+            if [ "$attempt" -eq 3 ]; then
+              echo "reproducible build failed after 3 attempts" >&2
+              exit 1
+            fi
+          done
+
+      - name: Build binary (standard)
+        if: ${{ !(matrix.platform.reproducible && matrix.binary.name == 'tempo') }}
         run: cargo build --locked --bin ${{ matrix.binary.name }} --features "${{ matrix.binary.features }}" --profile ${{ steps.profile.outputs.profile }} --target ${{ matrix.platform.target }}
         env:
           CARGO_TARGET_DIR: target
@@ -144,7 +173,14 @@ jobs:
             PROFILE_DIR="$PROFILE"
           fi
 
-          BINARY_PATH="target/${{ matrix.platform.target }}/${PROFILE_DIR}/${{ matrix.binary.name }}"
+          # Reproducible Docker builds output to out/tempo; standard builds
+          # output to the normal cargo target directory.
+          if [[ "${{ matrix.platform.reproducible }}" == "true" && "${{ matrix.binary.name }}" == "tempo" ]]; then
+            BINARY_PATH="out/tempo"
+          else
+            BINARY_PATH="target/${{ matrix.platform.target }}/${PROFILE_DIR}/${{ matrix.binary.name }}"
+          fi
+
           BINARY_NAME="${{ matrix.binary.name }}-${{ needs.get-version.outputs.version }}-${{ matrix.platform.target }}"
           ARCHIVE_NAME="${{ matrix.binary.name }}-${{ needs.get-version.outputs.version }}-${{ matrix.platform.target }}.tar.gz"
 
@@ -160,13 +196,24 @@ jobs:
         # Two separate single-line .sha256 files, one per artifact, so that
         # `sha256sum -c <file>.sha256` works cleanly for whichever artifact
         # the user downloaded. The bare-binary checksum is what an
-        # independent rebuilder will compare against once the reproducible
-        # build path lands (ELF hashes reproduce far more reliably than
+        # independent rebuilder will compare against via the reproducible
+        # build path (ELF hashes reproduce far more reliably than
         # tar/gzip hashes).
         run: |
           shasum -a 256 "${{ steps.prepare.outputs.archive_path }}" > "${{ steps.prepare.outputs.archive_name }}.sha256"
           shasum -a 256 "${{ steps.prepare.outputs.binary_name }}"  > "${{ steps.prepare.outputs.binary_name }}.sha256"
 
+      - name: Generate reproducible checksum sidecar
+        if: matrix.platform.reproducible && matrix.binary.name == 'tempo'
+        shell: bash
+        # The .reproducible.sha256 file is the checksum an independent
+        # rebuilder should compare against. For reproducible builds the
+        # binary checksum already matches, so we copy it under the canonical
+        # sidecar name that external verifiers look for.
+        run: |
+          cp "${{ steps.prepare.outputs.binary_name }}.sha256" \
+             "${{ steps.prepare.outputs.binary_name }}.reproducible.sha256"
+
       - name: GPG sign archive
         shell: bash
         env:
@@ -221,6 +268,7 @@ jobs:
             ${{ steps.prepare.outputs.archive_name }}.asc
             ${{ steps.prepare.outputs.archive_name }}.spdx.json
             ${{ steps.prepare.outputs.binary_name }}.sha256
+            ${{ steps.prepare.outputs.binary_name }}.reproducible.sha256
           if-no-files-found: error
 
   create-release:
diff --git a/.github/workflows/reproducible-build.yml b/.github/workflows/reproducible-build.yml
index 0de11423..a9143db3 100644
--- a/.github/workflows/reproducible-build.yml
+++ b/.github/workflows/reproducible-build.yml
@@ -1,6 +1,6 @@
 name: Reproducible Build
 
-# Builds the byte-deterministic `tempo` binary for x86_64-unknown-linux-gnu
+# Builds the byte-deterministic `tempo` binary for x86_64/aarch64-unknown-linux-gnu
 # from the pinned Dockerfile.reproducible recipe.
 
 permissions: {}
@@ -16,6 +16,19 @@ on:
   push:
     branches: [main]
 
+  workflow_call:
+    inputs:
+      ref:
+        description: "Git ref to build reproducibly"
+        type: string
+        required: false
+        default: ""
+      target:
+        description: "Rust target triple (x86_64-unknown-linux-gnu or aarch64-unknown-linux-gnu)"
+        type: string
+        required: false
+        default: "x86_64-unknown-linux-gnu"
+
   workflow_dispatch:
     inputs:
       ref:
@@ -23,11 +36,18 @@ on:
         type: string
         required: false
         default: "main"
+      target:
+        description: "Rust target triple"
+        type: choice
+        options:
+          - "x86_64-unknown-linux-gnu"
+          - "aarch64-unknown-linux-gnu"
+        default: "x86_64-unknown-linux-gnu"
 
 jobs:
   build:
     name: Build & hash
-    runs-on: depot-ubuntu-latest-16
+    runs-on: ${{ contains(inputs.target, 'aarch64') && 'depot-ubuntu-latest-arm-16' || 'depot-ubuntu-latest-16' }}
     permissions:
       contents: read
     steps:
@@ -36,22 +56,25 @@ jobs:
         with:
           # `inputs.ref` is empty on push events (push has no inputs);
           # checkout falls back to github.sha — the pushed commit. On
-          # workflow_dispatch it's the user-chosen ref (defaulted to main).
-          ref: ${{ inputs.ref }}
+          # workflow_dispatch / workflow_call it's the caller-chosen ref.
+          ref: ${{ inputs.ref || '' }}
           fetch-depth: 0
           persist-credentials: false
 
-      - name: Resolve version + artifact name
+      - name: Resolve target + runner config
         id: cfg
+        env:
+          INPUT_TARGET: ${{ inputs.target || 'x86_64-unknown-linux-gnu' }}
         run: |
           set -euo pipefail
           # Name the asset after the short SHA so the run can never be
           # confused with a real release asset (which uses v-prefixed tags).
           SHORT=$(git rev-parse --short=7 HEAD)
           SHA=$(git rev-parse HEAD)
-          echo "version=sha-$SHORT"                       >> "$GITHUB_OUTPUT"
-          echo "artifact=reproducible-hash-${SHA}"        >> "$GITHUB_OUTPUT"
-          echo "bin=tempo-sha-${SHORT}-x86_64-unknown-linux-gnu" >> "$GITHUB_OUTPUT"
+          echo "version=sha-$SHORT"                          >> "$GITHUB_OUTPUT"
+          echo "artifact=reproducible-hash-${SHA}-${INPUT_TARGET}" >> "$GITHUB_OUTPUT"
+          echo "bin=tempo-sha-${SHORT}-${INPUT_TARGET}"      >> "$GITHUB_OUTPUT"
+          echo "target=${INPUT_TARGET}"                      >> "$GITHUB_OUTPUT"
 
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
@@ -59,6 +82,7 @@ jobs:
       - name: Build reproducible binary
         env:
           VERSION: ${{ steps.cfg.outputs.version }}
+          TARGET: ${{ steps.cfg.outputs.target }}
         # Retry the docker build a few times to absorb the dominant transient
         # failure (snapshot.debian.org throttling/503s during apt-get inside
         # the Dockerfile).

TODO (future follow-up)

• GPG-sign the .reproducible.sha256 sidecar
• Wire release.yml to call reproducible-build.yml via workflow_call for independent verification

[github-actions]

This PR has been marked stale due to 7 days of inactivity.