Skip to content

fix(ci3): make redis_setexz a no-op instead of crashing bootstrap_ec2 when redis is unavailable#24823

Draft
AztecBot wants to merge 1 commit into
nextfrom
cb/redis-setexz-nightly-reland-0721
Draft

fix(ci3): make redis_setexz a no-op instead of crashing bootstrap_ec2 when redis is unavailable#24823
AztecBot wants to merge 1 commit into
nextfrom
cb/redis-setexz-nightly-reland-0721

Conversation

@AztecBot

Copy link
Copy Markdown
Collaborator

Replaces #24535 (auto-closed unmerged 2026-07-20 by the stale-draft reaper). Same one-line fix, re-derived and re-verified today.

⚠️ 2026-07-21 — this bug has now recurred 7×; the fix is trivial, correct, and was CI-green, but has NEVER landed. Nightly Debug Build run 29805523159 (run #118) failed again today with the identical gzip: stdout: Broken pipe at the first line of bootstrap_ec2 (~0s). The only reason this has never merged is process, not code: ClaudeBox can only open draft PRs and cannot mark them ready-for-review or merge a draft (merge_pr405 Pull Request is still a draft); the stale-draft reaper then auto-closes them after a few days. Please click "Ready for review" and merge this (or grant it non-draft/merge handling so the loop stops).


Problem

The nightly barretenberg "Debug Build" (barretenberg-nightly-debug-build.yml./ci3.sh barretenberg-debug, run on the aztec-claude mirror of this ci3/ source) fails immediately with:

--- Run barretenberg-debug CI ---
gzip: stdout: Broken pipe
##[error]Process completed with exit code 1.

Latest failing run: https://github.com/AztecProtocol/aztec-claude/actions/runs/29805523159 — the step dies in ~0s, before CI Log: is even printed and long before any EC2 builder is requested.

Root cause

ci.sh barretenberg-debug runs bootstrap_ec2, whose very first action logs a boot marker:

echo "CI booting..." | redis_setexz "$CI_LOG_ID" 300

redis_setexz was the only redis helper that did not guard on CI_REDIS_AVAILABLE:

function redis_setexz {
  gzip | redis_cli -x SETEX $1 $2 &>/dev/null
}

When redis is unreachable (CI_REDIS_AVAILABLE=0), redis_cli is deliberately a no-op that returns immediately without reading stdin. So gzip's downstream reader vanishes, gzip dies with EPIPE / "Broken pipe", and because the CI scripts run under set -euo pipefail, that failed pipeline element aborts the whole script on its first line. source_redis explicitly treats redis as optional ("Log and test cache will be disabled"), so the script is meant to keep running when redis is down — redis_setexz violated that contract while its siblings redis_cli/redis_publish already guard on the flag.

Redis is unavailable on this path because the run has no BUILD_INSTANCE_SSH_KEY (empty on the mirror), so source_redis can't open the bastion tunnel to the remote cache, nc localhost:6379 fails, and CI=1 disables the local-docker-redis fallback.

Fix

Guard redis_setexz on CI_REDIS_AVAILABLE, matching the other helpers; when redis is unavailable, drain stdin (cat >/dev/null) so the upstream producer never gets SIGPIPE. All callers (bootstrap_ec2, cache_log, denoise, run_test_cmd) feed the value via stdin, so draining is correct in every case.

function redis_setexz {
  if [ "${CI_REDIS_AVAILABLE:-0}" -eq 1 ]; then
    gzip | redis_cli -x SETEX $1 $2 &>/dev/null
  else
    cat >/dev/null
  fi
}

Verification (red/green)

Reproduced the pipeline under set -euo pipefail with CI_REDIS_AVAILABLE=0 (re-confirmed 2026-07-21):

  • before: echo ... | redis_setexz aborts on a broken pipe; the script never reaches the next line.
  • after: stdin is drained, the pipeline returns 0, execution continues — verified for both a small message and a large stream (no SIGPIPE to the producer).

bash -n ci3/source_redis passes.

History — please merge, don't let it auto-close

This identical fix has been proposed and auto-closed unmerged by the stale-draft reaper, never on technical grounds:

Because none landed, the bug is still in next and the nightly keeps failing.

Follow-ups (not in this PR)

  1. cache_persistent in ci3/source_cache has the identical cat "$tmpfile" | redis_cli -x SETEX ... pattern that would fail the same way under pipefail when redis is unavailable. Not on the nightly's pre-EC2 path, so it doesn't block this fix, but it should get the same CI_REDIS_AVAILABLE guard.
  2. For the nightly to fully complete on the aztec-claude mirror it additionally needs (a) its frozen next snapshot resynced to pick up this fix and (b) AWS credentials / BUILD_INSTANCE_SSH_KEY configured there to spin up the EC2 builder — repo-secrets/infra matters separate from this code fix.

Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant