Skip to content

Commit e82b478

Browse files
committed
fix(release): fail-fast dashboard release + harden cargo network
The dashboard release script polled in THREE sequential 60-minute loops (draft-exists → asset-count → workflow-conclusion) and only detected a failed build leg in the LAST one. So when one matrix leg died ~23s in (a transient crates.io reset fetching wasm-bindgen), the script still waited ~10min behind the slow legs before noticing. Collapse to ONE loop that polls the run and aborts the instant ANY leg reports failure/cancelled/timed_out — surfacing the break in seconds instead of minutes, and never publishing a run that didn't fully succeed. Replaces the attempt-counter loops with a wall-clock deadline. Root-cause hardening for the transient that triggered this: set CARGO_NET_RETRY=10, CARGO_NET_GIT_FETCH_WITH_CLI=true, and CARGO_HTTP_MULTIPLEXING=false on the build job so a spurious connection reset against the cargo download CDN retries instead of killing the leg. Kept fail-fast:false on the matrix (a release wants to see ALL broken platforms in one run) — the script-level fail-fast is what makes a single failure surface quickly now.
1 parent eab6f04 commit e82b478

2 files changed

Lines changed: 66 additions & 88 deletions

File tree

.github/workflows/dashboard-release.yml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,20 @@ env:
1515
jobs:
1616
build:
1717
name: Build Dashboard (${{ matrix.settings.label }})
18+
# Harden cargo against the transient crates.io drops we hit in the wild
19+
# ("download of … failed / curl … Connection reset by peer" mid-build, which
20+
# killed a single matrix leg ~23s in despite the code being fine). Retry hard
21+
# and disable HTTP/2 multiplexing — the documented workaround for spurious
22+
# connection resets against the cargo download CDN on hosted runners.
23+
env:
24+
CARGO_NET_RETRY: "10"
25+
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
26+
CARGO_HTTP_MULTIPLEXING: "false"
1827
strategy:
28+
# Keep building every platform leg even if one fails: a release wants to
29+
# surface ALL broken platforms in one run, not cancel the survivors. The
30+
# release SCRIPT fail-fasts instead (aborts the moment the run concludes
31+
# failure), so a single failed leg no longer hides behind the slow ones.
1932
fail-fast: false
2033
matrix:
2134
settings:

scripts/release-dashboard.sh

Lines changed: 53 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -192,112 +192,77 @@ echo " → CI is now building all platforms"
192192
echo " → Watch: https://github.com/cortexkit/magic-context/actions"
193193
echo ""
194194

195-
# Step 6: Wait for CI
196-
echo "→ Waiting for CI to create the draft release..."
197-
echo " (checking every 30s for up to 60 minutes)"
198-
ATTEMPTS=0
199-
MAX_ATTEMPTS=120
200-
while [[ $ATTEMPTS -lt $MAX_ATTEMPTS ]]; do
201-
RELEASE_STATE=$(gh release view "$TAG" --repo cortexkit/magic-context --json isDraft --jq '.isDraft' 2>/dev/null || echo "not_found")
202-
203-
if [[ "$RELEASE_STATE" == "true" ]]; then
204-
echo " ✓ Draft release found"
205-
break
206-
elif [[ "$RELEASE_STATE" == "false" ]]; then
207-
echo " ✓ Release already published"
208-
break
209-
fi
210-
211-
ATTEMPTS=$((ATTEMPTS + 1))
212-
if [[ $((ATTEMPTS % 4)) -eq 0 ]]; then
213-
echo " ... still waiting ($((ATTEMPTS * 30 / 60))m elapsed)"
214-
fi
215-
sleep 30
216-
done
217-
218-
if [[ $ATTEMPTS -ge $MAX_ATTEMPTS ]]; then
219-
echo " ⚠ Timed out waiting for release. Check CI manually."
220-
echo " → https://github.com/cortexkit/magic-context/actions"
221-
exit 0
222-
fi
223-
224-
# Step 7: Wait for all platform assets
225-
MIN_ASSETS=10
226-
echo ""
227-
echo "→ Waiting for all platform assets (expecting ≥$MIN_ASSETS)..."
228-
echo " (checking every 30s for up to 60 minutes)"
229-
ASSET_ATTEMPTS=0
230-
ASSET_MAX=120
231-
ASSET_COUNT=0
232-
while [[ $ASSET_ATTEMPTS -lt $ASSET_MAX ]]; do
233-
ASSET_COUNT=$(gh release view "$TAG" --repo cortexkit/magic-context --json assets --jq '.assets | length' 2>/dev/null || echo "0")
234-
235-
if [[ "$ASSET_COUNT" -ge "$MIN_ASSETS" ]]; then
236-
echo " ✓ Found $ASSET_COUNT assets — minimum asset threshold reached"
237-
break
238-
fi
239-
240-
ASSET_ATTEMPTS=$((ASSET_ATTEMPTS + 1))
241-
if [[ $((ASSET_ATTEMPTS % 4)) -eq 0 ]]; then
242-
ELAPSED=$((ASSET_ATTEMPTS * 30 / 60))
243-
echo " ... $ASSET_COUNT assets so far (${ELAPSED}m elapsed)"
244-
fi
245-
sleep 30
246-
done
247-
248-
if [[ "$ASSET_COUNT" -lt "$MIN_ASSETS" ]]; then
249-
echo " ⚠ Only $ASSET_COUNT assets after waiting. Some platforms may have failed."
250-
if have_tty; then
251-
read -r -p " Publish with $ASSET_COUNT assets? [y/N] " confirm </dev/tty
252-
else
253-
confirm=""
254-
fi
255-
if [[ "$confirm" != "y" && "$confirm" != "Y" ]]; then
256-
echo " Skipping publish. Run manually: gh release edit $TAG --repo cortexkit/magic-context --draft=false"
257-
exit 0
258-
fi
259-
fi
260-
261-
# Step 8: Wait for the release workflow to finish before publishing
262-
echo ""
263-
echo "→ Waiting for Dashboard Release workflow to complete..."
264-
echo " (checking every 30s for up to 60 minutes)"
265-
WORKFLOW_ATTEMPTS=0
266-
WORKFLOW_MAX=120
195+
# Step 6: Wait for the Dashboard Release workflow — FAIL FAST.
196+
#
197+
# The build is a 6-platform matrix (fail-fast:false, so every leg runs to its
198+
# own end). The release is only valid if the WHOLE run succeeds. The old code
199+
# had three sequential 60-min poll loops (draft-exists → asset-count → workflow-
200+
# conclusion) and only detected a failed leg in the LAST one — so a leg that
201+
# died 23s in still cost ~10min of waiting behind the slow legs before the script
202+
# noticed. Instead: poll the run's conclusion directly and ABORT THE INSTANT it
203+
# concludes failure, regardless of how many legs are still nominally "running"
204+
# (a concluded run means GitHub already stopped scheduling). One loop, one
205+
# concern: did the run succeed?
206+
echo "→ Waiting for the Dashboard Release workflow (fail-fast on any leg)..."
207+
echo " (checking every 15s for up to 60 minutes)"
267208
RUN_ID=""
268-
while [[ $WORKFLOW_ATTEMPTS -lt $WORKFLOW_MAX ]]; do
269-
RUN_INFO=$(gh run list --repo cortexkit/magic-context --workflow "Dashboard Release" --limit 20 --json databaseId,status,conclusion,headBranch --jq ".[] | select(.headBranch == \"$TAG\") | \"\(.databaseId) \(.status) \(.conclusion)\"" 2>/dev/null | head -n 1 || true)
209+
DEADLINE=$(( $(date +%s) + 3600 ))
210+
while [[ $(date +%s) -lt $DEADLINE ]]; do
211+
RUN_INFO=$(gh run list --repo cortexkit/magic-context --workflow "Dashboard Release" --limit 20 \
212+
--json databaseId,status,conclusion,headBranch \
213+
--jq ".[] | select(.headBranch == \"$TAG\") | \"\(.databaseId) \(.status) \(.conclusion)\"" 2>/dev/null | head -n 1 || true)
270214

271215
if [[ -n "$RUN_INFO" ]]; then
272216
read -r RUN_ID RUN_STATUS RUN_CONCLUSION <<<"$RUN_INFO"
217+
218+
# Surface a failed leg the moment the run reports it, even while sibling legs
219+
# are still finishing — `gh run view --json jobs` shows per-leg conclusions
220+
# before the overall run flips to completed.
221+
if [[ -n "$RUN_ID" ]]; then
222+
FAILED_LEG=$(gh run view "$RUN_ID" --repo cortexkit/magic-context --json jobs \
223+
--jq '.jobs[] | select(.conclusion == "failure" or .conclusion == "cancelled" or .conclusion == "timed_out") | .name' 2>/dev/null | head -n 1 || true)
224+
if [[ -n "$FAILED_LEG" ]]; then
225+
echo ""
226+
echo " ✗ Build leg failed: $FAILED_LEG"
227+
echo " → https://github.com/cortexkit/magic-context/actions/runs/$RUN_ID"
228+
echo " Release NOT published. Fix the leg (or re-run it if transient), then re-run this script."
229+
exit 1
230+
fi
231+
fi
232+
273233
if [[ "$RUN_STATUS" == "completed" ]]; then
274234
if [[ "$RUN_CONCLUSION" == "success" ]]; then
275235
echo " ✓ Workflow completed successfully"
276236
break
277237
fi
278-
echo " ✗ Workflow completed with conclusion: $RUN_CONCLUSION"
238+
echo ""
239+
echo " ✗ Workflow concluded: $RUN_CONCLUSION"
279240
echo " → https://github.com/cortexkit/magic-context/actions/runs/$RUN_ID"
280241
exit 1
281242
fi
282243
fi
283244

284-
WORKFLOW_ATTEMPTS=$((WORKFLOW_ATTEMPTS + 1))
285-
if [[ $((WORKFLOW_ATTEMPTS % 4)) -eq 0 ]]; then
286-
ELAPSED=$((WORKFLOW_ATTEMPTS * 30 / 60))
287-
if [[ -n "$RUN_ID" ]]; then
288-
echo " ... workflow $RUN_STATUS (${ELAPSED}m elapsed)"
289-
else
290-
echo " ... waiting for workflow run (${ELAPSED}m elapsed)"
291-
fi
292-
fi
293-
sleep 30
245+
printf "\r ... %s (run %s) " "${RUN_STATUS:-waiting for run}" "${RUN_ID:-?}"
246+
sleep 15
294247
done
295248

296-
if [[ $WORKFLOW_ATTEMPTS -ge $WORKFLOW_MAX ]]; then
297-
echo " ⚠ Timed out waiting for workflow. Skipping publish."
249+
if [[ $(date +%s) -ge $DEADLINE ]]; then
250+
echo ""
251+
echo " ⚠ Timed out waiting for workflow. Check CI manually."
298252
echo " → https://github.com/cortexkit/magic-context/actions"
299-
exit 0
253+
exit 1
254+
fi
255+
256+
# Sanity: the run succeeded, so every leg uploaded its asset. Confirm the count
257+
# before publishing (a defensive check, not a wait — the run is already green).
258+
ASSET_COUNT=$(gh release view "$TAG" --repo cortexkit/magic-context --json assets --jq '.assets | length' 2>/dev/null || echo "0")
259+
MIN_ASSETS=10
260+
if [[ "$ASSET_COUNT" -lt "$MIN_ASSETS" ]]; then
261+
echo " ⚠ Workflow succeeded but only $ASSET_COUNT/$MIN_ASSETS assets are attached."
262+
echo " → https://github.com/cortexkit/magic-context/releases/tag/$TAG"
263+
exit 1
300264
fi
265+
echo "$ASSET_COUNT assets attached"
301266

302267
# Step 9: Prompt for release notes
303268
echo ""

0 commit comments

Comments
 (0)