Skip to content

Commit 0a781f2

Browse files
authored
fix: remove continue-on-error masking sync workflow failures (#224) (#231)
Remove continue-on-error: true from sync-leaderboard.yml and replace with explicit failure tracking across 72 iterations. Track success/fail counts, print ::error:: annotations per failure, auto-create GitHub issue after 3 consecutive failures, and print final summary. Exit with code 1 if any iteration failed so the step shows red.
1 parent 40c593a commit 0a781f2

1 file changed

Lines changed: 45 additions & 2 deletions

File tree

.github/workflows/sync-leaderboard.yml

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ concurrency:
1111

1212
permissions:
1313
contents: write
14+
issues: write
1415

1516
jobs:
1617
run-fetch:
@@ -41,9 +42,15 @@ jobs:
4142
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
4243
4344
- name: Run sync loop
45+
env:
46+
GH_TOKEN: ${{ github.token }}
4447
run: |
4548
MAX_ITERATIONS=72
4649
INTERVAL=300
50+
SUCCESS_COUNT=0
51+
FAIL_COUNT=0
52+
CONSECUTIVE_FAILS=0
53+
ISSUE_CREATED=false
4754
4855
for i in $(seq 1 $MAX_ITERATIONS); do
4956
echo "--- Iteration $i/$MAX_ITERATIONS ---"
@@ -56,7 +63,30 @@ jobs:
5663
# 2. Run sync
5764
export DATA_DIR=db-repo
5865
export DATA_REPO_TOKEN=${{ secrets.DATA_REPO_TOKEN }}
59-
node scripts/sync-leaderboard.js
66+
67+
if node scripts/sync-leaderboard.js; then
68+
SUCCESS_COUNT=$((SUCCESS_COUNT + 1))
69+
CONSECUTIVE_FAILS=0
70+
else
71+
FAIL_COUNT=$((FAIL_COUNT + 1))
72+
CONSECUTIVE_FAILS=$((CONSECUTIVE_FAILS + 1))
73+
echo "::error::Iteration $i failed (#$FAIL_COUNT total, $CONSECUTIVE_FAILS consecutive)"
74+
75+
# Create a GitHub issue after 3 consecutive failures (once per run)
76+
if [ $CONSECUTIVE_FAILS -ge 3 ] && [ "$ISSUE_CREATED" = "false" ]; then
77+
ISSUE_CREATED=true
78+
gh issue create \
79+
--title "⚠️ Sync workflow failing consistently" \
80+
--label "type:devops" \
81+
--body "The sync-leaderboard workflow has failed ${CONSECUTIVE_FAILS} consecutive times.
82+
83+
Last attempt: Iteration $i / $MAX_ITERATIONS
84+
Timestamp: $(date -u)
85+
Total failures: $FAIL_COUNT
86+
87+
Workflow run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
88+
fi
89+
fi
6090
6191
# 3. Stage changes (everything except users.json)
6292
cd db-repo
@@ -88,4 +118,17 @@ jobs:
88118
sleep $INTERVAL
89119
fi
90120
done
91-
continue-on-error: true
121+
122+
# Final summary
123+
echo "=========================================="
124+
echo "Sync loop summary"
125+
echo " Iterations: $MAX_ITERATIONS"
126+
echo " Successful: $SUCCESS_COUNT"
127+
echo " Failed: $FAIL_COUNT"
128+
echo "=========================================="
129+
130+
# Fail the step if any iteration failed
131+
if [ $FAIL_COUNT -gt 0 ]; then
132+
echo "::error::$FAIL_COUNT out of $MAX_ITERATIONS iterations failed."
133+
exit 1
134+
fi

0 commit comments

Comments
 (0)