Skip to content

Commit 1e21fd1

Browse files
authored
fix(ci): make dead links workflow robust against silent curl failures (#2534)
The 'Scan Website for Dead Links' CI job was failing because it found 0 URLs in the sitemap. Root cause: curl -s was silently failing (no error output) and the pipeline had no error checking, so the step appeared to succeed with an empty urls.txt. Changes: - Add 'set -euo pipefail' for strict error handling - Download sitemap to a file with --compressed and --retry for reliability (fixes potential gzip encoding issues) - Check HTTP status code and fail explicitly on non-200 - Switch from grep -oP (PCRE) to grep -oE (POSIX ERE) for portability across runner environments - Validate URL count before proceeding to lychee scan - Remove extra single quotes around --accept value in lychee args that could cause parsing issues in YAML block scalars - Log sitemap size and dump content on failure for debuggability
1 parent b545a38 commit 1e21fd1

1 file changed

Lines changed: 31 additions & 5 deletions

File tree

.github/workflows/check-dead-links.yml

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,37 @@ jobs:
1919

2020
- name: Fetch sitemap URLs
2121
run: |
22-
curl -s https://microsoft.github.io/SynapseML/sitemap.xml \
23-
| grep -oP '<loc>[^<]+</loc>' \
24-
| sed 's/<\/?loc>//g' \
22+
set -euo pipefail
23+
24+
SITEMAP_URL="https://microsoft.github.io/SynapseML/sitemap.xml"
25+
26+
# Download sitemap to a file with retries and error reporting
27+
HTTP_CODE=$(curl --silent --show-error --compressed \
28+
--retry 3 --retry-delay 5 \
29+
-o sitemap.xml -w '%{http_code}' \
30+
"$SITEMAP_URL")
31+
32+
if [ "$HTTP_CODE" != "200" ]; then
33+
echo "::error::Failed to fetch sitemap from $SITEMAP_URL (HTTP $HTTP_CODE)"
34+
exit 1
35+
fi
36+
37+
SITEMAP_SIZE=$(wc -c < sitemap.xml)
38+
echo "Downloaded sitemap: $SITEMAP_SIZE bytes"
39+
40+
# Extract URLs using POSIX extended regex (portable, no PCRE dependency)
41+
grep -oE '<loc>[^<]+</loc>' sitemap.xml \
42+
| sed 's/<\/*loc>//g' \
2543
> urls.txt
26-
echo "Found $(wc -l < urls.txt) URLs in sitemap"
44+
45+
URL_COUNT=$(wc -l < urls.txt)
46+
echo "Found $URL_COUNT URLs in sitemap"
47+
48+
if [ "$URL_COUNT" -eq 0 ]; then
49+
echo "::error::Sitemap at $SITEMAP_URL contained no URLs (file was $SITEMAP_SIZE bytes)"
50+
head -c 500 sitemap.xml
51+
exit 1
52+
fi
2753
2854
- name: Scan for dead links
2955
uses: lycheeverse/lychee-action@v2
@@ -34,5 +60,5 @@ jobs:
3460
--max-retries 5
3561
--retry-wait-time 5
3662
--timeout 30
37-
--accept '100..=103,200..=299,503'
63+
--accept 100..=103,200..=299,503
3864
urls.txt

0 commit comments

Comments
 (0)