Skip to content

Commit 0cc7cf1

Browse files
Jimbo4350Jimbo4350
committed
Fix broken cross-package Haddock links on hosted docs site
Fixes #601. Cross-package hrefs emitted by cabal haddock-project are relative paths (e.g. href="../cardano-ledger-api-1.2.3-hash/Foo.html") that don't resolve on the published site — we only host cardano-api's own output, not its dependencies, so every cross-package reference 404s by default. Add scripts/fix-haddock-links.sh and wire it into the github-page workflow between haddock-project and the artifact upload. The script replaces each cross-package href with either an absolute URL on the upstream doc site or a tooltip-annotated unclickable <span>, so the published site has zero clickable 404s. Pipeline Phase 1: Scan filesystem, symlink versioned directories, fetch the CHaP index, grep HTML for cross-package link targets. Phase 2: Resolve each target and rewrite in place, or convert to an unclickable span if unresolvable. Phase 3: HEAD-validate rewritten URLs and annotate 404s as spans. Doc-site resolution for CHaP packages: two lookups, first hit wins: 1. Name-suffix heuristic under *.cardano.intersectmbo.org — strip trailing '-token' segments of the package name and HEAD-probe each candidate's doc-index.html. Covers cardano-ledger-*, plutus-*, ouroboros-*, etc. 2. Fixed fallback against a small IOG_DOC_BASES list — covers packages whose subdomain isn't a suffix of the package name (e.g. cardano-base at base.cardano.intersectmbo.org). Non-CHaP packages (bootlibs like base, bytestring, time) are NOT linked. Haddock's per-module URL structure doesn't line up cleanly with Hackage's (src/ source views, -inplace version suffixes from local rebuilds) so Hackage rewrites mostly produce 404s, and readers of cardano-api docs rarely click into bootlib internals. Rendered as unclickable spans, no outbound link, no validation noise.
1 parent 12b4591 commit 0cc7cf1

2 files changed

Lines changed: 389 additions & 0 deletions

File tree

‎.github/workflows/github-page.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ jobs:
3939
mkdir website
4040
cabal haddock-project --output=./website --internal --foreign-libraries
4141
42+
- name: Fix cross-package Haddock links
43+
run: |
44+
./scripts/fix-haddock-links.sh ./website
45+
4246
- name: Build typedoc documentation
4347
run: |
4448
nix build .#wasm-typedoc

‎scripts/fix-haddock-links.sh‎

Lines changed: 385 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,385 @@
1+
#!/usr/bin/env bash
2+
# fix-haddock-links.sh
3+
#
4+
# Post-processes Haddock HTML generated by `cabal haddock-project` to fix
5+
# cross-package links. Without this, links to external dependencies are
6+
# broken 404s because the hosted site only contains docs for packages in
7+
# this repo.
8+
#
9+
# Pipeline:
10+
# 1. Prepare: scan filesystem, create symlinks for versioned directories,
11+
# fetch the CHaP package index, grep HTML for cross-package link
12+
# targets.
13+
# 2. Resolve + rewrite: for each discovered target, probe candidate
14+
# doc-site URLs and rewrite HTML links in place (or mark unmapped
15+
# CHaP packages as unclickable).
16+
# 3. Validate + annotate: HEAD each rewritten URL; replace dead ones
17+
# with annotated plain-text spans so there are zero clickable 404s.
18+
#
19+
# When dead links are found (unmapped CHaP package or module-level 404),
20+
# the script prints an actionable summary — upstream module name, symbols
21+
# we're linking to, and the referring pages in our own docs — and exits 0
22+
# by default so the docs site can still deploy. Set
23+
# FIX_HADDOCK_LINKS_STRICT=1 to make the script fail instead.
24+
# In GitHub Actions, each dead link also produces a `::warning::`
25+
# annotation visible in the job summary.
26+
#
27+
# Usage: ./scripts/fix-haddock-links.sh <website-directory>
28+
29+
set -euo pipefail
30+
31+
WEBSITE_DIR="${1:?Usage: $0 <website-directory>}"
32+
[ -d "$WEBSITE_DIR" ] || { echo "Error: $WEBSITE_DIR is not a directory" >&2; exit 1; }
33+
34+
CHAP_PKGS_FILE="" URLS_FILE="" DEAD_URLS_FILE="" REWRITE_SCRIPT="" DEAD_SCRIPT=""
35+
trap 'rm -f "$CHAP_PKGS_FILE" "$URLS_FILE" "$DEAD_URLS_FILE" "$REWRITE_SCRIPT" "$DEAD_SCRIPT"' EXIT
36+
37+
# ============================================================================
38+
# Configuration and helpers
39+
# ============================================================================
40+
#
41+
# For each CHaP package, probe candidate doc-site URLs in order until one
42+
# returns 200 for doc-index.html (a standard Haddock asset every package
43+
# site serves; wrong-site probes reliably 404). Candidate order is:
44+
# 1. Name-suffix subdomain under *.cardano.intersectmbo.org, dropping
45+
# trailing "-<token>" segments of the package name — fast path for
46+
# families where the subdomain is a name-suffix (cardano-ledger-*,
47+
# ouroboros-*, plutus-*).
48+
# 2. IOG_DOC_BASES — known doc-site roots for irregular-subdomain
49+
# families (cardano-base at base.cardano…, network-mux at
50+
# ouroboros-network.cardano…, io-classes at input-output-hk.github.io/
51+
# io-sim, etc.).
52+
#
53+
# Adding a new IOG doc site = one line in IOG_DOC_BASES. Packages on
54+
# existing sites need no edit.
55+
56+
IOG_DOC_BASES=(
57+
"https://cardano-ledger.cardano.intersectmbo.org"
58+
"https://base.cardano.intersectmbo.org"
59+
"https://plutus.cardano.intersectmbo.org/haddock/latest"
60+
"https://ouroboros-consensus.cardano.intersectmbo.org/haddocks"
61+
"https://ouroboros-network.cardano.intersectmbo.org"
62+
"https://input-output-hk.github.io/io-sim"
63+
)
64+
65+
derive_name_candidates() {
66+
local name="$1"
67+
while true; do
68+
printf '%s\n' "https://${name}.cardano.intersectmbo.org"
69+
[[ "$name" == *-* ]] || break
70+
name="${name%-*}"
71+
done
72+
}
73+
74+
probe_site() {
75+
local code
76+
code=$(curl -sI -o /dev/null -w "%{http_code}" --connect-timeout 5 --max-time 10 \
77+
"${1}/${2}/doc-index.html" 2>/dev/null || echo "000")
78+
[[ "$code" == "200" || "$code" == "301" || "$code" == "302" ]]
79+
}
80+
81+
resolve_url() {
82+
local pkg="$1" is_chap="$2" base
83+
if [[ "$is_chap" != "yes" ]]; then echo "HACKAGE"; return; fi
84+
while IFS= read -r base; do
85+
probe_site "$base" "$pkg" && { echo "$base"; return; }
86+
done < <(derive_name_candidates "$pkg")
87+
for base in "${IOG_DOC_BASES[@]}"; do
88+
probe_site "$base" "$pkg" && { echo "$base"; return; }
89+
done
90+
echo "NONE"
91+
}
92+
93+
# Append a sed substitution that turns <a href="PATTERN...">X</a> into a
94+
# non-clickable <span class="dead-link" title="TITLE">X</span>, to the
95+
# named script file. We use file-based sed (-f) rather than -e on the
96+
# command line because full-repo builds can produce thousands of
97+
# substitutions whose combined argv hits xargs's limit.
98+
add_unclickable_cmd() {
99+
local file="$1" href_pattern="$2" title="$3"
100+
printf 's|<a href="%s[^"]*"[^>]*>\\([^<]*\\)</a>|<span class="dead-link" title="%s">\\1</span>|g\n' \
101+
"$href_pattern" "$title" >> "$file"
102+
}
103+
104+
# ============================================================================
105+
# Phase 1: Prepare
106+
# ============================================================================
107+
108+
echo "Phase 1: Preparing..."
109+
110+
# Pre-symlink directory set (used to match versioned-hash targets below)
111+
declare -A SHORT_DIRS
112+
for dir in "$WEBSITE_DIR"/*/; do
113+
SHORT_DIRS["$(basename "$dir")"]=1
114+
done
115+
116+
# Create short-name symlinks for local cabal packages
117+
# (cardano-api-10.26.0.0-inplace -> cardano-api)
118+
symlink_count=0
119+
for dir in "$WEBSITE_DIR"/*/; do
120+
name="$(basename "$dir")"
121+
[[ "$name" =~ -inplace-.+ ]] && continue
122+
if [[ "$name" =~ ^(.+)-[0-9]+\.[0-9]+.*-inplace$ ]]; then
123+
short="${BASH_REMATCH[1]}"
124+
if [[ ! -e "$WEBSITE_DIR/$short" ]]; then
125+
ln -s "$name" "$WEBSITE_DIR/$short"
126+
symlink_count=$((symlink_count + 1))
127+
fi
128+
fi
129+
done
130+
131+
# Post-symlink local-package set (used to skip re-rewriting our own docs)
132+
declare -A LOCAL_SET
133+
for dir in "$WEBSITE_DIR"/*/; do
134+
[ -d "$dir" ] && LOCAL_SET["$(basename "$dir")"]=1
135+
done
136+
137+
# Fetch CHaP index
138+
CHAP_PKGS_FILE=$(mktemp)
139+
curl -sL https://chap.intersectmbo.org/01-index.tar.gz \
140+
| tar -tz | grep -oP '^[^/]+' | sort -u > "$CHAP_PKGS_FILE"
141+
declare -A CHAP_SET
142+
while IFS= read -r pkg; do CHAP_SET["$pkg"]=1; done < "$CHAP_PKGS_FILE"
143+
144+
# Single HTML scan for all cross-package link targets
145+
DISCOVERED_PKGS=$(grep -rohP 'href="\.\./(\.\./)?\K[a-zA-Z][a-zA-Z0-9_.-]*(?=/)' \
146+
"$WEBSITE_DIR" 2>/dev/null | sort -u)
147+
discovered_total=$(echo "$DISCOVERED_PKGS" | grep -c . || true)
148+
149+
# Of those targets, symlink versioned-hash names whose stripped base
150+
# matches an existing short directory.
151+
# (href="../base-4.20.2.0-f074/..." + ./base/ exists -> base-4.20.2.0-f074 symlink)
152+
while IFS= read -r target; do
153+
[ -z "$target" ] && continue
154+
[[ -e "$WEBSITE_DIR/$target" ]] && continue
155+
if [[ "$target" =~ ^(.+)-[0-9]+\.[0-9] ]]; then
156+
short="${BASH_REMATCH[1]}"
157+
if [[ -v "SHORT_DIRS[$short]" ]]; then
158+
ln -s "$short" "$WEBSITE_DIR/$target"
159+
symlink_count=$((symlink_count + 1))
160+
fi
161+
fi
162+
done <<< "$DISCOVERED_PKGS"
163+
164+
echo " Local directories: ${#LOCAL_SET[@]}"
165+
echo " CHaP packages: $(wc -l < "$CHAP_PKGS_FILE")"
166+
echo " Cross-package link targets: $discovered_total"
167+
echo " Symlinks created: $symlink_count"
168+
169+
# ============================================================================
170+
# Phase 2: Resolve doc sites and rewrite links
171+
# ============================================================================
172+
173+
echo "Phase 2: Resolving doc sites and rewriting links..."
174+
175+
REWRITE_SCRIPT=$(mktemp)
176+
UNMAPPED_CHAP=() # CHaP packages we couldn't resolve to a doc site — gap in config
177+
NON_CHAP=() # non-CHaP packages (bootlibs etc.) we don't bother linking to
178+
skipped=0 rewrote_docsite=0
179+
180+
while IFS= read -r pkg; do
181+
[ -z "$pkg" ] && continue
182+
if [[ -v "LOCAL_SET[$pkg]" ]]; then skipped=$((skipped + 1)); continue; fi
183+
184+
# Haddock emits hrefs to locally-built deps as "<name>-<version>-inplace"
185+
# even when their docs weren't generated into the output tree (so they're
186+
# not in LOCAL_SET). Strip the suffix so classification sees the real
187+
# package name, for cleaner tooltips. Keep the original `pkg` for the sed
188+
# href pattern, which is what appears in the HTML.
189+
# NB: ordering matters — evaluate the negative match first so it doesn't
190+
# clobber BASH_REMATCH[1] set by the capturing regex.
191+
lookup="$pkg"
192+
if [[ ! "$pkg" =~ -inplace-.+ ]] && [[ "$pkg" =~ ^(.+)-[0-9]+\.[0-9]+.*-inplace$ ]]; then
193+
lookup="${BASH_REMATCH[1]}"
194+
fi
195+
if [[ "$lookup" != "$pkg" && -v "LOCAL_SET[$lookup]" ]]; then
196+
skipped=$((skipped + 1)); continue
197+
fi
198+
199+
is_chap="no"; [[ -v "CHAP_SET[$lookup]" ]] && is_chap="yes"
200+
url=$(resolve_url "$lookup" "$is_chap")
201+
202+
if [[ "$url" == "NONE" ]]; then
203+
UNMAPPED_CHAP+=("$lookup")
204+
add_unclickable_cmd "$REWRITE_SCRIPT" "\\.\\./${pkg}/" "No hosted documentation available for ${lookup}"
205+
add_unclickable_cmd "$REWRITE_SCRIPT" "\\.\\./\\.\\./${pkg}/" "No hosted documentation available for ${lookup}"
206+
continue
207+
fi
208+
209+
if [[ "$url" == "HACKAGE" ]]; then
210+
# Non-CHaP packages (bootlibs etc.) — don't link to Hackage. The links
211+
# to base/bytestring/time/... internals are rarely useful to readers of
212+
# cardano-api docs, and Haddock's per-module URL structure doesn't line
213+
# up cleanly with Hackage's rendering, so we'd mostly generate 404s.
214+
# Leave the symbol visible but unclickable.
215+
NON_CHAP+=("$lookup")
216+
add_unclickable_cmd "$REWRITE_SCRIPT" "\\.\\./${pkg}/" "${lookup} is not a CHaP package; docs not linked"
217+
add_unclickable_cmd "$REWRITE_SCRIPT" "\\.\\./\\.\\./${pkg}/" "${lookup} is not a CHaP package; docs not linked"
218+
continue
219+
fi
220+
221+
target="${url}/${lookup}/"
222+
rewrote_docsite=$((rewrote_docsite + 1))
223+
printf 's|href="\\.\\./%s/|href="%s|g\n' "$pkg" "$target" >> "$REWRITE_SCRIPT"
224+
printf 's|href="\\.\\./\\.\\./%s/|href="%s|g\n' "$pkg" "$target" >> "$REWRITE_SCRIPT"
225+
done <<< "$DISCOVERED_PKGS"
226+
227+
echo " Local (skipped): $skipped"
228+
echo " Rewritten (doc site): $rewrote_docsite"
229+
echo " Non-CHaP (unclickable): ${#NON_CHAP[@]}"
230+
echo " Unmapped CHaP: ${#UNMAPPED_CHAP[@]}"
231+
if [[ ${#UNMAPPED_CHAP[@]} -gt 0 ]]; then
232+
printf ' - %s\n' "${UNMAPPED_CHAP[@]}"
233+
fi
234+
235+
if [[ -s "$REWRITE_SCRIPT" ]]; then
236+
find "$WEBSITE_DIR" -name '*.html' -print0 | xargs -0 -P "$(nproc)" sed -i -f "$REWRITE_SCRIPT"
237+
fi
238+
239+
# ============================================================================
240+
# Phase 3: Validate external URLs and annotate dead ones
241+
# ============================================================================
242+
243+
echo "Phase 3: Validating external URLs..."
244+
245+
URLS_FILE=$(mktemp)
246+
DEAD_URLS_FILE=$(mktemp)
247+
grep -rohP 'href="\Khttps://[^"#]+\.html' "$WEBSITE_DIR" 2>/dev/null | sort -u > "$URLS_FILE"
248+
url_count=$(wc -l < "$URLS_FILE")
249+
250+
if [[ $url_count -gt 0 ]]; then
251+
# shellcheck disable=SC2016 # single quotes intentional: expansions evaluated by inner sh -c
252+
xargs -P 16 -I{} sh -c \
253+
'code=$(curl -sI -o /dev/null -w "%{http_code}" --connect-timeout 5 --max-time 10 "{}"); if [ "$code" != "200" ] && [ "$code" != "301" ] && [ "$code" != "302" ]; then echo "{}"; fi' \
254+
< "$URLS_FILE" > "$DEAD_URLS_FILE" 2>/dev/null
255+
fi
256+
dead_count=$(wc -l < "$DEAD_URLS_FILE" | tr -d ' ')
257+
258+
echo " URLs checked: $url_count"
259+
echo " Valid: $((url_count - dead_count))"
260+
echo " Dead: $dead_count"
261+
262+
# Per-URL context tables, populated BEFORE the <a>→<span> replacement so
263+
# the original anchor text and reference sites are still in the HTML.
264+
declare -A DEAD_PKG DEAD_MODULE DEAD_SYMBOLS DEAD_REFS
265+
266+
DEAD_SCRIPT=$(mktemp)
267+
if [[ $dead_count -gt 0 ]]; then
268+
while IFS= read -r dead_url; do
269+
[ -z "$dead_url" ] && continue
270+
271+
# Package + Haskell module name (Haddock encodes dots as dashes)
272+
if [[ "$dead_url" == *"hackage.haskell.org"* ]]; then
273+
pkg=$(echo "$dead_url" | grep -oP 'package/\K[^/]+')
274+
else
275+
pkg=$(echo "$dead_url" | grep -oP '[^/]+(?=/[^/]+\.html$)')
276+
fi
277+
module_file=$(basename "$dead_url" .html)
278+
DEAD_PKG[$dead_url]="$pkg"
279+
DEAD_MODULE[$dead_url]="${module_file//-/.}"
280+
281+
# Symbols the dead URL is trying to link to (text of each <a> tag)
282+
regex_url=$(printf '%s' "$dead_url" | sed 's|[][\\.^$*+?{}|()]|\\&|g')
283+
DEAD_SYMBOLS[$dead_url]=$(grep -rohP \
284+
"<a[^>]*href=\"${regex_url}[^\"]*\"[^>]*>\\K[^<]+" "$WEBSITE_DIR" 2>/dev/null \
285+
| sort -u | head -8 | paste -sd ', ' || echo "")
286+
287+
# Our own docs pages that reference it (the re-export sites we ship)
288+
DEAD_REFS[$dead_url]=$(grep -rlF "$dead_url" "$WEBSITE_DIR" 2>/dev/null \
289+
| sed "s|^${WEBSITE_DIR}/||" | sort -u | head -6 || echo "")
290+
291+
echo " - $dead_url ($pkg)"
292+
escaped=$(printf '%s' "$dead_url" | sed 's|[&/\]|\\&|g; s|\.|\\.|g')
293+
add_unclickable_cmd "$DEAD_SCRIPT" "$escaped" \
294+
"From ${pkg} — documentation not available at the expected URL"
295+
done < "$DEAD_URLS_FILE"
296+
if [[ -s "$DEAD_SCRIPT" ]]; then
297+
find "$WEBSITE_DIR" -name '*.html' -print0 | xargs -0 -P "$(nproc)" sed -i -f "$DEAD_SCRIPT"
298+
fi
299+
fi
300+
301+
# Inject dead-link CSS whenever any <span class="dead-link"> was produced
302+
if [[ ${#UNMAPPED_CHAP[@]} -gt 0 || ${#NON_CHAP[@]} -gt 0 || $dead_count -gt 0 ]]; then
303+
find "$WEBSITE_DIR" -name '*.html' -print0 | xargs -0 -P "$(nproc)" sed -i \
304+
's|</head>|<style>.dead-link { border-bottom: 1px dotted \#888; color: \#888; cursor: help; }</style></head>|'
305+
fi
306+
307+
# ============================================================================
308+
# Summary + strict-mode exit
309+
# ============================================================================
310+
311+
echo ""
312+
echo "=== fix-haddock-links summary ==="
313+
echo " Discovered: $discovered_total"
314+
echo " Local (skipped): $skipped"
315+
echo " Rewritten (doc site): $rewrote_docsite"
316+
echo " Non-CHaP (unclickable): ${#NON_CHAP[@]}"
317+
echo " Unmapped CHaP: ${#UNMAPPED_CHAP[@]}"
318+
echo " URLs validated: $url_count"
319+
echo " Dead links annotated: $dead_count"
320+
echo "================================="
321+
322+
total_dead=$(( ${#UNMAPPED_CHAP[@]} + dead_count ))
323+
if [[ $total_dead -eq 0 ]]; then
324+
exit 0
325+
fi
326+
327+
echo ""
328+
echo "=== fix-haddock-links: dead links detected ==="
329+
330+
if [[ ${#UNMAPPED_CHAP[@]} -gt 0 ]]; then
331+
echo ""
332+
echo "Unresolvable CHaP packages (${#UNMAPPED_CHAP[@]}):"
333+
for pkg in "${UNMAPPED_CHAP[@]}"; do
334+
echo " - $pkg"
335+
[[ -n "${GITHUB_ACTIONS:-}" ]] && \
336+
echo "::warning title=Unmapped CHaP package::$pkg has no known doc site. Add its base URL to IOG_DOC_BASES in scripts/fix-haddock-links.sh."
337+
done
338+
echo ""
339+
echo " Fix: find the package's published Haddocks (check its source repo for"
340+
echo " a gh-pages or CloudFront deployment), then append the base URL"
341+
echo " to IOG_DOC_BASES in scripts/fix-haddock-links.sh and re-run."
342+
fi
343+
344+
if [[ $dead_count -gt 0 ]]; then
345+
echo ""
346+
echo "Module-level 404s (${dead_count}):"
347+
while IFS= read -r dead_url; do
348+
[ -z "$dead_url" ] && continue
349+
echo ""
350+
echo " $dead_url"
351+
echo " Upstream: ${DEAD_MODULE[$dead_url]} (in package ${DEAD_PKG[$dead_url]})"
352+
[[ -n "${DEAD_SYMBOLS[$dead_url]:-}" ]] && \
353+
echo " Symbols we link to: ${DEAD_SYMBOLS[$dead_url]}"
354+
if [[ -n "${DEAD_REFS[$dead_url]:-}" ]]; then
355+
echo " Linked from our docs:"
356+
while IFS= read -r ref; do
357+
[ -z "$ref" ] && continue
358+
echo " - $ref"
359+
done <<< "${DEAD_REFS[$dead_url]}"
360+
fi
361+
[[ -n "${GITHUB_ACTIONS:-}" ]] && \
362+
echo "::warning title=Dead haddock link::${DEAD_MODULE[$dead_url]} (in ${DEAD_PKG[$dead_url]}) returned 404 at $dead_url"
363+
done < "$DEAD_URLS_FILE"
364+
echo ""
365+
echo " Fix:"
366+
echo " 1. Visit the upstream doc site and search for the module name."
367+
echo " If it's been renamed, moved, or removed, our re-exports or"
368+
echo " dependency bounds are out of date."
369+
echo " 2. The 'Linked from' HTML paths mirror our Haskell module layout"
370+
echo " (e.g. cardano-api/Cardano-Api-Ledger.html corresponds to the"
371+
echo " source module Cardano.Api.Ledger). Open those .hs files and"
372+
echo " grep for the listed symbols to find the re-export site."
373+
echo " 3. If a dep bound is pinned to a version whose module layout"
374+
echo " doesn't match the published docs, bump it."
375+
echo " 4. Otherwise the links are already annotated as greyed-out spans"
376+
echo " with tooltips — no action needed if that's acceptable."
377+
fi
378+
379+
echo ""
380+
if [[ "${FIX_HADDOCK_LINKS_STRICT:-0}" == "1" ]]; then
381+
echo "FIX_HADDOCK_LINKS_STRICT=1 — exiting 1."
382+
exit 1
383+
fi
384+
echo "Exiting 0 (warn-only default). Set FIX_HADDOCK_LINKS_STRICT=1 to fail the build."
385+
exit 0

0 commit comments

Comments
 (0)