Page-level HTML build cache backed by HF storage buckets

[mishig25]

What

Doc builds prerender every page even when a commit changed one file (transformers: ~400 en pages, 14 min + 18 min for other languages per main build, ~5 min of which is re-uploading pages that didn't meaningfully change). This adds an opt-in page-level HTML cache stored in an HF storage bucket:

doc-builder build ... --html --html_page_cache hf://buckets/hf-doc-build/doc-build-cache [--html_page_cache_write]

Design

• Key = sha256(generated mdx ‖ kit tree hash ‖ package ‖ language ‖ doc-builder version). Keying on the generated mdx (post autodoc + internal-link resolution) means docstring changes from library code invalidate exactly the affected pages, and any kit/dependency change invalidates everything.
• Version normalization: resolved internal links embed /docs/{pkg}/{version}/{lang}/, which would make otherwise-identical pages differ between main and pr_N builds. Keys normalize that segment, and on a cross-version reuse the page URLs are rewritten to the target version while asset URLs stay pinned to the source version — its content-hashed assets remain hosted (the doc bucket sync is additive) and chunk→chunk imports resolve relative to the chunk's own URL, so a pr_N preview page can safely load main's assets. Autodoc pages self-exclude from cross-version reuse (their GitHub source links embed version tags/line numbers).
• Storage: a manifest per (package, language) plus content-addressed HTML blobs; backends are a local directory (tests/local builds) or an HF bucket via huggingface_hub's bucket APIs.
• Never fatal: any cache error degrades to building the affected pages.
• Security: the shared workflows enable read+write on main builds (HF_TOKEN) and read-only on PR builds (which run without secrets against the public bucket) — untrusted PR code cannot poison pages that trusted builds would reuse.

The hf-doc-build/doc-build-cache bucket has been created (public).

Verification (accelerate e2e, local dir and a real bucket)

scenario	result
cold	0/48 reused, output byte-identical to an uncached build, 48 stored
warm	46/48 reused byte-identically — the 2 misses are the docstring pages containing nondeterministic <object object at 0x…> addresses, i.e. correct invalidation
one file changed	exactly that page (plus the 2 above) rebuilt; mixed output equivalent to a full fresh build
cross-version --version pr_5	35/48 reused; reused pages have all page links rewritten to pr_5 and all 15 asset refs pinned to main; misses = the 11 autodoc pages + 2 nondeterministic
real bucket backend	cold populate + warm reuse verified against hf://buckets/mishig/doc-cache-test

Also: pytest (incl. new tests/test_build_cache.py), ruff check/format clean.

Expected CI impact

• transformers other-languages job (12 langs, translations rarely change): mostly cache hits.
• main en builds: only pages whose docstrings/content changed rebuild; the Sync to bucket step (280 s today) also collapses since reused pages are byte-identical.
• PR builds: warm across pushes within a PR, plus cross-version reuse of non-autodoc pages from main's cache on first push.
• Consumer repos pinned to older workflow SHAs are unaffected (flag is opt-in via the shared workflows).

Follow-ups worth considering: a periodic GC job for old blobs (bucket rm by age), and extending the cache to the python conversion stage.

🤖 Generated with Claude Code