fix: make SPA routes crawlable at source (sitemap script + pre-render)#429
Merged
rdmueller merged 4 commits intoApr 13, 2026
Merged
Conversation
Root-cause fix for the recurring "workflow page is still dynamic" issue. ## The recurring bug Previous fix attempts (LLM-Coding#371, LLM-Coding#384) edited the output sitemap.xml directly but never touched scripts/generate-sitemap.js. Every time npm run build ran the script, it regenerated the sitemap with the broken hash URLs — silently reverting the fix. ## What changed ### 1. scripts/generate-sitemap.js — the actual root cause - Removed hash prefix (#/) from all generated URLs — they were not crawlable (hash fragments look like the homepage to search engines and pure HTTP fetchers like claude.ai). - Added all 11 router pages (was 3): Workflow, Brownfield, Contracts, Evaluations, Changelog, AgentSkill, Rejected Proposals, All Anchors. - Matches website/src/utils/router.js History API routes. ### 2. scripts/prerender-routes.js — the deeper problem Even with clean URLs, /workflow returned the SPA shell with empty #app. Crawlers and non-JS fetchers (claude.ai, curl, LLM retrievers) saw nothing. Only modern JS-executing crawlers (Googlebot) could eventually index the content. New post-build step generates per-route static HTML (dist/<route>/index.html) for all 9 doc routes by injecting the pre-rendered AsciiDoc fragment from render-docs.js into the Vite shell. Also updates the per-route <title>, meta description, and canonical URL. GitHub Pages serves dist/workflow/index.html when /workflow is requested. The SPA boots on top of the pre-rendered content, wipes #app, and re-renders normally — so interactive UX is unchanged. Crawlers see real content immediately. ### 3. website/package.json Chained the pre-render step into the build command so it can't be forgotten: `vite build && node ../scripts/prerender-routes.js`. ## Result - sitemap.xml: 141 crawlable URLs (11 pages + 130 anchors) - dist/workflow/index.html, dist/brownfield/index.html, dist/about/... each contain the real content, title, meta description, canonical URL - Unit tests pass (89/89) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 40 minutes and 24 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
WalkthroughDas Pull Request aktualisiert die Sitemap-Generierung für saubere URLs statt Hash-Fragmente, fügt ein neues Prerendering-Skript für statische HTML-Seiten pro Route hinzu und modifiziert die Build-Pipeline, um das Prerendering nach dem Vite-Build auszuführen. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/prerender-routes.js`:
- Around line 35-98: The ROUTES array in scripts/prerender-routes.js is missing
the '/contracts' entry so the prerender step doesn't produce
dist/contracts/index.html; add a matching route object (path: '/contracts',
fragment and metadata matching the existing route in
website/src/utils/router.js) into the ROUTES constant so prerendering generates
the static page, or alternatively remove '/contracts' from the crawlable list in
scripts/generate-sitemap.js to avoid advertising a non-prerendered URL;
reference the ROUTES constant and the '/contracts' path and coordinate with
website/src/utils/router.js and scripts/generate-sitemap.js when making the
change.
- Around line 136-139: The code currently only warns when a configured fragment
is missing (the fs.existsSync(fragmentPath) check) which allows builds to
succeed with incomplete prerendered routes; change this to fail fast by throwing
an Error or returning a rejected promise so the build exits non‑zero: replace
the console.warn/return false with a throw new Error(`Missing fragment for
${route.path}: ${route.fragment} (expected at ${fragmentPath})`) or ensure the
surrounding function (where fragmentPath and route are used) propagates the
error to the process so the build fails. Ensure the thrown error references
fragmentPath, route.path and route.fragment so logs show exact missing fragment
details.
In `@website/package.json`:
- Line 11: The build script ("build": "vite build && node
../scripts/prerender-routes.js") pulls in ../scripts/prerender-routes.js but
your lint/format gates still only target src/; update the project's lint/format
config and npm scripts so the ESLint and Prettier checks include the external
scripts (e.g., add ../scripts/**/*.js to the globs used by "lint", "lint:fix",
"format" and "format:check"), or move/duplicate the prerender script inside the
checked source; ensure the referenced symbols are "build", "lint", "lint:fix",
"format", "format:check" and the external script path
../scripts/prerender-routes.js so the new files are covered by Tier‑1 gates.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro
Run ID: 43ca92b7-2bc6-4b20-87a1-612811c9e6a6
📒 Files selected for processing (4)
scripts/generate-sitemap.jsscripts/prerender-routes.jswebsite/package.jsonwebsite/public/sitemap.xml
| "dev": "vite", | ||
| "prebuild": "node ../scripts/sync-anchors.js && node ../scripts/render-docs.js", | ||
| "build": "vite build", | ||
| "build": "vite build && node ../scripts/prerender-routes.js", |
Contributor
There was a problem hiding this comment.
Der neue Build-Pfad liegt außerhalb eurer Lint-/Prettier-Gates.
Mit node ../scripts/prerender-routes.js hängt der Produktionsbuild jetzt von zusätzlichem scripts/**/*.js ab, aber lint, lint:fix, format und format:check prüfen weiter nur src/. Syntax- und Formatfehler in den neuen Build-Skripten werden damit nicht von den Tier-1-Gates abgefangen.
Vorschlag
- "lint": "eslint src/",
- "lint:fix": "eslint src/ --fix",
- "format": "prettier --write src/",
- "format:check": "prettier --check src/"
+ "lint": "eslint src/ ../scripts/",
+ "lint:fix": "eslint src/ ../scripts/ --fix",
+ "format": "prettier --write src/ ../scripts/",
+ "format:check": "prettier --check src/ ../scripts/"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@website/package.json` at line 11, The build script ("build": "vite build &&
node ../scripts/prerender-routes.js") pulls in ../scripts/prerender-routes.js
but your lint/format gates still only target src/; update the project's
lint/format config and npm scripts so the ESLint and Prettier checks include the
external scripts (e.g., add ../scripts/**/*.js to the globs used by "lint",
"lint:fix", "format" and "format:check"), or move/duplicate the prerender script
inside the checked source; ensure the referenced symbols are "build", "lint",
"lint:fix", "format", "format:check" and the external script path
../scripts/prerender-routes.js so the new files are covered by Tier‑1 gates.
Addresses CodeRabbit pre-merge check for docstring coverage.
1. Drop /contracts from the sitemap. /contracts is a fully interactive JS page (localStorage, client-side filter UI, async data fetch) — it has no static content worth pre-rendering. Advertising it in the sitemap as a crawlable URL would return an empty SPA shell to non-JS fetchers (claude.ai, curl), which is exactly what this PR aims to avoid. Googlebot can still find it via regular navigation from the homepage. 2. Fail fast on missing fragments in prerender-routes.js. A console.warn + return false let the build ship with an incomplete set of static pages. Now throws a descriptive Error so the build exits non-zero if render-docs.js fails to produce a fragment this script expects.
6 tasks
This was referenced May 8, 2026
This was referenced Jun 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root-cause fix for the recurring "workflow page is still dynamic" issue. Two separate layers of the problem, both addressed:
Layer 1: The sitemap script was never fixed
Previous fixes (#371, #384) edited the output `sitemap.xml` directly but never touched `scripts/generate-sitemap.js`. Every `npm run build` ran the script and regenerated the sitemap with the broken hash URLs — silently reverting each fix. This is why the bug kept coming back.
Fixed: `scripts/generate-sitemap.js` now produces clean, crawlable URLs matching the History API router and includes all 11 pages (was 3). 141 URLs total.
Layer 2: Clean URLs alone don't help non-JS fetchers
Even with `/workflow` as a clean URL, GitHub Pages was serving the empty SPA shell — so pure HTTP fetchers (claude.ai, curl, basic crawlers that don't execute JS) saw nothing.
Fixed: New `scripts/prerender-routes.js` runs after `vite build` and generates `dist//index.html` for each of 9 doc routes. The pre-rendered AsciiDoc fragment from `render-docs.js` is injected into a static copy of the Vite shell with per-route `<title>`, ``, and canonical URL.
When GitHub Pages serves `dist/workflow/index.html` for `/workflow`:
Changes
Pre-rendered routes
/about, /workflow, /brownfield, /changelog, /contributing, /agentskill, /rejected-proposals, /all-anchors, /evaluations
Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit
Versionshinweise
New Features
Refactor