feat(web): docs SEO + LLM-discoverability — llms.txt, Pagefind search, FAQ JSON-LD, docs events (SQLR-36) (#172)

- /llms.txt (curated index) + /llms-full.txt (full docs+blog markdown) +
  /docs.md (raw-markdown docs), all force-static, referenced from a
  hand-rolled /robots.txt route (replaces the typed robots.ts)
- Pagefind docs search: postbuild index of .next/server/app scoped via
  data-pagefind-body, custom dark-themed UI on /docs
- Heading anchors: H2 helper on /docs, rehype-slug + autolink on blog MDX
- Blog articles: on-page ToC (>600 words), tag-ranked related posts
- /docs: Related footer (5 links) + helpful-vote widget
- Landing FAQ: 8 h3/p pairs mirrored into FAQPage JSON-LD
- PostHog custom events: docs-search-query, docs-helpful-vote (no-op
  without NEXT_PUBLIC_POSTHOG_KEY)
- Fixed the one bare code fence (ASCII diagram -> ```text)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: joaoh82

web/.gitignore

@@ -15,6 +15,9 @@
 /.next/
 /out/
 
+# pagefind search index — generated by the postbuild script
+/public/pagefind/
+
 # production
 /build
 

web/README.md

@@ -15,11 +15,13 @@ extracted into its own repository later without rewrites.
 
 ## Pages
 
-- `/` — landing (hero with animated REPL, features, architecture, roadmap, SDK switcher, SQL surface, desktop showcase, blog series, footer)
+- `/` — landing (hero with animated REPL, features, architecture, roadmap, SDK switcher, SQL surface, desktop showcase, blog series, FAQ with `FAQPage` JSON-LD, footer)
 - `/playground` — in-browser SQL playground: the full engine compiled to WebAssembly, with a CodeMirror editor, sample datasets, HNSW vector search, and OPFS session persistence. The WASM bundle is a pinned copy of `sdk/wasm/pkg/` vendored into `public/playground/pkg/`. See [`../examples/wasm-playground/README.md`](../examples/wasm-playground/README.md).
-- `/docs` — Getting Started page (sticky sidebar nav + on-page TOC)
+- `/docs` — Getting Started page (sticky sidebar nav + on-page TOC, Pagefind search box, heading anchor links, related-links footer, helpful-vote widget)
+- `/docs.md` — the same docs content served as raw `text/markdown` for AI crawlers that prefer markdown over rendered HTML
+- `/llms.txt` + `/llms-full.txt` — LLM-discoverability surfaces per the [llms.txt convention](https://llmstxt.org/); see "LLM surfaces" below
 - `/blog` — index of long-form posts pulled from `content/blog/*.mdx`
-- `/blog/[slug]` — per-post detail page (MDX rendered server-side, `Article` JSON-LD, breadcrumb JSON-LD, dynamic OG image, prev/next navigation)
+- `/blog/[slug]` — per-post detail page (MDX rendered server-side, `Article` JSON-LD, breadcrumb JSON-LD, dynamic OG image, on-page ToC, heading anchors, related posts, helpful-vote widget, prev/next navigation)
 - `/blog/tags/[tag]` — tag pages (one per unique frontmatter tag)
 - `/blog/rss.xml` — RSS 2.0 feed
 
@@ -51,14 +53,26 @@ Each public route ships full search/social metadata. The pieces:
   the apple icon are rasterized from the same play-glyph mark used in
   [`src/lib/og.tsx`](src/lib/og.tsx) and `.brand-mark`; regenerate them
   from `icon.svg` (e.g. with `sharp`) if the mark ever changes.
-- **`/sitemap.xml` + `/robots.txt`** — Next 15 metadata routes
-  ([`src/app/sitemap.ts`](src/app/sitemap.ts),
-  [`src/app/robots.ts`](src/app/robots.ts)). Add a route to the `ROUTES`
-  list when shipping a new page.
-- **JSON-LD structured data** — `SoftwareApplication` schema on the landing
-  page, `BreadcrumbList` on `/docs`, `Blog` on `/blog`, and
+- **`/sitemap.xml`** — Next 15 metadata route
+  ([`src/app/sitemap.ts`](src/app/sitemap.ts)). Add a route to the
+  `STATIC_ROUTES` list when shipping a new page.
+- **`/robots.txt`** — a hand-rolled route handler
+  ([`src/app/robots.txt/route.ts`](src/app/robots.txt/route.ts)) rather
+  than the typed metadata route, so it can reference the sitemap *and* the
+  `llms.txt` surfaces (Next's `MetadataRoute.Robots` can't emit comments).
+- **JSON-LD structured data** — `SoftwareApplication` + `FAQPage` schema on
+  the landing page, `BreadcrumbList` on `/docs`, `Blog` on `/blog`, and
   `BlogPosting` + `BreadcrumbList` on each `/blog/<slug>`. Validate via
   Google's [Rich Results Test](https://search.google.com/test/rich-results).
+  The FAQ lives in [`src/components/faq.tsx`](src/components/faq.tsx) — the
+  visible `<h3>`/`<p>` pairs and the JSON-LD are generated from the same
+  array, so they can't drift apart.
+- **Heading anchors** — every docs `h2` (via the local `H2` helper in
+  [`src/app/docs/page.tsx`](src/app/docs/page.tsx)) and every blog heading
+  (via `rehype-slug` + `rehype-autolink-headings` in
+  [`src/components/blog-mdx.tsx`](src/components/blog-mdx.tsx)) carries a
+  hover-visible `#` link, so sections are shareable and quotable with
+  anchors.
 - **Search Console verification** — fill in the placeholder tokens in
   `metadata.verification` ([`src/app/layout.tsx`](src/app/layout.tsx)) once
   Google Search Console + Bing Webmaster Tools issue them.
@@ -72,6 +86,49 @@ export lives in [`seo/keywords.md`](seo/keywords.md). When rewriting a
 page's headline or meta description, update the corresponding entry in
 that sheet so future rewrites stay coordinated.
 
+## Docs search (Pagefind)
+
+`/docs` ships a client-side search box backed by
+[Pagefind](https://pagefind.app/) — static, serverless, no account needed.
+
+- The index is generated by the `postbuild` script (`package.json`):
+  `pagefind --site .next/server/app --output-path public/pagefind`. It
+  indexes the **prerendered HTML** that `next build` leaves in
+  `.next/server/app`, so it runs after every production build (Vercel
+  included — npm runs `postbuild` automatically after `build`).
+- Only elements under a `data-pagefind-body` attribute are indexed — the
+  docs main column and blog articles opt in; everything else (landing,
+  playground, nav, footers) stays out of the index. `data-pagefind-ignore`
+  carves out the CTA/related/vote footers.
+- The UI is the custom [`src/components/docs-search.tsx`](src/components/docs-search.tsx)
+  (Pagefind's JS API, not its default UI) so it matches the site's dark
+  styling. Raw result URLs point at the `.html` files Pagefind saw
+  (`/docs.html`) and are mapped back to routes in `cleanUrl`.
+- `public/pagefind/` is gitignored — it's a build artifact. In `next dev`
+  there is no index; the box degrades to a hint to run a build.
+
+## LLM surfaces
+
+Three build-time-generated plain-text routes make the site legible to AI
+crawlers (and are referenced as comments from `robots.txt`):
+
+- **`/llms.txt`** ([`src/app/llms.txt/route.ts`](src/app/llms.txt/route.ts)) —
+  curated index per [llmstxt.org](https://llmstxt.org/): project name +
+  tagline, then sections of absolute links with one-line summaries (docs,
+  playground, every blog post, GitHub/docs.rs/registries).
+- **`/llms-full.txt`** ([`src/app/llms-full.txt/route.ts`](src/app/llms-full.txt/route.ts)) —
+  the full docs markdown + every blog post concatenated as one markdown
+  file.
+- **`/docs.md`** ([`src/app/docs.md/route.ts`](src/app/docs.md/route.ts)) —
+  the docs page as raw `text/markdown`.
+
+All three are `force-static` (generated at build time) and built by
+[`src/lib/llms.ts`](src/lib/llms.ts). The markdown source for the docs
+content is [`content/docs/getting-started.md`](content/docs/getting-started.md)
+— **it mirrors `src/app/docs/page.tsx`**; when you change a docs section,
+update the matching section there too. Blog content needs no mirroring (the
+MDX sources are the truth).
+
 ## Local development
 
 ```sh
@@ -93,7 +150,8 @@ npm run lint       # next lint (ESLint)
 ```
 web/
 ├── content/
-│   └── blog/                # MDX posts (one .mdx file per post; frontmatter at top)
+│   ├── blog/                # MDX posts (one .mdx file per post; frontmatter at top)
+│   └── docs/                # markdown mirror of /docs — feeds /docs.md + /llms-full.txt
 ├── seo/
 │   └── keywords.md          # keyword research + per-page primary/secondary registry (SQLR-33)
 ├── src/
@@ -105,10 +163,15 @@ web/
 │   │   ├── docs/page.tsx    # /docs
 │   │   ├── blog/            # /blog index, [slug] detail, tags/[tag], rss.xml
 │   │   ├── sitemap.ts       # /sitemap.xml — enumerates static + per-post + per-tag URLs
-│   │   └── robots.ts        # /robots.txt
+│   │   ├── robots.txt/      # /robots.txt route handler (sitemap + llms.txt refs)
+│   │   ├── llms.txt/        # /llms.txt — curated LLM index (llmstxt.org)
+│   │   ├── llms-full.txt/   # /llms-full.txt — full docs+blog markdown concat
+│   │   └── docs.md/         # /docs.md — docs page as raw markdown
 │   ├── components/          # one .tsx per landing section (hero, features, roadmap, …)
 │   └── lib/
+│       ├── analytics.ts     # useTrack() — PostHog capture, no-ops without the key
 │       ├── blog.ts          # MDX loader: frontmatter parsing, post enumeration, tag helpers
+│       ├── llms.ts          # builders for /llms.txt, /llms-full.txt, /docs.md
 │       ├── og.tsx           # shared OpenGraph frame
 │       ├── site.ts          # SITE constants (version, repo URL, social links)
 │       └── utils.ts         # shadcn cn() helper
@@ -277,6 +340,20 @@ PostHog is wired up in two places:
 If the env var is absent the provider is omitted and the middleware
 falls through to `NextResponse.next()`.
 
+Beyond pageviews/autocapture, two custom events (SQLR-36) flow through the
+`useTrack()` hook in [`src/lib/analytics.ts`](src/lib/analytics.ts), which
+no-ops when the key is absent:
+
+- **`docs-search-query`** — fired from the docs search box once typing
+  settles (~1.2 s debounce); properties: `query`, `results` (count). Zero-
+  result queries are the signal for missing docs.
+- **`docs-helpful-vote`** — fired by the "Was this page helpful?" widget on
+  `/docs` and every blog post; properties: `path`, `helpful` (boolean). No
+  PII either way.
+
+Worth a weekly skim in PostHog: most-visited docs pages, searches with
+`results = 0`, and pages accumulating `helpful = false` votes.
+
 ### Privacy / compliance follow-ups
 
 The current wiring uses PostHog defaults — autocapture + cookies — and

web/content/blog/shipping-concurrent-writes-mvcc-v010.mdx

@@ -229,7 +229,7 @@ distinguished from page frames by the sentinel `page_num = u32::MAX`
 The frame body encodes the commit timestamp plus a record stream of
 the resolved write-set:
 
-```
+```text
 ┌────────┬────────┬─────────────────────────────────────────────────┐
 │ offset │ length │ content                                         │
 ├────────┼────────┼─────────────────────────────────────────────────┤