Skip to content

Commit 19706b5

Browse files
lazarvlazarv
authored
feat: agent ready docs (#410)
## Summary Brings the docs site to the public agent-readiness bar (level 0 → level 5 on `isitagentready.com`) and ships the runtime support that other react-server applications need to do the same. Closes the gap that any pre-rendered react-server site has today: there is no way for an agent asking for `Accept: text/markdown` on the canonical URL to receive markdown, because the static layer always wins. The work splits cleanly into three buckets. The docs site grows a set of agent-facing surfaces (well-known endpoints, link headers, content negotiation, an MCP server). The runtime grows the primitives those surfaces depend on (`Accept`-aware static deferral across every adapter, header propagation across short-circuiting middlewares, a generic fix for rolldown's CJS-imports-JSON wrapping bug). The same primitives let any other application following this PR's patterns reach the same readiness level with a few hundred lines of glue. ## Runtime changes ### Accept-aware static deferral Every adapter in `@lazarv/react-server` previously gave static files unconditional priority over the worker. That makes browsers fast — and silently breaks content negotiation, because middleware never runs for paths whose pre-rendered HTML happens to match. This PR adds a shared `shouldDeferToServer(request)` / `isHtmlRoute(url)` pair under `adapters/shared/accept.mjs` and applies them at every static-first site so that browser navigation and asset traffic keep the fast path while agent traffic with `Accept: text/markdown` (or any concrete non-HTML media type) flows to the worker. The Cloudflare adapter checks both predicates before calling `env.ASSETS.fetch`. The Node-mode static handler in `lib/handlers/static.mjs` now defers when it would have served `text/html` and the client clearly prefers something else — covering Bun, Deno, Docker, Azure Functions, AWS Lambda, Firebase, and the singlefile adapter at once. The Vercel adapter gains a `has`-conditioned route that runs before `{ handle: "filesystem" }` and routes bare paths whose `Accept` header omits `text/html` to the function. The Netlify adapter drops the unconditional `preferStatic: true` from the function config so the in-process static handler can do the per-request decision; users who don't need content negotiation can opt back into the fast path via `adapterOptions.functions.config.preferStatic = true`. AWS Lambda's `tryServeStatic` and Docker's bespoke static-first server gain the same `isHtmlRoute && shouldDeferToServer` guard. The deferral predicate is deliberately narrow: it requires the URL to look like an HTML route (no extension, or `.html`/`.htm`) and the client to explicitly prefer a concrete non-HTML media type at higher q-value than `text/html` and `*/*`. Browsers always list `text/html` and never trigger it; image/CSS/JSON requests are excluded by the URL-extension filter; `curl` with the default `*/*` is treated as "anything is fine" and gets the static reply. ### Header propagation across short-circuiting middlewares react-server now supports running multiple middlewares in sequence — for example, an `agent-discovery` middleware that sets a `Link` header followed by a `content-negotiation` middleware that returns a `Response` directly. The old behaviour silently dropped headers set on the HTTP context whenever a later middleware short-circuited. A new shared helper at `lib/http/middleware-response.mjs#mergeContextHeaders` is now invoked from both `lib/start/ssr-handler.mjs` and `lib/dev/ssr-handler.mjs` and merges `setHeader` / `appendHeader` / `headers()` output onto the returned `Response` (the `Response`'s own headers win on conflict, so middlewares that explicitly set Content-Type or Cache-Control retain authority). ### Generic fix for the rolldown CJS-imports-JSON wrapping bug `@lazarv/react-server/mcp` pulls in `@modelcontextprotocol/sdk`, which transitively pulls in Express's CJS dependency tree (`statuses`, `mime-types`, `finalhandler`, `http-errors`). Several of those packages do `var data = require('./codes.json')` and then iterate `Object.keys(data).forEach(k => data[k].toLowerCase())`. The bundler's CJS-from-ESM interop wraps imported JSON as `{ __esModule: true, default: <getter> }` and never unwraps it back, so the iteration crashes on the boolean `__esModule` with `r.toLowerCase is not a function`. The bug reproduces in `examples/mcp` on `main` and is the reason any react-server project trying to host an MCP server has been broken in production. ## Docs site changes ### Discovery surface `/robots.txt` is now a proper file with a `User-agent: *` block, a `Content-Signal: search=yes, ai-input=yes, ai-train=yes` directive, and the sitemap. As an open-source documentation site, `react-server.dev` wants to be indexed and used by AI tools — the signal is intentionally permissive. A new `(agent-discovery).middleware.mjs` serves four well-known endpoints with the right content types: `/.well-known/api-catalog` (RFC 9727 linkset format), `/.well-known/agent-skills/index.json` (Agent Skills v0.2 index), `/.well-known/agent-skills/react-server/SKILL.md` (the canonical skill body, imported via `?raw` from the monorepo's `skills/react-server/SKILL.md`), and `/.well-known/mcp/server-card.json`. The same middleware sets RFC 8288 `Link` headers on every documentation page advertising the api-catalog, the MCP endpoint, llms.txt, the sitemap, and the agent-skills index. A new `(content-negotiation).middleware.mjs` handles `Accept: text/markdown` by rewriting requests to the existing `/md/[...slug]` route. The homepage has no `/md/` slug entry, so it returns the canonical `llms.txt` summary as `text/markdown` instead — that is the right "what is this site" answer for an agent landing at `/`. `Vary: Accept` is set on the response so HTML and markdown caches don't poison each other. The original `(i18n).middleware.mjs` was reduced back to pure locale resolution. With three middlewares now living side by side (alphabetically ordered: agent-discovery, content-negotiation, i18n), each file does one thing. ### MCP server and version source-of-truth The docs site now hosts a live MCP server at `/mcp`, dogfooding `@lazarv/react-server/mcp`. It exposes a `search_docs` tool (free-text query against the page index), a `read_doc` tool (fetches any docs page as markdown via the public URL — works on every runtime including Cloudflare Workers without filesystem access), a templated `docs-page` resource that lists every page, and an `explain-topic` prompt that orchestrates the two tools. A new `docs/src/version.mjs` strips the `react-server/` prefix off the package's namespaced version export, giving callers a clean semver. The `/mcp` server, the MCP server card, and the agent-skills index now all read from this single source — there is no longer a hardcoded `1.0.0` to drift out of sync with the package. ### WebMCP browser tools A `"use client"` `WebMCP.jsx` component mounts once in the root layout and registers `search_docs` and `get_docs_page` via `navigator.modelContext.registerTool`. Any in-browser agent (Claude, Cursor, ChatGPT Atlas, Cloudflare Browser-Use) interacting with a docs page can now search the docs and fetch any page as markdown without scraping HTML.
1 parent 5353c12 commit 19706b5

26 files changed

Lines changed: 964 additions & 38 deletions

‎docs/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,8 @@
3333
"remark-gfm": "^4.0.0",
3434
"remark-math": "^6.0.0",
3535
"three": "^0.183.1",
36-
"vite-plugin-svgr": "^4.5.0"
36+
"vite-plugin-svgr": "^4.5.0",
37+
"zod": "^3.23.8"
3738
},
3839
"devDependencies": {
3940
"@inlang/paraglide-js": "1.11.8",

‎docs/public/robots.txt‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,14 @@
1-
Sitemap: https://react-server.dev/sitemap.xml
1+
# @lazarv/react-server is an open-source React Server Components runtime.
2+
# We welcome AI agents — this site is documentation, and we want LLMs to
3+
# index it, cite it, and answer user questions about it accurately.
4+
#
5+
# Content-Signal (https://blog.cloudflare.com/content-signals/):
6+
# search — build a search index and link in search results
7+
# ai-input — use content as live input for AI answers (RAG, in-context)
8+
# ai-train — train or fine-tune AI models
9+
10+
User-agent: *
11+
Content-Signal: search=yes, ai-input=yes, ai-train=yes
12+
Allow: /
13+
14+
Sitemap: https://react-server.dev/sitemap.xml

‎docs/react-server.config.mjs‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,7 @@ import remarkMath from "remark-math";
77
export default {
88
root: "src/pages",
99
public: "public",
10-
adapter: [
11-
"cloudflare",
12-
{
13-
serverlessFunctions: false,
14-
},
15-
],
10+
adapter: "cloudflare",
1611
mdx: {
1712
remarkPlugins: [remarkGfm, remarkMath],
1813
rehypePlugins: [

‎docs/src/components/WebMCP.jsx‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
"use client";
2+
3+
import { useEffect } from "react";
4+
5+
/**
6+
* Register WebMCP tools so any browser-based agent (Claude, Cursor, ChatGPT
7+
* Atlas, Cloudflare Browser-Use) interacting with the docs page through
8+
* `navigator.modelContext` can search the docs and fetch any page as
9+
* markdown without scraping HTML.
10+
*
11+
* https://webmcp.org
12+
*/
13+
export default function WebMCP() {
14+
useEffect(() => {
15+
const nav = typeof navigator !== "undefined" ? navigator : null;
16+
if (!nav?.modelContext?.registerTool) return;
17+
18+
const registrations = [];
19+
20+
registrations.push(
21+
nav.modelContext.registerTool({
22+
name: "search_docs",
23+
description:
24+
"Search the @lazarv/react-server documentation for a query and return matching page paths with titles. Use this when the user asks how to do something with @lazarv/react-server.",
25+
inputSchema: {
26+
type: "object",
27+
properties: {
28+
query: {
29+
type: "string",
30+
description:
31+
"Free-text search query (e.g. 'file system router', 'use cache', 'cloudflare deploy').",
32+
},
33+
},
34+
required: ["query"],
35+
},
36+
annotations: { readOnlyHint: true, idempotentHint: true },
37+
async execute({ query }) {
38+
if (typeof query !== "string" || !query.trim()) {
39+
return { error: "Missing query" };
40+
}
41+
// Use the sitemap as a lightweight, cache-friendly index.
42+
const res = await fetch("/sitemap.xml", {
43+
headers: { Accept: "application/xml" },
44+
});
45+
if (!res.ok) return { error: `sitemap fetch failed: ${res.status}` };
46+
const xml = await res.text();
47+
const locs = [...xml.matchAll(/<loc>([^<]+)<\/loc>/g)].map(
48+
(m) => m[1]
49+
);
50+
const q = query.toLowerCase();
51+
const matches = locs
52+
.filter((u) => u.toLowerCase().includes(q))
53+
.slice(0, 20)
54+
.map((u) => ({
55+
url: u,
56+
markdown_url: `${u.replace(/\/$/, "")}.md`,
57+
}));
58+
return { matches, total: matches.length };
59+
},
60+
})
61+
);
62+
63+
registrations.push(
64+
nav.modelContext.registerTool({
65+
name: "get_docs_page",
66+
description:
67+
"Fetch a documentation page as markdown. Pass either a full URL on https://react-server.dev or a path like '/router/file-router'. Returns the page content as text/markdown.",
68+
inputSchema: {
69+
type: "object",
70+
properties: {
71+
path: {
72+
type: "string",
73+
description:
74+
"Page path (e.g. '/router/file-router') or full URL. The .md suffix is added automatically.",
75+
},
76+
},
77+
required: ["path"],
78+
},
79+
annotations: { readOnlyHint: true, idempotentHint: true },
80+
async execute({ path }) {
81+
if (typeof path !== "string" || !path) {
82+
return { error: "Missing path" };
83+
}
84+
let target = path;
85+
try {
86+
const u = new URL(path, location.origin);
87+
target = u.pathname;
88+
} catch {
89+
// not a URL, treat as path
90+
}
91+
if (!target.startsWith("/")) target = `/${target}`;
92+
target = target.replace(/\/$/, "");
93+
if (!target.endsWith(".md")) target = `${target}.md`;
94+
const res = await fetch(target, {
95+
headers: { Accept: "text/markdown" },
96+
});
97+
if (!res.ok) return { error: `fetch failed: ${res.status}` };
98+
const markdown = await res.text();
99+
return { path: target, markdown };
100+
},
101+
})
102+
);
103+
104+
return () => {
105+
for (const r of registrations) {
106+
if (typeof r === "function") {
107+
try {
108+
r();
109+
} catch {
110+
/* ignore */
111+
}
112+
} else if (r && typeof r.unregister === "function") {
113+
try {
114+
r.unregister();
115+
} catch {
116+
/* ignore */
117+
}
118+
}
119+
}
120+
};
121+
}, []);
122+
123+
return null;
124+
}

‎docs/src/pages/(agent-discovery).middleware.mjs‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
import { setHeader } from "@lazarv/react-server";
2+
import { useUrl } from "@lazarv/react-server";
3+
4+
import skillContent from "../../../skills/react-server/SKILL.md?raw";
5+
import { version } from "../version.mjs";
6+
7+
// ---------------------------------------------------------------------------
8+
// Agent-readiness payloads
9+
//
10+
// Implements the contracts checked by https://isitagentready.com:
11+
// - RFC 9727 API Catalog → /.well-known/api-catalog
12+
// - Agent Skills v0.2 index → /.well-known/agent-skills/index.json
13+
// - Agent Skill body → /.well-known/agent-skills/react-server/SKILL.md
14+
// - MCP Server Card → /.well-known/mcp/server-card.json
15+
// - RFC 8288 Link headers → on every documentation page
16+
// ---------------------------------------------------------------------------
17+
18+
const SITE = "https://react-server.dev";
19+
20+
const apiCatalog = {
21+
linkset: [
22+
{
23+
anchor: `${SITE}/.well-known/api-catalog`,
24+
item: [
25+
{ href: `${SITE}/mcp` },
26+
{ href: `${SITE}/llms.txt` },
27+
{ href: `${SITE}/sitemap.xml` },
28+
{ href: `${SITE}/schema.json` },
29+
],
30+
},
31+
{
32+
anchor: `${SITE}/mcp`,
33+
"service-doc": [
34+
{ href: `${SITE}/features/mcp`, type: "text/html" },
35+
{ href: `${SITE}/features/mcp.md`, type: "text/markdown" },
36+
],
37+
describedby: [
38+
{
39+
href: `${SITE}/.well-known/mcp/server-card.json`,
40+
type: "application/json",
41+
},
42+
],
43+
},
44+
{
45+
anchor: `${SITE}/llms.txt`,
46+
describedby: [{ href: `${SITE}/llms.txt`, type: "text/plain" }],
47+
},
48+
{
49+
anchor: `${SITE}/schema.json`,
50+
describedby: [
51+
{ href: `${SITE}/schema.json`, type: "application/schema+json" },
52+
],
53+
},
54+
],
55+
};
56+
57+
const agentSkillsIndex = {
58+
$schema: "https://agent-skills.dev/schema/v0.2.0.json",
59+
skills: [
60+
{
61+
name: "react-server",
62+
description:
63+
"Build applications with @lazarv/react-server — a React Server Components runtime built on Vite. Covers use directives, file-system router, HTTP hooks, caching, live components, workers, MCP, deployment, and all core APIs.",
64+
version,
65+
skill_url: `${SITE}/.well-known/agent-skills/react-server/SKILL.md`,
66+
homepage: SITE,
67+
license: "MIT",
68+
},
69+
],
70+
};
71+
72+
const mcpServerCard = {
73+
$schema:
74+
"https://modelcontextprotocol.io/schemas/draft/2025-09-29/server-card.json",
75+
name: "react-server-docs",
76+
title: "@lazarv/react-server Documentation",
77+
description:
78+
"Search and read the @lazarv/react-server documentation as Model Context Protocol resources and tools. Provides a search_docs tool and exposes every documentation page as a markdown resource.",
79+
version,
80+
homepage: SITE,
81+
documentation: `${SITE}/features/mcp`,
82+
endpoints: {
83+
streamable_http: `${SITE}/mcp`,
84+
},
85+
capabilities: {
86+
tools: { listChanged: false },
87+
resources: { listChanged: false, subscribe: false },
88+
prompts: { listChanged: false },
89+
},
90+
contact: {
91+
repository: "https://github.com/lazarv/react-server",
92+
},
93+
};
94+
95+
const wellKnown = {
96+
"/.well-known/api-catalog": () =>
97+
json(
98+
apiCatalog,
99+
'application/linkset+json; profile="https://www.rfc-editor.org/info/rfc9727"'
100+
),
101+
"/.well-known/agent-skills/index.json": () => json(agentSkillsIndex),
102+
"/.well-known/agent-skills/react-server/SKILL.md": () =>
103+
new Response(skillContent, {
104+
headers: {
105+
"Content-Type": "text/markdown; charset=utf-8",
106+
"Cache-Control": "public, max-age=3600",
107+
},
108+
}),
109+
"/.well-known/mcp/server-card.json": () => json(mcpServerCard),
110+
};
111+
112+
function json(body, contentType = "application/json; charset=utf-8") {
113+
return new Response(JSON.stringify(body, null, 2), {
114+
headers: {
115+
"Content-Type": contentType,
116+
"Cache-Control": "public, max-age=3600",
117+
},
118+
});
119+
}
120+
121+
// ---------------------------------------------------------------------------
122+
// Discovery Link headers (RFC 8288 / RFC 9727)
123+
//
124+
// Advertised on every documentation page so any HTTP client (including agents
125+
// that only do HEAD or GET on `/`) can discover the API catalog, MCP entry,
126+
// and human-/machine-readable documentation without crawling the whole site.
127+
// ---------------------------------------------------------------------------
128+
129+
const linkHeader = [
130+
'</.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json"',
131+
'</mcp>; rel="service-meta"; type="application/json"',
132+
'</llms.txt>; rel="describedby"; type="text/plain"',
133+
'</sitemap.xml>; rel="sitemap"; type="application/xml"',
134+
'</.well-known/agent-skills/index.json>; rel="https://agent-skills.dev/rel/index"; type="application/json"',
135+
].join(", ");
136+
137+
// Pathnames that should never receive the discovery Link header — they're
138+
// machine-only endpoints with their own headers/cache semantics.
139+
const SKIP_LINK_HEADER = (pathname) =>
140+
pathname === "/sitemap.xml" ||
141+
pathname === "/schema.json" ||
142+
pathname === "/mcp" ||
143+
pathname.startsWith("/mcp/") ||
144+
pathname.startsWith("/.well-known/") ||
145+
pathname.startsWith("/md/") ||
146+
pathname.endsWith(".md");
147+
148+
export default function AgentDiscovery() {
149+
const { pathname } = useUrl();
150+
151+
// 1. Static well-known endpoints — short-circuit with the right content type.
152+
const wellKnownHandler = wellKnown[pathname];
153+
if (wellKnownHandler) {
154+
return wellKnownHandler();
155+
}
156+
157+
// 2. Set discovery Link header on documentation pages.
158+
if (!SKIP_LINK_HEADER(pathname)) {
159+
setHeader("Link", linkHeader);
160+
}
161+
}

0 commit comments

Comments
 (0)