Skip to content

Commit 5d7792b

Browse files
critesjoshcritesjosh
authored
feat(docs): serve markdown + llms.txt for agent readiness (#24099)
## What Improves the docs site's [Fern agent score](https://buildwithfern.com/agent-score/company/aztec) (was 72/100, grade C, 6 failing checks) by making `docs.aztec.network` agent-readable. ## Changes - **Plugin swap**: `docusaurus-plugin-llms` → `@signalwire/docusaurus-plugin-llms-txt`. Covers all four docs instances (was developer-only), emits a `.md` sibling for every route + `llms-full.txt`, and points llms.txt links at `.md`. - **Content negotiation**: new Netlify edge function serves the `.md` sibling on `Accept: text/markdown`. - **Discovery directives**: a hidden `/llms.txt` link injected into every page `<body>` (`inject_llms_directive.js`), and an `llms.txt` pointer prepended to every generated page `.md` (`inject_md_directive.js`). - **Sitemap scoping**: dropped `augment_sitemap.js` so the ~2,500 auto-generated API HTML pages no longer inflate the coverage denominator; utility routes excluded. The API reference stays discoverable through the existing scoped hub-and-spoke `llms.txt` files. - **Idempotency**: `append_api_docs_to_llms.js` now strips before re-appending so re-runs don't duplicate sections. ## Checks addressed `markdown-url-support`, `content-negotiation`, `llms-txt-coverage`, `llms-txt-links-markdown`, `llms-txt-directive-html`, `llms-txt-directive-md`. ## Validation - Full `yarn build` green; coverage aligned (sitemap = page `.md` = llms.txt links; 0 API/tag/search pages in sitemap). - Edge function smoke-tested in the real Netlify Deno runtime (`netlify dev`): HTML stays HTML, `Accept: text/markdown` returns markdown for normal/trailing-slash/homepage routes, direct `.md` passes through, missing routes 404 cleanly, `Vary: Accept` set. - Production `llms.txt` ~31.6KB (under the 50KB threshold); local builds are larger only because they also include `dev` version routes. ## Notes - The HTML-minifier SSG warnings in the build log are pre-existing (SWC minifier from `future.faster`, mostly inline-SVG false positives) and unrelated to this change.
2 parents 403956b + 57021cc commit 5d7792b

10 files changed

Lines changed: 583 additions & 150 deletions

File tree

‎docs/docusaurus.config.js‎

Lines changed: 38 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,8 @@ function syncVersionsFromConfig(configFile, versionsFile, versionedDocsDir) {
2727
const configVersions = [
2828
...new Set(
2929
Object.values(config).filter(
30-
(v) => v && fs.existsSync(path.join(docsDir, `version-${v}`)),
31-
),
30+
(v) => v && fs.existsSync(path.join(docsDir, `version-${v}`))
31+
)
3232
),
3333
];
3434
const configVersionSet = new Set(Object.values(config).filter(Boolean));
@@ -41,23 +41,23 @@ function syncVersionsFromConfig(configFile, versionsFile, versionedDocsDir) {
4141
: [];
4242
fs.writeFileSync(
4343
path.join(__dirname, versionsFile),
44-
JSON.stringify([...configVersions, ...extraVersions], null, 2) + "\n",
44+
JSON.stringify([...configVersions, ...extraVersions], null, 2) + "\n"
4545
);
4646
return config;
4747
}
4848

4949
const developerVersionConfig = syncVersionsFromConfig(
5050
"./developer_version_config.json",
5151
"developer_versions.json",
52-
"developer_versioned_docs",
52+
"developer_versioned_docs"
5353
);
5454
const mainnetDeveloperVersion = developerVersionConfig.mainnet || null;
5555
const developerTestnetVersion = developerVersionConfig.testnet || null;
5656

5757
const networkVersionConfig = syncVersionsFromConfig(
5858
"./network_version_config.json",
5959
"network_versions.json",
60-
"network_versioned_docs",
60+
"network_versioned_docs"
6161
);
6262
const mainnetNetworkVersion = networkVersionConfig.mainnet || null;
6363
const testnetVersion = networkVersionConfig.testnet || null;
@@ -132,6 +132,11 @@ const config = {
132132
pages: {
133133
path: "src/pages",
134134
},
135+
// Keep utility routes out of the sitemap so they don't count against
136+
// llms.txt coverage (they are also excluded from the llms.txt index).
137+
sitemap: {
138+
ignorePatterns: ["/search", "/**/tags", "/**/tags/**"],
139+
},
135140
},
136141
],
137142
],
@@ -275,20 +280,36 @@ const config = {
275280
},
276281
],
277282
[
278-
"docusaurus-plugin-llms",
283+
"@signalwire/docusaurus-plugin-llms-txt",
279284
{
280-
generateLLMsTxt: true,
281-
generateLLMsFullTxt: true,
282-
docsDir: `developer_versioned_docs/version-${mainnetDeveloperVersion || developerTestnetVersion}`,
283-
title: "Aztec Protocol Documentation",
284-
excludeImports: true,
285-
version: mainnetDeveloperVersion || developerTestnetVersion,
286-
addMdExtension: false,
287-
pathTransformation: {
288-
ignorePaths: [
289-
`developer_versioned_docs/version-${mainnetDeveloperVersion || developerTestnetVersion}`,
285+
siteTitle: "Aztec Protocol Documentation",
286+
siteDescription:
287+
"Build private smart contracts on Ethereum's leading privacy-first L2 zkRollup.",
288+
content: {
289+
// Emit a .md sibling for every route so agents can fetch clean
290+
// markdown at PAGE.md, and a single-file llms-full.txt dump.
291+
enableMarkdownFiles: true,
292+
enableLlmsFullTxt: true,
293+
includeDocs: true,
294+
includePages: true,
295+
includeBlog: false,
296+
// In production the served docs ARE the versioned snapshots (the
297+
// current version is excluded), so they must be included or the index
298+
// covers nothing.
299+
includeVersionedDocs: true,
300+
// The auto-generated API reference (raw static HTML under
301+
// /aztec-nr-api and markdown under /typescript-api) is not part of
302+
// Docusaurus's routes. It is surfaced in llms.txt separately by
303+
// scripts/append_api_docs_to_llms.js, and deliberately kept out of the
304+
// sitemap, so exclude it here too. Utility routes are excluded for the
305+
// same reason.
306+
excludeRoutes: [
307+
"/search",
308+
"/**/tags",
309+
"/**/tags/**",
310+
"/aztec-nr-api/**",
311+
"/typescript-api/**",
290312
],
291-
addPaths: ["developers"],
292313
},
293314
},
294315
],

‎docs/netlify.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@
77
command = "yarn build"
88
functions = "netlify/functions"
99

10+
# The markdown content-negotiation edge function declares its own `path` via the
11+
# inline `config` export in netlify/edge-functions/markdown-negotiation.js, so it
12+
# does not need a [[edge_functions]] entry here (declaring both runs it twice).
13+
1014
# Install nargo only for preview deploys (not needed for production)
1115
[context.deploy-preview]
1216
command = "./scripts/netlify-preview-build.sh"

‎docs/netlify/edge-functions/markdown-negotiation.js‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
// Content negotiation for agents: when a request prefers markdown
2+
// (`Accept: text/markdown`), serve the pre-built .md sibling of the page that
3+
// the SignalWire llms-txt plugin emits at build time, instead of the HTML.
4+
//
5+
// Browsers send `Accept: text/html,...` and never `text/markdown`, so they fall
6+
// through to the normal HTML response untouched. Requests for assets (anything
7+
// with a file extension, including the .md files themselves) also fall through,
8+
// which both serves them directly and prevents this function from recursing on
9+
// the .md fetch below.
10+
11+
function prefersMarkdown(accept) {
12+
return accept.toLowerCase().includes("text/markdown");
13+
}
14+
15+
function toMarkdownPath(pathname) {
16+
let p = pathname;
17+
if (p.length > 1 && p.endsWith("/")) p = p.slice(0, -1);
18+
if (p === "" || p === "/") return "/index.md";
19+
return `${p}.md`;
20+
}
21+
22+
export default async (request, context) => {
23+
const accept = request.headers.get("accept") || "";
24+
if (!prefersMarkdown(accept)) return;
25+
26+
const url = new URL(request.url);
27+
28+
// Already a concrete file (asset, sitemap, llms.txt, or a .md page): let the
29+
// CDN serve it directly. This is also the recursion guard for the .md fetch
30+
// below. Match only known static-file extensions, not any trailing dot, so
31+
// doc routes whose last segment contains a dot (e.g. a changelog page like
32+
// ".../changelog/v2.0.2") still negotiate to their .md sibling.
33+
if (
34+
/\.(md|html?|xml|txt|json|js|mjs|cjs|map|css|png|jpe?g|gif|svg|webp|avif|ico|bmp|woff2?|ttf|otf|eot|pdf|zip|gz|wasm|mp4|webm|csv|ya?ml)$/i.test(
35+
url.pathname,
36+
)
37+
) {
38+
return;
39+
}
40+
41+
const markdownUrl = new URL(url.toString());
42+
markdownUrl.pathname = toMarkdownPath(url.pathname);
43+
44+
const markdown = await fetch(markdownUrl, {
45+
headers: { accept: "text/plain" },
46+
});
47+
if (!markdown.ok) return;
48+
49+
const headers = new Headers(markdown.headers);
50+
headers.set("content-type", "text/markdown; charset=utf-8");
51+
// Tell caches the representation depends on the Accept header so the markdown
52+
// response is never served to a browser asking for HTML.
53+
headers.set("vary", "Accept");
54+
55+
return new Response(markdown.body, { status: 200, headers });
56+
};
57+
58+
export const config = { path: "/*" };

‎docs/package.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
"version": "0.0.0",
44
"private": true,
55
"scripts": {
6-
"build": "yarn clean && yarn preprocess && yarn spellcheck && yarn preprocess:move && yarn validate:redirects && yarn validate:api-ref-links && yarn docusaurus build && node scripts/augment_sitemap.js && node scripts/append_api_docs_to_llms.js",
6+
"build": "yarn clean && yarn preprocess && yarn spellcheck && yarn preprocess:move && yarn validate:redirects && yarn validate:api-ref-links && yarn docusaurus build && node scripts/append_api_docs_to_llms.js && node scripts/inject_md_directive.js && node scripts/inject_llms_directive.js",
77
"validate:redirects": "./scripts/validate_redirect_targets.sh",
88
"validate:api-ref-links": "./scripts/validate_api_ref_links.sh",
99
"check:orphaned-urls": "./scripts/check_orphaned_urls.sh",
@@ -31,6 +31,7 @@
3131
"@docusaurus/theme-mermaid": "3.10.1",
3232
"@getbrevo/brevo": "^3.0.1",
3333
"@rspack/core": "^1.0.0",
34+
"@signalwire/docusaurus-plugin-llms-txt": "^1.2.2",
3435
"clsx": "^2.1.1",
3536
"docusaurus-theme-search-typesense": "0.25.0",
3637
"prism-react-renderer": "^2.4.1",
@@ -46,7 +47,6 @@
4647
"@docusaurus/tsconfig": "3.10.1",
4748
"@tsconfig/docusaurus": "^1.0.7",
4849
"cspell": "^8.19.4",
49-
"docusaurus-plugin-llms": "^0.4.0",
5050
"dotenv": "^16.6.1",
5151
"netlify-cli": "^24.3.0"
5252
},

‎docs/scripts/append_api_docs_to_llms.js‎

Lines changed: 36 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,8 @@ const path = require("path");
1515
const BUILD_DIR = path.join(__dirname, "..", "build");
1616
const STATIC_DIR = path.join(__dirname, "..", "static");
1717

18-
// Site URL used to build absolute links, matching the rest of llms.txt (which
19-
// docusaurus-plugin-llms generates with absolute URLs). Read from
20-
// docusaurus.config.js so it tracks the configured domain. Trailing slash
18+
// Site URL used to build absolute links for the API reference sections. Read
19+
// from docusaurus.config.js so it tracks the configured domain. Trailing slash
2120
// stripped so it can be concatenated with leading-slash paths.
2221
function loadSiteUrl() {
2322
try {
@@ -299,8 +298,9 @@ function findMarkdownFiles(dir) {
299298
}
300299

301300
/**
302-
* Get the absolute URL for a file, matching the absolute-URL style used by the
303-
* rest of llms.txt.
301+
* Get the absolute URL for an API reference file. API links stay absolute; the
302+
* page links the llms-txt plugin emits are root-relative. Both resolve on the
303+
* deployed site.
304304
*/
305305
function getUrlPath(filePath, staticDir) {
306306
const relativePath = path.relative(staticDir, filePath);
@@ -407,6 +407,23 @@ function insertAiToolingPointer(content) {
407407
return lines.join("\n").replace(/\n{3,}/g, "\n\n");
408408
}
409409

410+
/**
411+
* Truncate `content` at the earliest of the given sentinel strings, dropping it
412+
* and everything after. Returns the content unchanged when no sentinel is
413+
* present. Used to make the section appends below idempotent: a re-run strips
414+
* the sections a previous run added, then re-appends fresh ones, instead of
415+
* duplicating them. (`yarn build` always starts with `yarn clean`, so on the
416+
* normal path nothing is stripped; this only guards manual or repeated runs.)
417+
*/
418+
function stripFrom(content, ...sentinels) {
419+
let cut = -1;
420+
for (const sentinel of sentinels) {
421+
const i = content.indexOf(sentinel);
422+
if (i !== -1 && (cut === -1 || i < cut)) cut = i;
423+
}
424+
return cut === -1 ? content : content.slice(0, cut);
425+
}
426+
410427
/**
411428
* Main function to append API docs to llms.txt files.
412429
*/
@@ -425,6 +442,19 @@ function main() {
425442
? fs.readFileSync(llmsFullTxtPath, "utf-8")
426443
: "";
427444

445+
// Drop any sections a previous run of this script appended so re-running it
446+
// replaces rather than duplicates them (the top-of-file insertions below are
447+
// separately idempotent).
448+
llmsTxtContent = stripFrom(
449+
llmsTxtContent,
450+
"\n\n## API Reference\n\n",
451+
"\n\n## Optional\n\n",
452+
);
453+
llmsFullTxtContent = stripFrom(
454+
llmsFullTxtContent,
455+
"\n\n---\n\n## API Reference Documentation\n\n",
456+
);
457+
428458
// Point readers at the single-file dump up front. It sits right after the H1
429459
// title (and its optional blockquote summary) so an agent sees it before any
430460
// section, but stays out of the H1 to keep a single top-level heading.
@@ -435,7 +465,7 @@ function main() {
435465
let totalFiles = 0;
436466
let sectionsAdded = 0;
437467
// Per the llms.txt spec there is a single H1 (the project title, emitted by
438-
// docusaurus-plugin-llms). Everything we append is an H2 section. The
468+
// the llms-txt plugin). Everything we append is an H2 section. The
439469
// `## Optional` section has special meaning: its links may be skipped when a
440470
// shorter context is needed, so secondary community resources live there and
441471
// are ordered last. Community resources are independent of the generated API

‎docs/scripts/augment_sitemap.js‎

Lines changed: 0 additions & 103 deletions
This file was deleted.

0 commit comments

Comments
 (0)