Skip to content

Commit 200fe9c

Browse files
authored
docs: clarify ai api access instructions, integrate mcp and markdown pages (#5210)
1 parent e111990 commit 200fe9c

11 files changed

Lines changed: 56 additions & 40 deletions

File tree

fern/docs.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -523,6 +523,8 @@ redirects:
523523
destination: /learn/docs/ai-features/mcp-server
524524
- source: /learn/docs/seo/llms-txt
525525
destination: /learn/docs/ai-features/llms-txt
526+
- source: /learn/docs/developer-tools/view-markdown
527+
destination: /learn/docs/ai-features/markdown
526528
- source: /learn/docs/scribe-api/slack-scribe/get-scribe-slack-install-link
527529
destination: /learn/docs/ai-features/fern-writer
528530

fern/products/docs/docs.yml

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,9 @@ navigation:
178178
- page: AI-generated examples
179179
path: ./pages/ai/ai-examples.mdx
180180
slug: ai-examples
181+
- page: Markdown access
182+
path: ./pages/ai/markdown.mdx
183+
slug: markdown
181184
- section: llms-txt
182185
collapsed: open-by-default
183186
skip-slug: true
@@ -431,9 +434,7 @@ navigation:
431434
- page: GitLab
432435
path: ./pages/developer-tools/gitlab.mdx
433436
- page: Vale
434-
path: ./pages/developer-tools/vale.mdx
435-
- page: View Markdown
436-
path: ./pages/developer-tools/view-markdown.mdx
437+
path: ./pages/developer-tools/vale.mdx
437438
- page: Download OpenAPI spec
438439
path: ./pages/developer-tools/openapi-spec.mdx
439440
slug: openapi-spec

fern/products/docs/pages/ai/llms-txt/customize-llms-txt.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Within pages, use `<llms-only>` and `<llms-ignore>` tags to control what content
1111

1212
### Content for AI only
1313

14-
`<llms-only>` content appears only on the LLM-serving endpoints that external agents (like Claude, ChatGPT, or Cursor) fetch directly: `/llms.txt`, `/llms-full.txt`, and each page's [`.md`/`.mdx` source](/learn/docs/developer-tools/view-markdown). It's hidden from every human-facing surface, including the rendered page, [Copy page](/learn/docs/configuration/site-level-settings#page-actions-configuration), the search widget, and [Ask Fern](/learn/docs/ai-features/ask-fern/overview).
14+
`<llms-only>` content appears only on the LLM-serving endpoints that external agents (like Claude, ChatGPT, or Cursor) fetch directly: `/llms.txt`, `/llms-full.txt`, and each page's [`.md`/`.mdx` source](/learn/docs/ai-features/markdown). It's hidden from every human-facing surface, including the rendered page, [Copy page](/learn/docs/configuration/site-level-settings#page-actions-configuration), the search widget, and [Ask Fern](/learn/docs/ai-features/ask-fern/overview).
1515

1616
Use `<llms-only>` for:
1717
- Technical context that's verbose but helpful for AI, like implementation details or architecture notes
@@ -79,7 +79,7 @@ On the docs site, human readers see only the numbered UI steps — the `<llms-on
7979
8080
The guiding principle: **UI-only elements belong to human readers, and their programmatic equivalents belong to AI agents.** Wrap clickable cards and UI walkthroughs in `<llms-ignore>` so agents skip them, and pair each UI step with an `<llms-only>` block that gives the CLI or API equivalent.
8181
82-
To preview and debug what AI sees for any page, you can append `.md` or `.mdx` to the page URL to [view its Markdown source](/learn/docs/developer-tools/view-markdown).
82+
To preview and debug what AI sees for any page, you can append `.md` or `.mdx` to the page URL to [view its Markdown source](/learn/docs/ai-features/markdown).
8383
8484
## Filter endpoint output
8585

fern/products/docs/pages/ai/llms-txt/overview.mdx

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,7 @@ title: "`llms.txt` and `llms-full.txt`"
33
description: Fern automatically generates llms.txt and llms-full.txt Markdown files so AI tools can discover and index your documentation.
44
---
55

6-
[`llms.txt`](https://llmstxt.org/) is a standard for exposing website content to AI developer tools. Fern implements this standard, automatically generating and maintaining `llms.txt` and `llms-full.txt` Markdown files so AI tools can discover and index your documentation.
7-
8-
Fern also serves [clean Markdown instead of HTML](https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation) on any page URL when the request includes an `Accept: text/markdown` header, ensuring agents only receive the content they need. Together, these features reduce token consumption by 90%+.
6+
[`llms.txt`](https://llmstxt.org/) is a standard for exposing website content to AI developer tools. Fern implements this standard, automatically generating and maintaining `llms.txt` and `llms-full.txt` Markdown files so AI tools can discover and index your documentation. For single pages, agents can also fetch [Markdown directly](/learn/docs/ai-features/markdown).
97

108
<Frame>
119
<img
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
title: Markdown access
3+
description: Access your documentation as Markdown — through per-page URLs, `llms.txt`, and `llms-full.txt` — for AI agents and tooling.
4+
---
5+
6+
7+
Fern serves clean Markdown for any documentation page — including API Reference pages — so agents can consume your content efficiently. Agents fetch the source by appending `.md` or `.mdx` to a page URL, or by sending an `Accept: text/markdown` header via [content negotiation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation). Combined with [`llms.txt`](/learn/docs/ai-features/llms-txt), this reduces token consumption by 90%+ compared to HTML.
8+
9+
<Frame caption="For example, `https://buildwithfern.com/learn/docs/ai-features/markdown.md` displays the Markdown source for this page.">
10+
<img src="./markdown.png"
11+
alt="Example showing a page's underlying Markdown"
12+
className="rounded-lg border border-gray-200 dark:border-gray-700"
13+
/>
14+
</Frame>
15+
16+
The same Markdown is used everywhere — single pages, `llms.txt`, and `llms-full.txt` — and respects the same `<llms-only>` and `<llms-ignore>` content controls.
17+
18+
A [default per-page directive](/learn/docs/ai-features/agent-directives) is automatically prepended to every page's Markdown output when served to AI agents, pointing them to your `.md` URLs, `llms.txt`, and `llms-full.txt`. You can [override or disable this directive](/learn/docs/configuration/site-level-settings#agents-configuration) in `docs.yml`. The directive is only visible to agents — human-facing documentation is unaffected.
19+
20+
## Accessing protected docs
21+
22+
On sites with [authentication](/learn/docs/authentication/overview) enabled, agents must include a JWT on every Markdown request — whether for an individual page, `llms.txt`, or `llms-full.txt`. Exchange your [Fern token](/learn/cli-api/cli-reference/commands#fern-token) for a [JWT](/learn/docs/fern-api-reference/get-jwt):
23+
24+
```bash Get a JWT
25+
curl https://docs.example.com/api/fern-docs/get-jwt \
26+
-H "FERN_API_KEY: $FERN_TOKEN"
27+
# → { "fern_token": "eyJ...", "roles": [] }
28+
```
29+
30+
Send the returned JWT as the `FERN_TOKEN` header on subsequent requests:
31+
32+
```bash Fetch protected content
33+
curl https://docs.example.com/platform/overview \
34+
-H 'Accept: text/markdown' \
35+
-H "FERN_TOKEN: $JWT"
36+
```
37+
38+
JWTs are valid for 30 days — cache and refresh as needed.
39+
40+
## Markdown for troubleshooting
41+
42+
Viewing the Markdown directly is also useful for troubleshooting layout problems. A **View as Markdown** button is enabled by default on every page and can be configured through the [page actions configuration](/learn/docs/configuration/site-level-settings#page-actions-configuration).
458 KB
Loading

fern/products/docs/pages/ai/mcp-server.mdx

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,15 +16,7 @@ Your MCP server is available at `your-documentation-site.com/_mcp/server`. For e
1616

1717
Both buttons are enabled by default on sites with Ask Fern. For other clients (Claude Desktop, Windsurf, etc.), users can add `your-documentation-site.com/_mcp/server` to their MCP configuration.
1818

19-
## Direct API access for AI agents
19+
## Other ways agents can access your docs
2020

21-
AI agents can access your documentation directly via HTTP. Fern serves content as clean Markdown for token efficiency.
21+
In addition to MCP, agents can fetch documentation directly over HTTP. Fern serves clean Markdown via [per-page URLs, `llms.txt`, and `llms-full.txt`](/learn/docs/ai-features/markdown) — including on authenticated sites.
2222

23-
For authenticated sites, agents obtain a JWT via [`/api/fern-docs/get-jwt`](/learn/docs/fern-api-reference/get-jwt) with a Fern API key to access protected documentation content and search functionality.
24-
25-
```bash Access protected content
26-
curl https://docs.example.com/platform/overview \
27-
-H 'Accept: text/plain' \
28-
-H 'x-fern-host: docs.example.com' \
29-
-H 'FERN_TOKEN: your-jwt-here'
30-
```

fern/products/docs/pages/ai/overview.mdx

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,8 @@ AI helps keep your documentation current. Fern Writer is a Slack-based technical
4242

4343
Your site is automatically optimized for AI tools and search engines. Fern hosts `llms.txt` and `llms-full.txt` files so LLMs can index your documentation efficiently, and serves Markdown instead of HTML to AI agents. These features reduce token consumption and help agents process your content faster.
4444

45-
<CardGroup cols={2}>
45+
<CardGroup cols={3}>
46+
<Card title="Markdown access" icon="file-code" href="/learn/docs/ai-features/markdown" />
4647
<Card title="`llms.txt`" icon="file-text" href="/learn/docs/ai-features/llms-txt" />
4748
<Card title="Agent directives" icon="compass" href="/learn/docs/ai-features/agent-directives" />
4849
</CardGroup>

fern/products/docs/pages/developer-tools/view-markdown.mdx

Lines changed: 0 additions & 20 deletions
This file was deleted.
-342 KB
Binary file not shown.

0 commit comments

Comments
 (0)