Skip to content

Commit ff151bf

Browse files
sriramveeraghantasriramveeraghanta
committed
docs(spec): add agent discovery design
Brainstormed spec for three agent-discovery fixes: Content-Signal in robots.txt, Link response headers, and markdown content negotiation. Ephemeral — to be removed after implementation.
1 parent 04c50cc commit ff151bf

1 file changed

Lines changed: 137 additions & 0 deletions

File tree

‎superpowers/specs/2026-04-20-agent-discovery-design.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Agent Discovery Improvements — Design
2+
3+
**Date:** 2026-04-20
4+
**Status:** Draft — pending user review
5+
**Site:** developers.plane.so (VitePress static site on Vercel)
6+
7+
## Goal
8+
9+
Improve agent/AI-crawler discovery on the Plane developer docs by addressing three gaps flagged by an external audit:
10+
11+
1. No `Link` response headers advertising useful resources (RFC 8288).
12+
2. No `Content-Type: text/markdown` negotiation for agents that prefer markdown.
13+
3. No `Content-Signal` directives in `robots.txt` declaring AI usage preferences.
14+
15+
## Non-Goals
16+
17+
- Creating an OpenAPI spec or `/.well-known/api-catalog` (none exists today).
18+
- Flattening or cleaning VitePress custom components (`<ApiParam>`, `<CodePanel>`, etc.) in the markdown output. Agents receive source markdown with those tags present.
19+
- Changes to sidebar, navigation, or published content.
20+
- Edge-runtime logic. The design stays fully static.
21+
22+
## Current State
23+
24+
- VitePress static site with source in `docs/`, build output in `docs/.vitepress/dist/`.
25+
- Deployed on Vercel; config in `vercel.json` (already has `cleanUrls: true` and a `redirects` block).
26+
- `docs/public/robots.txt` exists and declares a sitemap and a (non-standard) `LLMs-txt:` line.
27+
- `docs/public/llms.txt` and `docs/public/llms-full.txt` already exist.
28+
- No `/.well-known/` directory. No OpenAPI file.
29+
30+
## Design
31+
32+
### Fix 1 — Content Signals in `robots.txt`
33+
34+
Append a single directive to `docs/public/robots.txt`:
35+
36+
```
37+
Content-Signal: search=yes, ai-train=yes, ai-input=yes
38+
```
39+
40+
**Policy rationale:** Plane's existing posture (publishing `llms.txt` and `llms-full.txt`) is maximally open. The Content Signal matches that — search indexing, model training, and AI-answer grounding are all permitted.
41+
42+
Placement: directly below the `User-agent: *` / `Allow:` block, before the `Sitemap:` line. One line, no other changes.
43+
44+
### Fix 2 — `Link` response headers (RFC 8288)
45+
46+
Add a `headers` block to `vercel.json` applied to all paths. A single `Link` header carries three relations, comma-separated per RFC 8288 §3.5:
47+
48+
```
49+
Link: </llms.txt>; rel="describedby"; type="text/plain",
50+
</api-reference/introduction>; rel="service-doc"; type="text/html",
51+
</sitemap.xml>; rel="sitemap"; type="application/xml"
52+
```
53+
54+
**Why each relation:**
55+
- `describedby` → `/llms.txt` — points agents to the LLM-friendly site map (canonical discovery).
56+
- `service-doc` → `/api-reference/introduction` — registered IANA relation for human-readable API docs.
57+
- `sitemap` → `/sitemap.xml` — widely supported, de facto standard.
58+
59+
Applied to `source: "/(.*)"` so the headers land on every page, not just the homepage. This lets any deep-link entry point still expose discovery metadata.
60+
61+
### Fix 3 — Markdown content negotiation (static approach)
62+
63+
Two pieces, both in files already owned by this repo:
64+
65+
**3a. Build hook in `docs/.vitepress/config.mts`**
66+
67+
Add a `buildEnd(siteConfig)` function that copies every source `.md` file from `siteConfig.srcDir` to the corresponding path under `siteConfig.outDir`, preserving directory structure. The hook:
68+
69+
- Walks `srcDir` recursively.
70+
- Skips anything under `.vitepress/` (config/theme/cache/dist).
71+
- For each `.md` file, computes the relative path and writes it into `outDir` at the same relative location, creating intermediate directories as needed.
72+
- Preserves file contents byte-for-byte. Frontmatter and VitePress custom components (`<ApiParam>`, `<CodePanel>`, etc.) are included verbatim — acknowledged limitation.
73+
74+
Result: after `pnpm build`, `dist/api-reference/introduction.md` exists alongside `dist/api-reference/introduction.html`.
75+
76+
**3b. Vercel rewrite in `vercel.json`**
77+
78+
Add a `rewrites` block that matches on the `Accept` header:
79+
80+
```json
81+
{
82+
"rewrites": [
83+
{
84+
"source": "/:path*",
85+
"has": [{ "type": "header", "key": "accept", "value": ".*text/markdown.*" }],
86+
"destination": "/:path*.md"
87+
}
88+
]
89+
}
90+
```
91+
92+
Vercel's default MIME mapping serves `.md` files as `text/markdown; charset=utf-8`. If preview-deployment testing shows a different `Content-Type` (e.g., `text/plain`), add an explicit header override to `vercel.json` for `source: "/(.*).md"` setting `Content-Type: text/markdown; charset=utf-8`. This is the only identified contingency.
93+
94+
Interaction with existing `cleanUrls: true`: `cleanUrls` strips `.html` from URLs; the `.md` rewrite only fires when the `Accept` header matches, so browsers still get HTML via the default behavior. The two directives don't conflict — `rewrites` run before `cleanUrls` resolution.
95+
96+
### Known Limitations (accepted)
97+
98+
- **Raw frontmatter in markdown responses.** Agents will see the YAML frontmatter block at the top of each page. Most agent parsers tolerate this.
99+
- **VitePress components in markdown responses.** Pages using `<ApiParam>`, `<CodePanel>`, `<ResponsePanel>`, `<Card>`, `<CardGroup>` will return those tags literally. Prose content is still machine-readable; component-rendered detail (e.g., parameter type badges) is not flattened.
100+
- **No content-type validation at build time.** If a page has no source `.md` (unlikely in VitePress), the rewrite returns a 404; the HTML version remains available for normal requests.
101+
102+
## File Changes (summary)
103+
104+
| File | Change |
105+
| ---------------------------------------- | ------------------------------------------------------------------ |
106+
| `docs/public/robots.txt` | Add one `Content-Signal:` line |
107+
| `vercel.json` | Add `headers` block (Fix 2) and `rewrites` block (Fix 3b) |
108+
| `docs/.vitepress/config.mts` | Add `buildEnd` hook that copies `.md` files into `dist` (Fix 3a) |
109+
110+
No new dependencies. No new directories.
111+
112+
## Testing
113+
114+
**Local (before merge):**
115+
116+
- `pnpm build` — verify no errors, then check that `docs/.vitepress/dist/` contains `.md` copies (spot-check `dist/api-reference/introduction.md`).
117+
- `pnpm check:format` — Prettier must pass.
118+
119+
**On Vercel preview deployment:**
120+
121+
- `curl -I https://<preview>.vercel.app/` — verify the `Link:` response header is present and contains all three relations.
122+
- `curl -H "Accept: text/markdown" https://<preview>.vercel.app/api-reference/introduction` — verify the body is markdown and `Content-Type: text/markdown; charset=utf-8`.
123+
- `curl https://<preview>.vercel.app/` (no custom Accept) — verify HTML is still returned (regression check).
124+
- `curl https://<preview>.vercel.app/robots.txt | grep Content-Signal` — verify the directive is present.
125+
126+
**Rationale for preview-only testing:** `pnpm preview` serves VitePress's built output directly and does not run Vercel's `rewrites` / `headers` engine. The header negotiation can only be validated on a Vercel deployment.
127+
128+
## Rollout
129+
130+
1. Implement all three fixes in a single PR branch (`feat/agent-discovery`).
131+
2. Push, let Vercel preview build, run the `curl` checks above against the preview URL.
132+
3. Merge to `master`; Vercel promotes to production.
133+
4. Re-run `isitagentready.com` (or the source audit) against production to confirm the three issues are resolved.
134+
135+
## Post-Implementation
136+
137+
Once the implementation PR is merged and verified in production, **delete the `superpowers/` directory** from the repo — the spec is ephemeral and does not belong in the long-lived tree.

0 commit comments

Comments
 (0)