Consider removing llms.txt files

[delucis]

In #11439 (April, 2025) we added the https://github.com/delucis/starlight-llms-txt plugin to build llms.txt and associated files because we considered this to be a potentially helpful resource, as proposed for adoption by https://llmstxt.org/.

I’m not sure this has panned out. Looking at data we have for the last 7 days, requests for files like this made up <0.1% of our traffic. There’s still no clear indication that major AI tools actually uses llms.txt files or that they have any value.¹²³ This is born out also by our attempts to gather any kind of feedback about it. Even people who were in favour of adding llms.txt to our docs weren’t able to provide much feedback that it was helpful and we don’t really have evidence of anyone using it.

Looking at some recent build logs, generating these files took ~23 seconds. That’s not a huge absolute number, but it’s still 10% of that build’s total, so significant. Even over a relatively quiet month like December, it likely accounted for ~2.5 hours of CI time (and we build both on GitHub Actions and Netlify, so that is duplicated across those.)

There is also maintenance overhead. We should in theory be keeping our custom llms.txt configuration in mind as we add content to the documentation site to make sure content is correctly categorised. I don’t think we currently do particularly, which speaks to our doubts about the value of the files.⁴ The other side of the maintenance is maintaining the Starlight plugin at all. I keep it ticking over primarily because we’re using it in docs, but would happily deprecate it or hand it off to another maintainer if we were no longer using it here.

In summary:

• llms.txt files are not used by AI tools
• llms.txt files are rarely requested from our docs site
• llms.txt files provide no value as far as we can tell
• llms.txt files increase deploy times
• llms.txt files add maintenance burden

Given the above, I’d like to propose we remove the plugin that generates llms.txt and update the “Building Astro sites with AI tools” page to reflect that.

That said, I’m open to hearing why these files are useful and are providing value. We just need some evidence for that.

Footnotes

1. Tushar Pol, “What Is LLMs.txt & Should You Use It?”, Semrush Blog (November 26, 2025) ↩

2. Ryan Law, “What Is llms.txt, and Should You Care About It?”, ahrefs blog (April 22, 2025) ↩

3. “The Case Against llms.txt: Why the Hype Outpaces the Reality”, daydream blog (June 5, 2025) ↩

4. https://github.com/withastro/docs/issues/13007 ↩

[florian-lefebvre]

For the record I made this PR because I wanted to build a mcp on top of it, but Matt ended up using kapa ai. So +1 for removing it

[Adammatthiesen]

I've had success with it and the MCP tied together. Helps prevent any weird hallucinations

[alvinometric]

I've had success with it, but the biggest win for me is being able to copy docs as markdown (hence my enthusiasm for this delucis/starlight-llms-txt#32) you can see it in action on the Expo docs

[ascorbic]

I agree. I think the best replacement would be markdown versions of pages. We could then replace llms.txt with a static file that just links to the top level nav items

[theurgi]

Thanks @ArmandPhilippot for bringing this to my attention and inviting feedback. I'm evidently part of the small minority that uses Astro's llms.txt endpoints, and I appreciate the request for input before discontinuing them.

I think Astro should do whatever's in its best interest. I agree that it doesn't make sense to retain the build task in its current form when comparing the cost to utilization. That said, I'll discuss some considerations that may be relevant to this decision, share how I use llms.txt, and propose what I think would be closer to an ideal solution.

Two Different Use Cases

The referenced articles evaluate llms.txt against a specific purpose: SEO and crawler visibility. The authors—Semrush, Daydream, and Ahrefs—are SEO-focused companies assessing whether llms.txt influences LLM training crawlers or improves "AI rankings". Consistent with their findings, it does appear that no major LLM provider has confirmed they use llms.txt files.

However, there's a distinct second use case: RAG for LLM-assisted code generation. This is what Astro's own docs describe:

This guide covers how to enhance AI tools with up-to-date Astro knowledge... Astro provides llms.txt and llms-full.txt files that contain the full docs content in a format optimized for AI consumption.

The conclusions "llms.txt files are not used by AI tools" and "provide no value" apply to crawler/training use cases. They don't address whether the files are useful for developers grounding their coding agents in current documentation.

Comparing the Alternatives

One way or another, LLMs need current context to generate correct code. The current alternatives to llms.txt are:

• HTML docs directly: Wastes tokens and pollutes the context window
• Third-party scraping services: Exa, Jina, Firecrawl, etc. use LLMs to strip HTML back down to text: markdown → HTML → markdown again
• Copy-paste into chat apps: Clunky and doesn't scale

For developer documentation—which is almost always authored in markdown—the current ecosystem often looks like:

Developers write MDX for documentation frameworks
        ↓
Frameworks build static sites with navigation, styling, components
        ↓
Scraper services use LLMs to strip it back down to plain text
        ↓
LLMs consume what's left

llms.txt short-circuits this unnecessary round-trip.

How I use llms.txt

I use llms.txt to provide local, authoritative context for LLM code generation.

My workflow:

1. Fetch all llms.txt endpoints (excluding llms-full.txt)
2. Split content by # headings into a local directory structure
3. Reference the main index in my AGENTS.md file
4. Instruct the agent to search these docs when needed

Here's the script I use for steps 1 and 2.

The result is a set of semantically organized, token-efficient "chunks" that can be progressively disclosed as needed.

For example:

docs/astro/
├── index.md                    # The main index
├── additional-guides/
│   ├── index.md                # Every section has an index
│   ├── authentication.md
│   ├── cloudinary-astro.md
│   ├── e-commerce.md
│   └── ...
├── api-reference/
│   └── ...
├── backend-services/
│   └── ...
├── build-a-blog-tutorial/
│   └── ...
├── cms-guides/
│   └── ...
├── core-documentation/ (Abridged documentation)
│   └── ...
├── deployment-guides/
│   └── ...
├── how-to-recipes/
│   └── ...
└── migration-guides/

This approach aligns with the Agent Skills protocol: contextually relevant files organized in folders, accessed on demand.

This approach has concrete benefits:

• Explicit context: I know exactly what the agent is working with
• Offline capability: No network dependency during generation
• No third-party intermediaries: No additional API calls, no data leaving my machine
• Learning benefit: Overseeing context retrieval from a structurally identical copy of the docs helps me learn the framework faster

Why Usage Is Low

I suspect low adoption stems from two factors:

1. Most developers don't know about these files or how to use them effectively
2. Most are using third-party solutions, likely remote MCP servers (Kapa, Exa, etc.)

These services are convenient, but they introduce dependencies, network latency, and security considerations that some of us prefer to avoid. For open-source projects especially, having documentation RAG depend on closed-source MCP servers feels like a step backward.

Optimizing Build Times

23 seconds does seem long for what's essentially MDX → Markdown.

Does the Starlight plugin inherit overhead from the Astro build pipeline? This might include module resolution, AST parsing, schema validation, component rendering, and similar processes.

Have you considered a dedicated script that just strips JSX tags and copies the markdown content directly? If parallelized across files you could probably achieve sub-second builds while preserving the same output.

Closer to an Ideal Solution

I wouldn't say that llms.txt in its current form is the solution.

What makes the most sense to me at this point is author-published LLM-optimized docs as a package.

I recently encountered an example of this in the form of Effect best-practices documentation: Effect Solutions

These docs are also available from the command line, which is especially useful for your agent.

This is a really cool idea.

But given the simplicity and wide adoption of Agent Skills, I think it makes more sense to have a standardized skill for doc retrieval that all documentation authors can adhere to.

This could look something like:

doc-search/
├── SKILL.md              # Skill instructions
├── docs/
│   ├── astro/
│   │   ├── index.md      # Entry point / overview
│   │   ├── api/
│   │   └── guides/
│   ├── preact/
│   └── turso/
└── registry.json         # Index of installed docs

Developers locally install what they need just like we do with source code.

I welcome any feedback on this.

In Summary

If the maintenance burden and build time aren't justifiable for <0.1% of traffic, I completely understand the decision. If the endpoints are discontinued, I'll likely create a script similar to the one I shared to source the MDX docs directly from GitHub.

[ascorbic]

Would the suggested option of markdown versions of individual pages do the job here? Maybe that could be used to generate a zipfile of all of them

[theurgi]

For my needs, the simple answer is yes.

The only difference I see between a zip of markdown files and llms.txt is the distribution method. Between these, I'm impartial.

How do you propose to efficiently generate the markdown files from the astro/mdx source? This seems to be the heart of this issue for Astro.

[alvinometric]

@theurgi it's been ready for a while, just not released delucis/starlight-llms-txt#32

[delucis]

Thanks for the detailed write-up @theurgi.

Does the Starlight plugin inherit overhead from the Astro build pipeline?

Yes, it renders to HTML then back to Markdown to ensure no content gets lost (some stuff in docs is can be written in components and e.g. leaving just <ImportantInfo /> in output would not be informative, it needs to be rendered to extract useful text content), so that’s fairly resource intensive basically rendering the entire English-language docs to HTML and back again.

The only difference I see between a zip of markdown files and llms.txt is the distribution method. Between these, I'm impartial.

Is there a reason not just to grab the source code from GitHub? I’d guess you could also go to https://github.com/withastro/docs/tree/main/src/content/docs/en and grab all the files from there?

You can even get a dedicated link for doing so: https://download-directory.github.io/?url=https%3A%2F%2Fgithub.com%2Fwithastro%2Fdocs%2Ftree%2Fmain%2Fsrc%2Fcontent%2Fdocs%2Fen (the Refined GitHub extension adds a link for this to the GitHub UI)

[theurgi]

@alvinometric So you're essentially saying that if individual .md files are accessible at the same URL path with .md appended, then more people would use them and Astro's build cost would be justified? Because this looks like it will have a similar build cost, perhaps more.

Is there a reason not just to grab the source code from GitHub?

This would be definitely be my solution if llms.txt was discontinued. SOTA LLMs can clearly infer the relevant context from the markup, but the reduction in accuracy, especially when accounting for the entire context window, is difficult to quantify. It's token inefficient in any case, for sure.

Right now I maintain scripts to fetch docs for Bun, Effect, and Astro. Each is different and I'll likely create more for additional dependencies.

And since I don't foresee LLMs going away or not needing to be augmented with current info (though I could be wrong), it makes sense to me for docs to be distributed like source-code is. Instead of developers having to juggle dozens of scripts or MCP plugins to retrieve docs published in a variety of formats, ideally there would be one standard that doc authors could conform to.

Developers could then simply do something like...

pm add @llm-docs/{astro,react,supabase,...}

...and the docs would be available locally for agents in an optimal form. Around such a standard, I would also expect an ecosystem of high-quality, drop-in tools to emerge: doc conversion, semantic search, etc. It's already the case that context is increasingly treated as source (Agent Skills) so there seems to be some inevitably about this.

I know this may exceed the scope of Astro's concern for this particular issue, but I haven't had the occasion to discuss this yet so it seemed like a good opportunity gather feedback from those confronting similar issues.

However, don't let me distract you! If clean markdown remains a desired end, and I were in your shoes, I'd probably have a hack at some Rust based regex solution. I don't know how well it would go, but hopefully I'd at least learn something in the process. 😅

Otherwise, if most Astro developers are happy using the MCP server, then I'd say just scrap the llms.txt build. Worst case, people who were using it will then speak up for themselves and you could go from there.

[theurgi]

Another two cents...

One feature that the .md endpoints could enable:

I suspect this would see a lot more use than the current llms.txt implementation on account of its visibility. I imagine that Starlight users would appreciate this capability as well.

This would push me in the direction of exploring the feasibility of a more efficient, perhaps auxiliary, build process if an increase in usage were not justification enough.

[ArmandPhilippot]

Thanks for the ideas, and the detailed write-up before! The issue you opened arrived at just the right time! 😄

There is a third party Starlight plugin for that https://github.com/corsfix/starlight-contextual-menu Maybe this doesn't include everything from the screenshot but I think it makes sense to have that as a plugin rather than built-in in Starlight.

Now, regarding Astro Docs, I'm not in the best position to judge because I don't use any of that and I'm not sure what people expect regarding AI. Maybe having this plugin would be more helpful than the llms.txt one... if this doesn't slow down more the build.

All I know is that people seem to be using MCP servers more than llms.txt files these days (no study, just an observation, I could be wrong) and so the build time might not be justified to satisfy only a minority.

[theurgi]

It looks like starlight-contextual-menu depends on starlight-markdown to do the the astro/mdx conversion, which is just this:

export async function GET({ params }) {
  const { path } = params

  const doc = await getEntry('docs', path || 'index')
  const markdown = `# ${doc.data.title}\n\n` + doc.body

  return new Response(markdown, {
    headers: {
      'content-type': 'text/markdown; charset=utf-8',
    },
  })
}

Correct me if I'm wrong, but this is not actually doing the expensive markdown conversion that starlight-llms-txt does, it's just concatenating a title to the MDX source. The results will still contain MDX and Astro components if they exist. So if the aim is to provide LLM-optimized, clean markdown docs, this doesn't get us any closer.

The emphasis of llms.txt as I understand it, even with the per-page .md endpoint implementation, is to provide LLMs with up-to-date, token/context efficient documentation. starlight-llms-txt does this, but it's expensive. A much more efficient solution is possible, but it might require a different approach than plugging into Astro's markdown pipeline.

All I know is that people seem to be using MCP servers more than llms.txt files these days

This is almost certainly true. The majority are very likely prioritizing convenience above computational-efficiency, security, and open-source. I'm not surprised by this, it's what I would expect in the current climate. However, I believe it's a technically inferior solution which has negative ramifications for the future of the open web.

But as I said upfront: I think Astro should do whatever it deems to be in its best interest. If providing clean markdown docs is not a top priority at the moment, I completely understand. Astro is an awesome framework with some of the best documentation I am aware of thanks to all of you. I trust you all know what's best for Astro.

[ArmandPhilippot]

Ah, I didn't checked how this was implemented. I thought you meant something like the contextual menu could be useful and it reminded me of that plugin. So, yes, maybe not the best implementation...

[theurgi]

LOL. Looks like Astro has compute on tap now. Congrats! 🎉

[delucis]

I revisited our numbers and worked out that our llms.txt files usage is <0.1% of our MCP server usage, which suggests it really does get very little usage. More details: #13538 (comment)

I’d like to push for us to merge #13538 removing the resource.

Thanks to everyone here who provided helpful feedback and apologies in advance to anyone who was relying on this. We will likely still revisit better AI support in the future, for example to add Markdown content for each individual page as suggested above.