feat: Add llms-full.txt

[elnelson575]

Closes #363

Auto-generated llms-full.txt following the llmstxt.org spec, so LLMs can discover and consume Shiny for Python documentation.

Key decisions

• llms-full.txt (572 KB full content) — includes cleaned page content for RAG/coding assistant use, matching the pattern used by Pydantic and others
• Content scope: Get Started, Concepts, Components (with examples), Layouts, API Reference, Templates. Gallery excluded (most of these are links to other repos or content made by others)
• Deterministic generation — output is a pure function of source files; only changed pages produce diffs
• CI freshness check — build fails with instructions if files are out of date
• _quarto.yml is the source of truth for site structure and page ordering; templates discovered by directory scan (not in sidebars)
• QMD cleaning strips frontmatter, Quarto directives, raw HTML, shinylive metadata; preserves prose and code blocks
• Page descriptions use title only (86/119 pages lack descriptions in frontmatter)
• @dataclass for site structure types — auto-generates __init__, __repr__, and structural __eq__; the last is essential for test assertions like assert entries == [SidebarEntry(...)]. NamedTuple was the alternative but feels semantically odd for types with mutable list fields.

Usage

make llms-full-txt          # generate both files
make all               # includes llms-txt in full build

Test plan

• 38 unit tests pass (python3 -m pytest tests/test_generate_llms_txt.py)
• Verify llms-full.txt has cleaned content (no :::, no frontmatter, no raw HTML)
• Verify determinism (run twice, diff is empty)
• Verify CI check catches stale files (https://github.com/posit-dev/py-shiny-site/actions/runs/23818115070/job/69423076433?pr=364)

---

Implementation plan

File Structure

File	Action	Responsibility
scripts/generate_llms_txt.py	Create	Main generation script — all logic lives here
tests/test_generate_llms_txt.py	Create	Unit tests for parsing and cleaning functions
Makefile	Modify	Add target, update all
_quarto.yml	Modify	Addllms-full.txt to resources (llms.txt is generated live from quarto from previous PR
.github/workflows/deploy-docs.yml	Modify	Add CI freshness check
llms-full.txt	Create (generated)	Full content file

[cpsievert]

I think it probably makes sense for this to come earlier in the workflow. In particular, make all takes a long time -- It would be frustrating to wait for the site to be built only for it to err.

[cpsievert]

It'd be helpful to have a docstring at the top of the file here to explain the intention behind the tests, when to run them, etc.

[cpsievert]

Do we not also want to run the tests after generation?

[gadenbuie]

To me, llms-full.txt seems like more of a build artifact than a source artifact. I'd personally prefer that it be built at render time and not checked into source.

It's not that I don't see the utility in tracking changes. If we were using a gh-pages branch, we'd have a copy of changes, which is nice.

But I have a strong aversion to having CI fail for things that could be automated. I have come to accept that we do this for build assets for reasons but it's not a pattern I'd recommend we follow for this file in particular.

[gadenbuie]

This whole section surprised me a bit. I would have expected a section like this to include code snippets showing the actual code to create the inputs.

Also the links seem reasonable, but they point to non-LLM-oriented pages. If we're making this whole system of documentation easier to navigate for LLMs, I'd think we'd prefer to link to LLM-ready pages.

I'm not saying that we have to have an LLM-ready markdown file for py/components/inputs/action-button/, but given how little other information is included in this section, we're basically telling the model to go read that file. So there's an asymmetry here.

[gadenbuie]

Did you look into using pandoc to handle this conversion? Quarto's markdown is really just pandoc markdown, and pandoc can handle moving between markdown variants.

It's more work, but I think much of this could be done through a Lua filter if the goal is to suppress Quarto-specific elements. Just in general, that stack is much better set up to handle parsing and rewriting markdown.

Frankly, I'm confident this function works right now but I wouldn't want to maintain it at all. Claude changes the calculus and all, but my larger worry than this particular function is that this code blends low-importance mechanical transformations with high-level assumptions about how we've laid out our docs, etc.

Relatedly, did you look into using something like markdownify or markdownify-rs? They work on HTML, but if this code moved away from dealing with the low-level details of parsing our source content I think it'd be more maintainable for us in the long run (not to mention more repeatable if we do something like this in other repos).

---

At one more level of abstraction higher, if we feel that this overall process is important I think it's worth looking into how to do this inside of Quarto. This kind of extension would be useful anywhere else in the ecosystem, like in quartodoc or Great Docs, and it'd probably be a lot easier to write and maintain if it were closer to the Quarto toolchain (i.e. pandoc and Lua filters).

OTOH, if we're just trying to get this done it seems like using something else that handles the HTML parsing for us would be beneficial even if the output isn't as tight as we'd like.