[TASK] Add llms.txt for AI documentation access

[NeoBlack]

Add an llms.txt file to provide guidance for AI systems and LLM-based documentation consumers.

• Define preferred SingleHTML documentation resources
• Provide guidance for selecting version-specific documentation
• Describe TYPO3 Core, system extension, library, and third-party documentation
• Define source priorities and crawling recommendations
• Document responsible usage and licensing requirements
• Reference robots.txt as the authoritative crawling policy

[CybotTM]

Something like:

Use the TYPO3 and extension versions found in the project's composer.lock or extensions composer.json

may be helpful.

[CybotTM]

llms.txt should mainly focus on generic rules, and reference to a catalog.json file for detailed versions and urls/pathes aso, which may be a static file now, but may be autom. updated in a later process.

[NeoBlack]

llms.txt should mainly focus on generic rules, and reference to a catalog.json file for detailed versions and urls/pathes aso, which may be a static file now, but may be autom. updated in a later process.

Yes, good idea, the catalog.json could maybe be generated from the docs inventory? I am not so deep in the rendering process, but I guess someone like Garvin now understands better.

[CybotTM]

Reference robots.txt as the authoritative crawling policy

I doubt that (coding) agents classify themselves as "crawlers" - as they are no crawlers technically.

[garvinhicking]

Yes, good idea, the catalog.json could maybe be generated from the docs inventory? I am not so deep in the rendering process, but I guess someone like Garvin now understands better.

Technically that probably would best be "active" PHP and maintained here:

https://github.com/TYPO3GmbH/site-intercept
https://github.com/TYPO3GmbH/site-intercept/tree/develop/legacy_hook/src

cc @andreaskienast so we would add a pseudo "container.json" URL, have active PHP in that which operates on the server.

Because the rendering only ever knows of ONE rendered repo. It has no knowledge of "all published docs" and cannot build a catalog. That would be a distinct app on the server.

OR it could be another TYPO3-Docs process that runs regularly / scheduled, and pushes an actual file to the server at intervals. I think the maintenance would be easier with a PHP Scrip on the server itself though, but that would at least need approval of the GmbH because it'd be active code on your infra.

[garvinhicking]

I'm taking "executive action" here and merge this. My idea is to improve on this in follow-up PRs, and it improves the current situation already.

Hope that is fine with everyone, else please do a revert.