docs.apify.com/index.md returns 404

[jirispilka]

https://docs.apify.com/index.md returns 404, while other .md URLs work fine:

• ✅ https://crawlee.dev/index.md → 200
• ✅ https://docs.apify.com/platform.md → 200
• ✅ https://docs.apify.com/academy.md → 200
• ❌ https://docs.apify.com/index.md → 404

I think it would be good if the https://docs.apify.com/index.md would work too

[jirispilka]

I assigned this to @patrikbraborec Feel free to reassign or close if you find it not relevant

[patrikbraborec]

Hi, I changed the assignee to @TC-MO as they can fix it within their docs team, or with the help of tooling team. Let me know if you need help.

[TC-MO]

Isn't this by design? In docusaurus.config.js L327-329 we explicitly exclude homepage from .md generation (I guess llms.txt should be treated as the homepage for .md versions?).

I guess @katzino to correct me if I'm wrong?

[katzino]

Isn't this by design? In docusaurus.config.js L327-329 we explicitly exclude homepage from .md generation (I guess llms.txt should be treated as the homepage for .md versions?).

I guess @katzino to correct me if I'm wrong?

I agree. Or at least from what I remember, we haven't added any index.md on purpose and instead we used llms.txt as entry point. For instance, https://apify.com/llms.txt instead of https://apify.com/index.md (which would have to serve as homepage markdown version).

[TC-MO]

I think we can close this one. The logic here is sound homepage.md would still be about routing to different parts of documentation and that is the same thing that llms.txt does.

I also took liberty to check other big docs sites (supabase, stripe, vercel to name a few) and none of them have this pattern either, while they have llms.txt set on homepage.

I would rather try & focus on improvements to llms.txt from what I've tested it seems to get truncated often as it is way too large and thus LLMs can miss existing pages/concepts. But I guess that should be separate issue ? @patrikbraborec FYI not sure if that would fall under DX? Docs? Acquisition or Tooling :D?

[jirispilka]

Thanks for the discussion, everyone.

For context, llms.txt turns out to not be widely used in practice; agents tend to ignore it.

What prompted this was that while testing the MCP server, I tried to get an .md version of https://docs.apify.com/ and got a 404. Then I noticed that https://crawlee.dev/index.md serves index.md, while docs.apify.com doesn't.

That made me think we should keep things consistent across our documentation sites.
That said, @TC-MO if you are ok with this inconsistency, let's keep this closed.

[TC-MO]

Good points, to add to that crawlee actually serves both at the same time, and maybe this is the approach? you can do either crawlee.dev/llms.txt or crawlee.dev/index.md

I agree re: llms.txt it's not only a failed standard but a bad one at that, as mentioned it gets ignored, and when it is not ignored it gets truncated so that it misses a lot and does not do it's job.

Maybe it would be worth to revisit this @patrikbraborec and I guess @B4nan could you shed a little more light on why index.md for crawlee landing page?

[B4nan]

I think (and git blame agrees) that the llms.txt/.md support in crawlee was done by the web team initially, no idea why it differs here, no strong opinions either. I guess it all boils down to what we want on that page, since the generated index.md will reflect what you have on the homepage, and that often doesn't provide much value on its own once converted to markdown.