fix: Improve Google search results and SEO for documentation

[manamnathtiwari]

Resolves #13267

• Fix robots.txt serving location to be at domain root
• Add meta description tags to all pages for search snippets
• Update sitemap reference to use HTTPS
• Improves search result visibility and snippet display

Checklist

• I have read the contributing guidelines
• I have personally reviewed, understood, and tested every change in this PR

[manamnathtiwari]

Hey @czepluch ,I hope you are doing well. I am waiting for your review and comments please let me know !

[czepluch]

I'll get to it next week.

[czepluch]

Thanks for the effort, but I'm going to close this, as it doesn't resolve #13267.

The pages that show up in Google without a summary are the ones we disallow in robots.txt (develop and old versions). As the issue thread explains, Disallow blocks Google from crawling the content, but the URL still gets indexed. That means the meta descriptions added here will never be read on the pages that actually have the problem. The fix goes the other way: noindex on non-canonical versions plus relaxing the Disallow rules so crawlers can see it, and/or rel="canonical" to /latest/.

A couple of things also make me doubt the "I have personally reviewed, understood, and tested every change" checkbox. The conf.py change just reformats html_extra_path into a multi-line list with the same single element, which is a no-op. And the description mentions fixing the robots.txt serving location, which isn't in the diff (it's already served at root).

Happy to see a fresh attempt along the noindex/canonical lines. Closing for now.