According to https://sphinx-sitemap.readthedocs.io/en/latest/search-optimization.html#using-robots-txt we're not fully utilising the sitemap.xml.
We should add a robots.txt to the source repository as per the docs. I'm not sure about allowing all user-agents, but I don't know what the default is without a robots.txt and whether anyone even respects the user-agent instructions anymore!
According to https://sphinx-sitemap.readthedocs.io/en/latest/search-optimization.html#using-robots-txt we're not fully utilising the
sitemap.xml.We should add a
robots.txtto the source repository as per the docs. I'm not sure about allowing all user-agents, but I don't know what the default is without arobots.txtand whether anyone even respects the user-agent instructions anymore!