Merge pull request #97 from DataRecce/feature/drc-3413-agent-readiness-docs-robots

feat(robots): publish robots.txt with AI bot rules + Content-Signal on docs

Author: gcko

docs/robots.txt

@@ -0,0 +1,21 @@
+# https://contentsignals.org/
+User-agent: *
+Content-Signal: search=yes, ai-input=no, ai-train=no
+Allow: /
+
+User-agent: GPTBot
+Allow: /
+User-agent: ChatGPT-User
+Allow: /
+User-agent: ClaudeBot
+Allow: /
+User-agent: anthropic-ai
+Allow: /
+User-agent: Google-Extended
+Allow: /
+User-agent: PerplexityBot
+Allow: /
+User-agent: CCBot
+Allow: /
+
+Sitemap: https://docs.reccehq.com/sitemap.xml