fix(seo): add robots.txt and llms.txt, deploy them to R2

- Create robots.txt allowing all crawlers with sitemap reference,
  blocking AI training crawlers (CCBot, Google-Extended, Bytespider,
  Applebot-Extended, meta-externalagent) while keeping retrieval bots
- Create llms.txt describing the dataset for LLM-friendly discovery
- Update deploy-site workflow to upload both files to R2

Author: caiopizzol

.github/workflows/deploy-site.yml

@@ -32,6 +32,8 @@ jobs:
           # Static files
           aws s3 cp apps/web/index.html "$R2_BUCKET/index.html" --endpoint-url "$R2_ENDPOINT"
           aws s3 cp apps/web/sitemap.xml "$R2_BUCKET/sitemap.xml" --endpoint-url "$R2_ENDPOINT"
+          aws s3 cp apps/web/robots.txt "$R2_BUCKET/robots.txt" --endpoint-url "$R2_ENDPOINT"
+          aws s3 cp apps/web/llms.txt "$R2_BUCKET/llms.txt" --endpoint-url "$R2_ENDPOINT"
           aws s3 cp apps/web/public/ "$R2_BUCKET/public/" --recursive --endpoint-url "$R2_ENDPOINT"
 
           # Generated type pages (clean URLs: /types/legal, not /types/legal.html)

apps/web/llms.txt

@@ -0,0 +1,44 @@
+# docx-corpus
+
+> The largest classified corpus of Word documents on the public web.
+
+docx-corpus is an open dataset of 736K+ .docx files collected from the public web, classified into 10 document types and 9 topics across 46+ languages. It is built for document processing research, NLP benchmarking, and training models that work with real-world Word documents.
+
+Documents are classified using ModernBERT with an average confidence of 82%.
+
+Built by [SuperDoc](https://www.superdoc.dev), the document rendering engine.
+
+## Document Types
+
+- [Legal](https://docxcorp.us/types/legal): Contracts, agreements, legal notices, court filings
+- [Forms](https://docxcorp.us/types/forms): Application forms, surveys, questionnaires, fillable templates
+- [Educational](https://docxcorp.us/types/educational): Course materials, syllabi, assignments, lecture notes
+- [Administrative](https://docxcorp.us/types/administrative): Meeting minutes, agendas, organizational documents
+- [Policies](https://docxcorp.us/types/policies): Policy documents, procedures, guidelines, handbooks
+- [Correspondence](https://docxcorp.us/types/correspondence): Letters, memos, formal communications
+- [Reports](https://docxcorp.us/types/reports): Annual reports, research reports, financial reports
+- [Reference](https://docxcorp.us/types/reference): Reference materials, glossaries, directories, catalogs
+- [Technical](https://docxcorp.us/types/technical): Technical documentation, specifications, user manuals
+- [Creative](https://docxcorp.us/types/creative): Creative writing, marketing materials, newsletters
+
+## Topics
+
+- [Government](https://docxcorp.us/topics/government): Public administration and civic organizations
+- [Education](https://docxcorp.us/topics/education): Schools, universities, research institutions
+- [Healthcare](https://docxcorp.us/topics/healthcare): Hospitals, clinics, health organizations
+- [General](https://docxcorp.us/topics/general): Cross-sector documents
+- [Legal / Judicial](https://docxcorp.us/topics/legal_judicial): Law firms, courts, regulatory bodies
+- [Finance](https://docxcorp.us/topics/finance): Banks, investment firms, insurance
+- [Environment](https://docxcorp.us/topics/environment): Environmental agencies, sustainability
+- [Nonprofit](https://docxcorp.us/topics/nonprofit): NGOs, charities, foundations
+- [Technology](https://docxcorp.us/topics/technology): Tech companies, software, IT
+
+## Links
+
+- Homepage: https://docxcorp.us
+- Browse all types: https://docxcorp.us/types
+- Browse all topics: https://docxcorp.us/topics
+- GitHub: https://github.com/superdoc-dev/docx-corpus
+- HuggingFace: https://huggingface.co/datasets/superdoc-dev/docx-corpus
+- API: https://api.docxcorp.us
+- Takedown requests: help@docxcorp.us

apps/web/robots.txt

@@ -0,0 +1,24 @@
+# https://docxcorp.us/robots.txt
+
+User-agent: *
+Allow: /
+Sitemap: https://docxcorp.us/sitemap.xml
+
+# AI-friendly content available at /llms.txt
+# See https://llmstxt.org for the specification
+
+# Block AI training crawlers (we allow retrieval bots like GPTBot, ClaudeBot, Amazonbot)
+User-agent: CCBot
+Disallow: /
+
+User-agent: Google-Extended
+Disallow: /
+
+User-agent: Bytespider
+Disallow: /
+
+User-agent: Applebot-Extended
+Disallow: /
+
+User-agent: meta-externalagent
+Disallow: /