feat(seo): add llms.txt and expand AI crawler allowlist in robots.txt

Author: tw93

public/llms.txt

@@ -0,0 +1,23 @@
+# 潮流周刊 (Tw93 Weekly)
+
+> Weekly curation of tech discoveries and lifestyle by Tw93 (Tang). 266+ issues since 2021.
+
+## About
+
+潮流周刊 is a Chinese-language weekly newsletter curating interesting tech tools, design inspiration, and lifestyle discoveries. Published every week by Tw93, a product engineer based in Hangzhou.
+
+## Content
+
+- 266+ weekly issues
+- Bilingual: Chinese original + English translation
+- RSS: https://weekly.tw93.fun/rss.xml
+
+## Author
+
+Tw93 is a product engineer who builds open source developer tools (Pake, MiaoYan, Kaku, Mole). 130K+ GitHub stars.
+
+## Links
+
+- GitHub: https://github.com/tw93
+- Blog: https://tw93.fun
+- AI Profile: https://yobi.tw93.fun/llms-full.txt

public/robots.txt

@@ -1,17 +1,37 @@
 # Allow all search engines
 User-agent: *
 Allow: /
+Disallow: /pagefind/
+Disallow: /_astro/
+
+# Sitemap
+Sitemap: https://weekly.tw93.fun/sitemap-index.xml
 
-# Specific rules for major search engines
+# Search engines
 User-agent: Googlebot
 Allow: /
 
 User-agent: Baiduspider
 Allow: /
 
-# Sitemap
-Sitemap: https://weekly.tw93.fun/sitemap-index.xml
+# AI crawlers
+User-agent: GPTBot
+Allow: /
 
-# Disallow crawling of these directories
-Disallow: /pagefind/
-Disallow: /_astro/
+User-agent: ChatGPT-User
+Allow: /
+
+User-agent: ClaudeBot
+Allow: /
+
+User-agent: anthropic-ai
+Allow: /
+
+User-agent: PerplexityBot
+Allow: /
+
+User-agent: Google-Extended
+Allow: /
+
+User-agent: CCBot
+Allow: /