feat: drop jina.ai dependency - direct .md for support, freeze blog

lroolle · claude · lroolle · commit 0fd798c4f122 · 2026-07-16T01:52:41.000+08:00
support.claude.com now serves a .md variant for every article
(verified 2026-07-08: 200 text/markdown on all sitemap URLs), so
support articles fetch directly like platform/code/mcp docs - no
proxy, no rate limits, 355/355 instead of 4/355 through jina.

anthropic.com blog (engineering/research/news/product) remains
HTML-only (no llms.txt, no .md, no content negotiation), and the
jina path had been ~95% dead from rate limiting anyway. Freeze it:
content/blog/ stays as a static archive, sources disabled in
sources.json with dated notes, --section choices trimmed.

Full run after: 3231/3231 downloaded, 0 failed (was 2907/3638,
731 failed). Also removes the JINA_API_KEY workflow plumbing added
earlier on this branch - no secret needed anymore.

Co-Authored-By: Claude Fable 5 &lt;noreply@anthropic.com&gt;
diff --git a/.github/workflows/fetch-claude-docs.yml b/.github/workflows/fetch-claude-docs.yml
@@ -45,7 +45,6 @@ jobs:
       - name: Fetch all Anthropic docs
         env:
           GITHUB_TOKEN: ${{ steps.app-token.outputs.token }}
-          JINA_API_KEY: ${{ secrets.JINA_API_KEY }}
         run: uv run scripts/fetcher.py
 
       - name: Setup GitHub CLI and Git
@@ -104,11 +103,9 @@ jobs:
             - `content/en/api/` - API reference (1500+ docs)
             - `content/en/build-with-claude/` - Platform features
             - `content/mcp/` - MCP protocol spec
-            - `content/blog/engineering/` - Anthropic engineering posts
-            - `content/blog/research/` - Research papers
-            - `content/blog/news/` - Model releases, announcements
+            - `content/blog/` - FROZEN archive (anthropic.com is HTML-only; no longer fetched)
             - `content/github/` - GitHub repos (cookbooks, skills, plugins, courses)
-            - `content/support/` - Support articles
+            - `content/support/` - Support articles (sitemap + .md)
 
             ## YOUR WORKFLOW
 
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -5,15 +5,17 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ## Repository Purpose
 
 Comprehensive archive of everything Anthropic publishes for building with
-Claude. 2,900+ docs from 11 sources, auto-updated every 6 hours.
+Claude. 2,900+ docs from 11 sources; active sources auto-updated four
+times daily (blog archive frozen, see Fetcher).
 
 ## Fetcher
 
 `scripts/fetcher.py` -- single-file multi-source fetcher.
 
 Sources: code.claude.com, platform.claude.com, modelcontextprotocol.io,
-anthropic.com (engineering/research/news), support.claude.com,
-github.com/anthropics/* (10 repos).
+support.claude.com (sitemap + .md), github.com/anthropics/* (10 repos).
+anthropic.com blog (engineering/research/news) is a FROZEN archive as of
+2026-07: the site is HTML-only and the jina.ai proxy path was removed.
 
 ```bash
 uv run scripts/fetcher.py                    # Fetch everything
@@ -22,8 +24,7 @@ uv run scripts/fetcher.py --tree             # Show sources
 uv run scripts/fetcher.py --discover         # Probe for new sources
 ```
 
-Sections: `claude-code`, `api`, `platform`, `mcp`, `blog`, `engineering`,
-`research`, `news`, `github`, `support`, `all`
+Sections: `claude-code`, `api`, `platform`, `mcp`, `github`, `support`, `all`
 
 Source registry: `sources.json`
 Architecture: `REFACTOR.md`
@@ -105,7 +106,7 @@ Use these paths to reference documentation when helping users:
 - `content/mcp/seps/` - Specification Enhancement Proposals
 - `content/mcp/community/` - Governance, working groups
 
-#### Engineering & Research (from anthropic.com)
+#### Engineering & Research (from anthropic.com) — FROZEN archive, not auto-updated
 - `content/blog/engineering/` - "Building Effective Agents", tool use, harness design
 - `content/blog/research/` - Research papers
 - `content/blog/news/` - Model releases, announcements
diff --git a/scripts/fetcher.py b/scripts/fetcher.py
@@ -6,8 +6,9 @@
   - platform.claude.com     -> API/platform docs (sitemap + .md suffix)
   - code.claude.com         -> Claude Code + Agent SDK (llms.txt + .md suffix)
   - modelcontextprotocol.io -> MCP spec (sitemap + .md suffix)
-  - anthropic.com           -> Engineering, research, news (jina.ai proxy)
-  - support.claude.com      -> Help articles (jina.ai proxy)
+  - support.claude.com      -> Help articles (sitemap + .md suffix)
+  - anthropic.com blog      -> FROZEN 2026-07 (HTML-only, no .md variant;
+                               the jina.ai proxy path was removed)
   - github.com/anthropics/* -> Repos (raw.githubusercontent.com)
 
 Usage:
@@ -57,12 +58,6 @@
     ("anthropics/anthropic-sdk-typescript","main",   [".md"]),
 ]
 
-NEWS_KEYWORDS = [
-    "claude", "model", "api", "agent", "mcp", "sdk", "opus", "sonnet",
-    "haiku", "code", "bedrock", "vertex", "foundry", "tool", "safety",
-    "computer-use",
-]
-
 DISCOVER_DOMAINS = [
     ("anthropic.com",           "Main site"),
     ("platform.claude.com",     "API platform docs"),
@@ -89,7 +84,6 @@ def __init__(
 
         self.platform_sitemap_url = "https://platform.claude.com/sitemap.xml"
         self.claude_code_llms_url = "https://code.claude.com/docs/llms.txt"
-        self.anthropic_sitemap_url = "https://www.anthropic.com/sitemap.xml"
         self.mcp_sitemap_url = "https://modelcontextprotocol.io/sitemap.xml"
         self.support_sitemap_url = "https://support.claude.com/sitemap.xml"
 
@@ -107,20 +101,11 @@ async def fetch_text(self, session: aiohttp.ClientSession, url: str) -> str:
             r.raise_for_status()
             return await r.text()
 
-    async def fetch_bytes(
-        self, session: aiohttp.ClientSession, url: str,
-        headers: Optional[Dict] = None,
-    ) -> bytes:
-        async with session.get(url, headers=headers) as r:
+    async def fetch_bytes(self, session: aiohttp.ClientSession, url: str) -> bytes:
+        async with session.get(url) as r:
             r.raise_for_status()
             return await r.read()
 
-    def _jina_headers(self) -> Optional[Dict]:
-        # Without a key, r.jina.ai rate-limits hard (~95% failures at 10
-        # concurrent). Set JINA_API_KEY to lift blog/support success rates.
-        key = os.environ.get("JINA_API_KEY")
-        return {"Authorization": f"Bearer {key}"} if key else None
-
     def extract_sitemap_urls(self, xml: str, must_contain: str = "") -> List[str]:
         urls = []
         for line in xml.split("\n"):
@@ -139,26 +124,9 @@ async def fetch_claude_code_urls(self, session: aiohttp.ClientSession) -> List[s
             urls.append(match[1:-4])  # strip parens and .md
         return urls
 
-    def filter_blog_urls(self, sitemap_xml: str) -> Dict[str, List[str]]:
-        result = {"engineering": [], "research": [], "news": [], "product": []}
-        for line in sitemap_xml.split("\n"):
-            if "<loc>" not in line:
-                continue
-            url = line.split("<loc>")[1].split("</loc>")[0]
-            if "/engineering/" in url and url != "https://www.anthropic.com/engineering":
-                result["engineering"].append(url)
-            elif "/product/" in url:
-                result["product"].append(url)
-            elif "/news/" in url and url != "https://www.anthropic.com/news":
-                slug = url.rsplit("/", 1)[-1].lower()
-                if any(kw in slug for kw in NEWS_KEYWORDS):
-                    result["news"].append(url)
-            elif "/research/" in url and url != "https://www.anthropic.com/research":
-                if "/research/team/" not in url:
-                    result["research"].append(url)
-        return result
-
     def extract_support_urls(self, sitemap_xml: str) -> List[str]:
+        # Articles serve a .md variant directly (since ~2026-07), so plain
+        # download_doc applies; sitemap covers more articles than llms.txt.
         return [
             url for url in self.extract_sitemap_urls(sitemap_xml)
             if "/en/articles/" in url
@@ -210,55 +178,6 @@ async def download_doc(self, session, url, semaphore) -> Dict:
                 self.stats["failed"] += 1
                 return {"url": url, "status": "failed", "error": str(e)}
 
-    async def download_via_jina(self, session, url, output_subdir, semaphore) -> Dict:
-        async with semaphore:
-            slug = url.split("anthropic.com/")[1] if "anthropic.com" in url else url.rsplit("/", 1)[-1]
-            output_path = self.output_dir / "blog" / f"{slug}.md"
-            if output_subdir:
-                output_path = self.output_dir / output_subdir / f"{slug}.md"
-            if self.incremental and output_path.exists():
-                self.stats["skipped"] += 1
-                return {"url": url, "status": "skipped"}
-            try:
-                content = await self.fetch_bytes(
-                    session, f"https://r.jina.ai/{url}", headers=self._jina_headers())
-                output_path.parent.mkdir(parents=True, exist_ok=True)
-                async with aiofiles.open(output_path, "wb") as f:
-                    await f.write(content)
-                self.stats["downloaded"] += 1
-                return {
-                    "url": url, "status": "success",
-                    "path": str(output_path.relative_to(self.output_dir)),
-                    "sha256": hashlib.sha256(content).hexdigest(),
-                    "size": len(content),
-                }
-            except Exception as e:
-                self.stats["failed"] += 1
-                return {"url": url, "status": "failed", "error": str(e)}
-
-    async def download_support_article(self, session, url, semaphore) -> Dict:
-        async with semaphore:
-            output_path = self.get_output_path(url)
-            if self.incremental and output_path.exists():
-                self.stats["skipped"] += 1
-                return {"url": url, "status": "skipped"}
-            try:
-                content = await self.fetch_bytes(
-                    session, f"https://r.jina.ai/{url}", headers=self._jina_headers())
-                output_path.parent.mkdir(parents=True, exist_ok=True)
-                async with aiofiles.open(output_path, "wb") as f:
-                    await f.write(content)
-                self.stats["downloaded"] += 1
-                return {
-                    "url": url, "status": "success",
-                    "path": str(output_path.relative_to(self.output_dir)),
-                    "sha256": hashlib.sha256(content).hexdigest(),
-                    "size": len(content),
-                }
-            except Exception as e:
-                self.stats["failed"] += 1
-                return {"url": url, "status": "failed", "error": str(e)}
-
     async def download_github_file(self, session, repo, branch, filepath, semaphore) -> Dict:
         async with semaphore:
             repo_short = repo.split("/")[1]
@@ -345,7 +264,6 @@ async def fetch_all(self):
 
             tasks = []
             semaphore = asyncio.Semaphore(self.jobs)
-            jina_semaphore = asyncio.Semaphore(min(self.jobs, 10))
             counts = {}
 
             # -- Platform docs --
@@ -382,20 +300,9 @@ async def fetch_all(self):
                 for url in urls:
                     tasks.append(self.download_doc(session, url, semaphore))
 
-            # -- Blog: engineering, research, news, product --
-            if self.want("blog", "engineering", "research", "news"):
-                print("Source: anthropic.com/sitemap.xml")
-                xml = await self.fetch_text(session, self.anthropic_sitemap_url)
-                blog = self.filter_blog_urls(xml)
-
-                for category, urls in blog.items():
-                    if not self.want("blog", category):
-                        continue
-                    counts[f"blog/{category}"] = len(urls)
-                    print(f"  {len(urls)} {category}")
-                    for url in urls:
-                        tasks.append(self.download_via_jina(
-                            session, url, None, jina_semaphore))
+            # -- Blog (anthropic.com): FROZEN 2026-07 --
+            # HTML-only upstream (no llms.txt / .md variant); the jina.ai
+            # proxy path was removed. content/blog/ stays as a static archive.
 
             # -- Support articles --
             if self.want("support"):
@@ -405,8 +312,7 @@ async def fetch_all(self):
                 counts["support"] = len(urls)
                 print(f"  {len(urls)} articles")
                 for url in urls:
-                    tasks.append(self.download_support_article(
-                        session, url, jina_semaphore))
+                    tasks.append(self.download_doc(session, url, semaphore))
 
             # -- GitHub repos --
             if self.want("github"):
@@ -558,10 +464,8 @@ async def show_tree(self):
             cc_urls = await self.fetch_claude_code_urls(session)
             mcp_urls = self.extract_sitemap_urls(
                 await self.fetch_text(session, self.mcp_sitemap_url))
-            anthropic_xml = await self.fetch_text(session, self.anthropic_sitemap_url)
-            blog = self.filter_blog_urls(anthropic_xml)
-            support_xml = await self.fetch_text(session, self.support_sitemap_url)
-            support_urls = self.extract_support_urls(support_xml)
+            support_urls = self.extract_support_urls(
+                await self.fetch_text(session, self.support_sitemap_url))
 
         def show_grouped(title, urls, strip_prefix):
             print(f"{title} ({len(urls)})")
@@ -579,16 +483,12 @@ def show_grouped(title, urls, strip_prefix):
         show_grouped("platform.claude.com", platform_urls, "https://platform.claude.com/docs/en/")
         show_grouped("modelcontextprotocol.io", mcp_urls, "https://modelcontextprotocol.io/")
 
-        print(f"anthropic.com blog")
-        print("-" * 50)
-        for cat, urls in blog.items():
-            print(f"  {cat}: {len(urls)}")
-        print()
         print(f"support.claude.com: {len(support_urls)} articles")
+        print("anthropic.com blog: frozen archive (not fetched)")
         print(f"GitHub repos: {len(GITHUB_REPOS)} repos configured")
         print()
 
-        total = len(cc_urls) + len(platform_urls) + len(mcp_urls) + sum(len(v) for v in blog.values()) + len(support_urls)
+        total = len(cc_urls) + len(platform_urls) + len(mcp_urls) + len(support_urls)
         print(f"Total fetchable: {total}+ (excludes GitHub repos)")
 
     # -- Discovery ---------------------------------------------------------
@@ -702,19 +602,18 @@ async def main():
   claude-code   Claude Code + Agent SDK docs (code.claude.com)
   api/platform  API and platform docs (platform.claude.com)
   mcp           MCP protocol spec (modelcontextprotocol.io)
-  blog          All blog content (engineering + research + news + product)
-  engineering   Engineering blog only
-  research      Research posts only
-  news          Filtered news (model releases, Claude updates)
   github        All configured GitHub repos
-  support       Support articles (support.claude.com)
+  support       Support articles (support.claude.com, sitemap + .md)
   all           Everything (default)
 
+Note: content/blog/ (anthropic.com engineering/research/news) is a
+frozen archive as of 2026-07 — the site is HTML-only and the jina.ai
+proxy path was removed.
+
 Examples:
   fetcher.py                               Fetch everything
   fetcher.py --section mcp                 MCP spec only
   fetcher.py --section github              GitHub repos only
-  fetcher.py --section engineering          Engineering blog only
   fetcher.py --tree                         Show all sources
   fetcher.py --discover                     Probe domains for new sources
   fetcher.py --incremental                  Skip existing files
@@ -728,7 +627,6 @@ async def main():
         "--section", "-s",
         choices=[
             "claude-code", "api", "platform", "mcp",
-            "blog", "engineering", "research", "news",
             "github", "support", "all",
         ],
     )
diff --git a/sources.json b/sources.json
