fix(fetch): fall back when Readability strips hidden SSR content

[Christian-Sidak]

Summary

• Adds a three-stage fallback to extract_content_from_html() so that pages using progressive SSR (hidden pre-hydration markup) are not silently reduced to a single line of loading-shell text
• Stage 1: Readability (existing behavior, unchanged for normal sites)
• Stage 2: readabilipy without Readability JS (less aggressive, does not filter by CSS visibility)
• Stage 3: Raw markdownify conversion (last resort)
• Fallback only activates when Readability output is shorter than 1% of the input HTML

Motivation

Sites using progressive server-side rendering (Next.js streaming, Remix deferred, custom Lambda SSR) deliver content in two phases: a small visible loading shell, then the real content in a hidden container (visibility:hidden; position:absolute; top:-9999px) that becomes visible after client-side hydration. Mozilla Readability treats hidden elements as non-content and strips them entirely, causing mcp-server-fetch to return only the loading shell text with no indication that content was lost.

For example, fetching https://runtimeweb.com returns just "Unified Serverless Framework for Full-Stack TypeScript Applications" instead of the full page content.

Changes

• src/fetch/src/mcp_server_fetch/server.py: Modified extract_content_from_html() to try three extraction stages, falling back only when the previous stage produces disproportionately little text
• src/fetch/tests/test_server.py: Added 6 unit tests covering all fallback paths, threshold behavior, and no-regression for normal pages

Breaking Changes

None. The fix only activates when Readability extracts less than 1% of the HTML size as text. Normal sites where Readability works correctly are completely unaffected.

Test plan

• All 6 new fallback tests pass
• Existing tests unaffected (they test Readability directly via Node.js, independent of this change)
• No new dependencies added

Fixes #3878

[olaservo]

This is the strongest of the three Readability fallback PRs (#3879, #3894, #3922):

• 3-stage pipeline (Readability → readabilipy without Readability → raw markdownify) gives a good quality gradient
• Proportional 1% threshold scales with page size, unlike a fixed constant
• Preserves the <error> return for truly empty pages (no behavior change)
• 6 fully-mocked deterministic tests with good edge case coverage
• Smallest diff to production code (only 4 lines removed)

We'll close the other two PRs with credit to @morozow for filing the original issue (#3878).

---

This review was assisted by Claude Code.