Skip to content

fix: keep exact clutter selectors from removing majority-content containers#337

Open
mvanhorn wants to merge 3 commits into
kepano:mainfrom
mvanhorn:fix/283-clutter-selector-swallows-main-content
Open

fix: keep exact clutter selectors from removing majority-content containers#337
mvanhorn wants to merge 3 commits into
kepano:mainfrom
mvanhorn:fix/283-clutter-selector-swallows-main-content

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Pages whose main content lives inside a container with an ad-like class (like <app-root class="ad">) no longer come back empty. The exact-selector clutter pass now skips removing an ambiguous ad-like element when it holds the large majority of the page's text, while explicit clutter (#comments, footer, dialogs, nav) and unsafe tags (script, style, noscript, iframe) stay removable as before.

Why this matters

The reporter's page in #283 extracts to zero words. The root cause: EXACT_SELECTORS' .ad entry matches the SPA's root element app-root class="ad", which holds essentially all of the page text, and the display:none print copy is removed by hidden-element handling, so nothing survives. Any site whose framework root or wrapper happens to carry an ad-like class hits the same total-loss failure, and there is no retry path that disables exact selectors.

The guard is deliberately narrow: it only protects elements whose class matches ambiguous ad-like tokens, only when they hold >=80% of the main content's text (minimum 200 chars), and never protects unsafe tags or high-confidence clutter selectors, so existing fixtures are unaffected.

Testing

  • New fixture + expected output pair issues--283-clutter-selector-removes-content reproduces the reporter's page shape (SPA root with class="ad", hidden print copy, large inline script): extraction goes from 0 words to the full article, and leaked script HTML is asserted absent.
  • TZ=UTC npx vitest run: 20 files, 371 tests pass.

Fixes #283

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extraction fails for frontend-heavy Angular SPA HTML with large inline style/font blocks

1 participant