diff --git a/CLAUDE.md b/CLAUDE.md index 2ee21af..911de0d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,7 +43,9 @@ extraction, Meilisearch indexing with scheduled refresh, and four MCP tools │ ├── UrlMatcher.java # Ant-glob include/exclude matching against URL paths │ ├── ContentLocaleResolver.java # path-prefix locale rule (/de -> German, else English) │ ├── DiscoveredItem.java # a discovered URL + lastmod change marker -│ └── SitemapCrawler.java # sitemap/index discovery + bounded fallback crawl +│ ├── SitemapCrawler.java # sitemap/index discovery + bounded fallback crawl +│ ├── ContentExtractor.java # jsoup body + metadata extraction +│ └── ExtractedContent.java # extractor output (body + metadata) ├── src/main/resources/application.yaml # datasource, JPA, OAuth2, MCP, Meilisearch, content config ├── src/test/java/com/openelements/content/ # behavior tests (context, MCP enabled/disabled, search-down, jsoup) └── docs/ @@ -80,7 +82,11 @@ returns `DiscoveredItem`s filtered by `UrlMatcher`; it is fault-tolerant per sit (the fetch stage) performs polite HTTP GETs — bot `User-Agent`, conditional `If-None-Match`/ `If-Modified-Since` (→ `304` handling), a `max-body-bytes` cap, per-host rate limiting (`HostRateLimiter`), and retry-with-backoff on transient failures — returning a classified -`FetchResult` (`OK`/`NOT_MODIFIED`/`NOT_FOUND`/`ERROR`). +`FetchResult` (`OK`/`NOT_MODIFIED`/`NOT_FOUND`/`ERROR`). `ContentExtractor` (the extract stage) turns +fetched HTML into an `ExtractedContent` via jsoup: it selects the main container by the source's +`contentSelector` (with a readability fallback), always strips scripts/styles, applies +`contentExclude`, whitespace-normalizes the body, and reads metadata (title/excerpt/date/author/ +categories/preview image/locale) from OpenGraph/Article ``, JSON-LD, and ``. > **Key gotcha:** the library ships no Spring Boot auto-configuration and couples MCP to a JPA > datasource, so `@Import({ McpConfiguration, SearchConfig })` alone does **not** boot. See diff --git a/docs/specs/006-content-extractor/steps.md b/docs/specs/006-content-extractor/steps.md new file mode 100644 index 0000000..eaa5962 --- /dev/null +++ b/docs/specs/006-content-extractor/steps.md @@ -0,0 +1,72 @@ +# Implementation Steps: Content Extractor + +## Step 1: `ExtractedContent` record + +- [x] Record with `title`, `excerpt`, `body`, `author`, `categories`, `publishedDate`, `previewImage`, `locale`; `categories` normalized to non-null + +**Related behaviors:** all (return shape) + +--- + +## Step 2: `ContentExtractor` — body extraction + +- [x] `@Component` taking `ContentLocaleResolver` + `ObjectMapper` +- [x] Read metadata first, then strip `script`/`style`/`noscript`/`template` + comments (so JSON-LD survives) +- [x] Container selection: `contentSelector` comma list (first non-empty match), `body` = whole document, readability fallback (semantic tag, else largest div/section) when nothing matches +- [x] Remove `contentExclude` selectors from the container +- [x] Whitespace-normalized, block-separated body (`\n\n` between paragraphs); fallback to collapsed container text + +**Related behaviors:** selector container; comma fallback; body selector; scripts/styles removed; contentExclude; whitespace normalized; readability fallback; empty container + +--- + +## Step 3: `ContentExtractor` — metadata extraction + +- [x] title: `og:title` → `` → first `` +- [x] excerpt: `description` → `og:description` +- [x] previewImage: `og:image` +- [x] publishedDate: `article:published_time` → `` → JSON-LD `datePublished` +- [x] author: `author`/`article:author` meta → JSON-LD `author.name` +- [x] categories: `article:tag` metas → JSON-LD `keywords` +- [x] locale: from `ContentLocaleResolver` (path rule) +- [x] JSON-LD parsed via Jackson, following arrays and `@graph` +- [x] Graceful degradation to null when fields are absent + +**Related behaviors:** OpenGraph metadata; JSON-LD date/author; article tags; locale; metadata independent of selector; missing metadata + +--- + +## Step 4: Tests + +- [x] `ContentExtractorTest` (14 cases) — body selection/cleaning + metadata + edge cases + +**Acceptance criteria:** +- [x] All tests pass (`mvn test`); build green + +--- + +## Behavior Coverage + +| Scenario | Layer | Covered in Step | +|----------|-------|-----------------| +| Selector picks the article container | Backend | Step 4 (`selectorPicksArticleContainer`) | +| Comma-separated fallback list, first match wins | Backend | Step 4 (`commaSeparatedFallbackFirstMatchWins`) | +| body selector takes the whole document | Backend | Step 4 (`bodySelectorTakesWholeDocument`) | +| Scripts and styles are always removed | Backend | Step 4 (`scriptsAndStylesAreAlwaysRemoved`) | +| contentExclude removes boilerplate | Backend | Step 4 (`contentExcludeRemovesBoilerplate`) | +| Whitespace is normalized | Backend | Step 4 (`whitespaceIsNormalized`) | +| Readability fallback when no selector matches | Backend | Step 4 (`readabilityFallbackWhenNoSelectorMatches`) | +| OpenGraph metadata is read | Backend | Step 4 (`openGraphMetadataIsRead`) | +| JSON-LD fallback for date/author | Backend | Step 4 (`jsonLdFallbackForDateAndAuthor`) | +| Categories from article tags | Backend | Step 4 (`categoriesFromArticleTags`) | +| Locale derived from path | Backend | Step 4 (`localeDerivedFromPath`) | +| Metadata read regardless of content selector | Backend | Step 4 (`metadataReadRegardlessOfContentSelector`) | +| Missing metadata degrades gracefully | Backend | Step 4 (`missingMetadataDegradesGracefully`) | +| Empty content container | Backend | Step 4 (`emptyContentContainerFallsBack`) | + +All scenarios are backend; there is no frontend in this spec. + +## Notes + +- Excerpt open question: kept `null` when no `description`/`og:description` (no body-snippet fallback) — the behavior allows null. +- Markdown vs plaintext: plaintext body kept (no `flexmark` dependency added). diff --git a/docs/specs/INDEX.md b/docs/specs/INDEX.md index 3c8a320..b1d7587 100644 --- a/docs/specs/INDEX.md +++ b/docs/specs/INDEX.md @@ -11,7 +11,7 @@ roadmap step, to be implemented sequentially (each builds on the previous). | 003 | 003-content-document-index | Content document & index | backend, search, data-model | `ContentDocument` record + Meilisearch `IndexSettings` bean | #5 | done | | 004 | 004-sitemap-crawler | Sitemap crawler | backend, crawler | `SitemapCrawler` — sitemap discovery of URLs + lastmod | #7 | done | | 005 | 005-page-fetcher | Page fetcher | backend, crawler | `PageFetcher` — robust HTTP fetch (ETag/IMS, rate limit, retry) | #9 | done | -| 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | — | open | +| 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | #11 | done | | 007 | 007-source-strategy | Source strategy | backend, architecture | `ContentSourceStrategy` interface + `WebsiteSourceStrategy` | — | open | | 008 | 008-content-indexer | Content indexer | backend, search, crawler | `ContentIndexer` — orchestration, diff, upsert/delete | — | open | | 009 | 009-content-bootstrap-step | Content bootstrap step | backend, search | `ContentBootstrapStep` — initial reindex via `SearchIndexBootstrapStep` | — | open | diff --git a/src/main/java/com/openelements/content/ContentExtractor.java b/src/main/java/com/openelements/content/ContentExtractor.java new file mode 100644 index 0000000..345a459 --- /dev/null +++ b/src/main/java/com/openelements/content/ContentExtractor.java @@ -0,0 +1,328 @@ +package com.openelements.content; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Deque; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; +import org.jsoup.Jsoup; +import org.jsoup.nodes.Comment; +import org.jsoup.nodes.Document; +import org.jsoup.nodes.Element; +import org.jsoup.nodes.Node; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +/** + * Extracts a clean body and metadata from raw HTML using jsoup. + * + * Body: metadata is read first (before any {@code Visible text.
Body: metadata is read first (before any {@code
Visible text.