Merge pull request #12 from OpenElementsLabs/feat/006-content-extractor

Content extractor — jsoup content + metadata extraction

Author: hendrikebbers

CLAUDE.md

@@ -43,7 +43,9 @@ extraction, Meilisearch indexing with scheduled refresh, and four MCP tools
 │   ├── UrlMatcher.java                      # Ant-glob include/exclude matching against URL paths
 │   ├── ContentLocaleResolver.java           # path-prefix locale rule (/de -> German, else English)
 │   ├── DiscoveredItem.java                  # a discovered URL + lastmod change marker
-│   └── SitemapCrawler.java                  # sitemap/index discovery + bounded fallback crawl
+│   ├── SitemapCrawler.java                  # sitemap/index discovery + bounded fallback crawl
+│   ├── ContentExtractor.java                # jsoup body + metadata extraction
+│   └── ExtractedContent.java                # extractor output (body + metadata)
 ├── src/main/resources/application.yaml      # datasource, JPA, OAuth2, MCP, Meilisearch, content config
 ├── src/test/java/com/openelements/content/  # behavior tests (context, MCP enabled/disabled, search-down, jsoup)
 └── docs/
@@ -80,7 +82,11 @@ returns `DiscoveredItem`s filtered by `UrlMatcher`; it is fault-tolerant per sit
 (the fetch stage) performs polite HTTP GETs — bot `User-Agent`, conditional `If-None-Match`/
 `If-Modified-Since` (→ `304` handling), a `max-body-bytes` cap, per-host rate limiting
 (`HostRateLimiter`), and retry-with-backoff on transient failures — returning a classified
-`FetchResult` (`OK`/`NOT_MODIFIED`/`NOT_FOUND`/`ERROR`).
+`FetchResult` (`OK`/`NOT_MODIFIED`/`NOT_FOUND`/`ERROR`). `ContentExtractor` (the extract stage) turns
+fetched HTML into an `ExtractedContent` via jsoup: it selects the main container by the source's
+`contentSelector` (with a readability fallback), always strips scripts/styles, applies
+`contentExclude`, whitespace-normalizes the body, and reads metadata (title/excerpt/date/author/
+categories/preview image/locale) from OpenGraph/Article `<meta>`, JSON-LD, and `<time>`.
 
 > **Key gotcha:** the library ships no Spring Boot auto-configuration and couples MCP to a JPA
 > datasource, so `@Import({ McpConfiguration, SearchConfig })` alone does **not** boot. See

docs/specs/006-content-extractor/steps.md

@@ -0,0 +1,72 @@
+# Implementation Steps: Content Extractor
+
+## Step 1: `ExtractedContent` record
+
+- [x] Record with `title`, `excerpt`, `body`, `author`, `categories`, `publishedDate`, `previewImage`, `locale`; `categories` normalized to non-null
+
+**Related behaviors:** all (return shape)
+
+---
+
+## Step 2: `ContentExtractor` — body extraction
+
+- [x] `@Component` taking `ContentLocaleResolver` + `ObjectMapper`
+- [x] Read metadata first, then strip `script`/`style`/`noscript`/`template` + comments (so JSON-LD survives)
+- [x] Container selection: `contentSelector` comma list (first non-empty match), `body` = whole document, readability fallback (semantic tag, else largest div/section) when nothing matches
+- [x] Remove `contentExclude` selectors from the container
+- [x] Whitespace-normalized, block-separated body (`\n\n` between paragraphs); fallback to collapsed container text
+
+**Related behaviors:** selector container; comma fallback; body selector; scripts/styles removed; contentExclude; whitespace normalized; readability fallback; empty container
+
+---
+
+## Step 3: `ContentExtractor` — metadata extraction
+
+- [x] title: `og:title` → `<title>` → first `<h1>`
+- [x] excerpt: `description` → `og:description`
+- [x] previewImage: `og:image`
+- [x] publishedDate: `article:published_time` → `<time datetime>` → JSON-LD `datePublished`
+- [x] author: `author`/`article:author` meta → JSON-LD `author.name`
+- [x] categories: `article:tag` metas → JSON-LD `keywords`
+- [x] locale: from `ContentLocaleResolver` (path rule)
+- [x] JSON-LD parsed via Jackson, following arrays and `@graph`
+- [x] Graceful degradation to null when fields are absent
+
+**Related behaviors:** OpenGraph metadata; JSON-LD date/author; article tags; locale; metadata independent of selector; missing metadata
+
+---
+
+## Step 4: Tests
+
+- [x] `ContentExtractorTest` (14 cases) — body selection/cleaning + metadata + edge cases
+
+**Acceptance criteria:**
+- [x] All tests pass (`mvn test`); build green
+
+---
+
+## Behavior Coverage
+
+| Scenario | Layer | Covered in Step |
+|----------|-------|-----------------|
+| Selector picks the article container | Backend | Step 4 (`selectorPicksArticleContainer`) |
+| Comma-separated fallback list, first match wins | Backend | Step 4 (`commaSeparatedFallbackFirstMatchWins`) |
+| body selector takes the whole document | Backend | Step 4 (`bodySelectorTakesWholeDocument`) |
+| Scripts and styles are always removed | Backend | Step 4 (`scriptsAndStylesAreAlwaysRemoved`) |
+| contentExclude removes boilerplate | Backend | Step 4 (`contentExcludeRemovesBoilerplate`) |
+| Whitespace is normalized | Backend | Step 4 (`whitespaceIsNormalized`) |
+| Readability fallback when no selector matches | Backend | Step 4 (`readabilityFallbackWhenNoSelectorMatches`) |
+| OpenGraph metadata is read | Backend | Step 4 (`openGraphMetadataIsRead`) |
+| JSON-LD fallback for date/author | Backend | Step 4 (`jsonLdFallbackForDateAndAuthor`) |
+| Categories from article tags | Backend | Step 4 (`categoriesFromArticleTags`) |
+| Locale derived from path | Backend | Step 4 (`localeDerivedFromPath`) |
+| Metadata read regardless of content selector | Backend | Step 4 (`metadataReadRegardlessOfContentSelector`) |
+| Missing metadata degrades gracefully | Backend | Step 4 (`missingMetadataDegradesGracefully`) |
+| Empty content container | Backend | Step 4 (`emptyContentContainerFallsBack`) |
+
+All scenarios are backend; there is no frontend in this spec.
+
+## Notes
+
+- Excerpt open question: kept `null` when no `description`/`og:description` (no body-snippet fallback) — the behavior allows null.
+- Markdown vs plaintext: plaintext body kept (no `flexmark` dependency added).

docs/specs/INDEX.md

@@ -11,7 +11,7 @@ roadmap step, to be implemented sequentially (each builds on the previous).
 | 003 | 003-content-document-index | Content document & index | backend, search, data-model | `ContentDocument` record + Meilisearch `IndexSettings` bean | #5 | done |
 | 004 | 004-sitemap-crawler | Sitemap crawler | backend, crawler | `SitemapCrawler` — sitemap discovery of URLs + lastmod | #7 | done |
 | 005 | 005-page-fetcher | Page fetcher | backend, crawler | `PageFetcher` — robust HTTP fetch (ETag/IMS, rate limit, retry) | #9 | done |
-| 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | — | open |
+| 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | #11 | done |
 | 007 | 007-source-strategy | Source strategy | backend, architecture | `ContentSourceStrategy` interface + `WebsiteSourceStrategy` | — | open |
 | 008 | 008-content-indexer | Content indexer | backend, search, crawler | `ContentIndexer` — orchestration, diff, upsert/delete | — | open |
 | 009 | 009-content-bootstrap-step | Content bootstrap step | backend, search | `ContentBootstrapStep` — initial reindex via `SearchIndexBootstrapStep` | — | open |