From e609113f1cfcd658729f953b28e94b6c66865d17 Mon Sep 17 00:00:00 2001
From: Hendrik Ebbers <hendrik.ebbers@open-elements.com>
Date: Sun, 12 Jul 2026 14:33:08 +0200
Subject: [PATCH 1/3] chore(spec-006): mark content-extractor in progress, link
 issue #11

Refs #11
---
 docs/specs/INDEX.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/specs/INDEX.md b/docs/specs/INDEX.md
index 3c8a320..fee3da8 100644
--- a/docs/specs/INDEX.md
+++ b/docs/specs/INDEX.md
@@ -11,7 +11,7 @@ roadmap step, to be implemented sequentially (each builds on the previous).
 | 003 | 003-content-document-index | Content document & index | backend, search, data-model | `ContentDocument` record + Meilisearch `IndexSettings` bean | #5 | done |
 | 004 | 004-sitemap-crawler | Sitemap crawler | backend, crawler | `SitemapCrawler` — sitemap discovery of URLs + lastmod | #7 | done |
 | 005 | 005-page-fetcher | Page fetcher | backend, crawler | `PageFetcher` — robust HTTP fetch (ETag/IMS, rate limit, retry) | #9 | done |
-| 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | — | open |
+| 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | #11 | in progress |
 | 007 | 007-source-strategy | Source strategy | backend, architecture | `ContentSourceStrategy` interface + `WebsiteSourceStrategy` | — | open |
 | 008 | 008-content-indexer | Content indexer | backend, search, crawler | `ContentIndexer` — orchestration, diff, upsert/delete | — | open |
 | 009 | 009-content-bootstrap-step | Content bootstrap step | backend, search | `ContentBootstrapStep` — initial reindex via `SearchIndexBootstrapStep` | — | open |

From 8352cfbbfcd8a96f8f2e947058dc8396c11e198e Mon Sep 17 00:00:00 2001
From: Hendrik Ebbers <hendrik.ebbers@open-elements.com>
Date: Sun, 12 Jul 2026 14:38:07 +0200
Subject: [PATCH 2/3] feat(spec-006): jsoup content + metadata extractor

- ExtractedContent record (body + title/excerpt/author/categories/
  publishedDate/previewImage/locale; categories normalized).
- ContentExtractor (@Component, ContentLocaleResolver + ObjectMapper):
  reads metadata before stripping scripts (JSON-LD survives), then strips
  script/style/noscript/template + comments; selects the container via
  contentSelector (comma fallback, body = whole doc, readability fallback),
  removes contentExclude, and whitespace-normalizes to block-separated body.
  Metadata from OpenGraph/Article meta, JSON-LD (arrays/@graph), and
  <time datetime>; locale via the path rule.
- ContentExtractorTest: 14 cases (selection/cleaning + metadata + edges).
  66 tests green.
- steps.md; refresh CLAUDE.md.

Refs #11
---
 CLAUDE.md                                     |  10 +-
 docs/specs/006-content-extractor/steps.md     |  72 ++++
 .../content/ContentExtractor.java             | 328 ++++++++++++++++++
 .../content/ExtractedContent.java             |  36 ++
 .../content/ContentExtractorTest.java         | 215 ++++++++++++
 5 files changed, 659 insertions(+), 2 deletions(-)
 create mode 100644 docs/specs/006-content-extractor/steps.md
 create mode 100644 src/main/java/com/openelements/content/ContentExtractor.java
 create mode 100644 src/main/java/com/openelements/content/ExtractedContent.java
 create mode 100644 src/test/java/com/openelements/content/ContentExtractorTest.java

diff --git a/CLAUDE.md b/CLAUDE.md
index 2ee21af..911de0d 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -43,7 +43,9 @@ extraction, Meilisearch indexing with scheduled refresh, and four MCP tools
 │   ├── UrlMatcher.java                      # Ant-glob include/exclude matching against URL paths
 │   ├── ContentLocaleResolver.java           # path-prefix locale rule (/de -> German, else English)
 │   ├── DiscoveredItem.java                  # a discovered URL + lastmod change marker
-│   └── SitemapCrawler.java                  # sitemap/index discovery + bounded fallback crawl
+│   ├── SitemapCrawler.java                  # sitemap/index discovery + bounded fallback crawl
+│   ├── ContentExtractor.java                # jsoup body + metadata extraction
+│   └── ExtractedContent.java                # extractor output (body + metadata)
 ├── src/main/resources/application.yaml      # datasource, JPA, OAuth2, MCP, Meilisearch, content config
 ├── src/test/java/com/openelements/content/  # behavior tests (context, MCP enabled/disabled, search-down, jsoup)
 └── docs/
@@ -80,7 +82,11 @@ returns `DiscoveredItem`s filtered by `UrlMatcher`; it is fault-tolerant per sit
 (the fetch stage) performs polite HTTP GETs — bot `User-Agent`, conditional `If-None-Match`/
 `If-Modified-Since` (→ `304` handling), a `max-body-bytes` cap, per-host rate limiting
 (`HostRateLimiter`), and retry-with-backoff on transient failures — returning a classified
-`FetchResult` (`OK`/`NOT_MODIFIED`/`NOT_FOUND`/`ERROR`).
+`FetchResult` (`OK`/`NOT_MODIFIED`/`NOT_FOUND`/`ERROR`). `ContentExtractor` (the extract stage) turns
+fetched HTML into an `ExtractedContent` via jsoup: it selects the main container by the source's
+`contentSelector` (with a readability fallback), always strips scripts/styles, applies
+`contentExclude`, whitespace-normalizes the body, and reads metadata (title/excerpt/date/author/
+categories/preview image/locale) from OpenGraph/Article `<meta>`, JSON-LD, and `<time>`.
 
 > **Key gotcha:** the library ships no Spring Boot auto-configuration and couples MCP to a JPA
 > datasource, so `@Import({ McpConfiguration, SearchConfig })` alone does **not** boot. See
diff --git a/docs/specs/006-content-extractor/steps.md b/docs/specs/006-content-extractor/steps.md
new file mode 100644
index 0000000..eaa5962
--- /dev/null
+++ b/docs/specs/006-content-extractor/steps.md
@@ -0,0 +1,72 @@
+# Implementation Steps: Content Extractor
+
+## Step 1: `ExtractedContent` record
+
+- [x] Record with `title`, `excerpt`, `body`, `author`, `categories`, `publishedDate`, `previewImage`, `locale`; `categories` normalized to non-null
+
+**Related behaviors:** all (return shape)
+
+---
+
+## Step 2: `ContentExtractor` — body extraction
+
+- [x] `@Component` taking `ContentLocaleResolver` + `ObjectMapper`
+- [x] Read metadata first, then strip `script`/`style`/`noscript`/`template` + comments (so JSON-LD survives)
+- [x] Container selection: `contentSelector` comma list (first non-empty match), `body` = whole document, readability fallback (semantic tag, else largest div/section) when nothing matches
+- [x] Remove `contentExclude` selectors from the container
+- [x] Whitespace-normalized, block-separated body (`\n\n` between paragraphs); fallback to collapsed container text
+
+**Related behaviors:** selector container; comma fallback; body selector; scripts/styles removed; contentExclude; whitespace normalized; readability fallback; empty container
+
+---
+
+## Step 3: `ContentExtractor` — metadata extraction
+
+- [x] title: `og:title` → `<title>` → first `<h1>`
+- [x] excerpt: `description` → `og:description`
+- [x] previewImage: `og:image`
+- [x] publishedDate: `article:published_time` → `<time datetime>` → JSON-LD `datePublished`
+- [x] author: `author`/`article:author` meta → JSON-LD `author.name`
+- [x] categories: `article:tag` metas → JSON-LD `keywords`
+- [x] locale: from `ContentLocaleResolver` (path rule)
+- [x] JSON-LD parsed via Jackson, following arrays and `@graph`
+- [x] Graceful degradation to null when fields are absent
+
+**Related behaviors:** OpenGraph metadata; JSON-LD date/author; article tags; locale; metadata independent of selector; missing metadata
+
+---
+
+## Step 4: Tests
+
+- [x] `ContentExtractorTest` (14 cases) — body selection/cleaning + metadata + edge cases
+
+**Acceptance criteria:**
+- [x] All tests pass (`mvn test`); build green
+
+---
+
+## Behavior Coverage
+
+| Scenario | Layer | Covered in Step |
+|----------|-------|-----------------|
+| Selector picks the article container | Backend | Step 4 (`selectorPicksArticleContainer`) |
+| Comma-separated fallback list, first match wins | Backend | Step 4 (`commaSeparatedFallbackFirstMatchWins`) |
+| body selector takes the whole document | Backend | Step 4 (`bodySelectorTakesWholeDocument`) |
+| Scripts and styles are always removed | Backend | Step 4 (`scriptsAndStylesAreAlwaysRemoved`) |
+| contentExclude removes boilerplate | Backend | Step 4 (`contentExcludeRemovesBoilerplate`) |
+| Whitespace is normalized | Backend | Step 4 (`whitespaceIsNormalized`) |
+| Readability fallback when no selector matches | Backend | Step 4 (`readabilityFallbackWhenNoSelectorMatches`) |
+| OpenGraph metadata is read | Backend | Step 4 (`openGraphMetadataIsRead`) |
+| JSON-LD fallback for date/author | Backend | Step 4 (`jsonLdFallbackForDateAndAuthor`) |
+| Categories from article tags | Backend | Step 4 (`categoriesFromArticleTags`) |
+| Locale derived from path | Backend | Step 4 (`localeDerivedFromPath`) |
+| Metadata read regardless of content selector | Backend | Step 4 (`metadataReadRegardlessOfContentSelector`) |
+| Missing metadata degrades gracefully | Backend | Step 4 (`missingMetadataDegradesGracefully`) |
+| Empty content container | Backend | Step 4 (`emptyContentContainerFallsBack`) |
+
+All scenarios are backend; there is no frontend in this spec.
+
+## Notes
+
+- Excerpt open question: kept `null` when no `description`/`og:description` (no body-snippet fallback) — the behavior allows null.
+- Markdown vs plaintext: plaintext body kept (no `flexmark` dependency added).
diff --git a/src/main/java/com/openelements/content/ContentExtractor.java b/src/main/java/com/openelements/content/ContentExtractor.java
new file mode 100644
index 0000000..345a459
--- /dev/null
+++ b/src/main/java/com/openelements/content/ContentExtractor.java
@@ -0,0 +1,328 @@
+package com.openelements.content;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import org.jsoup.Jsoup;
+import org.jsoup.nodes.Comment;
+import org.jsoup.nodes.Document;
+import org.jsoup.nodes.Element;
+import org.jsoup.nodes.Node;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.stereotype.Component;
+
+/**
+ * Extracts a clean body and metadata from raw HTML using jsoup.
+ *
+ * <p>Body: metadata is read first (before any {@code <script>} is stripped, so JSON-LD survives),
+ * then {@code script}/{@code style}/{@code noscript}/{@code template} elements and comments are
+ * removed. The main container is chosen by the source's {@code contentSelector} (a comma-separated
+ * fallback list; {@code body} means the whole document; a readability fallback applies when nothing
+ * matches), {@code contentExclude} elements are removed, and the text is whitespace-normalized with
+ * paragraph separation preserved.
+ *
+ * <p>Metadata is read independently of the content selector, preferring structured sources:
+ * OpenGraph/Article {@code <meta>} tags, JSON-LD, and {@code <time datetime>}.
+ */
+@Component
+public class ContentExtractor {
+
+    private static final Logger log = LoggerFactory.getLogger(ContentExtractor.class);
+
+    private static final String STRIP_SELECTOR = "script, style, noscript, template";
+    private static final String BLOCK_SELECTOR = "p, li, h1, h2, h3, h4, h5, h6, blockquote, pre";
+
+    private final ContentLocaleResolver localeResolver;
+    private final ObjectMapper objectMapper;
+
+    public ContentExtractor(ContentLocaleResolver localeResolver, ObjectMapper objectMapper) {
+        this.localeResolver = localeResolver;
+        this.objectMapper = objectMapper;
+    }
+
+    /**
+     * Extracts the body and metadata of a page.
+     *
+     * @param source the source (supplies {@code contentSelector}/{@code contentExclude})
+     * @param url    the page URL (used for locale derivation)
+     * @param html   the raw HTML
+     * @return the extracted content
+     */
+    public ExtractedContent extract(ContentSource source, String url, String html) {
+        Document doc = Jsoup.parse(html, url);
+
+        // Read metadata before stripping scripts (JSON-LD lives in a <script> tag).
+        Metadata metadata = extractMetadata(doc);
+        String locale = localeResolver.resolve(url).getLanguage();
+
+        // Now strip non-content nodes for body extraction.
+        doc.select(STRIP_SELECTOR).remove();
+        removeComments(doc);
+        String body = extractBody(doc, source);
+
+        return new ExtractedContent(
+            metadata.title, metadata.excerpt, body, metadata.author,
+            metadata.categories, metadata.publishedDate, metadata.previewImage, locale);
+    }
+
+    // ---- body extraction ----
+
+    private String extractBody(Document doc, ContentSource source) {
+        Element container = selectContainer(doc, source);
+        if (container == null) {
+            return "";
+        }
+        for (String exclude : source.contentExclude()) {
+            container.select(exclude).remove();
+        }
+        return normalizeText(container);
+    }
+
+    private Element selectContainer(Document doc, ContentSource source) {
+        String selector = source.contentSelector();
+        if (selector != null && !selector.isBlank()) {
+            if (selector.strip().equals("body")) {
+                return doc.body();
+            }
+            for (String part : selector.split(",")) {
+                String candidate = part.strip();
+                if (candidate.isEmpty()) {
+                    continue;
+                }
+                Element element = doc.selectFirst(candidate);
+                if (element != null && !element.text().isBlank()) {
+                    return element;
+                }
+            }
+        }
+        return readabilityFallback(doc);
+    }
+
+    /**
+     * Simple readability fallback: prefer a semantic content element, otherwise the {@code <div>} or
+     * {@code <section>} carrying the most text.
+     */
+    private Element readabilityFallback(Document doc) {
+        for (String semantic : List.of("article", "main", "[role=main]")) {
+            Element element = doc.selectFirst(semantic);
+            if (element != null && !element.text().isBlank()) {
+                return element;
+            }
+        }
+        Element best = null;
+        int bestLength = -1;
+        for (Element element : doc.select("div, section")) {
+            int length = element.text().length();
+            if (length > bestLength) {
+                bestLength = length;
+                best = element;
+            }
+        }
+        return best != null ? best : doc.body();
+    }
+
+    /**
+     * Extracts block-separated text: each block element's collapsed text becomes a paragraph, joined
+     * by blank lines. Falls back to the container's collapsed text when it has no block elements.
+     */
+    private String normalizeText(Element container) {
+        List<String> paragraphs = new ArrayList<>();
+        for (Element block : container.select(BLOCK_SELECTOR)) {
+            String text = block.text().strip();
+            if (!text.isEmpty()) {
+                paragraphs.add(text);
+            }
+        }
+        if (paragraphs.isEmpty()) {
+            return container.text().strip();
+        }
+        return String.join("\n\n", paragraphs);
+    }
+
+    private static void removeComments(Document doc) {
+        List<Node> comments = new ArrayList<>();
+        for (Element element : doc.getAllElements()) {
+            for (Node child : element.childNodes()) {
+                if (child instanceof Comment) {
+                    comments.add(child);
+                }
+            }
+        }
+        comments.forEach(Node::remove);
+    }
+
+    // ---- metadata extraction ----
+
+    private Metadata extractMetadata(Document doc) {
+        Metadata metadata = new Metadata();
+
+        metadata.title = firstNonBlank(
+            metaContent(doc, "meta[property=og:title]"),
+            blankToNull(doc.title()),
+            textOf(doc.selectFirst("h1")));
+        metadata.excerpt = firstNonBlank(
+            metaContent(doc, "meta[name=description]"),
+            metaContent(doc, "meta[property=og:description]"));
+        metadata.previewImage = metaContent(doc, "meta[property=og:image]");
+
+        String metaPublished = firstNonBlank(
+            metaContent(doc, "meta[property=article:published_time]"),
+            attrOf(doc.selectFirst("time[datetime]"), "datetime"));
+        String metaAuthor = firstNonBlank(
+            metaContent(doc, "meta[name=author]"),
+            metaContent(doc, "meta[property=article:author]"));
+
+        Set<String> categories = new LinkedHashSet<>();
+        doc.select("meta[property=article:tag]").forEach(tag -> {
+            String value = tag.attr("content").strip();
+            if (!value.isEmpty()) {
+                categories.add(value);
+            }
+        });
+
+        JsonLd jsonLd = readJsonLd(doc);
+        metadata.publishedDate = firstNonBlank(metaPublished, jsonLd.datePublished);
+        metadata.author = firstNonBlank(metaAuthor, jsonLd.author);
+        if (categories.isEmpty()) {
+            categories.addAll(jsonLd.keywords);
+        }
+        metadata.categories = List.copyOf(categories);
+
+        return metadata;
+    }
+
+    private JsonLd readJsonLd(Document doc) {
+        JsonLd result = new JsonLd();
+        for (Element script : doc.select("script[type=application/ld+json]")) {
+            JsonNode root;
+            try {
+                root = objectMapper.readTree(script.data());
+            } catch (Exception e) {
+                log.debug("Ignoring unparseable JSON-LD block: {}", e.toString());
+                continue;
+            }
+            for (JsonNode node : flatten(root)) {
+                if (result.datePublished == null && node.hasNonNull("datePublished")) {
+                    result.datePublished = node.get("datePublished").asText();
+                }
+                if (result.author == null && node.has("author")) {
+                    result.author = authorName(node.get("author"));
+                }
+                if (result.keywords.isEmpty() && node.has("keywords")) {
+                    result.keywords.addAll(keywords(node.get("keywords")));
+                }
+            }
+        }
+        return result;
+    }
+
+    private static String authorName(JsonNode author) {
+        if (author.isObject()) {
+            return blankToNull(author.path("name").asText(""));
+        }
+        if (author.isArray() && !author.isEmpty()) {
+            return authorName(author.get(0));
+        }
+        if (author.isTextual()) {
+            return blankToNull(author.asText());
+        }
+        return null;
+    }
+
+    private static List<String> keywords(JsonNode keywords) {
+        List<String> values = new ArrayList<>();
+        if (keywords.isArray()) {
+            keywords.forEach(k -> {
+                String value = k.asText("").strip();
+                if (!value.isEmpty()) {
+                    values.add(value);
+                }
+            });
+        } else if (keywords.isTextual()) {
+            for (String part : keywords.asText().split(",")) {
+                String value = part.strip();
+                if (!value.isEmpty()) {
+                    values.add(value);
+                }
+            }
+        }
+        return values;
+    }
+
+    /** Flattens JSON-LD into candidate objects, following arrays and {@code @graph} containers. */
+    private static List<JsonNode> flatten(JsonNode root) {
+        List<JsonNode> out = new ArrayList<>();
+        Deque<JsonNode> stack = new ArrayDeque<>();
+        stack.push(root);
+        while (!stack.isEmpty()) {
+            JsonNode node = stack.pop();
+            if (node == null) {
+                continue;
+            }
+            if (node.isArray()) {
+                node.forEach(stack::push);
+            } else {
+                out.add(node);
+                if (node.has("@graph")) {
+                    stack.push(node.get("@graph"));
+                }
+            }
+        }
+        return out;
+    }
+
+    // ---- small helpers ----
+
+    private static String metaContent(Document doc, String cssQuery) {
+        return attrOf(doc.selectFirst(cssQuery), "content");
+    }
+
+    private static String attrOf(Element element, String attribute) {
+        if (element == null) {
+            return null;
+        }
+        return blankToNull(element.attr(attribute));
+    }
+
+    private static String textOf(Element element) {
+        return element == null ? null : blankToNull(element.text());
+    }
+
+    private static String firstNonBlank(String... values) {
+        for (String value : values) {
+            if (value != null && !value.isBlank()) {
+                return value.strip();
+            }
+        }
+        return null;
+    }
+
+    private static String blankToNull(String value) {
+        if (value == null) {
+            return null;
+        }
+        String stripped = value.strip();
+        return stripped.isEmpty() ? null : stripped;
+    }
+
+    private static final class Metadata {
+        String title;
+        String excerpt;
+        String author;
+        String publishedDate;
+        String previewImage;
+        List<String> categories = List.of();
+    }
+
+    private static final class JsonLd {
+        String datePublished;
+        String author;
+        final List<String> keywords = new ArrayList<>();
+    }
+}
diff --git a/src/main/java/com/openelements/content/ExtractedContent.java b/src/main/java/com/openelements/content/ExtractedContent.java
new file mode 100644
index 0000000..9c4a7f2
--- /dev/null
+++ b/src/main/java/com/openelements/content/ExtractedContent.java
@@ -0,0 +1,36 @@
+package com.openelements.content;
+
+import java.util.List;
+
+/**
+ * The result of extracting one page with {@link ContentExtractor}: the cleaned body plus the
+ * metadata read from the document. The indexer (spec 008) maps this together with the URL and source
+ * into a {@link ContentDocument}.
+ *
+ * <p>{@code categories} is normalized to a non-null list; the remaining metadata fields may be
+ * {@code null} when the page does not provide them.
+ *
+ * @param title         the page title
+ * @param excerpt       a short summary, or {@code null}
+ * @param body          the cleaned, whitespace-normalized main text
+ * @param author        the author name, or {@code null}
+ * @param categories    category/tag values (never {@code null})
+ * @param publishedDate the published date/time as found in the page, or {@code null}
+ * @param previewImage  a preview image URL, or {@code null}
+ * @param locale        the derived locale language code (e.g. {@code en}, {@code de})
+ */
+public record ExtractedContent(
+    String title,
+    String excerpt,
+    String body,
+    String author,
+    List<String> categories,
+    String publishedDate,
+    String previewImage,
+    String locale
+) {
+
+    public ExtractedContent {
+        categories = categories == null ? List.of() : List.copyOf(categories);
+    }
+}
diff --git a/src/test/java/com/openelements/content/ContentExtractorTest.java b/src/test/java/com/openelements/content/ContentExtractorTest.java
new file mode 100644
index 0000000..010b89e
--- /dev/null
+++ b/src/test/java/com/openelements/content/ContentExtractorTest.java
@@ -0,0 +1,215 @@
+package com.openelements.content;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.util.List;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Unit tests for {@link ContentExtractor}: body selection/cleaning and metadata extraction.
+ */
+@DisplayName("Content extractor")
+class ContentExtractorTest {
+
+    private final ContentExtractor extractor =
+        new ContentExtractor(new ContentLocaleResolver(), new ObjectMapper());
+
+    private static ContentSource source(String contentSelector, List<String> contentExclude) {
+        return new ContentSource(
+            "test", SourceType.WEBSITE, "https://ex.com",
+            List.of(), List.of("/**"), List.of(), contentSelector, contentExclude, true);
+    }
+
+    private ExtractedContent extract(String contentSelector, List<String> exclude, String html) {
+        return extractor.extract(source(contentSelector, exclude), "https://ex.com/posts/x", html);
+    }
+
+    // ---- body extraction ----
+
+    @Test
+    @DisplayName("the content selector picks the article container, excluding nav/header/footer")
+    void selectorPicksArticleContainer() {
+        String html = "<html><body>"
+            + "<nav><p>Navigation</p></nav><header><p>Site header</p></header>"
+            + "<article><p>The real article body.</p></article>"
+            + "<footer><p>Footer stuff</p></footer>"
+            + "</body></html>";
+
+        String body = extract("article", List.of(), html).body();
+
+        assertThat(body).contains("The real article body.");
+        assertThat(body).doesNotContain("Navigation", "Site header", "Footer stuff");
+    }
+
+    @Test
+    @DisplayName("a comma-separated selector list uses the first match")
+    void commaSeparatedFallbackFirstMatchWins() {
+        String html = "<html><body><main><p>Main content here.</p></main></body></html>";
+
+        String body = extract("article, main, .prose", List.of(), html).body();
+
+        assertThat(body).contains("Main content here.");
+    }
+
+    @Test
+    @DisplayName("the body selector takes the whole document")
+    void bodySelectorTakesWholeDocument() {
+        String html = "<html><body><div>Section one.</div><div>Section two.</div></body></html>";
+
+        String body = extract("body", List.of(), html).body();
+
+        assertThat(body).contains("Section one.").contains("Section two.");
+    }
+
+    @Test
+    @DisplayName("scripts and styles are always removed, even with the body selector")
+    void scriptsAndStylesAreAlwaysRemoved() {
+        String html = "<html><head><style>.a{color:red}</style></head><body>"
+            + "<script>alert('x')</script><p>Visible text.</p></body></html>";
+
+        String body = extract("body", List.of(), html).body();
+
+        assertThat(body).contains("Visible text.");
+        assertThat(body).doesNotContain("alert", "color:red");
+    }
+
+    @Test
+    @DisplayName("contentExclude removes boilerplate elements")
+    void contentExcludeRemovesBoilerplate() {
+        String html = "<html><body>"
+            + "<nav>Navigation</nav><div class=\"cookie-banner\">Accept cookies</div>"
+            + "<p>Actual content.</p><footer>Footer</footer>"
+            + "</body></html>";
+
+        String body = extract("body", List.of("nav", "footer", ".cookie-banner"), html).body();
+
+        assertThat(body).contains("Actual content.");
+        assertThat(body).doesNotContain("Navigation", "Accept cookies", "Footer");
+    }
+
+    @Test
+    @DisplayName("whitespace is collapsed and paragraphs are separated")
+    void whitespaceIsNormalized() {
+        String html = "<html><body><article>"
+            + "<p>First    paragraph\n  with   irregular\twhitespace.</p>"
+            + "<p>Second paragraph.</p>"
+            + "</article></body></html>";
+
+        String body = extract("article", List.of(), html).body();
+
+        assertThat(body).isEqualTo("First paragraph with irregular whitespace.\n\nSecond paragraph.");
+    }
+
+    @Test
+    @DisplayName("a readability fallback is used when no selector matches")
+    void readabilityFallbackWhenNoSelectorMatches() {
+        String html = "<html><body>"
+            + "<nav><p>Nav</p></nav>"
+            + "<article><p>The main readable article content.</p></article>"
+            + "<footer><p>Footer</p></footer>"
+            + "</body></html>";
+
+        // selector matches nothing -> fallback should find the <article>
+        String body = extract(".does-not-exist", List.of(), html).body();
+
+        assertThat(body).contains("The main readable article content.");
+        assertThat(body).doesNotContain("Nav", "Footer");
+    }
+
+    // ---- metadata extraction ----
+
+    @Test
+    @DisplayName("OpenGraph metadata populates title, excerpt, image and published date")
+    void openGraphMetadataIsRead() {
+        String html = "<html><head>"
+            + "<meta property=\"og:title\" content=\"OG Title\">"
+            + "<meta property=\"og:description\" content=\"OG summary\">"
+            + "<meta property=\"og:image\" content=\"https://ex.com/img.png\">"
+            + "<meta property=\"article:published_time\" content=\"2026-03-12T10:00:00Z\">"
+            + "</head><body><article><p>Body</p></article></body></html>";
+
+        ExtractedContent result = extract("article", List.of(), html);
+
+        assertThat(result.title()).isEqualTo("OG Title");
+        assertThat(result.excerpt()).isEqualTo("OG summary");
+        assertThat(result.previewImage()).isEqualTo("https://ex.com/img.png");
+        assertThat(result.publishedDate()).isEqualTo("2026-03-12T10:00:00Z");
+    }
+
+    @Test
+    @DisplayName("JSON-LD supplies date and author when article meta is absent")
+    void jsonLdFallbackForDateAndAuthor() {
+        String html = "<html><head><title>T</title>"
+            + "<script type=\"application/ld+json\">"
+            + "{\"@type\":\"Article\",\"datePublished\":\"2026-03-12\",\"author\":{\"name\":\"Hendrik\"}}"
+            + "</script></head><body><article><p>Body</p></article></body></html>";
+
+        ExtractedContent result = extract("article", List.of(), html);
+
+        assertThat(result.publishedDate()).isEqualTo("2026-03-12");
+        assertThat(result.author()).isEqualTo("Hendrik");
+    }
+
+    @Test
+    @DisplayName("categories are read from multiple article:tag meta entries")
+    void categoriesFromArticleTags() {
+        String html = "<html><head>"
+            + "<meta property=\"article:tag\" content=\"ai\">"
+            + "<meta property=\"article:tag\" content=\"web3\">"
+            + "</head><body><article><p>Body</p></article></body></html>";
+
+        ExtractedContent result = extract("article", List.of(), html);
+
+        assertThat(result.categories()).containsExactly("ai", "web3");
+    }
+
+    @Test
+    @DisplayName("locale is derived from the URL path")
+    void localeDerivedFromPath() {
+        ContentSource source = source("article", List.of());
+        String html = "<html><body><article><p>Inhalt</p></article></body></html>";
+
+        ExtractedContent result = extractor.extract(source, "https://ex.com/de/posts/x", html);
+
+        assertThat(result.locale()).isEqualTo("de");
+    }
+
+    @Test
+    @DisplayName("metadata is read from <head> regardless of the content selector")
+    void metadataReadRegardlessOfContentSelector() {
+        String html = "<html><head><meta property=\"og:title\" content=\"Head Title\"></head>"
+            + "<body><article><span id=\"tiny\">x</span><p>Body</p></article></body></html>";
+
+        ExtractedContent result = extract("#tiny", List.of(), html);
+
+        assertThat(result.title()).isEqualTo("Head Title");
+    }
+
+    // ---- edge cases ----
+
+    @Test
+    @DisplayName("missing metadata degrades to null without error")
+    void missingMetadataDegradesGracefully() {
+        String html = "<html><body><article><p>Just a body, no meta.</p></article></body></html>";
+
+        ExtractedContent result = extract("article", List.of(), html);
+
+        assertThat(result.excerpt()).isNull();
+        assertThat(result.author()).isNull();
+        assertThat(result.previewImage()).isNull();
+        assertThat(result.categories()).isEmpty();
+        assertThat(result.body()).contains("Just a body, no meta.");
+    }
+
+    @Test
+    @DisplayName("an empty matched container falls back and yields an empty body without error")
+    void emptyContentContainerFallsBack() {
+        String html = "<html><body><div id=\"empty\"></div></body></html>";
+
+        String body = extract("#empty", List.of(), html).body();
+
+        assertThat(body).isEmpty();
+    }
+}

From adaae994fa3606f13e84ee80db96d76e72c9f8dd Mon Sep 17 00:00:00 2001
From: Hendrik Ebbers <hendrik.ebbers@open-elements.com>
Date: Sun, 12 Jul 2026 14:38:55 +0200
Subject: [PATCH 3/3] chore(spec-006): mark content-extractor done

Refs #11
---
 docs/specs/INDEX.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/specs/INDEX.md b/docs/specs/INDEX.md
index fee3da8..b1d7587 100644
--- a/docs/specs/INDEX.md
+++ b/docs/specs/INDEX.md
@@ -11,7 +11,7 @@ roadmap step, to be implemented sequentially (each builds on the previous).
 | 003 | 003-content-document-index | Content document & index | backend, search, data-model | `ContentDocument` record + Meilisearch `IndexSettings` bean | #5 | done |
 | 004 | 004-sitemap-crawler | Sitemap crawler | backend, crawler | `SitemapCrawler` — sitemap discovery of URLs + lastmod | #7 | done |
 | 005 | 005-page-fetcher | Page fetcher | backend, crawler | `PageFetcher` — robust HTTP fetch (ETag/IMS, rate limit, retry) | #9 | done |
-| 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | #11 | in progress |
+| 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | #11 | done |
 | 007 | 007-source-strategy | Source strategy | backend, architecture | `ContentSourceStrategy` interface + `WebsiteSourceStrategy` | — | open |
 | 008 | 008-content-indexer | Content indexer | backend, search, crawler | `ContentIndexer` — orchestration, diff, upsert/delete | — | open |
 | 009 | 009-content-bootstrap-step | Content bootstrap step | backend, search | `ContentBootstrapStep` — initial reindex via `SearchIndexBootstrapStep` | — | open |