Source strategy — ContentSourceStrategy + WebsiteSourceStrategy

CLAUDE.md

@@ -45,7 +45,11 @@ extraction, Meilisearch indexing with scheduled refresh, and four MCP tools
 │   ├── DiscoveredItem.java                  # a discovered URL + lastmod change marker
 │   ├── SitemapCrawler.java                  # sitemap/index discovery + bounded fallback crawl
 │   ├── ContentExtractor.java                # jsoup body + metadata extraction
-│   └── ExtractedContent.java                # extractor output (body + metadata)
+│   ├── ExtractedContent.java                # extractor output (body + metadata)
+│   ├── ContentSourceStrategy.java           # per-source-type discover/fetch seam
+│   ├── WebsiteSourceStrategy.java           # website strategy: crawler + fetcher + extractor
+│   ├── FetchOutcome.java                    # INDEX/UNCHANGED/DELETE/SKIP + optional document
+│   └── SourceStrategyRegistry.java          # selects a strategy by ContentSource.type()
 ├── src/main/resources/application.yaml      # datasource, JPA, OAuth2, MCP, Meilisearch, content config
 ├── src/test/java/com/openelements/content/  # behavior tests (context, MCP enabled/disabled, search-down, jsoup)
 └── docs/
@@ -87,6 +91,11 @@ fetched HTML into an `ExtractedContent` via jsoup: it selects the main container
 `contentSelector` (with a readability fallback), always strips scripts/styles, applies
 `contentExclude`, whitespace-normalizes the body, and reads metadata (title/excerpt/date/author/
 categories/preview image/locale) from OpenGraph/Article `<meta>`, JSON-LD, and `<time>`.
+`ContentSourceStrategy` is the per-source-type seam that composes these stages: `WebsiteSourceStrategy`
+wires crawler → fetcher → extractor and maps the fetch result to a `FetchOutcome`
+(`INDEX`/`UNCHANGED`/`DELETE`/`SKIP`); `SourceStrategyRegistry` selects the strategy by
+`ContentSource.type()`, so a future `git` strategy (spec 016) plugs in as a bean with no indexer
+changes.
 
 > **Key gotcha:** the library ships no Spring Boot auto-configuration and couples MCP to a JPA
 > datasource, so `@Import({ McpConfiguration, SearchConfig })` alone does **not** boot. See

docs/specs/007-source-strategy/steps.md

@@ -0,0 +1,61 @@
+# Implementation Steps: Source Strategy
+
+## Step 1: `FetchOutcome`
+
+- [x] Record `FetchOutcome(Result, ContentDocument)` with nested `Result {INDEX, UNCHANGED, DELETE, SKIP}` and factory helpers
+
+**Related behaviors:** all fetch-outcome scenarios (return shape)
+
+---
+
+## Step 2: `ContentSourceStrategy` interface
+
+- [x] `type()`, `discover(source)`, `fetch(source, item)` — the seam the indexer depends on
+
+**Related behaviors:** all (contract)
+
+---
+
+## Step 3: `WebsiteSourceStrategy`
+
+- [x] `@Component` implementing the interface for `SourceType.WEBSITE`
+- [x] `discover` delegates to `SitemapCrawler`
+- [x] `fetch` maps `FetchResult` → `FetchOutcome`: 304 → UNCHANGED, 404/410 → DELETE, error → SKIP, OK → extract → INDEX (empty extraction → SKIP)
+- [x] Builds the `ContentDocument` from `ExtractedContent` + url + source (lastmod = the discovery change marker)
+
+**Related behaviors:** discover delegation; INDEX; UNCHANGED; DELETE; SKIP (error and empty)
+
+---
+
+## Step 4: `SourceStrategyRegistry`
+
+- [x] `@Component` indexing injected `List<ContentSourceStrategy>` by `type()`; `forSource` throws a clear error for unknown types; duplicate types fail fast
+
+**Related behaviors:** website resolves; unknown type fails clearly; new strategy auto-discovered
+
+---
+
+## Step 5: Tests
+
+- [x] `SourceStrategyRegistryTest` — resolve, unknown-type error, auto-discovery, duplicate guard (test-double strategies)
+- [x] `WebsiteSourceStrategyTest` — real `SitemapCrawler`/`PageFetcher`/`ContentExtractor` + `MockRestServiceServer`: type, discover delegation, INDEX/UNCHANGED/DELETE/SKIP(error)/SKIP(empty)
+
+**Acceptance criteria:**
+- [x] All tests pass (`mvn test`); build green
+
+---
+
+## Behavior Coverage
+
+| Scenario | Layer | Covered in Step |
+|----------|-------|-----------------|
+| Website source resolves to website strategy | Backend | Step 5 (`SourceStrategyRegistryTest.websiteSourceResolvesToWebsiteStrategy`) |
+| Unknown type fails clearly | Backend | Step 5 (`SourceStrategyRegistryTest.unknownTypeFailsClearly`) |
+| discover delegates to the sitemap crawler | Backend | Step 5 (`WebsiteSourceStrategyTest.discoverDelegatesToCrawler`) |
+| Successful fetch produces an INDEX outcome | Backend | Step 5 (`WebsiteSourceStrategyTest.successfulFetchProducesIndex`) |
+| 304 produces UNCHANGED | Backend | Step 5 (`WebsiteSourceStrategyTest.notModifiedProducesUnchanged`) |
+| 404 produces DELETE | Backend | Step 5 (`WebsiteSourceStrategyTest.notFoundProducesDelete`) |
+| Extraction failure produces SKIP | Backend | Step 5 (`WebsiteSourceStrategyTest.emptyContentProducesSkip`, `fetchErrorProducesSkip`) |
+| New strategy is auto-discovered | Backend | Step 5 (`SourceStrategyRegistryTest.newStrategyIsAutoDiscovered`) |
+
+All scenarios are backend; there is no frontend in this spec.

docs/specs/INDEX.md

@@ -12,7 +12,7 @@ roadmap step, to be implemented sequentially (each builds on the previous).
 | 004 | 004-sitemap-crawler | Sitemap crawler | backend, crawler | `SitemapCrawler` — sitemap discovery of URLs + lastmod | #7 | done |
 | 005 | 005-page-fetcher | Page fetcher | backend, crawler | `PageFetcher` — robust HTTP fetch (ETag/IMS, rate limit, retry) | #9 | done |
 | 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | #11 | done |
-| 007 | 007-source-strategy | Source strategy | backend, architecture | `ContentSourceStrategy` interface + `WebsiteSourceStrategy` | — | open |
+| 007 | 007-source-strategy | Source strategy | backend, architecture | `ContentSourceStrategy` interface + `WebsiteSourceStrategy` | #13 | done |
 | 008 | 008-content-indexer | Content indexer | backend, search, crawler | `ContentIndexer` — orchestration, diff, upsert/delete | — | open |
 | 009 | 009-content-bootstrap-step | Content bootstrap step | backend, search | `ContentBootstrapStep` — initial reindex via `SearchIndexBootstrapStep` | — | open |
 | 010 | 010-content-refresh-scheduler | Content refresh scheduler | backend, scheduling | `ContentRefreshScheduler` — `@Scheduled` incremental re-crawl | — | open |

src/main/java/com/openelements/content/ContentSourceStrategy.java

@@ -0,0 +1,33 @@
+package com.openelements.content;
+
+import java.util.List;
+
+/**
+ * Strategy for discovering and fetching the content of one {@link SourceType}.
+ *
+ * <p>The indexer (spec 008) depends only on this interface, so a new source type (e.g. {@code git},
+ * spec 016) is added as a new bean without changing the orchestration or the Meilisearch flow. One
+ * implementation exists per {@link SourceType}; {@link SourceStrategyRegistry} selects the right one.
+ */
+public interface ContentSourceStrategy {
+
+    /** @return the source type this strategy handles */
+    SourceType type();
+
+    /**
+     * Discovers the candidate items of a source.
+     *
+     * @param source the source to discover
+     * @return the discovered items (URL + change marker)
+     */
+    List<DiscoveredItem> discover(ContentSource source);
+
+    /**
+     * Fetches a single discovered item and decides what the indexer should do with it.
+     *
+     * @param source the owning source
+     * @param item   the item to fetch
+     * @return the outcome (index/unchanged/delete/skip)
+     */
+    FetchOutcome fetch(ContentSource source, DiscoveredItem item);
+}

src/main/java/com/openelements/content/FetchOutcome.java

@@ -0,0 +1,42 @@
+package com.openelements.content;
+
+/**
+ * What a {@link ContentSourceStrategy} determined should happen with one discovered item, plus the
+ * document when there is one to index.
+ *
+ * <p>This widens the design's {@code ContentDocument fetch(item)} sketch so the non-happy paths
+ * (unchanged, deleted, skipped) are first-class rather than encoded as {@code null} or exceptions.
+ *
+ * @param result   the action the indexer should take
+ * @param document the document to upsert (non-null only for {@link Result#INDEX})
+ */
+public record FetchOutcome(Result result, ContentDocument document) {
+
+    /** The action the indexer should take for a discovered item. */
+    public enum Result {
+        /** A document to upsert into the index. */
+        INDEX,
+        /** Unchanged since last seen (e.g. 304); nothing to do. */
+        UNCHANGED,
+        /** The item no longer exists (e.g. 404/410); remove it from the index. */
+        DELETE,
+        /** Fetch or extraction failed for this item; log and continue with the batch. */
+        SKIP
+    }
+
+    static FetchOutcome index(ContentDocument document) {
+        return new FetchOutcome(Result.INDEX, document);
+    }
+
+    static FetchOutcome unchanged() {
+        return new FetchOutcome(Result.UNCHANGED, null);
+    }
+
+    static FetchOutcome delete() {
+        return new FetchOutcome(Result.DELETE, null);
+    }
+
+    static FetchOutcome skip() {
+        return new FetchOutcome(Result.SKIP, null);
+    }
+}

src/main/java/com/openelements/content/SourceStrategyRegistry.java

@@ -0,0 +1,45 @@
+package com.openelements.content;
+
+import java.util.EnumMap;
+import java.util.List;
+import java.util.Map;
+import org.springframework.stereotype.Component;
+
+/**
+ * Selects the {@link ContentSourceStrategy} for a {@link ContentSource} by its {@link SourceType}.
+ *
+ * <p>All strategy beans are injected and indexed by {@link ContentSourceStrategy#type()}, so a new
+ * strategy is auto-discovered simply by being a bean. Two strategies claiming the same type is a
+ * configuration error and fails fast at startup.
+ */
+@Component
+public class SourceStrategyRegistry {
+
+    private final Map<SourceType, ContentSourceStrategy> byType;
+
+    public SourceStrategyRegistry(List<ContentSourceStrategy> strategies) {
+        Map<SourceType, ContentSourceStrategy> map = new EnumMap<>(SourceType.class);
+        for (ContentSourceStrategy strategy : strategies) {
+            ContentSourceStrategy previous = map.putIfAbsent(strategy.type(), strategy);
+            if (previous != null) {
+                throw new IllegalStateException(
+                    "Multiple content source strategies registered for type " + strategy.type());
+            }
+        }
+        this.byType = map;
+    }
+
+    /**
+     * @param source the source whose strategy is needed
+     * @return the strategy handling the source's type
+     * @throws IllegalArgumentException if no strategy is registered for the source's type
+     */
+    public ContentSourceStrategy forSource(ContentSource source) {
+        ContentSourceStrategy strategy = byType.get(source.type());
+        if (strategy == null) {
+            throw new IllegalArgumentException(
+                "No content source strategy registered for type " + source.type());
+        }
+        return strategy;
+    }
+}

src/main/java/com/openelements/content/WebsiteSourceStrategy.java

@@ -0,0 +1,80 @@
+package com.openelements.content;
+
+import java.util.List;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.stereotype.Component;
+
+/**
+ * {@link ContentSourceStrategy} for {@link SourceType#WEBSITE}: discovery via {@link SitemapCrawler},
+ * fetching via {@link PageFetcher}, and extraction via {@link ContentExtractor}.
+ *
+ * <p>The fetcher's classified {@link FetchResult} is mapped to a {@link FetchOutcome}: {@code 304} →
+ * {@code UNCHANGED}, {@code 404}/{@code 410} → {@code DELETE}, transient/other errors → {@code SKIP},
+ * and a successful fetch with extractable content → {@code INDEX} carrying a {@link ContentDocument}.
+ * An empty extraction is treated as {@code SKIP}.
+ */
+@Component
+public class WebsiteSourceStrategy implements ContentSourceStrategy {
+
+    private static final Logger log = LoggerFactory.getLogger(WebsiteSourceStrategy.class);
+
+    private final SitemapCrawler sitemapCrawler;
+    private final PageFetcher pageFetcher;
+    private final ContentExtractor contentExtractor;
+
+    public WebsiteSourceStrategy(SitemapCrawler sitemapCrawler, PageFetcher pageFetcher,
+                                 ContentExtractor contentExtractor) {
+        this.sitemapCrawler = sitemapCrawler;
+        this.pageFetcher = pageFetcher;
+        this.contentExtractor = contentExtractor;
+    }
+
+    @Override
+    public SourceType type() {
+        return SourceType.WEBSITE;
+    }
+
+    @Override
+    public List<DiscoveredItem> discover(ContentSource source) {
+        return sitemapCrawler.discover(source);
+    }
+
+    @Override
+    public FetchOutcome fetch(ContentSource source, DiscoveredItem item) {
+        // Prior ETag/Last-Modified for the conditional GET are supplied by the indexer's diff
+        // (spec 008); until then this passes null and relies on the sitemap lastmod diff.
+        FetchResult result = pageFetcher.fetch(item.url(), null, null);
+        return switch (result.status()) {
+            case NOT_MODIFIED -> FetchOutcome.unchanged();
+            case NOT_FOUND -> FetchOutcome.delete();
+            case ERROR -> {
+                log.warn("Skipping {}: fetch failed (http {})", item.url(), result.httpStatus());
+                yield FetchOutcome.skip();
+            }
+            case OK -> toIndexOutcome(source, item, result);
+        };
+    }
+
+    private FetchOutcome toIndexOutcome(ContentSource source, DiscoveredItem item, FetchResult result) {
+        ExtractedContent extracted = contentExtractor.extract(source, item.url(), result.html());
+        if (extracted.body() == null || extracted.body().isBlank()) {
+            log.warn("Skipping {}: no extractable content", item.url());
+            return FetchOutcome.skip();
+        }
+        ContentDocument document = new ContentDocument(
+            ContentDocument.id(source.id(), item.url()),
+            source.id(),
+            extracted.locale(),
+            item.url(),
+            extracted.title(),
+            extracted.excerpt(),
+            extracted.body(),
+            extracted.author(),
+            extracted.categories(),
+            extracted.publishedDate(),
+            item.lastmod(),
+            extracted.previewImage());
+        return FetchOutcome.index(document);
+    }
+}

src/test/java/com/openelements/content/SourceStrategyRegistryTest.java

@@ -0,0 +1,77 @@
+package com.openelements.content;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.util.List;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Unit tests for {@link SourceStrategyRegistry} using lightweight test-double strategies.
+ */
+@DisplayName("Source strategy registry")
+class SourceStrategyRegistryTest {
+
+    private static ContentSource source(SourceType type) {
+        return new ContentSource(
+            "s", type, "https://ex.com", List.of(), List.of("/**"), List.of(), "article", List.of(), true);
+    }
+
+    private static ContentSourceStrategy strategyFor(SourceType type) {
+        return new ContentSourceStrategy() {
+            @Override
+            public SourceType type() {
+                return type;
+            }
+
+            @Override
+            public List<DiscoveredItem> discover(ContentSource src) {
+                return List.of();
+            }
+
+            @Override
+            public FetchOutcome fetch(ContentSource src, DiscoveredItem item) {
+                return FetchOutcome.skip();
+            }
+        };
+    }
+
+    @Test
+    @DisplayName("a WEBSITE source resolves to the website strategy")
+    void websiteSourceResolvesToWebsiteStrategy() {
+        ContentSourceStrategy website = strategyFor(SourceType.WEBSITE);
+        SourceStrategyRegistry registry = new SourceStrategyRegistry(List.of(website));
+
+        assertThat(registry.forSource(source(SourceType.WEBSITE))).isSameAs(website);
+    }
+
+    @Test
+    @DisplayName("an unregistered type fails with a clear error naming the type")
+    void unknownTypeFailsClearly() {
+        SourceStrategyRegistry registry = new SourceStrategyRegistry(List.of(strategyFor(SourceType.WEBSITE)));
+
+        assertThatThrownBy(() -> registry.forSource(source(SourceType.GIT)))
+            .isInstanceOf(IllegalArgumentException.class)
+            .hasMessageContaining("GIT");
+    }
+
+    @Test
+    @DisplayName("an additional strategy bean is auto-discovered and indexed by its type")
+    void newStrategyIsAutoDiscovered() {
+        ContentSourceStrategy git = strategyFor(SourceType.GIT);
+        SourceStrategyRegistry registry =
+            new SourceStrategyRegistry(List.of(strategyFor(SourceType.WEBSITE), git));
+
+        assertThat(registry.forSource(source(SourceType.GIT))).isSameAs(git);
+    }
+
+    @Test
+    @DisplayName("two strategies for the same type fail fast")
+    void duplicateStrategiesFailFast() {
+        assertThatThrownBy(() -> new SourceStrategyRegistry(
+            List.of(strategyFor(SourceType.WEBSITE), strategyFor(SourceType.WEBSITE))))
+            .isInstanceOf(IllegalStateException.class)
+            .hasMessageContaining("WEBSITE");
+    }
+}