diff --git a/CLAUDE.md b/CLAUDE.md index 4fa022e..52e7a1e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -49,7 +49,11 @@ extraction, Meilisearch indexing with scheduled refresh, and four MCP tools │ ├── ContentSourceStrategy.java # per-source-type discover/fetch seam │ ├── WebsiteSourceStrategy.java # website strategy: crawler + fetcher + extractor │ ├── FetchOutcome.java # INDEX/UNCHANGED/DELETE/SKIP + optional document -│ └── SourceStrategyRegistry.java # selects a strategy by ContentSource.type() +│ ├── SourceStrategyRegistry.java # selects a strategy by ContentSource.type() +│ ├── ContentIndexer.java # orchestrates discover→diff→fetch→upsert/delete +│ ├── IndexReport.java # per-pass counters +│ ├── ContentIndexStore.java # index read/write seam (StoredDocument) +│ └── MeilisearchContentIndexStore.java # ContentIndexStore backed by MeilisearchClient ├── src/main/resources/application.yaml # datasource, JPA, OAuth2, MCP, Meilisearch, content config ├── src/test/java/com/openelements/content/ # behavior tests (context, MCP enabled/disabled, search-down, jsoup) └── docs/ @@ -95,7 +99,13 @@ categories/preview image/locale) from OpenGraph/Article ``, JSON-LD, and ` wires crawler → fetcher → extractor and maps the fetch result to a `FetchOutcome` (`INDEX`/`UNCHANGED`/`DELETE`/`SKIP`); `SourceStrategyRegistry` selects the strategy by `ContentSource.type()`, so a future `git` strategy (spec 016) plugs in as a bean with no indexer -changes. +changes. `ContentIndexer` is the orchestration engine: discover → diff discovered `lastmod` against +the state read from the index (`ContentIndexStore`; the index *is* the state) → fetch only new/changed +items via the strategy → batch-upsert, and delete documents that 404 or vanished from discovery. +`MeilisearchContentIndexStore` implements the store over `MeilisearchClient` (paged `multiSearch` to +read state, `addDocuments`+`waitForTask` to upsert, `deleteDocument`), since the library's +`BatchWriter` is package-private. The indexer is the reusable engine for the bootstrap step (009) and +refresh scheduler (010). > **Key gotcha:** the library ships no Spring Boot auto-configuration and couples MCP to a JPA > datasource, so `@Import({ McpConfiguration, SearchConfig })` alone does **not** boot. See diff --git a/docs/specs/008-content-indexer/steps.md b/docs/specs/008-content-indexer/steps.md new file mode 100644 index 0000000..ee53661 --- /dev/null +++ b/docs/specs/008-content-indexer/steps.md @@ -0,0 +1,61 @@ +# Implementation Steps: Content Indexer + +## Step 1: `IndexReport` + +- [x] Record `IndexReport(discovered, upserted, unchanged, deleted, skipped)` + +**Related behaviors:** all (report counters) + +--- + +## Step 2: `ContentIndexStore` seam + +- [x] Interface `loadState(source) → id→StoredDocument`, `upsert(docs) → count`, `delete(id)`; nested `StoredDocument(url, lastmod)` +- [x] `MeilisearchContentIndexStore` production impl: paged filtered `multiSearch` (testable `parseHits`), batched `addDocuments` + `waitForTask` (BatchWriter is package-private), `deleteDocument`; gated on `meilisearch.enabled` + +**Related behaviors:** underpins all (diff/upsert/delete) + +--- + +## Step 3: `ContentIndexer` + +- [x] `@Component` (gated on `meilisearch.enabled`) taking `SourceStrategyRegistry` + `ContentIndexStore` +- [x] `indexSource`: discover → load stored state → per item: unchanged (lastmod equal) skip; else strategy.fetch → INDEX(collect)/UNCHANGED/DELETE/SKIP; batch upsert; delete stored-but-not-discovered +- [x] `streamAllDocuments`: lazy discover→fetch→INDEX map stream for bootstrap +- [x] Idempotent (upsert by stable id); fault-tolerant (SKIP one, continue) + +**Related behaviors:** new/unchanged/changed/304/vanished/404/idempotent/skip/dedupe/stream + +--- + +## Step 4: Tests + +- [x] `ContentIndexerTest` (10) — in-memory store + stub strategy: every behavior +- [x] `MeilisearchContentIndexStoreTest` (3) — `parseHits` parsing (hits, empty, missing id) + +**Acceptance criteria:** +- [x] All tests pass (`mvn test`); build green + +--- + +## Behavior Coverage + +| Scenario | Layer | Covered in Step | +|----------|-------|-----------------| +| New pages are upserted | Backend | Step 4 (`newPagesAreUpserted`) | +| Unchanged pages are skipped | Backend | Step 4 (`unchangedPagesAreSkipped`) | +| Changed page is re-fetched and upserted | Backend | Step 4 (`changedPageIsReFetched`) | +| Conditional GET short-circuits a lastmod false-positive | Backend | Step 4 (`conditionalGetShortCircuits`) | +| Vanished URL is deleted | Backend | Step 4 (`vanishedUrlIsDeleted`) | +| 404 on fetch deletes the document | Backend | Step 4 (`notFoundDeletesDocument`) | +| Re-running is idempotent | Backend | Step 4 (`reRunIsIdempotent`) | +| One failing page does not abort the batch | Backend | Step 4 (`oneFailingPageIsSkipped`) | +| Stable id prevents duplicates | Backend | Step 4 (`stableIdPreventsDuplicates`) | +| streamAllDocuments yields every in-scope document | Backend | Step 4 (`streamAllDocumentsYieldsAll`) | + +All scenarios are backend; there is no frontend in this spec. + +## Notes / deviations + +- The design suggested `BatchWriter.write(...)`, but `BatchWriter` is package-private in `spring-services` and not reusable from this package. Upsert uses `MeilisearchClient.addDocuments` in batches with `waitForTask`. +- A `ContentIndexStore` seam was introduced (not in the original design) to encapsulate the Meilisearch read/write and make the orchestration unit-testable without a live instance. The production impl's `parseHits` is unit-tested; the `multiSearch` paging / `addDocuments` batching is thin delegation exercised end-to-end by specs 009/010 against real Meilisearch. diff --git a/docs/specs/INDEX.md b/docs/specs/INDEX.md index 40bf92e..e732467 100644 --- a/docs/specs/INDEX.md +++ b/docs/specs/INDEX.md @@ -13,7 +13,7 @@ roadmap step, to be implemented sequentially (each builds on the previous). | 005 | 005-page-fetcher | Page fetcher | backend, crawler | `PageFetcher` — robust HTTP fetch (ETag/IMS, rate limit, retry) | #9 | done | | 006 | 006-content-extractor | Content extractor | backend, crawler | `ContentExtractor` — jsoup content + metadata extraction | #11 | done | | 007 | 007-source-strategy | Source strategy | backend, architecture | `ContentSourceStrategy` interface + `WebsiteSourceStrategy` | #13 | done | -| 008 | 008-content-indexer | Content indexer | backend, search, crawler | `ContentIndexer` — orchestration, diff, upsert/delete | — | open | +| 008 | 008-content-indexer | Content indexer | backend, search, crawler | `ContentIndexer` — orchestration, diff, upsert/delete | #15 | done | | 009 | 009-content-bootstrap-step | Content bootstrap step | backend, search | `ContentBootstrapStep` — initial reindex via `SearchIndexBootstrapStep` | — | open | | 010 | 010-content-refresh-scheduler | Content refresh scheduler | backend, scheduling | `ContentRefreshScheduler` — `@Scheduled` incremental re-crawl | — | open | | 011 | 011-content-search-service | Content search service | backend, search | `ContentSearchService` — `multiSearch` + highlighting facade | — | open | diff --git a/src/main/java/com/openelements/content/ContentIndexStore.java b/src/main/java/com/openelements/content/ContentIndexStore.java new file mode 100644 index 0000000..dc8d642 --- /dev/null +++ b/src/main/java/com/openelements/content/ContentIndexStore.java @@ -0,0 +1,47 @@ +package com.openelements.content; + +import java.util.List; +import java.util.Map; + +/** + * The persistence seam the {@link ContentIndexer} uses to read and write index state. + * + *

The Meilisearch content index is the state: {@link #loadState(String)} returns the + * stored change-marker per document so the indexer can diff without a separate datastore. Isolating + * these three operations keeps the indexer's orchestration logic testable with an in-memory + * implementation, independent of a running Meilisearch. + */ +public interface ContentIndexStore { + + /** + * Loads the currently-indexed documents for a source. + * + * @param source the source id + * @return a map of document id → {@link StoredDocument} (its URL and change marker) + */ + Map loadState(String source); + + /** + * Upserts documents (keyed by their {@code id}); existing documents with the same id are replaced. + * + * @param documents document attribute maps (from {@link ContentDocument#toMap()}) + * @return the number of documents written + */ + int upsert(List> documents); + + /** + * Removes a document from the index. + * + * @param id the stable document id + */ + void delete(String id); + + /** + * The stored projection of an indexed document needed for diffing. + * + * @param url the document URL + * @param lastmod the stored change marker, or {@code null} + */ + record StoredDocument(String url, String lastmod) { + } +} diff --git a/src/main/java/com/openelements/content/ContentIndexer.java b/src/main/java/com/openelements/content/ContentIndexer.java new file mode 100644 index 0000000..dc9e387 --- /dev/null +++ b/src/main/java/com/openelements/content/ContentIndexer.java @@ -0,0 +1,121 @@ +package com.openelements.content; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.stream.Stream; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.stereotype.Component; + +/** + * Orchestrates the ingestion pipeline for one source: Discover → Diff → Fetch → Extract → Index. + * + *

Discovery yields URLs with change markers; the indexer diffs these against the state stored in + * the index (via {@link ContentIndexStore}) and only fetches new or changed items. Fetched documents + * are upserted; items reported gone (404) or no longer present in discovery are deleted. Upserts are + * keyed by the stable document id, so re-running is idempotent, and a single failing page is skipped + * without aborting the pass. + * + *

This is the reusable engine invoked by the bootstrap step (spec 009) and the refresh scheduler + * (spec 010). It is only created when the Meilisearch stack is enabled. + */ +@Component +@ConditionalOnProperty(prefix = "openelements.meilisearch", name = "enabled", havingValue = "true") +public class ContentIndexer { + + private static final Logger log = LoggerFactory.getLogger(ContentIndexer.class); + + private final SourceStrategyRegistry strategyRegistry; + private final ContentIndexStore indexStore; + + public ContentIndexer(SourceStrategyRegistry strategyRegistry, ContentIndexStore indexStore) { + this.strategyRegistry = strategyRegistry; + this.indexStore = indexStore; + } + + /** + * Runs a full incremental pass over one source. + * + * @param source the source to index + * @return a report of what happened + */ + public IndexReport indexSource(ContentSource source) { + ContentSourceStrategy strategy = strategyRegistry.forSource(source); + List discovered = strategy.discover(source); + Map stored = indexStore.loadState(source.id()); + + List> toUpsert = new ArrayList<>(); + Set discoveredIds = new HashSet<>(); + int upserted = 0; + int unchanged = 0; + int deleted = 0; + int skipped = 0; + + for (DiscoveredItem item : discovered) { + String id = ContentDocument.id(source.id(), item.url()); + discoveredIds.add(id); + ContentIndexStore.StoredDocument existing = stored.get(id); + + if (existing != null && isUnchanged(existing.lastmod(), item.lastmod())) { + unchanged++; + continue; + } + + FetchOutcome outcome = strategy.fetch(source, item); + switch (outcome.result()) { + case INDEX -> { + toUpsert.add(outcome.document().toMap()); + upserted++; + } + case UNCHANGED -> unchanged++; + case DELETE -> { + if (existing != null) { + indexStore.delete(id); + deleted++; + } + } + case SKIP -> skipped++; + } + } + + if (!toUpsert.isEmpty()) { + indexStore.upsert(toUpsert); + } + + // Documents still in the index but no longer discovered have been removed at the source. + for (String storedId : stored.keySet()) { + if (!discoveredIds.contains(storedId)) { + indexStore.delete(storedId); + deleted++; + } + } + + IndexReport report = new IndexReport(discovered.size(), upserted, unchanged, deleted, skipped); + log.info("Indexed source {}: {}", source.id(), report); + return report; + } + + /** + * Lazily fetches and yields every in-scope document of a source, without diffing against the + * index — used for a full (re)build by the bootstrap step (spec 009). + * + * @param source the source + * @return a lazy stream of document attribute maps ready for the index + */ + public Stream> streamAllDocuments(ContentSource source) { + ContentSourceStrategy strategy = strategyRegistry.forSource(source); + return strategy.discover(source).stream() + .map(item -> strategy.fetch(source, item)) + .filter(outcome -> outcome.result() == FetchOutcome.Result.INDEX) + .map(outcome -> outcome.document().toMap()); + } + + /** A discovered item is unchanged only when it carries a change marker equal to the stored one. */ + private static boolean isUnchanged(String storedLastmod, String discoveredLastmod) { + return discoveredLastmod != null && discoveredLastmod.equals(storedLastmod); + } +} diff --git a/src/main/java/com/openelements/content/IndexReport.java b/src/main/java/com/openelements/content/IndexReport.java new file mode 100644 index 0000000..adc3f76 --- /dev/null +++ b/src/main/java/com/openelements/content/IndexReport.java @@ -0,0 +1,13 @@ +package com.openelements.content; + +/** + * Summary of one indexing pass over a source. + * + * @param discovered number of items discovery returned + * @param upserted documents fetched/extracted and written to the index + * @param unchanged items skipped because they were unchanged (by {@code lastmod} diff or a 304) + * @param deleted documents removed (vanished from discovery, or 404 on fetch) + * @param skipped items that failed to fetch or extract and were skipped + */ +public record IndexReport(int discovered, int upserted, int unchanged, int deleted, int skipped) { +} diff --git a/src/main/java/com/openelements/content/MeilisearchContentIndexStore.java b/src/main/java/com/openelements/content/MeilisearchContentIndexStore.java new file mode 100644 index 0000000..d31f287 --- /dev/null +++ b/src/main/java/com/openelements/content/MeilisearchContentIndexStore.java @@ -0,0 +1,133 @@ +package com.openelements.content; + +import com.fasterxml.jackson.databind.JsonNode; +import com.openelements.spring.base.services.search.MeilisearchClient; +import com.openelements.spring.base.services.search.MeilisearchProperties; +import com.openelements.spring.base.services.search.TaskOutcome; +import java.time.Duration; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.stereotype.Component; + +/** + * {@link ContentIndexStore} backed by the library's {@link MeilisearchClient}. + * + *

State is read with a filtered {@code multiSearch} (retrieving only {@code id}/{@code url}/ + * {@code lastmod}, paged); upserts use {@code addDocuments} in batches (the library {@code BatchWriter} + * is package-private and not reusable here), waiting for each task; deletes use {@code deleteDocument}. + * Write failures are logged and do not abort the pass. + * + *

Only created when the Meilisearch stack is enabled. + */ +@Component +@ConditionalOnProperty(prefix = "openelements.meilisearch", name = "enabled", havingValue = "true") +public class MeilisearchContentIndexStore implements ContentIndexStore { + + private static final Logger log = LoggerFactory.getLogger(MeilisearchContentIndexStore.class); + + private static final int PAGE_SIZE = 1000; + private static final int BATCH_SIZE = 500; + private static final Duration TASK_WAIT = Duration.ofSeconds(10); + + private final MeilisearchClient client; + private final String indexUid; + + public MeilisearchContentIndexStore(MeilisearchClient client, MeilisearchProperties properties) { + this.client = client; + this.indexUid = properties.resolveIndex("content"); + } + + @Override + public Map loadState(String source) { + Map state = new LinkedHashMap<>(); + int offset = 0; + while (true) { + Map query = new LinkedHashMap<>(); + query.put("indexUid", indexUid); + query.put("q", ""); + query.put("filter", "source = \"" + source.replace("\"", "\\\"") + "\""); + query.put("attributesToRetrieve", List.of("id", "url", "lastmod")); + query.put("limit", PAGE_SIZE); + query.put("offset", offset); + + JsonNode response = client.multiSearch(Map.of("queries", List.of(query))); + Map page = parseHits(response); + state.putAll(page); + if (page.size() < PAGE_SIZE) { + break; + } + offset += PAGE_SIZE; + } + return state; + } + + @Override + public int upsert(List> documents) { + int written = 0; + for (int start = 0; start < documents.size(); start += BATCH_SIZE) { + List> batch = documents.subList(start, Math.min(start + BATCH_SIZE, documents.size())); + try { + long task = client.addDocuments(indexUid, batch); + TaskOutcome outcome = client.waitForTask(task, TASK_WAIT); + if (outcome == TaskOutcome.SUCCEEDED) { + written += batch.size(); + } else { + log.warn("Upsert batch of {} did not succeed: {}", batch.size(), outcome); + } + } catch (Exception e) { + log.warn("Failed to upsert a batch of {} documents: {}", batch.size(), e.toString()); + } + } + return written; + } + + @Override + public void delete(String id) { + try { + long task = client.deleteDocument(indexUid, id); + client.waitForTask(task, TASK_WAIT); + } catch (Exception e) { + log.warn("Failed to delete document {}: {}", id, e.toString()); + } + } + + /** + * Parses the hits of a {@code multiSearch} response (first query result) into a document-id map. + * Package-private and static so it can be unit-tested without a Meilisearch instance. + * + * @param response the raw multiSearch response + * @return document id → stored projection + */ + static Map parseHits(JsonNode response) { + Map result = new LinkedHashMap<>(); + JsonNode results = response.path("results"); + if (!results.isArray() || results.isEmpty()) { + return result; + } + JsonNode hits = results.get(0).path("hits"); + if (!hits.isArray()) { + return result; + } + for (JsonNode hit : hits) { + String id = textOrNull(hit, "id"); + if (id == null) { + continue; + } + result.put(id, new StoredDocument(textOrNull(hit, "url"), textOrNull(hit, "lastmod"))); + } + return result; + } + + private static String textOrNull(JsonNode node, String field) { + JsonNode value = node.get(field); + if (value == null || value.isNull()) { + return null; + } + String text = value.asText(); + return text.isBlank() ? null : text; + } +} diff --git a/src/test/java/com/openelements/content/ContentIndexerTest.java b/src/test/java/com/openelements/content/ContentIndexerTest.java new file mode 100644 index 0000000..a33b645 --- /dev/null +++ b/src/test/java/com/openelements/content/ContentIndexerTest.java @@ -0,0 +1,235 @@ +package com.openelements.content; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.function.Function; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * Unit tests for {@link ContentIndexer} using an in-memory {@link ContentIndexStore} and a + * test-double {@link ContentSourceStrategy}, so the diff/upsert/delete orchestration is verified + * without a running Meilisearch or any mocks. + */ +@DisplayName("Content indexer") +class ContentIndexerTest { + + private static final ContentSource SOURCE = new ContentSource( + "oe", SourceType.WEBSITE, "https://ex.com", List.of(), List.of("/**"), List.of(), "article", List.of(), true); + + private final InMemoryContentIndexStore store = new InMemoryContentIndexStore(); + private final StubStrategy strategy = new StubStrategy(); + private final ContentIndexer indexer = + new ContentIndexer(new SourceStrategyRegistry(List.of(strategy)), store); + + private static DiscoveredItem item(String path, String lastmod) { + return new DiscoveredItem("https://ex.com" + path, lastmod); + } + + private static ContentDocument document(String path, String lastmod) { + String url = "https://ex.com" + path; + return new ContentDocument( + ContentDocument.id("oe", url), "oe", "en", url, "T", "E", "body text", "a", + List.of("x"), "2026-01-01", lastmod, null); + } + + @Test + @DisplayName("new pages are fetched and upserted") + void newPagesAreUpserted() { + strategy.discovered = List.of(item("/a", "L"), item("/b", "L"), item("/c", "L")); + strategy.fetchFn = it -> FetchOutcome.index(document(pathOf(it), "L")); + + IndexReport report = indexer.indexSource(SOURCE); + + assertThat(report.discovered()).isEqualTo(3); + assertThat(report.upserted()).isEqualTo(3); + assertThat(store.loadState("oe")).hasSize(3); + } + + @Test + @DisplayName("unchanged pages are skipped without fetching") + void unchangedPagesAreSkipped() { + store.upsert(List.of(document("/a", "L").toMap(), document("/b", "L").toMap(), document("/c", "L").toMap())); + strategy.discovered = List.of(item("/a", "L"), item("/b", "L"), item("/c", "L")); + strategy.fetchFn = it -> { + throw new AssertionError("fetch must not be called for unchanged items"); + }; + + IndexReport report = indexer.indexSource(SOURCE); + + assertThat(report.unchanged()).isEqualTo(3); + assertThat(report.upserted()).isZero(); + assertThat(strategy.fetched).isEmpty(); + } + + @Test + @DisplayName("a page whose lastmod changed is re-fetched and upserted") + void changedPageIsReFetched() { + store.upsert(List.of(document("/a", "old").toMap())); + strategy.discovered = List.of(item("/a", "new")); + strategy.fetchFn = it -> FetchOutcome.index(document("/a", "new")); + + IndexReport report = indexer.indexSource(SOURCE); + + assertThat(report.upserted()).isEqualTo(1); + assertThat(strategy.fetched).containsExactly("https://ex.com/a"); + assertThat(store.loadState("oe").get(ContentDocument.id("oe", "https://ex.com/a")).lastmod()) + .isEqualTo("new"); + } + + @Test + @DisplayName("a 304 (UNCHANGED) after a lastmod change causes no upsert") + void conditionalGetShortCircuits() { + store.upsert(List.of(document("/a", "old").toMap())); + strategy.discovered = List.of(item("/a", "new")); + strategy.fetchFn = it -> FetchOutcome.unchanged(); + + IndexReport report = indexer.indexSource(SOURCE); + + assertThat(report.unchanged()).isEqualTo(1); + assertThat(report.upserted()).isZero(); + assertThat(strategy.fetched).containsExactly("https://ex.com/a"); // fetch was attempted + } + + @Test + @DisplayName("a document no longer present in discovery is deleted") + void vanishedUrlIsDeleted() { + store.upsert(List.of(document("/a", "L").toMap(), document("/b", "L").toMap())); + strategy.discovered = List.of(item("/a", "L")); + + IndexReport report = indexer.indexSource(SOURCE); + + assertThat(report.unchanged()).isEqualTo(1); + assertThat(report.deleted()).isEqualTo(1); + assertThat(store.loadState("oe")).containsOnlyKeys(ContentDocument.id("oe", "https://ex.com/a")); + } + + @Test + @DisplayName("a 404 (DELETE) removes the document by stable id") + void notFoundDeletesDocument() { + store.upsert(List.of(document("/a", "old").toMap())); + strategy.discovered = List.of(item("/a", "new")); + strategy.fetchFn = it -> FetchOutcome.delete(); + + IndexReport report = indexer.indexSource(SOURCE); + + assertThat(report.deleted()).isEqualTo(1); + assertThat(store.loadState("oe")).isEmpty(); + } + + @Test + @DisplayName("re-running with no changes is idempotent") + void reRunIsIdempotent() { + strategy.discovered = List.of(item("/a", "L"), item("/b", "L")); + strategy.fetchFn = it -> FetchOutcome.index(document(pathOf(it), "L")); + indexer.indexSource(SOURCE); + strategy.fetched.clear(); + + IndexReport second = indexer.indexSource(SOURCE); + + assertThat(second.unchanged()).isEqualTo(2); + assertThat(second.upserted()).isZero(); + assertThat(second.deleted()).isZero(); + assertThat(store.loadState("oe")).hasSize(2); + } + + @Test + @DisplayName("one failing page does not abort the batch") + void oneFailingPageIsSkipped() { + strategy.discovered = List.of( + item("/a", "L"), item("/b", "L"), item("/c", "L"), item("/d", "L"), item("/e", "L")); + strategy.fetchFn = it -> pathOf(it).equals("/e") + ? FetchOutcome.skip() + : FetchOutcome.index(document(pathOf(it), "L")); + + IndexReport report = indexer.indexSource(SOURCE); + + assertThat(report.upserted()).isEqualTo(4); + assertThat(report.skipped()).isEqualTo(1); + assertThat(store.loadState("oe")).hasSize(4); + } + + @Test + @DisplayName("the stable id prevents duplicates across runs") + void stableIdPreventsDuplicates() { + strategy.discovered = List.of(item("/a", "L1")); + strategy.fetchFn = it -> FetchOutcome.index(document("/a", "L1")); + indexer.indexSource(SOURCE); + + strategy.discovered = List.of(item("/a", "L2")); + strategy.fetchFn = it -> FetchOutcome.index(document("/a", "L2")); + indexer.indexSource(SOURCE); + + assertThat(store.loadState("oe")).hasSize(1); + } + + @Test + @DisplayName("streamAllDocuments lazily yields every in-scope document") + void streamAllDocumentsYieldsAll() { + strategy.discovered = List.of(item("/a", "L"), item("/b", "L"), item("/c", "L")); + strategy.fetchFn = it -> FetchOutcome.index(document(pathOf(it), "L")); + + List> documents = indexer.streamAllDocuments(SOURCE).toList(); + + assertThat(documents).hasSize(3); + assertThat(documents).allSatisfy(doc -> assertThat(doc).containsKey("id")); + } + + private static String pathOf(DiscoveredItem item) { + return item.url().substring("https://ex.com".length()); + } + + /** In-memory {@link ContentIndexStore}: documents keyed by id. */ + private static final class InMemoryContentIndexStore implements ContentIndexStore { + private final Map> documents = new LinkedHashMap<>(); + + @Override + public Map loadState(String source) { + Map state = new LinkedHashMap<>(); + documents.forEach((id, doc) -> { + if (source.equals(doc.get("source"))) { + state.put(id, new StoredDocument((String) doc.get("url"), (String) doc.get("lastmod"))); + } + }); + return state; + } + + @Override + public int upsert(List> docs) { + docs.forEach(doc -> documents.put((String) doc.get("id"), doc)); + return docs.size(); + } + + @Override + public void delete(String id) { + documents.remove(id); + } + } + + /** Test-double strategy with configurable discovery and fetch behavior, recording fetches. */ + private static final class StubStrategy implements ContentSourceStrategy { + private List discovered = List.of(); + private Function fetchFn = it -> FetchOutcome.skip(); + private final List fetched = new ArrayList<>(); + + @Override + public SourceType type() { + return SourceType.WEBSITE; + } + + @Override + public List discover(ContentSource source) { + return discovered; + } + + @Override + public FetchOutcome fetch(ContentSource source, DiscoveredItem item) { + fetched.add(item.url()); + return fetchFn.apply(item); + } + } +} diff --git a/src/test/java/com/openelements/content/MeilisearchContentIndexStoreTest.java b/src/test/java/com/openelements/content/MeilisearchContentIndexStoreTest.java new file mode 100644 index 0000000..6050e9c --- /dev/null +++ b/src/test/java/com/openelements/content/MeilisearchContentIndexStoreTest.java @@ -0,0 +1,62 @@ +package com.openelements.content; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import java.util.Map; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * Unit tests for {@link MeilisearchContentIndexStore#parseHits(JsonNode)} — the parsing of a + * Meilisearch multi-search response, verifiable without a running Meilisearch. + */ +@DisplayName("Meilisearch index store — hit parsing") +class MeilisearchContentIndexStoreTest { + + private final ObjectMapper objectMapper = new ObjectMapper(); + + private JsonNode json(String raw) throws Exception { + return objectMapper.readTree(raw); + } + + @Test + @DisplayName("hits are parsed into id -> (url, lastmod), tolerating a missing lastmod") + void parsesHits() throws Exception { + JsonNode response = json(""" + {"results":[{"hits":[ + {"id":"oe_abc","url":"https://ex.com/a","lastmod":"2026-01-01"}, + {"id":"oe_def","url":"https://ex.com/b"} + ]}]}"""); + + Map state = MeilisearchContentIndexStore.parseHits(response); + + assertThat(state).hasSize(2); + assertThat(state.get("oe_abc")).isEqualTo( + new ContentIndexStore.StoredDocument("https://ex.com/a", "2026-01-01")); + assertThat(state.get("oe_def")).isEqualTo( + new ContentIndexStore.StoredDocument("https://ex.com/b", null)); + } + + @Test + @DisplayName("an empty result set yields an empty map") + void parsesEmptyResults() throws Exception { + assertThat(MeilisearchContentIndexStore.parseHits(json("{\"results\":[{\"hits\":[]}]}"))).isEmpty(); + assertThat(MeilisearchContentIndexStore.parseHits(json("{\"results\":[]}"))).isEmpty(); + } + + @Test + @DisplayName("hits without an id are ignored") + void ignoresHitsWithoutId() throws Exception { + JsonNode response = json(""" + {"results":[{"hits":[ + {"url":"https://ex.com/no-id"}, + {"id":"oe_ok","url":"https://ex.com/ok"} + ]}]}"""); + + Map state = MeilisearchContentIndexStore.parseHits(response); + + assertThat(state).containsOnlyKeys("oe_ok"); + } +}