Content refresh scheduler — scheduled incremental re-crawl

CLAUDE.md

@@ -54,7 +54,8 @@ extraction, Meilisearch indexing with scheduled refresh, and four MCP tools
 │   ├── IndexReport.java                     # per-pass counters
 │   ├── ContentIndexStore.java               # index read/write seam (StoredDocument)
 │   ├── MeilisearchContentIndexStore.java    # ContentIndexStore backed by MeilisearchClient
-│   └── ContentBootstrapStep.java            # SearchIndexBootstrapStep: startup full reindex
+│   ├── ContentBootstrapStep.java            # SearchIndexBootstrapStep: startup full reindex
+│   └── ContentRefreshScheduler.java         # @Scheduled incremental re-crawl (cron, guarded)
 ├── src/main/resources/application.yaml      # datasource, JPA, OAuth2, MCP, Meilisearch, content config
 ├── src/test/java/com/openelements/content/  # behavior tests (context, MCP enabled/disabled, search-down, jsoup)
 └── docs/
@@ -109,7 +110,10 @@ read state, `addDocuments`+`waitForTask` to upsert, `deleteDocument`), since the
 refresh scheduler (010). `ContentBootstrapStep` implements the library's `SearchIndexBootstrapStep`:
 at startup the library's `MeilisearchBootstrapRunner` discovers it, consumes its lazy `documents()`
 stream (over all enabled sources, via `ContentIndexer.streamAllDocuments`) in batches into Meilisearch,
-and toggles `SearchReadinessState`.
+and toggles `SearchReadinessState`. `ContentRefreshScheduler` is a `@Scheduled` bean (cron
+`open-elements.content.refresh-cron`, default hourly, from `@EnableScheduling` in spec 001) that
+re-runs `ContentIndexer.indexSource` over enabled sources — guarded to skip while disabled or
+bootstrapping and to never overlap, with per-source fault isolation.
 
 > **Key gotcha:** the library ships no Spring Boot auto-configuration and couples MCP to a JPA
 > datasource, so `@Import({ McpConfiguration, SearchConfig })` alone does **not** boot. See

docs/specs/010-content-refresh-scheduler/steps.md

@@ -0,0 +1,45 @@
+# Implementation Steps: Content Refresh Scheduler
+
+## Step 1: `ContentRefreshScheduler`
+
+- [x] `@Component` (gated on `meilisearch.enabled`) with a `@Scheduled(cron = "${open-elements.content.refresh-cron:0 0 * * * *}")` `refresh()`
+- [x] Iterates enabled sources, calling `ContentIndexer.indexSource` and logging each `IndexReport`
+- [x] Guards: skip when globally disabled; skip while `SearchReadinessState.isBootstrapping()`; non-overlap via `AtomicBoolean`
+- [x] Per-source try/catch so one failing source does not stop the others
+
+**Acceptance criteria:**
+- [x] Project builds; behaviors verified by tests
+
+**Related behaviors:** all
+
+---
+
+## Step 2: Tests
+
+- [x] `ContentRefreshSchedulerTest` (real `ContentIndexer` + stub strategy + in-memory store + real `SearchReadinessState`): per-source, disabled skip, bootstrapping/global-disabled guards, non-overlap (re-entrancy), fault isolation, cron externalization (reflection), new/deleted propagation
+
+**Acceptance criteria:**
+- [x] All tests pass (`mvn test`); build green
+
+---
+
+## Behavior Coverage
+
+| Scenario | Layer | Covered in Step |
+|----------|-------|-----------------|
+| Refresh runs each enabled source on tick | Backend | Step 2 (`runsEachEnabledSource`) |
+| Disabled source is skipped | Backend | Step 2 (`disabledSourceIsSkipped`) |
+| Cron is externally configurable | Backend | Step 2 (`cronIsExternallyConfigurable`) |
+| Skips while bootstrapping | Backend | Step 2 (`skipsWhileBootstrapping`) |
+| Skips when globally disabled | Backend | Step 2 (`skipsWhenGloballyDisabled`) |
+| Non-overlapping runs | Backend | Step 2 (`nonOverlappingRuns`) |
+| New page appears after refresh | Backend | Step 2 (`newPageIsAdded`) |
+| Deleted page removed after refresh | Backend | Step 2 (`deletedPageIsRemoved`) |
+| One source failing does not stop the others | Backend | Step 2 (`oneSourceFailingDoesNotStopOthers`) |
+
+All scenarios are backend; there is no frontend in this spec.
+
+## Notes
+
+- `SearchReadinessState` reports `isBootstrapping()==true` until the runner finishes; the scheduler correctly skips during that window (tests mark bootstrap complete before asserting a run).
+- Non-overlap is verified deterministically by re-entrantly calling `refresh()` during a run (no threads/sleeps).

docs/specs/INDEX.md

@@ -15,7 +15,7 @@ roadmap step, to be implemented sequentially (each builds on the previous).
 | 007 | 007-source-strategy | Source strategy | backend, architecture | `ContentSourceStrategy` interface + `WebsiteSourceStrategy` | #13 | done |
 | 008 | 008-content-indexer | Content indexer | backend, search, crawler | `ContentIndexer` — orchestration, diff, upsert/delete | #15 | done |
 | 009 | 009-content-bootstrap-step | Content bootstrap step | backend, search | `ContentBootstrapStep` — initial reindex via `SearchIndexBootstrapStep` | #17 | done |
-| 010 | 010-content-refresh-scheduler | Content refresh scheduler | backend, scheduling | `ContentRefreshScheduler` — `@Scheduled` incremental re-crawl | — | open |
+| 010 | 010-content-refresh-scheduler | Content refresh scheduler | backend, scheduling | `ContentRefreshScheduler` — `@Scheduled` incremental re-crawl | #19 | done |
 | 011 | 011-content-search-service | Content search service | backend, search | `ContentSearchService` — `multiSearch` + highlighting facade | — | open |
 | 012 | 012-content-mcp-tools | Content MCP tools | backend, mcp, api | `ContentMcpToolProvider` — the 4 MCP tools | — | open |
 | 013 | 013-scoped-search-key | Scoped search key | backend, search, security | Read-only scoped Meilisearch key for the content index | — | open |

src/main/java/com/openelements/content/ContentRefreshScheduler.java

@@ -0,0 +1,73 @@
+package com.openelements.content;
+
+import com.openelements.spring.base.services.search.SearchReadinessState;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.scheduling.annotation.Scheduled;
+import org.springframework.stereotype.Component;
+
+/**
+ * Periodically re-runs incremental ingestion over all enabled sources, reusing {@link ContentIndexer}
+ * so scheduled refresh and startup bootstrap (spec 009) share identical semantics.
+ *
+ * <p>Driven by the configurable {@code open-elements.content.refresh-cron} (default hourly). A tick is
+ * a no-op while the content pipeline is disabled or Meilisearch is still bootstrapping, and runs never
+ * overlap — if a refresh is still in progress when the next tick fires, that tick is skipped. A source
+ * that fails to index is logged and does not stop the others.
+ *
+ * <p>Only created when the Meilisearch stack is enabled (it depends on {@link ContentIndexer}).
+ */
+@Component
+@ConditionalOnProperty(prefix = "openelements.meilisearch", name = "enabled", havingValue = "true")
+public class ContentRefreshScheduler {
+
+    private static final Logger log = LoggerFactory.getLogger(ContentRefreshScheduler.class);
+
+    private final ContentSourceProperties properties;
+    private final ContentIndexer indexer;
+    private final SearchReadinessState readinessState;
+    private final AtomicBoolean running = new AtomicBoolean(false);
+
+    public ContentRefreshScheduler(ContentSourceProperties properties, ContentIndexer indexer,
+                                   SearchReadinessState readinessState) {
+        this.properties = properties;
+        this.indexer = indexer;
+        this.readinessState = readinessState;
+    }
+
+    /**
+     * Refreshes every enabled source once. Bound to the {@code refresh-cron} property (default hourly).
+     */
+    @Scheduled(cron = "${open-elements.content.refresh-cron:0 0 * * * *}")
+    public void refresh() {
+        if (!properties.enabled()) {
+            log.debug("Content pipeline disabled; skipping refresh");
+            return;
+        }
+        if (readinessState.isBootstrapping()) {
+            log.debug("Bootstrap in progress; skipping refresh");
+            return;
+        }
+        if (!running.compareAndSet(false, true)) {
+            log.info("Previous refresh still running; skipping this tick");
+            return;
+        }
+        try {
+            for (ContentSource source : properties.sources()) {
+                if (!source.enabled()) {
+                    continue;
+                }
+                try {
+                    IndexReport report = indexer.indexSource(source);
+                    log.info("Refreshed source {}: {}", source.id(), report);
+                } catch (Exception e) {
+                    log.warn("Refresh failed for source {}: {}", source.id(), e.toString());
+                }
+            }
+        } finally {
+            running.set(false);
+        }
+    }
+}

src/main/resources/application.yaml

@@ -6,14 +6,21 @@ spring:
     # server depends on. The content MCP keeps no relational state of its own; this in-memory H2
     # instance backs only the library's tables. Override with a persistent DB for production if the
     # library's api-key/user data must survive restarts.
-    url: ${SPRING_DATASOURCE_URL:jdbc:h2:mem:content-mcp;DB_CLOSE_DELAY=-1;MODE=PostgreSQL}
+    url: ${SPRING_DATASOURCE_URL:jdbc:h2:mem:content-mcp;DB_CLOSE_DELAY=-1;MODE=PostgreSQL;INIT=CREATE SCHEMA IF NOT EXISTS OE_SPRING_SERVICES}
     username: ${SPRING_DATASOURCE_USERNAME:sa}
     password: ${SPRING_DATASOURCE_PASSWORD:}
     driver-class-name: org.h2.Driver
   jpa:
     hibernate:
       ddl-auto: update
     open-in-view: false
+    properties:
+      hibernate:
+        # spring-services' entities live in the OE_SPRING_SERVICES schema; without this Hibernate
+        # tries to CREATE TABLE in a schema it never created (fails on a fresh H2). This makes it
+        # create entity-declared schemas first.
+        hbm2ddl:
+          create_namespaces: true
   security:
     oauth2:
       resourceserver:

src/test/java/com/openelements/content/ContentRefreshSchedulerTest.java

@@ -0,0 +1,209 @@
+package com.openelements.content;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import com.openelements.spring.base.services.search.SearchReadinessState;
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.BiFunction;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.springframework.scheduling.annotation.Scheduled;
+import org.springframework.util.unit.DataSize;
+
+/**
+ * Unit tests for {@link ContentRefreshScheduler} using a real {@link ContentIndexer} driven by a
+ * test-double strategy and an in-memory store, plus a real {@link SearchReadinessState}. No mocks or
+ * live Meilisearch; non-overlap is verified deterministically via re-entrancy rather than threads.
+ */
+@DisplayName("Content refresh scheduler")
+class ContentRefreshSchedulerTest {
+
+    private final StubStrategy strategy = new StubStrategy();
+    private final InMemoryContentIndexStore store = new InMemoryContentIndexStore();
+    private final ContentIndexer indexer =
+        new ContentIndexer(new SourceStrategyRegistry(List.of(strategy)), store);
+    private final SearchReadinessState readiness = new SearchReadinessState();
+
+    @BeforeEach
+    void markBootstrapComplete() {
+        // A fresh SearchReadinessState reports isBootstrapping()==true until bootstrap finishes;
+        // simulate a completed startup bootstrap so scheduled refreshes are allowed to run.
+        readiness.markBootstrappingFinished();
+    }
+
+    private static ContentSource source(String id, boolean enabled) {
+        return new ContentSource(
+            id, SourceType.WEBSITE, "https://ex.com", List.of(), List.of("/**"), List.of(), "article", List.of(), enabled);
+    }
+
+    private static ContentSourceProperties properties(boolean enabled, ContentSource... sources) {
+        return new ContentSourceProperties(
+            enabled, null, "UA", 2.0, Duration.ofSeconds(10), DataSize.ofMegabytes(5), List.of(sources));
+    }
+
+    private ContentRefreshScheduler scheduler(ContentSourceProperties properties) {
+        return new ContentRefreshScheduler(properties, indexer, readiness);
+    }
+
+    private static ContentDocument document(String source, String url, String lastmod) {
+        return new ContentDocument(
+            ContentDocument.id(source, url), source, "en", url, "T", "E", "body", "a",
+            List.of(), "2026-01-01", lastmod, null);
+    }
+
+    @Test
+    @DisplayName("each enabled source is indexed on a tick")
+    void runsEachEnabledSource() {
+        scheduler(properties(true, source("a", true), source("b", true))).refresh();
+
+        assertThat(strategy.discoveredSources).containsExactly("a", "b");
+    }
+
+    @Test
+    @DisplayName("a disabled source is skipped")
+    void disabledSourceIsSkipped() {
+        scheduler(properties(true, source("a", true), source("b", false))).refresh();
+
+        assertThat(strategy.discoveredSources).containsExactly("a");
+    }
+
+    @Test
+    @DisplayName("refresh is skipped while bootstrapping")
+    void skipsWhileBootstrapping() {
+        readiness.markBootstrappingStarted();
+
+        scheduler(properties(true, source("a", true))).refresh();
+
+        assertThat(strategy.discoveredSources).isEmpty();
+    }
+
+    @Test
+    @DisplayName("refresh does nothing when the pipeline is globally disabled")
+    void skipsWhenGloballyDisabled() {
+        scheduler(properties(false, source("a", true))).refresh();
+
+        assertThat(strategy.discoveredSources).isEmpty();
+    }
+
+    @Test
+    @DisplayName("a re-entrant tick does not overlap a running refresh")
+    void nonOverlappingRuns() {
+        ContentRefreshScheduler scheduler = scheduler(properties(true, source("a", true), source("b", true)));
+        strategy.reentrantOnce = scheduler::refresh; // fired during the first discover of the outer run
+
+        scheduler.refresh();
+
+        // The re-entrant call is skipped, so each source is discovered exactly once.
+        assertThat(strategy.discoveredSources).containsExactly("a", "b");
+    }
+
+    @Test
+    @DisplayName("one source failing does not stop the others")
+    void oneSourceFailingDoesNotStopOthers() {
+        strategy.throwOnDiscover.add("a");
+
+        scheduler(properties(true, source("a", true), source("b", true))).refresh();
+
+        assertThat(strategy.discoveredSources).containsExactly("a", "b");
+    }
+
+    @Test
+    @DisplayName("the cron expression is externalized with an hourly default")
+    void cronIsExternallyConfigurable() throws Exception {
+        Method refresh = ContentRefreshScheduler.class.getMethod("refresh");
+        Scheduled scheduled = refresh.getAnnotation(Scheduled.class);
+
+        assertThat(scheduled).isNotNull();
+        assertThat(scheduled.cron()).isEqualTo("${open-elements.content.refresh-cron:0 0 * * * *}");
+    }
+
+    @Test
+    @DisplayName("a new page is added to the index on refresh")
+    void newPageIsAdded() {
+        strategy.itemsBySource.put("a", List.of(new DiscoveredItem("https://ex.com/a", "L")));
+
+        scheduler(properties(true, source("a", true))).refresh();
+
+        assertThat(store.loadState("a")).containsKey(ContentDocument.id("a", "https://ex.com/a"));
+    }
+
+    @Test
+    @DisplayName("a vanished page is deleted from the index on refresh")
+    void deletedPageIsRemoved() {
+        store.upsert(List.of(document("a", "https://ex.com/gone", "L").toMap()));
+        strategy.itemsBySource.put("a", List.of()); // no longer discovered
+
+        scheduler(properties(true, source("a", true))).refresh();
+
+        assertThat(store.loadState("a")).isEmpty();
+    }
+
+    /** Test-double strategy: records discovered sources, can throw or fire a one-shot re-entrant action. */
+    private static final class StubStrategy implements ContentSourceStrategy {
+        private final List<String> discoveredSources = new ArrayList<>();
+        private final Set<String> throwOnDiscover = new HashSet<>();
+        private final Map<String, List<DiscoveredItem>> itemsBySource = new HashMap<>();
+        private BiFunction<ContentSource, DiscoveredItem, FetchOutcome> fetchFn =
+            (source, item) -> FetchOutcome.index(document(source.id(), item.url(), item.lastmod()));
+        private Runnable reentrantOnce;
+        private boolean reentrantFired;
+
+        @Override
+        public SourceType type() {
+            return SourceType.WEBSITE;
+        }
+
+        @Override
+        public List<DiscoveredItem> discover(ContentSource source) {
+            discoveredSources.add(source.id());
+            if (reentrantOnce != null && !reentrantFired) {
+                reentrantFired = true;
+                reentrantOnce.run();
+            }
+            if (throwOnDiscover.contains(source.id())) {
+                throw new IllegalStateException("boom " + source.id());
+            }
+            return itemsBySource.getOrDefault(source.id(), List.of());
+        }
+
+        @Override
+        public FetchOutcome fetch(ContentSource source, DiscoveredItem item) {
+            return fetchFn.apply(source, item);
+        }
+    }
+
+    /** In-memory {@link ContentIndexStore}: documents keyed by id. */
+    private static final class InMemoryContentIndexStore implements ContentIndexStore {
+        private final Map<String, Map<String, Object>> documents = new HashMap<>();
+
+        @Override
+        public Map<String, StoredDocument> loadState(String source) {
+            Map<String, StoredDocument> state = new HashMap<>();
+            documents.forEach((id, doc) -> {
+                if (source.equals(doc.get("source"))) {
+                    state.put(id, new StoredDocument((String) doc.get("url"), (String) doc.get("lastmod")));
+                }
+            });
+            return state;
+        }
+
+        @Override
+        public int upsert(List<Map<String, Object>> docs) {
+            docs.forEach(doc -> documents.put((String) doc.get("id"), doc));
+            return docs.size();
+        }
+
+        @Override
+        public void delete(String id) {
+            documents.remove(id);
+        }
+    }
+}