Merge pull request #34 from OpenElementsLabs/feat/017-search-enhancements

Search enhancements — synonyms/stop-words + facet surfacing

Author: hendrikebbers

CLAUDE.md

@@ -16,8 +16,9 @@ rate limit, retries), extracts content/metadata with jsoup, indexes into Meilise
 bootstrap + scheduled incremental refresh), and exposes four MCP tools on `/mcp` (`search_content`,
 `list_posts`, `get_post`, `list_categories`) backed by a scoped Meilisearch key. Three website sources are
 configured (spec 015): `open-elements` (`/posts/**`), `hiero` (`/blog/**`), and `support-and-care`
-(`/en/support-care*`). Phase 3+ (see [`docs/roadmap.md`](docs/roadmap.md)): Git/Markdown sources and
-search enhancements.
+(`/en/support-care*`), plus a disabled example `git` (Markdown) source (spec 016). Search supports
+optional synonyms/stop-words and category facets (spec 017). **The full roadmap (specs 001–017) is
+implemented.** The one intentionally-deferred item is semantic/embeddings search (a future spec).
 
 ### Tech Stack
 
@@ -62,7 +63,9 @@ search enhancements.
 │   ├── ContentSearchService.java            # read facade: multiSearch + Highlighter (+ result records)
 │   ├── ContentMcpToolProvider.java          # McpToolProvider: the 4 tools on /mcp
 │   ├── RobotsPolicy.java / HttpRobotsPolicy.java  # robots.txt allow/disallow + Crawl-delay
-│   └── GitSourceStrategy.java / GitConfig.java    # type: git — GitHub Markdown source (spec 016)
+│   ├── GitSourceStrategy.java / GitConfig.java    # type: git — GitHub Markdown source (spec 016)
+│   ├── ContentSearchProperties.java         # optional synonyms/stop-words config (spec 017)
+│   └── SearchSettingsInitializer.java       # pushes synonyms/stop-words via updateSettings
 ├── src/main/resources/application.yaml      # datasource, JPA, OAuth2, MCP, Meilisearch, content config
 ├── src/test/java/com/openelements/content/  # behavior tests (context, MCP enabled/disabled, search-down, jsoup)
 └── docs/
@@ -112,7 +115,11 @@ changes. `GitSourceStrategy` (`SourceType.GIT`, `GitConfig`) indexes Markdown fr
 Trees-API discovery filtered by `paths` globs (blob SHA = change marker), raw-content fetch with a
 server-side bearer token, YAML-frontmatter + Markdown extraction (Hugo shortcodes cleaned), path →
 canonical-URL mapping, and locale from the filename suffix — producing the same `ContentDocument` as
-websites, so downstream is unchanged. `ContentIndexer` is the orchestration engine: discover → diff discovered `lastmod` against
+websites, so downstream is unchanged. Search enhancements (spec 017): `ContentSearchProperties`
+(`open-elements.content.search.synonyms`/`stop-words`) are pushed to the index at startup by
+`SearchSettingsInitializer` via `updateSettings` (no reindex; no-op when unset), and `search_content`
+returns category facet counts (`SearchHits.facets`). Semantic/embeddings search is intentionally
+deferred to a future spec. `ContentIndexer` is the orchestration engine: discover → diff discovered `lastmod` against
 the state read from the index (`ContentIndexStore`; the index *is* the state) → fetch only new/changed
 items via the strategy → batch-upsert, and delete documents that 404 or vanished from discovery.
 `MeilisearchContentIndexStore` implements the store over `MeilisearchClient` (paged `multiSearch` to

docs/specs/017-search-enhancements/steps.md

@@ -0,0 +1,58 @@
+# Implementation Steps: Search Enhancements
+
+Scope for this spec (chosen from the design's optional menu): **synonyms & stop words** and **facet
+surfacing**. **Semantic/hybrid search is deferred** to a separate spec (heavy: embedder provider +
+indexing cost), per the design's open question.
+
+## Step 1: Synonyms & stop words
+
+- [x] `ContentSearchProperties` (`@ConfigurationProperties("open-elements.content.search")`): `synonyms` (map) + `stopWords` (list); registered on `ContentConfig`
+- [x] `SearchSettingsInitializer` (`ApplicationRunner`, `@Order(40)`, gated on `meilisearch.enabled`): pushes `synonyms`/`stopWords` via `MeilisearchClient.updateSettings` (no data reindex); no-op when unset; failure logged, not fatal
+- [x] Commented example in `application.yaml`
+
+**Related behaviors:** synonym expands a query; stop word ignored; settings applied without reindex disruption
+
+---
+
+## Step 2: Facet surfacing
+
+- [x] `SearchHits` gains a `facets` (`List<CategoryCount>`) field
+- [x] `ContentSearchService.search` requests `facets: ["categories"]` and populates `SearchHits.facets` from `facetDistribution`; `list_categories` unchanged (already faceted)
+
+**Related behaviors:** search response includes facet distribution; list_categories reflects enriched facets
+
+---
+
+## Step 3: Semantic search (deferred)
+
+- [x] Not implemented — with it absent/off, no embeddings are generated and keyword search is unchanged (the "semantic is opt-in / off = unchanged" behavior holds by default). A future spec would add an embedder + hybrid mode.
+
+**Related behaviors:** semantic opt-in (off by default → unchanged); hybrid search — deferred
+
+---
+
+## Step 4: Tests
+
+- [x] `SearchSettingsInitializerTest` — settings pushed via `updateSettings`, no-op when empty, `buildSettings` keys
+- [x] `ContentSearchServiceTest.searchSurfacesFacets` — search requests facets and returns counts
+- [x] Non-regression: the full existing suite (155 → 159) still passes with enhancements present but unconfigured
+
+**Acceptance criteria:**
+- [x] All tests pass (`mvn test`); build green
+
+---
+
+## Behavior Coverage
+
+| Scenario | Layer | Covered in Step |
+|----------|-------|-----------------|
+| Synonym expands a query | Backend | Step 4 (settings pushed); live expansion is Meilisearch's, verified against a live instance |
+| Stop word is ignored in ranking | Backend | Step 4 (settings pushed); live effect is Meilisearch's |
+| Settings applied without reindex disruption | Backend | Step 4 (`SearchSettingsInitializerTest` — `updateSettings`, not addDocuments) |
+| Search response includes facet distribution | Backend | Step 4 (`searchSurfacesFacets`) |
+| list_categories reflects enriched facets | Backend | Existing `ContentSearchService.categoryFacets` (spec 011) |
+| Hybrid search returns semantically related results | Backend | Deferred (Step 3) — separate spec |
+| Semantic search is opt-in | Backend | Step 3 — off by default → keyword search unchanged |
+| Enhancements do not break core search | Backend | Step 4 (full suite green; enhancements additive/gated) |
+
+All scenarios are backend; there is no frontend in this spec.

docs/specs/INDEX.md

@@ -22,4 +22,4 @@ roadmap step, to be implemented sequentially (each builds on the previous).
 | 014 | 014-ops-robustness | Ops & robustness | backend, infrastructure, observability | robots.txt handling, fault tolerance, logging/metrics | #27 | done |
 | 015 | 015-additional-web-sources | Additional web sources | backend, configuration | hiero.org/blog + Support & Care as config-only sources | #29 | done |
 | 016 | 016-git-markdown-source | Git markdown source | backend, architecture, security | `type: git` source + `GitSourceStrategy` (GitHub Markdown) | #31 | done |
-| 017 | 017-search-enhancements | Search enhancements | backend, search | Facets, synonyms/stop words, optional semantic search | — | open |
+| 017 | 017-search-enhancements | Search enhancements | backend, search | Facets, synonyms/stop words, optional semantic search | #33 | done |

src/main/java/com/openelements/content/ContentConfig.java

@@ -20,7 +20,7 @@
  * {@code @Bean} definitions (indexer, search service, MCP tools).
  */
 @Configuration
-@EnableConfigurationProperties(ContentSourceProperties.class)
+@EnableConfigurationProperties({ContentSourceProperties.class, ContentSearchProperties.class})
 public class ContentConfig {
 
     /**

src/main/java/com/openelements/content/ContentSearchProperties.java

@@ -0,0 +1,31 @@
+package com.openelements.content;
+
+import java.util.List;
+import java.util.Map;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+
+/**
+ * Optional search-tuning settings bound from {@code open-elements.content.search.*} (spec 017).
+ *
+ * <p>Applied to the Meilisearch index at startup by {@link SearchSettingsInitializer}. Both are
+ * empty by default, in which case nothing is pushed.
+ *
+ * @param synonyms word → equivalent terms (Meilisearch {@code synonyms}); e.g. {@code ai: [artificial intelligence]}
+ * @param stopWords words ignored in ranking/matching (Meilisearch {@code stopWords})
+ */
+@ConfigurationProperties("open-elements.content.search")
+public record ContentSearchProperties(
+    Map<String, List<String>> synonyms,
+    List<String> stopWords
+) {
+
+    public ContentSearchProperties {
+        synonyms = synonyms == null ? Map.of() : Map.copyOf(synonyms);
+        stopWords = stopWords == null ? List.of() : List.copyOf(stopWords);
+    }
+
+    /** @return {@code true} if there is anything to push to the index */
+    public boolean hasSettings() {
+        return !synonyms.isEmpty() || !stopWords.isEmpty();
+    }
+}

src/main/java/com/openelements/content/ContentSearchService.java

@@ -57,6 +57,7 @@ public SearchHits search(String query, ContentFilters filters, int page, int siz
         query0.put("highlightPreTag", Highlighter.PRE_MARK);
         query0.put("highlightPostTag", Highlighter.POST_MARK);
         query0.put("showRankingScore", true);
+        query0.put("facets", List.of("categories"));
         return parseHits(client.multiSearch(wrap(query0)));
     }
 
@@ -170,7 +171,7 @@ static SearchHits parseHits(JsonNode response) {
         for (JsonNode hit : result.path("hits")) {
             hits.add(toHit(hit));
         }
-        return new SearchHits(hits, result.path("estimatedTotalHits").asLong(0));
+        return new SearchHits(hits, result.path("estimatedTotalHits").asLong(0), categoryFacetsOf(result));
     }
 
     private static SearchHit toHit(JsonNode hit) {
@@ -184,9 +185,10 @@ private static SearchHit toHit(JsonNode hit) {
 
     static List<CategoryCount> parseFacets(JsonNode response) {
         JsonNode result = firstResult(response);
-        if (result == null) {
-            return List.of();
-        }
+        return result == null ? List.of() : categoryFacetsOf(result);
+    }
+
+    private static List<CategoryCount> categoryFacetsOf(JsonNode result) {
         List<CategoryCount> counts = new ArrayList<>();
         result.path("facetDistribution").path("categories").fields()
             .forEachRemaining(entry -> counts.add(new CategoryCount(entry.getKey(), entry.getValue().asLong(0))));

src/main/java/com/openelements/content/SearchHits.java

@@ -3,18 +3,20 @@
 import java.util.List;
 
 /**
- * A page of search/list results.
+ * A page of search/list results, optionally with category facet counts.
  *
  * @param hits           the results on this page
  * @param estimatedTotal Meilisearch's estimate of the total matching documents
+ * @param facets         category facet counts for the query (empty unless facets were requested)
  */
-public record SearchHits(List<SearchHit> hits, long estimatedTotal) {
+public record SearchHits(List<SearchHit> hits, long estimatedTotal, List<CategoryCount> facets) {
 
     public SearchHits {
         hits = hits == null ? List.of() : List.copyOf(hits);
+        facets = facets == null ? List.of() : List.copyOf(facets);
     }
 
     static SearchHits empty() {
-        return new SearchHits(List.of(), 0);
+        return new SearchHits(List.of(), 0, List.of());
     }
 }

src/main/java/com/openelements/content/SearchSettingsInitializer.java

@@ -0,0 +1,67 @@
+package com.openelements.content;
+
+import com.openelements.spring.base.services.search.MeilisearchClient;
+import com.openelements.spring.base.services.search.MeilisearchProperties;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.ApplicationArguments;
+import org.springframework.boot.ApplicationRunner;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.core.annotation.Order;
+import org.springframework.stereotype.Component;
+
+/**
+ * Applies optional {@code synonyms}/{@code stopWords} settings to the content index at startup, via
+ * {@code MeilisearchClient.updateSettings} — which the library's {@code IndexSettings} bean
+ * (spec 003) deliberately omits. These take effect on subsequent searches without a data reindex.
+ *
+ * <p>Runs after the library's settings/bootstrap initializers ({@code @Order} 20/30) and is a no-op
+ * when no synonyms or stop words are configured. Only created when the Meilisearch stack is enabled;
+ * a failure to reach Meilisearch is logged, not fatal.
+ */
+@Component
+@Order(40)
+@ConditionalOnProperty(prefix = "openelements.meilisearch", name = "enabled", havingValue = "true")
+public class SearchSettingsInitializer implements ApplicationRunner {
+
+    private static final Logger log = LoggerFactory.getLogger(SearchSettingsInitializer.class);
+
+    private final MeilisearchClient client;
+    private final ContentSearchProperties searchProperties;
+    private final String indexUid;
+
+    public SearchSettingsInitializer(MeilisearchClient client, ContentSearchProperties searchProperties,
+                                     MeilisearchProperties meilisearchProperties) {
+        this.client = client;
+        this.searchProperties = searchProperties;
+        this.indexUid = meilisearchProperties.resolveIndex("content");
+    }
+
+    @Override
+    public void run(ApplicationArguments args) {
+        if (!searchProperties.hasSettings()) {
+            return;
+        }
+        try {
+            client.updateSettings(indexUid, buildSettings(searchProperties));
+            log.info("Applied search settings to {}: {} synonym(s), {} stop word(s)",
+                indexUid, searchProperties.synonyms().size(), searchProperties.stopWords().size());
+        } catch (Exception e) {
+            log.warn("Could not apply search settings to {}: {}", indexUid, e.toString());
+        }
+    }
+
+    /** Builds the Meilisearch settings payload for the configured synonyms/stop words. */
+    static Map<String, Object> buildSettings(ContentSearchProperties properties) {
+        Map<String, Object> settings = new LinkedHashMap<>();
+        if (!properties.synonyms().isEmpty()) {
+            settings.put("synonyms", properties.synonyms());
+        }
+        if (!properties.stopWords().isEmpty()) {
+            settings.put("stopWords", properties.stopWords());
+        }
+        return settings;
+    }
+}

src/main/resources/application.yaml

@@ -55,6 +55,12 @@ open-elements:
     rate-limit-per-host: 2
     request-timeout: 10s
     max-body-bytes: 5MB
+    # Optional search tuning (spec 017), applied via updateSettings at startup (no reindex).
+    # Empty by default. Example:
+    # search:
+    #   synonyms:
+    #     ai: [ "artificial intelligence", "machine learning" ]
+    #   stop-words: [ "the", "a", "an" ]
     sources:
       - id: open-elements
         type: website

src/test/java/com/openelements/content/ContentMcpToolProviderTest.java

@@ -140,7 +140,7 @@ void listCategoriesDelegates() {
     }
 
     private void searchContentResult() {
-        searchService.hits = new SearchHits(List.of(new SearchHit("T", "u", "2026-01-01", "snip", 0.9)), 1);
+        searchService.hits = new SearchHits(List.of(new SearchHit("T", "u", "2026-01-01", "snip", 0.9)), 1, List.of());
     }
 
     private static ContentDocument document() {