Merge pull request #30 from OpenElementsLabs/feat/015-additional-web-sources

Additional website sources — hiero.org/blog + Support & Care (config-only)

Author: hendrikebbers

CLAUDE.md

@@ -14,9 +14,10 @@ Current state: **Phase 1 complete** (specs 001–014). The app crawls configured
 (sitemap discovery + bounded fallback, robots.txt-aware), fetches politely (conditional GET, per-host
 rate limit, retries), extracts content/metadata with jsoup, indexes into Meilisearch (startup
 bootstrap + scheduled incremental refresh), and exposes four MCP tools on `/mcp` (`search_content`,
-`list_posts`, `get_post`, `list_categories`) backed by a scoped Meilisearch key. Phase 2+ (see
-[`docs/roadmap.md`](docs/roadmap.md)): additional website sources, Git/Markdown sources, and search
-enhancements.
+`list_posts`, `get_post`, `list_categories`) backed by a scoped Meilisearch key. Three website sources are
+configured (spec 015): `open-elements` (`/posts/**`), `hiero` (`/blog/**`), and `support-and-care`
+(`/en/support-care*`). Phase 3+ (see [`docs/roadmap.md`](docs/roadmap.md)): Git/Markdown sources and
+search enhancements.
 
 ### Tech Stack
 

docs/specs/015-additional-web-sources/steps.md

@@ -0,0 +1,35 @@
+# Implementation Steps: Additional Website Sources
+
+## Step 1: Resolve open questions (live checks)
+
+- [x] **hiero.org sitemap:** `https://hiero.org/sitemap.xml` exists — a flat `<urlset>` with 66 `<loc>`s, 62 of them `/blog/<post>`. So `/sitemap.xml` + `url-include: [/blog/**]` covers all posts; the no-sitemap fallback crawl is not needed.
+- [x] **hiero.org robots.txt:** served as a Next.js HTML catch-all (no `User-agent`/`Disallow` rules) → treated as allow-all by `HttpRobotsPolicy`.
+- [x] **hiero post markup:** posts are wrapped in `<main><article>` → selector `main article`. No OpenGraph/Article meta, so `title` falls back to `<title>`/`<h1>` and `publishedDate` may be null (metadata is limited for hiero).
+- [x] **Support & Care:** two pages — `/en/support-care` and `/en/support-care-maven` — listed in `/en/sitemap.xml` (under `/en/`, not `/support-care` as the design assumed). OE `robots.txt` is `Allow: /`.
+
+## Step 2: Configuration (no code)
+
+- [x] Add `hiero` source: `base-url https://hiero.org`, `sitemaps [/sitemap.xml]`, `url-include [/blog/**]`, `content-selector "main article"`
+- [x] Add `support-and-care` source: `base-url https://open-elements.com`, `sitemaps [/en/sitemap.xml]`, `url-include ["/en/support-care*", "/en/support-care/**"]` (matches both current pages), `content-selector "body"` + `content-exclude [nav, header, footer, .cookie-banner, aside]`
+
+## Step 3: Tests
+
+- [x] `ConfiguredSourcesTest` — the three sources bind from `application.yaml` with the expected type/URLs/selectors (no code)
+
+**Acceptance criteria:**
+- [x] All tests pass (`mvn test`); build green
+
+## Behavior Coverage
+
+| Scenario | Layer | Covered in Step |
+|----------|-------|-----------------|
+| New source needs no code | Backend | Step 3 (`ConfiguredSourcesTest.allSourcesAreConfigured`) |
+| Blog posts are indexed (/blog/** only) | Backend | Step 3 (hiero `urlInclude` `[/blog/**]`); live index verified manually |
+| Sitemap-absent fallback | Backend | N/A for hiero — the sitemap exists and lists posts (finding); generic fallback is covered by spec 004 |
+| Clean extraction with the article selector | Backend | Step 1 finding (`main article`); live extraction verified manually |
+| robots.txt honored for hiero.org | Backend | Covered by spec 014 (`HttpRobotsPolicy`); hiero has no rules → allow-all (finding) |
+| Body-selector with excludes yields clean text | Backend | Step 2 config; live extraction verified manually |
+| URL scope matches single vs. multi page | Backend | Step 3 (`supportAndCareSourceIsConfigured`) — multi-page `/en/support-care*` |
+| Source filter isolates results | Backend | Covered by spec 011 (`ContentSearchServiceTest` source filter) |
+
+Config-only spec: binding is unit-tested; live crawl/extraction against the third-party sites is verified manually against a dev index (not run in CI, which must not hit external sites).

docs/specs/INDEX.md

@@ -20,6 +20,6 @@ roadmap step, to be implemented sequentially (each builds on the previous).
 | 012 | 012-content-mcp-tools | Content MCP tools | backend, mcp, api | `ContentMcpToolProvider` — the 4 MCP tools | #23 | done |
 | 013 | 013-scoped-search-key | Scoped search key | backend, search, security | Read-only scoped Meilisearch key for the content index | #25 | done |
 | 014 | 014-ops-robustness | Ops & robustness | backend, infrastructure, observability | robots.txt handling, fault tolerance, logging/metrics | #27 | done |
-| 015 | 015-additional-web-sources | Additional web sources | backend, configuration | hiero.org/blog + Support & Care as config-only sources | — | open |
+| 015 | 015-additional-web-sources | Additional web sources | backend, configuration | hiero.org/blog + Support & Care as config-only sources | #29 | done |
 | 016 | 016-git-markdown-source | Git markdown source | backend, architecture, security | `type: git` source + `GitSourceStrategy` (GitHub Markdown) | — | open |
 | 017 | 017-search-enhancements | Search enhancements | backend, search | Facets, synonyms/stop words, optional semantic search | — | open |

src/main/resources/application.yaml

@@ -63,3 +63,22 @@ open-elements:
         url-include: [ "/posts/**", "/de/posts/**" ]
         content-selector: "article"
         enabled: true
+      # hiero.org has a real /sitemap.xml (flat urlset) listing every /blog/<post>; robots.txt is a
+      # Next.js HTML catch-all (no rules -> allow-all). Posts are wrapped in <main><article>.
+      - id: hiero
+        type: website
+        base-url: https://hiero.org
+        sitemaps: [ /sitemap.xml ]
+        url-include: [ "/blog/**" ]
+        content-selector: "main article"
+        enabled: true
+      # Support & Care is two pages under /en/ (support-care, support-care-maven), listed in
+      # /en/sitemap.xml. They lack a clean article container, so take the <body> minus boilerplate.
+      - id: support-and-care
+        type: website
+        base-url: https://open-elements.com
+        sitemaps: [ /en/sitemap.xml ]
+        url-include: [ "/en/support-care*", "/en/support-care/**" ]
+        content-selector: "body"
+        content-exclude: [ "nav", "header", "footer", ".cookie-banner", "aside" ]
+        enabled: true

src/test/java/com/openelements/content/ConfiguredSourcesTest.java

@@ -0,0 +1,59 @@
+package com.openelements.content;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.test.context.SpringBootTest;
+
+/**
+ * Verifies that the website sources are bound from {@code application.yaml} with no code — the point
+ * of spec 015 (hiero.org/blog and Support & Care are added purely as configuration).
+ *
+ * <p>Shares the application context of the other {@code api-key.enabled=false} tests. The actual crawl
+ * and extraction against the live sites are verified manually against a dev index.
+ */
+@SpringBootTest(properties = "openelements.mcp.auth.api-key.enabled=false")
+@DisplayName("Configured website sources")
+class ConfiguredSourcesTest {
+
+    @Autowired
+    private ContentSourceProperties properties;
+
+    private Map<String, ContentSource> sourcesById() {
+        return properties.sources().stream()
+            .collect(Collectors.toMap(ContentSource::id, Function.identity()));
+    }
+
+    @Test
+    @DisplayName("open-elements, hiero and support-and-care are all configured")
+    void allSourcesAreConfigured() {
+        assertThat(sourcesById()).containsKeys("open-elements", "hiero", "support-and-care");
+    }
+
+    @Test
+    @DisplayName("the hiero source targets /blog via the sitemap with the article selector")
+    void hieroSourceIsConfigured() {
+        ContentSource hiero = sourcesById().get("hiero");
+        assertThat(hiero.type()).isEqualTo(SourceType.WEBSITE);
+        assertThat(hiero.baseUrl()).isEqualTo("https://hiero.org");
+        assertThat(hiero.sitemaps()).containsExactly("/sitemap.xml");
+        assertThat(hiero.urlInclude()).containsExactly("/blog/**");
+        assertThat(hiero.contentSelector()).isEqualTo("main article");
+        assertThat(hiero.enabled()).isTrue();
+    }
+
+    @Test
+    @DisplayName("the support-and-care source scopes to the /en/support-care pages with body + excludes")
+    void supportAndCareSourceIsConfigured() {
+        ContentSource support = sourcesById().get("support-and-care");
+        assertThat(support.baseUrl()).isEqualTo("https://open-elements.com");
+        assertThat(support.urlInclude()).contains("/en/support-care*");
+        assertThat(support.contentSelector()).isEqualTo("body");
+        assertThat(support.contentExclude()).contains("nav", "header", "footer", ".cookie-banner");
+    }
+}