From 338408bf8ca01feb9c58d5b54c259e3ef2f6e017 Mon Sep 17 00:00:00 2001 From: Hendrik Ebbers Date: Mon, 13 Jul 2026 21:56:22 +0200 Subject: [PATCH 1/3] chore(spec-015): mark additional-web-sources in progress, link issue #29 Refs #29 --- docs/specs/INDEX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specs/INDEX.md b/docs/specs/INDEX.md index 453ada1..870463e 100644 --- a/docs/specs/INDEX.md +++ b/docs/specs/INDEX.md @@ -20,6 +20,6 @@ roadmap step, to be implemented sequentially (each builds on the previous). | 012 | 012-content-mcp-tools | Content MCP tools | backend, mcp, api | `ContentMcpToolProvider` — the 4 MCP tools | #23 | done | | 013 | 013-scoped-search-key | Scoped search key | backend, search, security | Read-only scoped Meilisearch key for the content index | #25 | done | | 014 | 014-ops-robustness | Ops & robustness | backend, infrastructure, observability | robots.txt handling, fault tolerance, logging/metrics | #27 | done | -| 015 | 015-additional-web-sources | Additional web sources | backend, configuration | hiero.org/blog + Support & Care as config-only sources | — | open | +| 015 | 015-additional-web-sources | Additional web sources | backend, configuration | hiero.org/blog + Support & Care as config-only sources | #29 | in progress | | 016 | 016-git-markdown-source | Git markdown source | backend, architecture, security | `type: git` source + `GitSourceStrategy` (GitHub Markdown) | — | open | | 017 | 017-search-enhancements | Search enhancements | backend, search | Facets, synonyms/stop words, optional semantic search | — | open | From 3c1a8541696775ee1e525c1433b6085266d2e185 Mon Sep 17 00:00:00 2001 From: Hendrik Ebbers Date: Mon, 13 Jul 2026 21:58:23 +0200 Subject: [PATCH 2/3] feat(spec-015): add hiero.org/blog + Support & Care sources (config-only) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolved open questions against the live sites: - hiero.org has /sitemap.xml (flat urlset, 62 /blog/ entries); robots is a Next.js HTML catch-all (allow-all); posts use
, no OG meta. -> hiero source: /sitemap.xml, /blog/**, selector "main article". - Support & Care is 2 pages under /en/ (support-care, support-care-maven) in /en/sitemap.xml. -> support-and-care source: /en/support-care*, selector "body" + content-exclude. No code changes — both added to application.yaml. ConfiguredSourcesTest verifies binding. 145 tests green. steps.md records the findings; CLAUDE.md updated. Refs #29 --- CLAUDE.md | 7 ++- .../specs/015-additional-web-sources/steps.md | 35 +++++++++++ src/main/resources/application.yaml | 19 ++++++ .../content/ConfiguredSourcesTest.java | 59 +++++++++++++++++++ 4 files changed, 117 insertions(+), 3 deletions(-) create mode 100644 docs/specs/015-additional-web-sources/steps.md create mode 100644 src/test/java/com/openelements/content/ConfiguredSourcesTest.java diff --git a/CLAUDE.md b/CLAUDE.md index 7eb4dd1..876c475 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,9 +14,10 @@ Current state: **Phase 1 complete** (specs 001–014). The app crawls configured (sitemap discovery + bounded fallback, robots.txt-aware), fetches politely (conditional GET, per-host rate limit, retries), extracts content/metadata with jsoup, indexes into Meilisearch (startup bootstrap + scheduled incremental refresh), and exposes four MCP tools on `/mcp` (`search_content`, -`list_posts`, `get_post`, `list_categories`) backed by a scoped Meilisearch key. Phase 2+ (see -[`docs/roadmap.md`](docs/roadmap.md)): additional website sources, Git/Markdown sources, and search -enhancements. +`list_posts`, `get_post`, `list_categories`) backed by a scoped Meilisearch key. Three website sources are +configured (spec 015): `open-elements` (`/posts/**`), `hiero` (`/blog/**`), and `support-and-care` +(`/en/support-care*`). Phase 3+ (see [`docs/roadmap.md`](docs/roadmap.md)): Git/Markdown sources and +search enhancements. ### Tech Stack diff --git a/docs/specs/015-additional-web-sources/steps.md b/docs/specs/015-additional-web-sources/steps.md new file mode 100644 index 0000000..18d1d91 --- /dev/null +++ b/docs/specs/015-additional-web-sources/steps.md @@ -0,0 +1,35 @@ +# Implementation Steps: Additional Website Sources + +## Step 1: Resolve open questions (live checks) + +- [x] **hiero.org sitemap:** `https://hiero.org/sitemap.xml` exists — a flat `` with 66 ``s, 62 of them `/blog/`. So `/sitemap.xml` + `url-include: [/blog/**]` covers all posts; the no-sitemap fallback crawl is not needed. +- [x] **hiero.org robots.txt:** served as a Next.js HTML catch-all (no `User-agent`/`Disallow` rules) → treated as allow-all by `HttpRobotsPolicy`. +- [x] **hiero post markup:** posts are wrapped in `
` → selector `main article`. No OpenGraph/Article meta, so `title` falls back to ``/`<h1>` and `publishedDate` may be null (metadata is limited for hiero). +- [x] **Support & Care:** two pages — `/en/support-care` and `/en/support-care-maven` — listed in `/en/sitemap.xml` (under `/en/`, not `/support-care` as the design assumed). OE `robots.txt` is `Allow: /`. + +## Step 2: Configuration (no code) + +- [x] Add `hiero` source: `base-url https://hiero.org`, `sitemaps [/sitemap.xml]`, `url-include [/blog/**]`, `content-selector "main article"` +- [x] Add `support-and-care` source: `base-url https://open-elements.com`, `sitemaps [/en/sitemap.xml]`, `url-include ["/en/support-care*", "/en/support-care/**"]` (matches both current pages), `content-selector "body"` + `content-exclude [nav, header, footer, .cookie-banner, aside]` + +## Step 3: Tests + +- [x] `ConfiguredSourcesTest` — the three sources bind from `application.yaml` with the expected type/URLs/selectors (no code) + +**Acceptance criteria:** +- [x] All tests pass (`mvn test`); build green + +## Behavior Coverage + +| Scenario | Layer | Covered in Step | +|----------|-------|-----------------| +| New source needs no code | Backend | Step 3 (`ConfiguredSourcesTest.allSourcesAreConfigured`) | +| Blog posts are indexed (/blog/** only) | Backend | Step 3 (hiero `urlInclude` `[/blog/**]`); live index verified manually | +| Sitemap-absent fallback | Backend | N/A for hiero — the sitemap exists and lists posts (finding); generic fallback is covered by spec 004 | +| Clean extraction with the article selector | Backend | Step 1 finding (`main article`); live extraction verified manually | +| robots.txt honored for hiero.org | Backend | Covered by spec 014 (`HttpRobotsPolicy`); hiero has no rules → allow-all (finding) | +| Body-selector with excludes yields clean text | Backend | Step 2 config; live extraction verified manually | +| URL scope matches single vs. multi page | Backend | Step 3 (`supportAndCareSourceIsConfigured`) — multi-page `/en/support-care*` | +| Source filter isolates results | Backend | Covered by spec 011 (`ContentSearchServiceTest` source filter) | + +Config-only spec: binding is unit-tested; live crawl/extraction against the third-party sites is verified manually against a dev index (not run in CI, which must not hit external sites). diff --git a/src/main/resources/application.yaml b/src/main/resources/application.yaml index df57ee3..e41c4d4 100644 --- a/src/main/resources/application.yaml +++ b/src/main/resources/application.yaml @@ -63,3 +63,22 @@ open-elements: url-include: [ "/posts/**", "/de/posts/**" ] content-selector: "article" enabled: true + # hiero.org has a real /sitemap.xml (flat urlset) listing every /blog/<post>; robots.txt is a + # Next.js HTML catch-all (no rules -> allow-all). Posts are wrapped in <main><article>. + - id: hiero + type: website + base-url: https://hiero.org + sitemaps: [ /sitemap.xml ] + url-include: [ "/blog/**" ] + content-selector: "main article" + enabled: true + # Support & Care is two pages under /en/ (support-care, support-care-maven), listed in + # /en/sitemap.xml. They lack a clean article container, so take the <body> minus boilerplate. + - id: support-and-care + type: website + base-url: https://open-elements.com + sitemaps: [ /en/sitemap.xml ] + url-include: [ "/en/support-care*", "/en/support-care/**" ] + content-selector: "body" + content-exclude: [ "nav", "header", "footer", ".cookie-banner", "aside" ] + enabled: true diff --git a/src/test/java/com/openelements/content/ConfiguredSourcesTest.java b/src/test/java/com/openelements/content/ConfiguredSourcesTest.java new file mode 100644 index 0000000..2dbbeda --- /dev/null +++ b/src/test/java/com/openelements/content/ConfiguredSourcesTest.java @@ -0,0 +1,59 @@ +package com.openelements.content; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.Map; +import java.util.function.Function; +import java.util.stream.Collectors; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; + +/** + * Verifies that the website sources are bound from {@code application.yaml} with no code — the point + * of spec 015 (hiero.org/blog and Support & Care are added purely as configuration). + * + * <p>Shares the application context of the other {@code api-key.enabled=false} tests. The actual crawl + * and extraction against the live sites are verified manually against a dev index. + */ +@SpringBootTest(properties = "openelements.mcp.auth.api-key.enabled=false") +@DisplayName("Configured website sources") +class ConfiguredSourcesTest { + + @Autowired + private ContentSourceProperties properties; + + private Map<String, ContentSource> sourcesById() { + return properties.sources().stream() + .collect(Collectors.toMap(ContentSource::id, Function.identity())); + } + + @Test + @DisplayName("open-elements, hiero and support-and-care are all configured") + void allSourcesAreConfigured() { + assertThat(sourcesById()).containsKeys("open-elements", "hiero", "support-and-care"); + } + + @Test + @DisplayName("the hiero source targets /blog via the sitemap with the article selector") + void hieroSourceIsConfigured() { + ContentSource hiero = sourcesById().get("hiero"); + assertThat(hiero.type()).isEqualTo(SourceType.WEBSITE); + assertThat(hiero.baseUrl()).isEqualTo("https://hiero.org"); + assertThat(hiero.sitemaps()).containsExactly("/sitemap.xml"); + assertThat(hiero.urlInclude()).containsExactly("/blog/**"); + assertThat(hiero.contentSelector()).isEqualTo("main article"); + assertThat(hiero.enabled()).isTrue(); + } + + @Test + @DisplayName("the support-and-care source scopes to the /en/support-care pages with body + excludes") + void supportAndCareSourceIsConfigured() { + ContentSource support = sourcesById().get("support-and-care"); + assertThat(support.baseUrl()).isEqualTo("https://open-elements.com"); + assertThat(support.urlInclude()).contains("/en/support-care*"); + assertThat(support.contentSelector()).isEqualTo("body"); + assertThat(support.contentExclude()).contains("nav", "header", "footer", ".cookie-banner"); + } +} From c0804367b240fcdb4372cca8e655b69b314529a6 Mon Sep 17 00:00:00 2001 From: Hendrik Ebbers <hendrik.ebbers@open-elements.com> Date: Mon, 13 Jul 2026 21:58:52 +0200 Subject: [PATCH 3/3] chore(spec-015): mark additional-web-sources done Refs #29 --- docs/specs/INDEX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specs/INDEX.md b/docs/specs/INDEX.md index 870463e..bf29b31 100644 --- a/docs/specs/INDEX.md +++ b/docs/specs/INDEX.md @@ -20,6 +20,6 @@ roadmap step, to be implemented sequentially (each builds on the previous). | 012 | 012-content-mcp-tools | Content MCP tools | backend, mcp, api | `ContentMcpToolProvider` — the 4 MCP tools | #23 | done | | 013 | 013-scoped-search-key | Scoped search key | backend, search, security | Read-only scoped Meilisearch key for the content index | #25 | done | | 014 | 014-ops-robustness | Ops & robustness | backend, infrastructure, observability | robots.txt handling, fault tolerance, logging/metrics | #27 | done | -| 015 | 015-additional-web-sources | Additional web sources | backend, configuration | hiero.org/blog + Support & Care as config-only sources | #29 | in progress | +| 015 | 015-additional-web-sources | Additional web sources | backend, configuration | hiero.org/blog + Support & Care as config-only sources | #29 | done | | 016 | 016-git-markdown-source | Git markdown source | backend, architecture, security | `type: git` source + `GitSourceStrategy` (GitHub Markdown) | — | open | | 017 | 017-search-enhancements | Search enhancements | backend, search | Facets, synonyms/stop words, optional semantic search | — | open |