Merge pull request #32 from OpenElementsLabs/feat/016-git-markdown-source

Git Markdown source — type: git + GitSourceStrategy

Author: hendrikebbers

CLAUDE.md

@@ -62,6 +62,7 @@ search enhancements.
 │   ├── ContentSearchService.java            # read facade: multiSearch + Highlighter (+ result records)
 │   ├── ContentMcpToolProvider.java          # McpToolProvider: the 4 tools on /mcp
 │   ├── RobotsPolicy.java / HttpRobotsPolicy.java  # robots.txt allow/disallow + Crawl-delay
+│   └── GitSourceStrategy.java / GitConfig.java    # type: git — GitHub Markdown source (spec 016)
 ├── src/main/resources/application.yaml      # datasource, JPA, OAuth2, MCP, Meilisearch, content config
 ├── src/test/java/com/openelements/content/  # behavior tests (context, MCP enabled/disabled, search-down, jsoup)
 └── docs/
@@ -106,8 +107,12 @@ categories/preview image/locale) from OpenGraph/Article `<meta>`, JSON-LD, and `
 `ContentSourceStrategy` is the per-source-type seam that composes these stages: `WebsiteSourceStrategy`
 wires crawler → fetcher → extractor and maps the fetch result to a `FetchOutcome`
 (`INDEX`/`UNCHANGED`/`DELETE`/`SKIP`); `SourceStrategyRegistry` selects the strategy by
-`ContentSource.type()`, so a future `git` strategy (spec 016) plugs in as a bean with no indexer
-changes. `ContentIndexer` is the orchestration engine: discover → diff discovered `lastmod` against
+`ContentSource.type()`, so the `git` strategy (spec 016) plugs in as a bean with no indexer
+changes. `GitSourceStrategy` (`SourceType.GIT`, `GitConfig`) indexes Markdown from GitHub repos:
+Trees-API discovery filtered by `paths` globs (blob SHA = change marker), raw-content fetch with a
+server-side bearer token, YAML-frontmatter + Markdown extraction (Hugo shortcodes cleaned), path →
+canonical-URL mapping, and locale from the filename suffix — producing the same `ContentDocument` as
+websites, so downstream is unchanged. `ContentIndexer` is the orchestration engine: discover → diff discovered `lastmod` against
 the state read from the index (`ContentIndexStore`; the index *is* the state) → fetch only new/changed
 items via the strategy → batch-upsert, and delete documents that 404 or vanished from discovery.
 `MeilisearchContentIndexStore` implements the store over `MeilisearchClient` (paged `multiSearch` to

docs/specs/016-git-markdown-source/steps.md

@@ -0,0 +1,51 @@
+# Implementation Steps: Git Markdown Source
+
+## Step 1: Config model
+
+- [x] `GitConfig` record (`provider`, `repo`, `ref`, `paths`, `token`) with `hasToken()`
+- [x] Add optional `git` field to `ContentSource` (present only for `type: git`); existing call sites pass `null`
+
+**Related behaviors:** private-repo token; config binding
+
+---
+
+## Step 2: `GitSourceStrategy`
+
+- [x] `@Component implements ContentSourceStrategy` for `SourceType.GIT` (auto-routed by `SourceStrategyRegistry`)
+- [x] `discover`: GitHub Trees API (`/repos/{repo}/git/trees/{ref}?recursive=1`), keep blobs matching `paths` globs (`AntPathMatcher`), SHA as change marker; a hard failure (auth) throws so the source is isolated (not mass-deleted)
+- [x] `fetch`: raw content (`raw.githubusercontent.com`), split YAML frontmatter from Markdown body, clean Hugo shortcodes, map path → canonical URL, derive locale from filename suffix, build `ContentDocument` (SHA = `lastmod`)
+- [x] Secrets: bearer token applied server-side only; never logged; empty-default env placeholder
+
+**Related behaviors:** discovery/filter; incremental SHA; frontmatter/shortcode/URL/locale extraction; token handling
+
+---
+
+## Step 3: Config + tests
+
+- [x] `application.yaml`: example `oe-website-markdown` git source (disabled by default; token via `GITHUB_TOKEN_OE_WEBSITE`, empty default)
+- [x] `GitSourceStrategyTest` (MockRestServiceServer): discovery+filter+SHA, bearer token, discovery-failure, fetch extraction (frontmatter/shortcode/URL/locale), fetch-failure skip, and pure `mapToUrl`/`localeFromPath`/`parseMarkdown`
+- [x] `ConfiguredSourcesTest`: the git source binds as `SourceType.GIT`
+
+**Acceptance criteria:**
+- [x] All tests pass (`mvn test`); build green
+
+---
+
+## Behavior Coverage
+
+| Scenario | Layer | Covered in Step |
+|----------|-------|-----------------|
+| Files discovered via the trees API and filtered by paths | Backend | Step 3 (`discoveryFiltersByPaths`) |
+| Non-matching files excluded | Backend | Step 3 (`discoveryFiltersByPaths` — README/png excluded) |
+| Unchanged file (same SHA) is skipped | Backend | SHA returned as `lastmod` (`discoveryFiltersByPaths`); the unchanged/changed diff is `ContentIndexer`'s (spec 008 tests) |
+| Changed file (new SHA) is re-indexed | Backend | As above — diff handled by `ContentIndexer`; strategy supplies the SHA |
+| Frontmatter becomes metadata | Backend | Step 3 (`fetchExtractsFrontmatterAndBody`) |
+| Hugo shortcodes are cleaned | Backend | Step 3 (`fetchExtractsFrontmatterAndBody` — no `{{<`) |
+| Path maps to canonical URL | Backend | Step 3 (`datedFilenameMapsToCanonicalUrl`, `fetchExtractsFrontmatterAndBody`) |
+| Locale from filename suffix | Backend | Step 3 (`localeFromFilenameSuffix`) |
+| Private repo accessed with token | Backend | Step 3 (`tokenIsSentForPrivateRepo`) |
+| Token is never logged or served | Backend | Design: token applied only as a request header, never logged (auth logs use `repo`, not token); MCP serves only indexed docs |
+| Missing token for private repo fails clearly | Backend | Step 3 (`missingTokenFailsClearly`) |
+| Git-sourced docs behave like website docs | Backend | Downstream unchanged — same `ContentDocument`/indexer/tools (specs 008/011/012) |
+
+All scenarios are backend; there is no frontend in this spec.

docs/specs/INDEX.md

@@ -21,5 +21,5 @@ roadmap step, to be implemented sequentially (each builds on the previous).
 | 013 | 013-scoped-search-key | Scoped search key | backend, search, security | Read-only scoped Meilisearch key for the content index | #25 | done |
 | 014 | 014-ops-robustness | Ops & robustness | backend, infrastructure, observability | robots.txt handling, fault tolerance, logging/metrics | #27 | done |
 | 015 | 015-additional-web-sources | Additional web sources | backend, configuration | hiero.org/blog + Support & Care as config-only sources | #29 | done |
-| 016 | 016-git-markdown-source | Git markdown source | backend, architecture, security | `type: git` source + `GitSourceStrategy` (GitHub Markdown) | — | open |
+| 016 | 016-git-markdown-source | Git markdown source | backend, architecture, security | `type: git` source + `GitSourceStrategy` (GitHub Markdown) | #31 | done |
 | 017 | 017-search-enhancements | Search enhancements | backend, search | Facets, synonyms/stop words, optional semantic search | — | open |

src/main/java/com/openelements/content/ContentSource.java

@@ -24,6 +24,7 @@
  * @param contentSelector CSS selector for the main content element (applied in spec 006)
  * @param contentExclude CSS selectors removed before text extraction (applied in spec 006)
  * @param enabled        whether the source is active; consumers in later specs skip disabled sources
+ * @param git            Git configuration; present only for {@link SourceType#GIT} sources (spec 016)
  */
 public record ContentSource(
     String id,
@@ -34,7 +35,8 @@ public record ContentSource(
     List<String> urlExclude,
     String contentSelector,
     List<String> contentExclude,
-    boolean enabled
+    boolean enabled,
+    GitConfig git
 ) {
 
     /** The include pattern applied when a source declares no {@code urlInclude}: match everything. */

src/main/java/com/openelements/content/GitConfig.java

@@ -0,0 +1,31 @@
+package com.openelements.content;
+
+import java.util.List;
+
+/**
+ * Configuration for a {@link SourceType#GIT} source, bound from the {@code git:} sub-object of a
+ * content source. Present only when {@code type: git}.
+ *
+ * @param provider the git provider ({@code github}); only GitHub is supported for now
+ * @param repo     the repository in {@code owner/name} form
+ * @param ref      the branch, tag, or commit to read (e.g. {@code main})
+ * @param paths    Ant-glob patterns selecting which files to index (e.g. {@code content/posts/**​/*.md})
+ * @param token    an access token for private repos; supplied via the environment, never in plaintext YAML
+ */
+public record GitConfig(
+    String provider,
+    String repo,
+    String ref,
+    List<String> paths,
+    String token
+) {
+
+    public GitConfig {
+        paths = paths == null ? List.of() : List.copyOf(paths);
+    }
+
+    /** @return {@code true} if an access token is configured (private repo) */
+    public boolean hasToken() {
+        return token != null && !token.isBlank();
+    }
+}