Operations & robustness — robots.txt, crawl-delay, hardening

CLAUDE.md

@@ -10,11 +10,13 @@ The **Open Elements Content MCP** is an MCP server that will crawl, index, and m
 searchable the content (primarily blog posts) of Open-Elements-adjacent websites, exposing
 that content to AI agents over the standard `/mcp` streamable-HTTP endpoint.
 
-Current state: **project skeleton** (spec `001-project-skeleton`). The app boots and exposes
-`/mcp` with no content-specific tools yet. Planned capabilities (see
-[`docs/roadmap.md`](docs/roadmap.md)): sitemap crawling, HTTP page fetching, jsoup content
-extraction, Meilisearch indexing with scheduled refresh, and four MCP tools
-(`search_content`, `list_posts`, `get_post`, `list_categories`).
+Current state: **Phase 1 complete** (specs 001–014). The app crawls configured website sources
+(sitemap discovery + bounded fallback, robots.txt-aware), fetches politely (conditional GET, per-host
+rate limit, retries), extracts content/metadata with jsoup, indexes into Meilisearch (startup
+bootstrap + scheduled incremental refresh), and exposes four MCP tools on `/mcp` (`search_content`,
+`list_posts`, `get_post`, `list_categories`) backed by a scoped Meilisearch key. Phase 2+ (see
+[`docs/roadmap.md`](docs/roadmap.md)): additional website sources, Git/Markdown sources, and search
+enhancements.
 
 ### Tech Stack
 
@@ -57,7 +59,8 @@ extraction, Meilisearch indexing with scheduled refresh, and four MCP tools
 │   ├── ContentBootstrapStep.java            # SearchIndexBootstrapStep: startup full reindex
 │   ├── ContentRefreshScheduler.java         # @Scheduled incremental re-crawl (cron, guarded)
 │   ├── ContentSearchService.java            # read facade: multiSearch + Highlighter (+ result records)
-│   └── ContentMcpToolProvider.java          # McpToolProvider: the 4 tools on /mcp
+│   ├── ContentMcpToolProvider.java          # McpToolProvider: the 4 tools on /mcp
+│   ├── RobotsPolicy.java / HttpRobotsPolicy.java  # robots.txt allow/disallow + Crawl-delay
 ├── src/main/resources/application.yaml      # datasource, JPA, OAuth2, MCP, Meilisearch, content config
 ├── src/test/java/com/openelements/content/  # behavior tests (context, MCP enabled/disabled, search-down, jsoup)
 └── docs/
@@ -127,7 +130,11 @@ as thin adapters over `ContentSearchService`, using the house helpers for schema
 `ContentConfig` registers a `ScopedKeySpec` bean so the library exchanges the master key for one scoped
 to the content index with the minimal actions the service uses (it both reads and writes); MCP api-key
 auth on `/mcp` defaults to **enabled** (`MCP_API_KEY_AUTH_ENABLED`), with the `dev` profile
-(`application-dev.yaml`) disabling it for local use.
+(`application-dev.yaml`) disabling it for local use. Robustness (spec 014): `HttpRobotsPolicy` honors
+each host's `robots.txt` (cached per host) — `SitemapCrawler` drops disallowed URLs and `PageFetcher`
+respects `Crawl-delay` via the per-host rate limiter. The pipeline is fault-tolerant end-to-end (one
+failing page/source never aborts a run) and logs an `IndexReport` per source; Micrometer metrics are
+deferred until a metrics backend is wired.
 
 > **Build note:** `spring-services` is pinned to a specific snapshot timestamp
 > (`1.3.0-20260712.175350-13`) in `pom.xml`, not the floating `1.3.0-SNAPSHOT`, because a later

docs/specs/014-ops-robustness/steps.md

@@ -0,0 +1,59 @@
+# Implementation Steps: Operations & Robustness
+
+## Step 1: robots.txt
+
+- [x] `RobotsPolicy` interface (`isAllowed`, `crawlDelay`) with an `ALLOW_ALL` no-op for tests
+- [x] `HttpRobotsPolicy` (`@Component`): fetches `robots.txt` per host over HTTPS, caches with a TTL (injectable clock), selects the group for our user agent (falling back to `*`), longest-prefix matching with `Allow` winning ties, missing robots → allow-all, parses `Crawl-delay`
+- [x] `SitemapCrawler` filters discovered URLs (sitemap + fallback crawl) through `RobotsPolicy`, logging skips
+- [x] `PageFetcher` honors `Crawl-delay` via `HostRateLimiter.acquire(host, minInterval)` (stricter of config vs. crawl-delay)
+- [x] Hardening: `SitemapCrawler` only follows child sitemaps on the same host (a sitemap index cannot point the crawler at arbitrary/internal hosts) — closes the SSRF surface flagged in earlier specs
+
+**Related behaviors:** disallowed skipped; allowed fetched; cached per host; missing → allow-all; crawl-delay tightens the rate limit
+
+---
+
+## Step 2: Fault tolerance (audit)
+
+- [x] Verified end-to-end (covered by existing tests): one failing page → `SKIP` (`WebsiteSourceStrategyTest`, `ContentIndexerTest.oneFailingPageIsSkipped`); one failing source isolated (`ContentBootstrapStepTest.failingSourceDoesNotBlockOthers`, `ContentRefreshSchedulerTest.oneSourceFailingDoesNotStopOthers`); idempotent upserts (`ContentIndexerTest`)
+
+**Related behaviors:** failing page isolated; failing source isolated
+
+---
+
+## Step 3: Observability (logging; metrics deferred)
+
+- [x] Structured logging: `IndexReport` per source (`ContentIndexer`/`ContentRefreshScheduler`); robots-skip (`SitemapCrawler`/`HttpRobotsPolicy`); non-success Meilisearch task outcomes logged (`MeilisearchContentIndexStore`)
+- [ ] Micrometer metrics — **deferred** per the design's open question: no metrics backend is wired (only `micrometer-observation`/`-commons` are on the classpath, not `micrometer-core`/a `MeterRegistry`). Structured logging is shipped now; add counters/timers when a metrics backend exists.
+
+**Related behaviors:** IndexReport logged per source; failed Meilisearch task surfaced (logging half); metrics behaviors deferred
+
+---
+
+## Step 4: Tests
+
+- [x] `HttpRobotsPolicyTest` (allow/disallow/allow-override/agent-group/cache/missing/crawl-delay via `MockRestServiceServer`)
+- [x] `HostRateLimiterTest.crawlDelayTightensInterval`
+- [x] `SitemapCrawlerTest.robotsDisallowedUrlsAreDropped`
+- [x] Existing crawler/fetcher/strategy tests updated to `RobotsPolicy.ALLOW_ALL`
+
+**Acceptance criteria:**
+- [x] All tests pass (`mvn test`); build green
+
+---
+
+## Behavior Coverage
+
+| Scenario | Layer | Covered in Step |
+|----------|-------|-----------------|
+| Disallowed path is skipped | Backend | Step 4 (`HttpRobotsPolicyTest.disallowedPathIsRejected`, `SitemapCrawlerTest.robotsDisallowedUrlsAreDropped`) |
+| Allowed path is fetched | Backend | Step 4 (`HttpRobotsPolicyTest.disallowedPathIsRejected` asserts allowed too) |
+| robots.txt is cached per host | Backend | Step 4 (`HttpRobotsPolicyTest.robotsIsCachedPerHost`) |
+| Missing robots.txt means allow-all | Backend | Step 4 (`HttpRobotsPolicyTest.missingRobotsAllowsAll`) |
+| Crawl-delay tightens the rate limit | Backend | Step 4 (`HttpRobotsPolicyTest.crawlDelayIsParsed` + `HostRateLimiterTest.crawlDelayTightensInterval`) |
+| Failing page is isolated | Backend | Step 2 (existing `WebsiteSourceStrategyTest`/`ContentIndexerTest`) |
+| Failing source is isolated | Backend | Step 2 (existing `ContentBootstrapStepTest`/`ContentRefreshSchedulerTest`) |
+| IndexReport is logged per source | Backend | Step 3 (structured logging in `ContentIndexer`/`ContentRefreshScheduler`) |
+| Metrics record fetch and index activity | Backend | **Deferred** (Step 3) — no metrics backend wired; design open question |
+| Failed Meilisearch task is surfaced | Backend | Step 3 (logged in `MeilisearchContentIndexStore`; metric part deferred) |
+
+All scenarios are backend; there is no frontend in this spec.

docs/specs/INDEX.md

@@ -19,7 +19,7 @@ roadmap step, to be implemented sequentially (each builds on the previous).
 | 011 | 011-content-search-service | Content search service | backend, search | `ContentSearchService` — `multiSearch` + highlighting facade | #21 | done |
 | 012 | 012-content-mcp-tools | Content MCP tools | backend, mcp, api | `ContentMcpToolProvider` — the 4 MCP tools | #23 | done |
 | 013 | 013-scoped-search-key | Scoped search key | backend, search, security | Read-only scoped Meilisearch key for the content index | #25 | done |
-| 014 | 014-ops-robustness | Ops & robustness | backend, infrastructure, observability | robots.txt handling, fault tolerance, logging/metrics | — | open |
+| 014 | 014-ops-robustness | Ops & robustness | backend, infrastructure, observability | robots.txt handling, fault tolerance, logging/metrics | #27 | done |
 | 015 | 015-additional-web-sources | Additional web sources | backend, configuration | hiero.org/blog + Support & Care as config-only sources | — | open |
 | 016 | 016-git-markdown-source | Git markdown source | backend, architecture, security | `type: git` source + `GitSourceStrategy` (GitHub Markdown) | — | open |
 | 017 | 017-search-enhancements | Search enhancements | backend, search | Facets, synonyms/stop words, optional semantic search | — | open |

src/main/java/com/openelements/content/HostRateLimiter.java

@@ -1,5 +1,6 @@
 package com.openelements.content;
 
+import java.time.Duration;
 import java.util.HashMap;
 import java.util.Map;
 import java.util.function.LongConsumer;
@@ -38,7 +39,20 @@ public HostRateLimiter(double permitsPerSecond, LongSupplier clockNanos, LongCon
      * @param host the target host (an empty or {@code null} host shares a single bucket)
      */
     public void acquire(String host) {
-        if (minIntervalNanos == 0L) {
+        acquire(host, Duration.ZERO);
+    }
+
+    /**
+     * Like {@link #acquire(String)}, but enforces at least {@code minInterval} between requests to the
+     * host — used to honor a {@code robots.txt} {@code Crawl-delay}, taking the stricter of the
+     * configured rate and the crawl delay.
+     *
+     * @param host        the target host
+     * @param minInterval a minimum interval to enforce (e.g. from {@code Crawl-delay}); may be zero
+     */
+    public void acquire(String host, Duration minInterval) {
+        long effectiveIntervalNanos = Math.max(minIntervalNanos, minInterval == null ? 0L : minInterval.toNanos());
+        if (effectiveIntervalNanos == 0L) {
             return;
         }
         String key = host == null ? "" : host;
@@ -47,7 +61,7 @@ public void acquire(String host) {
             long now = clockNanos.getAsLong();
             long start = Math.max(now, nextAllowedNanos.getOrDefault(key, now));
             waitNanos = start - now;
-            nextAllowedNanos.put(key, start + minIntervalNanos);
+            nextAllowedNanos.put(key, start + effectiveIntervalNanos);
         }
         if (waitNanos > 0) {
             sleeperMillis.accept(Math.max(1, Math.round(waitNanos / 1_000_000.0)));

src/main/java/com/openelements/content/HttpRobotsPolicy.java

@@ -0,0 +1,209 @@
+package com.openelements.content;
+
+import java.net.URI;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.LongSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+import org.springframework.web.client.RestClient;
+
+/**
+ * {@link RobotsPolicy} that fetches and caches each host's {@code robots.txt}.
+ *
+ * <p>Rules are fetched once per host (over HTTPS) and cached with a TTL. The group matching our bot
+ * {@code User-Agent} is used, falling back to the {@code *} group; a missing or unreachable
+ * {@code robots.txt} allows everything. Path matching is longest-prefix with {@code Allow} winning
+ * ties (the common robots semantics). The clock is injectable so cache expiry is testable.
+ */
+@Component
+public class HttpRobotsPolicy implements RobotsPolicy {
+
+    private static final Logger log = LoggerFactory.getLogger(HttpRobotsPolicy.class);
+    private static final Duration CACHE_TTL = Duration.ofHours(1);
+
+    private final RestClient restClient;
+    private final String userAgent;
+    private final LongSupplier clockNanos;
+    private final Map<String, CachedRules> cache = new ConcurrentHashMap<>();
+
+    @Autowired
+    public HttpRobotsPolicy(RestClient.Builder restClientBuilder, ContentSourceProperties properties) {
+        this(restClientBuilder, properties, System::nanoTime);
+    }
+
+    HttpRobotsPolicy(RestClient.Builder restClientBuilder, ContentSourceProperties properties, LongSupplier clockNanos) {
+        this.restClient = restClientBuilder.build();
+        this.userAgent = properties.userAgent();
+        this.clockNanos = clockNanos;
+    }
+
+    @Override
+    public boolean isAllowed(String url) {
+        String host = hostOf(url);
+        if (host.isEmpty()) {
+            return true;
+        }
+        boolean allowed = rulesFor(host).allows(pathOf(url));
+        if (!allowed) {
+            log.info("robots.txt disallows {} for {}", url, userAgent);
+        }
+        return allowed;
+    }
+
+    @Override
+    public Duration crawlDelay(String host) {
+        return host == null || host.isEmpty() ? Duration.ZERO : rulesFor(host).crawlDelay();
+    }
+
+    private RobotsRules rulesFor(String host) {
+        CachedRules cached = cache.get(host);
+        if (cached != null && cached.expiryNanos() > clockNanos.getAsLong()) {
+            return cached.rules();
+        }
+        RobotsRules rules = fetchAndParse(host);
+        cache.put(host, new CachedRules(rules, clockNanos.getAsLong() + CACHE_TTL.toNanos()));
+        return rules;
+    }
+
+    private RobotsRules fetchAndParse(String host) {
+        try {
+            String body = restClient.get()
+                .uri(URI.create("https://" + host + "/robots.txt"))
+                .header("User-Agent", userAgent)
+                .retrieve()
+                .body(String.class);
+            return parse(body == null ? "" : body, userAgent);
+        } catch (Exception e) {
+            // Missing/unreachable robots.txt (e.g. 404) means allow everything.
+            log.debug("No usable robots.txt for {} ({}); allowing all", host, e.toString());
+            return RobotsRules.ALLOW_ALL;
+        }
+    }
+
+    /** Parses robots.txt, selecting the group for our user agent (falling back to {@code *}). */
+    static RobotsRules parse(String body, String userAgent) {
+        List<Group> groups = new ArrayList<>();
+        Group current = null;
+        boolean lastWasRule = false;
+        for (String rawLine : body.split("\n")) {
+            String line = stripComment(rawLine).trim();
+            if (line.isEmpty()) {
+                continue;
+            }
+            int colon = line.indexOf(':');
+            if (colon < 0) {
+                continue;
+            }
+            String field = line.substring(0, colon).trim().toLowerCase(Locale.ROOT);
+            String value = line.substring(colon + 1).trim();
+            if (field.equals("user-agent")) {
+                if (current == null || lastWasRule) {
+                    current = new Group();
+                    groups.add(current);
+                }
+                current.agents.add(value.toLowerCase(Locale.ROOT));
+                lastWasRule = false;
+            } else if (current != null) {
+                switch (field) {
+                    case "disallow" -> current.disallow.add(value);
+                    case "allow" -> current.allow.add(value);
+                    case "crawl-delay" -> current.crawlDelay = parseDelay(value);
+                    default -> { /* ignore other directives */ }
+                }
+                lastWasRule = true;
+            }
+        }
+        return selectGroup(groups, userAgent).toRules();
+    }
+
+    private static Group selectGroup(List<Group> groups, String userAgent) {
+        String ua = userAgent.toLowerCase(Locale.ROOT);
+        Group wildcard = null;
+        for (Group group : groups) {
+            for (String agent : group.agents) {
+                if (agent.equals("*")) {
+                    wildcard = group;
+                } else if (!agent.isEmpty() && ua.contains(agent)) {
+                    return group; // a group naming our agent wins over the wildcard
+                }
+            }
+        }
+        return wildcard == null ? new Group() : wildcard;
+    }
+
+    private static Duration parseDelay(String value) {
+        try {
+            return Duration.ofMillis(Math.round(Double.parseDouble(value) * 1000));
+        } catch (NumberFormatException e) {
+            return Duration.ZERO;
+        }
+    }
+
+    private static String stripComment(String line) {
+        int hash = line.indexOf('#');
+        return hash >= 0 ? line.substring(0, hash) : line;
+    }
+
+    private static String hostOf(String url) {
+        try {
+            String host = URI.create(url).getHost();
+            return host == null ? "" : host;
+        } catch (IllegalArgumentException e) {
+            return "";
+        }
+    }
+
+    private static String pathOf(String url) {
+        try {
+            String path = URI.create(url).getPath();
+            return path == null || path.isEmpty() ? "/" : path;
+        } catch (IllegalArgumentException e) {
+            return "/";
+        }
+    }
+
+    private static final class Group {
+        private final List<String> agents = new ArrayList<>();
+        private final List<String> disallow = new ArrayList<>();
+        private final List<String> allow = new ArrayList<>();
+        private Duration crawlDelay = Duration.ZERO;
+
+        private RobotsRules toRules() {
+            return new RobotsRules(List.copyOf(disallow), List.copyOf(allow), crawlDelay);
+        }
+    }
+
+    private record CachedRules(RobotsRules rules, long expiryNanos) {
+    }
+
+    /** Parsed robots rules for the applicable group. */
+    record RobotsRules(List<String> disallow, List<String> allow, Duration crawlDelay) {
+
+        static final RobotsRules ALLOW_ALL = new RobotsRules(List.of(), List.of(), Duration.ZERO);
+
+        boolean allows(String path) {
+            int longestDisallow = longestMatch(disallow, path);
+            if (longestDisallow < 0) {
+                return true;
+            }
+            return longestMatch(allow, path) >= longestDisallow; // Allow wins ties
+        }
+
+        private static int longestMatch(List<String> rules, String path) {
+            int longest = -1;
+            for (String rule : rules) {
+                if (!rule.isEmpty() && path.startsWith(rule) && rule.length() > longest) {
+                    longest = rule.length();
+                }
+            }
+            return longest;
+        }
+    }
+}

src/main/java/com/openelements/content/PageFetcher.java

@@ -36,20 +36,22 @@ public class PageFetcher {
     private final String userAgent;
     private final long maxBodyBytes;
     private final HostRateLimiter rateLimiter;
+    private final RobotsPolicy robotsPolicy;
     private final LongConsumer backoffSleeper;
 
     @Autowired
     public PageFetcher(RestClient contentRestClient, ContentSourceProperties properties,
-                       HostRateLimiter rateLimiter) {
-        this(contentRestClient, properties, rateLimiter, PageFetcher::sleep);
+                       HostRateLimiter rateLimiter, RobotsPolicy robotsPolicy) {
+        this(contentRestClient, properties, rateLimiter, robotsPolicy, PageFetcher::sleep);
     }
 
     PageFetcher(RestClient restClient, ContentSourceProperties properties,
-                HostRateLimiter rateLimiter, LongConsumer backoffSleeper) {
+                HostRateLimiter rateLimiter, RobotsPolicy robotsPolicy, LongConsumer backoffSleeper) {
         this.restClient = restClient;
         this.userAgent = properties.userAgent();
         this.maxBodyBytes = properties.maxBodyBytes().toBytes();
         this.rateLimiter = rateLimiter;
+        this.robotsPolicy = robotsPolicy;
         this.backoffSleeper = backoffSleeper;
     }
 
@@ -83,7 +85,9 @@ public FetchResult fetch(String url, String priorEtag, String priorLastmod) {
     }
 
     private FetchResult attemptOnce(String url, String priorEtag, String priorLastmod) {
-        rateLimiter.acquire(hostOf(url));
+        String host = hostOf(url);
+        // Honor robots.txt Crawl-delay for this host, taking the stricter of it and the configured rate.
+        rateLimiter.acquire(host, robotsPolicy.crawlDelay(host));
         return restClient.get()
             .uri(URI.create(url))
             .header(HttpHeaders.USER_AGENT, userAgent)