Sitemap crawler — URL discovery via sitemaps with bounded fallback crawl

[herbie-bot]

Summary

SitemapCrawler discovers the candidate URLs of a website source by reading its sitemap index and child sitemaps via Spring RestClient, collecting every <loc> + <lastmod>, and filtering against the source's Ant-glob urlInclude/urlExclude (spec 002). When a source has no sitemap, it falls back to a bounded link-following crawl from baseUrl. Output: DiscoveredItem(url, lastmod) for the fetch stage.

Spec

• Spec folder: docs/specs/004-sitemap-crawler/
• Design: docs/specs/004-sitemap-crawler/design.md
• Behaviors: docs/specs/004-sitemap-crawler/behaviors.md

Acceptance criteria (from behaviors.md)

• Flat <urlset> collected into DiscoveredItems (url + lastmod)
• <sitemapindex> followed recursively; union of children returned
• Missing <lastmod> → null (tolerated)
• Only urlInclude URLs returned; urlExclude URLs dropped
• No sitemap → bounded fallback crawl from baseUrl; out-of-host links ignored
• Fallback crawl terminates on cyclic links (visited-set)
• Unreachable sitemap (404/timeout) logged; other sitemaps still contribute
• Malformed XML logged; that sitemap contributes no URLs; no exception propagates

Notes

• Roadmap Phase 1 step 4; design doc §5.1, §6. Builds on spec 002 (UrlMatcher).