Fix GHA crawl: remove 404 section seeds, add --full to weekly crawl

Section-level seed URLs like /search/10.3.2512 and /get-started/10.2
return HTTP 404 on help.splunk.com. They accumulated as 'failed' in
crawl_state and were re-attempted on every GHA run. Landing page BFS
already discovers all pages without them, so they are removed.

Add --full to the GHA crawl step so the weekly cron actually re-fetches
content. Without --full, all seeds are in crawl_state as 'fetched' and
every run prints 'Nothing to crawl' -- the index never updates.
--full re-fetches all pages; the content hash check skips re-embedding
unchanged pages.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: jwindley

.github/workflows/crawl-and-release.yml

@@ -53,7 +53,12 @@ jobs:
             splunk-db-${{ matrix.source }}-
 
       - name: Crawl ${{ matrix.source }}
-        run: uv run splunk-crawl --sources ${{ matrix.source }} --db data/${{ matrix.source }}.db
+        # --full re-fetches all pages on every run so the index stays current.
+        # Pages whose raw HTML hasn't changed are skipped at the extract/embed
+        # stage (hash comparison), so only genuinely updated content is re-indexed.
+        # Without --full the cached crawl_state causes "Nothing to crawl" on every
+        # run after the first, meaning the index is never updated.
+        run: uv run splunk-crawl --sources ${{ matrix.source }} --db data/${{ matrix.source }}.db --full
 
       - name: Save per-source DB cache
         if: always()

src/splunk_docs_mcp/config.py

@@ -185,14 +185,11 @@ class CrawlSource:
         display_name="Splunk Enterprise 10.2",
         version="10.2",
         seed_urls=[
-            # Landing page is the primary reliable entry point
+            # Landing page only — BFS discovers all section pages from here.
+            # Section-level seeds like /{section}/10.2 return HTTP 404 on
+            # help.splunk.com and were removed to avoid accumulating dead URLs
+            # in crawl_state that get re-attempted on every run.
             "https://help.splunk.com/en/splunk-enterprise/",
-            # Section-level seeds — may redirect to a deeper page; the redirect
-            # bug fix (2026-04-18) ensures relative links resolve correctly.
-            *[
-                f"https://help.splunk.com/en/splunk-enterprise/{s}/10.2"
-                for s in _ENTERPRISE_SECTIONS
-            ],
         ],
         url_prefix="https://help.splunk.com/en/splunk-enterprise/",
         blocked_path_prefixes=_HELP_BLOCKED,
@@ -202,13 +199,8 @@ class CrawlSource:
         display_name="Splunk Cloud Platform 10.3.2512",
         version="10.3.2512",
         seed_urls=[
-            # Landing page is the primary reliable entry point
+            # Landing page only — same reasoning as splunk-enterprise above.
             "https://help.splunk.com/en/splunk-cloud-platform/",
-            # Section-level seeds
-            *[
-                f"https://help.splunk.com/en/splunk-cloud-platform/{s}/10.3.2512"
-                for s in _CLOUD_SECTIONS
-            ],
         ],
         url_prefix="https://help.splunk.com/en/splunk-cloud-platform/",
         blocked_path_prefixes=_HELP_BLOCKED,
@@ -219,10 +211,6 @@ class CrawlSource:
         version="10.1",
         seed_urls=[
             "https://help.splunk.com/en/splunk-enterprise/",
-            *[
-                f"https://help.splunk.com/en/splunk-enterprise/{s}/10.1"
-                for s in _ENTERPRISE_SECTIONS
-            ],
         ],
         url_prefix="https://help.splunk.com/en/splunk-enterprise/",
         blocked_path_prefixes=_HELP_BLOCKED,
@@ -233,10 +221,6 @@ class CrawlSource:
         version="10.2",
         seed_urls=[
             "https://help.splunk.com/en/splunk-cloud-platform/",
-            *[
-                f"https://help.splunk.com/en/splunk-cloud-platform/{s}/10.2"
-                for s in _CLOUD_SECTIONS
-            ],
         ],
         url_prefix="https://help.splunk.com/en/splunk-cloud-platform/",
         blocked_path_prefixes=_HELP_BLOCKED,