docs: govern historical artifacts

Author: mschwar

AGENTS.md

@@ -18,6 +18,7 @@ Start with `CURRENT_STATE.md` for the current audit summary, then read `docs/age
 - `scripts/render_heatmap.py`: source-to-raster rebuild script
 - `scripts/check_homepage.py`: homepage smoke test script
 - `docs/heatmap-provenance.md`: provenance and rebuild notes
+- `docs/historical-artifacts.md`: policy for retained historical snapshots
 
 ## Commands To Run Before And After Changes
 
@@ -56,7 +57,7 @@ Hand-authored:
 
 Generated or historical artifact:
 
-- `README.CRAWL.md`: historical crawl output with stale machine-local metadata; do not treat as canonical truth
+- `README.CRAWL.md`: historical crawl output with stale machine-local metadata; see `docs/historical-artifacts.md`
 
 ## Data, Citation, And Provenance Rules
 
@@ -77,6 +78,7 @@ Generated or historical artifact:
 - Do not hand-edit generated artifacts if a real generator is introduced later.
 - Prefer edits that preserve GitHub Pages compatibility.
 - Avoid deleting historical artifacts unless their role is documented and the removal is clearly safe.
+- Keep retained historical snapshots documented in `docs/historical-artifacts.md`.
 
 ## Known Project-Specific Traps
 

CURRENT_STATE.md

@@ -15,6 +15,7 @@ This repository is a small GitHub Pages static site that publishes a single oral
 - `scripts/check_homepage.py` provides a non-UI smoke test for the rendered homepage.
 - The Jekyll config excludes `vendor/`, so the bundled gem tree is kept out of the public site build.
 - The validate workflow uses Node 24-compatible `actions/checkout@v6` and `actions/setup-python@v6`.
+- `README.CRAWL.md` is explicitly documented as a retained historical artifact in `docs/historical-artifacts.md`.
 - A repo-local validation command now checks required files, local links, key config values, and tracked OS junk.
 - The live GitHub Pages deployment returned HTTP 200 and served the expected page content in this session.
 
@@ -64,6 +65,6 @@ The pinned build path is now committed in-repo, so local build instructions are
 
 ## Immediate Next Moves
 
-1. Implement historical artifact governance in the next feature branch, focusing on the final treatment of `README.CRAWL.md` and any similar repo-only files.
+1. QA the historical artifact governance branch by confirming `README.CRAWL.md` stays out of the public site and the new policy doc stays discoverable.
 2. Keep `scripts/render_heatmap.py` and `Figure2-Teeth_v4.1.pdf` in sync if the figure changes.
 3. Run `python scripts/check_homepage.py` against `_site/index.html` after a local build if you need a homepage smoke check.

README.md

@@ -41,6 +41,12 @@ A 5-point color scale indicates the health status of each tooth, from healthy (b
 *   `scripts/render_heatmap.py`: Rebuild script for the published heatmap PNG.
 *   `scripts/check_homepage.py`: Smoke test for the rendered homepage HTML.
 *   `docs/heatmap-provenance.md`: Source artifact and rebuild notes.
+*   `docs/historical-artifacts.md`: Policy for retained historical snapshots.
+*   `README.CRAWL.md`: Historical crawl snapshot retained for auditability only.
+
+## Historical Artifacts
+
+`README.CRAWL.md` is kept as a historical artifact, not as canonical documentation. It is excluded from the public site, and the repo-level policy for retained snapshots lives in [`docs/historical-artifacts.md`](docs/historical-artifacts.md).
 
 ## Validate The Repo
 

docs/agentic-overhaul/2026-05-audit.md

@@ -14,6 +14,7 @@ Primary files:
 - `assets/css/style.scss.css`: site styling overrides
 - `oral-health-heatmap.png`: core visualization image
 - `README.CRAWL.md`: historical directory crawl artifact, not canonical documentation
+- `docs/historical-artifacts.md`: policy for retained historical snapshots
 
 Infrastructure added in this audit:
 
@@ -54,12 +55,12 @@ Scale: 0 = absent, 1 = fragile, 2 = basic, 3 = workable, 4 = strong, 5 = excelle
 - Local build tooling is now pinned with `Gemfile` and `.ruby-version`, and the build is wired into CI.
 - The heatmap source artifact is now committed as `Figure2-Teeth_v4.1.pdf`, with `scripts/render_heatmap.py` and `docs/heatmap-provenance.md` documenting the rebuild path.
 - The live GitHub Pages site returned `200 OK` and served the expected heatmap page in this session.
+- `README.CRAWL.md` remains at the repository root as a historical artifact, and `docs/historical-artifacts.md` records the policy for keeping it out of the public site.
 
 ## P1 Backlog
 
 - Add a provenance note describing what inputs and scripts produced the final heatmap, if those materials can be published safely.
 - Add a screenshot or rendered-page check in CI if a stable local build path is introduced.
-- Clarify whether `README.CRAWL.md` should remain as an artifact, move under `docs/`, or be removed.
 
 ## P2 Backlog
 

docs/historical-artifacts.md

@@ -0,0 +1,15 @@
+# Historical Artifacts
+
+This repository intentionally retains `README.CRAWL.md` as a historical crawl snapshot.
+
+## Policy
+
+- `README.CRAWL.md` is not canonical documentation.
+- It stays excluded from the public GitHub Pages site through `_config.yml`.
+- The file remains at the repository root so future audits can compare it against current repository truth.
+- Any future historical snapshots should follow the same pattern: document their purpose, keep them out of the public site, and point agents to the policy doc instead of the artifact itself.
+
+## Current Scope
+
+- The only retained historical artifact is `README.CRAWL.md`.
+- When in doubt, use `CURRENT_STATE.md`, `README.md`, and `AGENTS.md` as the authoritative operational docs.

scripts/validate_repo.py

@@ -17,6 +17,7 @@
     ROOT / "AGENTS.md",
     ROOT / "docs" / "agentic-first-buildout-plan.md",
     ROOT / "docs" / "heatmap-provenance.md",
+    ROOT / "docs" / "historical-artifacts.md",
     ROOT / "docs" / "agentic-overhaul" / "2026-05-audit.md",
 ]
 REQUIRED_FILES = [
@@ -33,6 +34,7 @@
     ROOT / "scripts" / "check_homepage.py",
     ROOT / "docs" / "agentic-first-buildout-plan.md",
     ROOT / "docs" / "heatmap-provenance.md",
+    ROOT / "docs" / "historical-artifacts.md",
     ROOT / "CURRENT_STATE.md",
     ROOT / "AGENTS.md",
     ROOT / "docs" / "agentic-overhaul" / "2026-05-audit.md",
@@ -137,6 +139,10 @@ def check_readme(warnings: list[str]) -> None:
         warnings.append("README.md does not document the homepage smoke test command.")
     if "Figure2-Teeth_v4.1.pdf" not in readme:
         warnings.append("README.md does not mention the committed heatmap source PDF.")
+    if "docs/historical-artifacts.md" not in readme:
+        warnings.append("README.md does not document the historical artifact policy.")
+    if "README.CRAWL.md" not in readme:
+        warnings.append("README.md does not mention the retained historical crawl snapshot.")
     crawl = ROOT / "README.CRAWL.md"
     if crawl.exists():
         crawl_text = crawl.read_text(encoding="utf-8")