feat(polish): tutorial code-fence static check (Phase 4 of polish-fact-check) (#37)

Phase 4 adds a static analysis pass over Python code fences in
polished tutorials. Every ```python fence is extracted, parsed
with ast.parse for syntax errors, and type-checked with
`mypy --strict` as a subprocess per fence. Findings land in the
same `## Unresolved references` block as Phase 1 fact-check
findings.

Scope is limited to `docs/tutorials/*.md`. Other doc kinds may
carry code samples but the reader-follow-along expectation is
highest for tutorials, so they're the highest-value target.

New module: src/attune_author/fact_check/tutorial_static_check.py
  - _FENCE_PATTERN regex extracts ```python ... ``` bodies plus
    line offsets so findings map back to absolute tutorial lines
  - _run_mypy: subprocess wrapper with 10s timeout; handles
    TimeoutExpired, FileNotFoundError, and unexpected exit codes
    by returning an empty findings list. The static check never
    blocks the polish pipeline.
  - _parse_mypy_output: regex parse of `<path>:<line>: <sev>:
    <msg>` lines, rewriting mypy line numbers to absolute file
    position via `base_line + mypy_line - 1`
  - _strip_skip_directive: honors `# attune-author: skip-mypy`
    as the first non-blank line of a fence (only first-line is
    recognized; trailing directives are intentionally preserved)
  - strip_skip_directives_in_file: removes the directive from
    the written tutorial so readers don't see it

Wiring:
  - check_tutorial_static toggle on FactCheckConfig (default True)
  - check_polished_file routes to tutorial_static_check.check only
    when is_tutorial_path(polished_path) returns True
  - apply_polish_results calls strip_skip_directives_in_file on
    the final content of tutorial-kind templates before writing

Phase 4.2 (sample EXECUTION — running the code, not just
type-checking it) is explicitly OUT OF SCOPE for this PR. Static
analysis only. Execution-tier design tracked as a follow-up.

Tests: 16 new tests covering syntax-error path, mypy happy path
with mocked subprocess, skip directive (in-fence + file-level
strip), tutorial-path routing in check_polished_file, and every
subprocess failure mode. Full suite: 942 passed (was 926), 37
pre-existing skips.

Real-fixture acceptance tests (4.8, 4.9, 4.11) require the
ops-dashboard tutorial fixture + a real mypy install. Unit-level
coverage with mocked subprocess + syntax-error path is in place;
the integration test lands alongside Phase 2/3's live-LLM run.

Spec: docs/specs/polish-fact-check/

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

Author: silversurfer562

CHANGELOG.md

@@ -15,6 +15,31 @@ changes land, not at tag time.
 
 ### Added
 
+- **Polish fact-check Phase 4 — tutorial code-fence static
+  check.** Polished tutorials (`docs/tutorials/*.md`) now have
+  their ```python code fences extracted, syntax-checked with
+  `ast.parse`, and type-checked with `mypy --strict` as a
+  subprocess per fence. Failures land in the same
+  `## Unresolved references` block as Phase 1 findings. The
+  static check only runs on the tutorial doc kind — other kinds
+  (how-to, reference, architecture) are skipped.
+  - New module: `src/attune_author/fact_check/tutorial_static_check.py`.
+  - `# attune-author: skip-mypy` as the first line of a fence
+    opts that fence out of the mypy pass and is stripped from
+    the published tutorial. Trailing directives are preserved
+    (only first-line is recognized).
+  - Subprocess robustness: missing `mypy`, 10-second timeout,
+    and unexpected exit codes all degrade silently. The check
+    never blocks the polish pipeline.
+  - Config: `check_tutorial_static` toggle on
+    `FactCheckConfig` (defaults `True`); routes only when the
+    polished path lives under `tutorials/`.
+  - 16 new tests under `tests/unit/fact_check/test_tutorial_static_check.py`.
+  - Phase 4.2 (sample *execution* — running the code, not just
+    type-checking it) is explicitly **out of scope** for this
+    PR. Static check ships now; execution-tier design is
+    deferred per the spec.
+
 - **Polish fact-check Phase 3 — faithfulness judge.** Wraps
   `attune_rag.eval.faithfulness.FaithfulnessJudge` as a
   post-polish step: scores each polished file's claims against

README.md

@@ -193,6 +193,43 @@ End-of-run telemetry (call count, skip count, total estimated
 USD) logs at INFO level after `attune-author regenerate`. Set
 `ATTUNE_AUTHOR_FAITHFULNESS=off` to disable for a single run.
 
+## Tutorial static check (Phase 4)
+
+Polished tutorials get an additional pass that targets their
+embedded code samples. Every ```python fence is extracted, parsed
+with `ast.parse` for syntax errors, and type-checked with `mypy
+--strict` as a subprocess. Findings land in the same
+`## Unresolved references` block as the Phase 1 fact-check
+findings.
+
+The check only runs on tutorials (`docs/tutorials/<feature>.md`).
+Other doc kinds (how-to, reference, architecture) may carry code
+samples, but the reader-follow-along expectation is highest for
+tutorials, so they're the highest-value target.
+
+For samples that intentionally use unresolved types
+(illustrative pseudocode, packages that don't exist yet), add a
+directive as the **first line** of the fence:
+
+````markdown
+```python
+# attune-author: skip-mypy
+some_function_we_havent_built_yet()
+```
+````
+
+The directive is stripped from the published tutorial before it
+reaches readers. Trailing directives (later lines) are
+intentionally preserved — only first-line directives opt the
+fence out.
+
+Subprocess robustness: a missing `mypy`, a 10-second timeout, or
+any unexpected exit code degrades silently. The check never
+blocks the polish pipeline.
+
+Phase 4.2 — actual *execution* of code samples — is explicitly
+out of scope for this release. Static analysis only.
+
 ## Polish cache
 
 `attune-author` caches LLM polish responses on disk so re-generating an

docs/specs/polish-fact-check/decisions.md

@@ -108,3 +108,44 @@ To be filled in during Phase 3 implementation:
     default and the calibration is scheduled to land alongside
     the live-LLM Phase 2 acceptance run so a single real-API
     cycle covers both phases' open work.
+- 2026-05-16 — Phase 4 shipped. New decisions captured during
+  implementation:
+  - **Scope limited to tutorials**: `is_tutorial_path` gates the
+    routing in `check_polished_file` so only `docs/tutorials/`
+    files invoke the static check. Other doc kinds may carry
+    code samples but the reader-follow-along expectation is
+    highest for tutorials.
+  - **Subprocess robustness over feature richness**: a missing
+    `mypy`, a TimeoutExpired, or any unexpected exit code returns
+    an empty findings list rather than raising. The static check
+    must never block the polish pipeline — matches the Phase 1
+    fact-check and Phase 3 judge contracts.
+  - **First-line skip directive only**: `# attune-author:
+    skip-mypy` is recognized as the first non-blank line of a
+    fence and ignored elsewhere. Trailing directives are
+    intentionally preserved so authors can document the policy
+    without changing it.
+  - **Strip directives at write time, not read time**: the
+    polish writer calls `strip_skip_directives_in_file` on the
+    final content before writing tutorials only. The directive
+    therefore appears in the source corpus (visible in PR
+    review) but not in the published tutorial (readers don't
+    see it).
+  - **Sub-table config deferred**: the spec called for
+    `[tool.attune-author.fact-check.tutorial_static]` with
+    `enabled`, `mypy_args`, `timeout_seconds`. Only the top-level
+    `check_tutorial_static` toggle landed for v1; the
+    `mypy_args`/`timeout_seconds` knobs ship when a real consumer
+    asks for them. The default 10s timeout + `--strict
+    --no-error-summary --no-color-output` works for the test
+    suite and the ops-dashboard fixture.
+  - **No execution (Phase 4.2 deferred)**: this PR ships static
+    analysis only — `ast.parse` + `mypy --strict`. Executing
+    LLM-generated code samples is a separate design question
+    (security + perf) tracked for a follow-up. Tracked in
+    tasks.md as task 4.13.
+  - **Real-fixture acceptance deferred**: tasks 4.8, 4.9, 4.11
+    require the ops-dashboard tutorial fixture + a real `mypy`
+    installation. Unit-level coverage with mocked subprocess +
+    syntax-error path is in place; the integration test lands
+    alongside Phase 2/3's live-LLM run.

docs/specs/polish-fact-check/tasks.md

@@ -136,26 +136,28 @@ plumbing.
 
 | # | Task | Layer | Status | Notes |
 |---|------|-------|--------|-------|
-| 4.1 | Add `tutorial_static_check.check(polished_path, project_root)` to `fact_check/` package | attune-author | todo | Operates only on `docs/tutorials/*.md` |
-| 4.2 | Code-fence extractor: pull all ```python fences with line numbers | attune-author | todo | Skip fences with `# attune-author: skip-mypy` first line |
-| 4.3 | `ast.parse` each fence; collect syntax errors as findings | attune-author | todo | Cheap pre-check before invoking mypy |
-| 4.4 | Run `mypy --strict --no-error-summary` per fence | attune-author | todo | Subprocess; timeout 10s; capture stderr |
-| 4.5 | Parse mypy output into findings | attune-author | todo | Map line numbers back to original fence position |
-| 4.6 | Strip `# attune-author: skip-mypy` directives before publication | attune-author | todo | Apply only to the file written; preserve in source if any |
-| 4.7 | Add `[tool.attune-author.fact-check.tutorial_static]` config | attune-author | todo | `enabled`, `mypy_args`, `timeout_seconds` |
-| 4.8 | Test: pre-fix `tutorials/ops-dashboard.md` flags `_readers` + `_models` imports | attune-author | todo | The headline acceptance gate |
-| 4.9 | Test: post-fix version produces zero errors | attune-author | todo | |
-| 4.10 | Test: `skip-mypy` directive is honored and stripped from output | attune-author | todo | |
-| 4.11 | Test: total static-check time per tutorial < 10s | attune-author | todo | Bench against the ops-dashboard tutorial |
-| 4.12 | Update CHANGELOG + README | attune-author | todo | Note Phase 4.2 (execution) explicitly deferred |
-| 4.13 | Add design.md follow-up section on Phase 4.2 execution tiers | attune-author | todo | Reference the security + perf walkthrough from spec discussion |
+| 4.1 | Add `tutorial_static_check.check(polished_path, project_root)` to `fact_check/` package | attune-author | **done** | `is_tutorial_path(...)` heuristic gates routing in `check_polished_file` so only `docs/tutorials/` files run the static check |
+| 4.2 | Code-fence extractor: pull all ```python fences with line numbers | attune-author | **done** | `_FENCE_PATTERN` regex; line numbers derived from `_line_of_offset` |
+| 4.3 | `ast.parse` each fence; collect syntax errors as findings | attune-author | **done** | SyntaxError caught with the `exc.lineno` mapped back to absolute file line |
+| 4.4 | Run `mypy --strict --no-error-summary` per fence | attune-author | **done** | `_run_mypy` via temp file + subprocess; 10s timeout; handles TimeoutExpired + FileNotFoundError + unexpected exit codes by returning `[]` |
+| 4.5 | Parse mypy output into findings | attune-author | **done** | `_parse_mypy_output` regex; line numbers rewritten to absolute file position via `base_line + mypy_line - 1` |
+| 4.6 | Strip `# attune-author: skip-mypy` directives before publication | attune-author | **done** | `strip_skip_directives_in_file` invoked in `apply_polish_results` for `tutorial` depth only; first-line directive only, trailing directives intentionally preserved |
+| 4.7 | Add `[tool.attune-author.fact-check.tutorial_static]` config | attune-author | **partial** | Top-level `check_tutorial_static` toggle on `FactCheckConfig` (defaults True). Sub-table for `mypy_args` / `timeout_seconds` deferred — current constants match the spec; expose only when a real consumer needs the knob. |
+| 4.8 | Test: pre-fix `tutorials/ops-dashboard.md` flags `_readers` + `_models` imports | attune-author | deferred | Requires the ops-dashboard regression fixture + real mypy run. Unit-level coverage via mocked mypy + syntax-error path is in place; the integration test lands alongside the live-LLM acceptance run. |
+| 4.9 | Test: post-fix version produces zero errors | attune-author | deferred | Same gate as 4.8 |
+| 4.10 | Test: `skip-mypy` directive is honored and stripped from output | attune-author | **done** | `test_check_skips_fence_with_directive` + `test_strip_skip_directive_first_line` |
+| 4.11 | Test: total static-check time per tutorial < 10s | attune-author | deferred | Enforced indirectly via the 10s mypy subprocess timeout; a real bench lands with 4.8 |
+| 4.12 | Update CHANGELOG + README | attune-author | **done** | CHANGELOG under Unreleased; README adds "Tutorial static check (Phase 4)" subsection. 4.2 execution explicitly noted as out of scope. |
+| 4.13 | Add design.md follow-up section on Phase 4.2 execution tiers | attune-author | deferred | Tracked separately; not gating Phase 4.1 ship |
 
 ### Phase 4 exit checklist
 
-- [ ] Tasks 4.1–4.13 done
-- [ ] Pre-fix fixture flagged correctly; post-fix clean
-- [ ] Per-tutorial check time < 10s
-- [ ] Spec status updated; full umbrella spec marked `complete`
+- [x] Tasks 4.1–4.7, 4.10, 4.12 done (16 new tests)
+- [x] Spec status updated
+- [ ] Tasks 4.8, 4.9, 4.11 (real-fixture mypy runs) — deferred to the
+      same live-LLM cycle that closes Phase 2/3's open items
+- [ ] Task 4.13 (Phase 4.2 execution-tier design follow-up) — tracked
+      separately; Phase 4.1 ships without it
 
 ---
 

src/attune_author/fact_check/__init__.py

@@ -12,7 +12,7 @@
 
 from pathlib import Path
 
-from . import cli_refs, md_links, numeric_refs, python_refs
+from . import cli_refs, md_links, numeric_refs, python_refs, tutorial_static_check
 from .config import load_config
 from .report import (
     CHECK_CLI_REFS,
@@ -26,6 +26,7 @@
     Severity,
     format_unresolved_block,
 )
+from .tutorial_static_check import CHECK_TUTORIAL_STATIC
 
 
 def check_polished_file(
@@ -68,6 +69,10 @@ def check_polished_file(
         report.extend(md_links.check(polished_path))
     if cfg.is_check_enabled(CHECK_NUMERIC_REFS, rel_path):
         report.extend(numeric_refs.check(polished_path, project_root))
+    if cfg.is_check_enabled(
+        CHECK_TUTORIAL_STATIC, rel_path
+    ) and tutorial_static_check.is_tutorial_path(polished_path):
+        report.extend(tutorial_static_check.check(polished_path))
 
     return report
 
@@ -93,6 +98,7 @@ def apply_soft_fail(polished_path: Path, report: FactCheckReport) -> bool:
     "CHECK_MD_LINKS",
     "CHECK_NUMERIC_REFS",
     "CHECK_PYTHON_REFS",
+    "CHECK_TUTORIAL_STATIC",
     "FactCheckConfig",
     "FactCheckError",
     "FactCheckReport",

src/attune_author/fact_check/config.py

@@ -48,6 +48,7 @@ def _bool(key: str, default: bool) -> bool:
         check_cli_refs=_bool("check_cli_refs", True),
         check_md_links=_bool("check_md_links", True),
         check_numeric_refs=_bool("check_numeric_refs", True),
+        check_tutorial_static=_bool("check_tutorial_static", True),
     )
     if isinstance(skip, dict):
         cfg.skip = {

src/attune_author/fact_check/report.py

@@ -18,6 +18,7 @@
 CHECK_CLI_REFS = "check_cli_refs"
 CHECK_MD_LINKS = "check_md_links"
 CHECK_NUMERIC_REFS = "check_numeric_refs"
+CHECK_TUTORIAL_STATIC = "check_tutorial_static"
 
 
 class FactCheckError(Exception):
@@ -82,6 +83,7 @@ class FactCheckConfig:
     check_cli_refs: bool = True
     check_md_links: bool = True
     check_numeric_refs: bool = True
+    check_tutorial_static: bool = True
     #: Mapping of file path (relative to project root, posix) to
     #: a list of check-id strings that should be skipped for
     #: that file.