feat(polish): faithfulness judge integration (Phase 3 of polish-fact-check)

[silversurfer562]

Summary

Stacked on #35 (Phase 2) — please review and merge that first.

• Ship Phase 3 of the polish-fact-check spec: wrap attune_rag.eval.faithfulness.FaithfulnessJudge as a post-polish step. Every polished file gets scored against its source files; below-threshold scores append a ## Faithfulness review block listing the unsupported claims and the judge's reasoning.
• New src/attune_author/faithfulness/ package: judge wrapper (sync-bridged via asyncio.run), FaithfulnessConfig dataclass, char-based cost estimator that gates the call before paying for it, and a format_review_block / apply_review_block pair matching the Phase 1 soft-fail shape.
• Wired into generator.apply_polish_results after the Phase 1 fact-check pass. Per-process cost telemetry counters reset at run_maintenance start and summary-logged at end.
• Opt-in by default (enabled = false in config). The judge is best-effort: missing attune-rag[claude], missing ANTHROPIC_API_KEY, over-budget cost estimates, and transient API failures all degrade silently rather than blocking the polish. CI lanes can opt into strict mode with block_polish_on_unavailable = true.

Motivation

Phase 1 catches mistakes after generation (regex/AST matching for known-bad shapes — invented imports, unknown CLI flags, broken links, wrong counts). Phase 2 prevents them during generation (ground-truth context injection). Phase 3 covers the middle ground: an LLM-as-judge that catches shapes Phase 1 can't pattern-match (e.g. the missing-security-callout for 0.0.0.0 from attune-ai PR #351's fixture) and that Phase 2 can't fully prevent.

Configuration

[tool.attune-author.fact-check.faithfulness]
enabled = true                # opt-in; defaults to false
threshold = 0.95              # below this triggers a review block
budget_per_file_usd = 0.10    # skip if estimated cost exceeds cap
model = "claude-sonnet-4-6"   # haiku is ~1/3 the cost
block_polish_on_unavailable = false  # set true in strict CI lanes

End-of-run telemetry log:

INFO Faithfulness judge: 11 call(s), 2 skipped, estimated cost $0.0537

Test plan

• Unit tests: 30 new tests under tests/unit/faithfulness/ covering — the budget gate, every skip path (disabled, missing file, no sources, over-budget, missing extra, missing key, transient failure), the happy path with mocked FaithfulnessJudge, the below-threshold review-block append, env-var override (ATTUNE_AUTHOR_FAITHFULNESS=off), telemetry counters + reset, and unexpected-exception swallowing
• Full attune-author suite: 926 passed, 37 pre-existing skips
• ruff check clean across all touched files
• Live-LLM acceptance (gated): threshold calibration against the ops-dashboard pre-fix / post-fix fixture (tasks 3.3 + 3.4) — scheduled to land alongside Phase 2's live-LLM acceptance run so a single real-API cycle covers both phases' open items. The placeholder default threshold=0.95 is documented as pre-calibration in decisions.md.

Notes for review

• uv.lock is intentionally excluded — pre-existing drift (lockfile recorded attune-author 0.6.1), separate cleanup PR.
• Why opt-in: the judge makes real Anthropic API calls. We shouldn't bill users for it silently on first run after install. Phase 1 (no API calls) defaults on; Phase 3 (real API calls) defaults off. Easy to flip per-project via pyproject.toml.
• Why asyncio.run: the existing polish pipeline is synchronous and FaithfulnessJudge.score is async. The bridge is at one call site; expanding to native-async polish is a future change if drift surfaces.
• Why char-based cost estimate vs real tokenizer: the budget gate cares about ~$0.10 precision; chars/4 token estimate is accurate to ~20% which is well inside that gate. Documented in decisions.md.

🤖 Generated with Claude Code