Surface failed + zero-page sources and add citadel ingest --retry

[MarkusNeusinger]

Problem

Two ways a source can silently contribute nothing to the wiki were too easy to miss:

• a failed source (errored / timed-out / unreadable) is persisted in the failures catalog, but once the console scrolls it takes a deliberate citadel status to notice — and retrying meant collecting paths for --force by hand;
• a source that "succeeded" with zero page changes was worse: marked done in the manifest, never revisited, and recorded nowhere — indistinguishable from a healthy ingest even in status.

What changed

Visibility

• IngestReport.no_pages: a fresh source (plain ingest of a new file/repo — not a reconcile, not a delete cleanup, where an unchanged verdict is legitimate) that changes nothing now lands in a WARNING — ingested but produced NO wiki changes section of the run report, plus a permanent log.md line. The Errors section now also points at the retry lane.
• The live progress renders the no changes verdict in yellow instead of dim.
• citadel status marks every ingested source that no wiki page cites with NO PAGES (nothing cites this source) (computed via the existing store.citing_pages_map — the same verdict as sources/index.md's "Referenced by" column, best-effort so a broken wiki degrades to no markers instead of a failed status). --json carries it as "uncited". The table closes with a retry hint whenever anything is failed or uncited.

Easy retry

• New citadel ingest --retry: computes its own bounded set — every failed source still on disk (deliberate same-basename duplicate skips and page-keyed curate records excluded) plus every ingested-but-uncited source as a forced reconcile — prints it, then runs it. Refuses explicit paths and --force (exit 2, keeping --force's no-accidental-corpus-wide-run discipline intact: the set is bounded and explicit either way), and exits 0 with Nothing to retry on a healthy corpus.

Tests

tests/test_retry.py (14 new tests, all offline on the shared tmp_citadel/fake_agent/seed_page fixtures): the no_pages flag on fresh ingests and its absence on reconciles, the report/log/render surfaces, retry_candidates() bucketing (vanished/duplicate/curate/cited exclusions), the CLI refusals, the spy-verified forced run over the computed set, an end-to-end retry that clears the stuck source, and the status markers/hints in table + JSON.

Gates: pytest (1301 passed, 1 skipped), ruff check, ruff format --check, and the beverages-workspace citadel lint all green; citadel status on the committed beverages showcase shows no false NO PAGES markers. No rules-tree or prompt changes (the ingest diff is report bookkeeping only), so no verify-corpus run was needed. Docs updated (CLAUDE.md + regenerated copilot twin, CHANGELOG, maintenance + troubleshooting).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Cy9pETjieaS5z48Ku1tArB

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR improves ingest observability and recovery by surfacing (1) failed sources and (2) “successful” ingests that produced zero wiki entries, and by adding a one-command retry flow (citadel ingest --retry) to re-run the bounded “stuck” set.

Changes:

• Add IngestReport.no_pages + reporting/logging so fresh ingests that produce zero wiki changes are visible and actionable.
• Add citadel ingest --retry (CLI + ingest support) to retry failed sources on disk and force-reconcile ingested-but-uncited sources.
• Extend citadel status to mark ingested sources with no citing pages as NO PAGES (and expose "uncited" in --json), plus add a retry hint.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
citadel/ingest.py	Tracks and renders no_pages, persists log entries for zero-page sources, and adds retry_candidates() backing --retry.
citadel/cli.py	Adds --retry flag to ingest command, including usage constraints and computed retry-set execution.
citadel/status.py	Adds uncited marker to ingested rows via store.citing_pages_map and prints a one-line retry hint when relevant.
citadel/progress.py	Changes “no changes” verdict styling to yellow for higher visibility.
tests/test_retry.py	Adds offline tests covering no_pages, retry_candidates(), --retry CLI behavior, and status uncited markers.
docs/troubleshooting.md	Documents NO PAGES in citadel status and introduces citadel ingest --retry.
docs/maintenance.md	Documents the new NO PAGES/no-pages surfaces and the retry workflow.
CLAUDE.md	Updates contributor docs to include --retry and the enhanced status behavior.
CHANGELOG.md	Records the new visibility and retry features in the changelog.
.github/copilot-instructions.md	Regenerated docs reflecting the --retry and status updates.

Comments suppressed due to low confidence (1)

citadel/ingest.py:3240

• The log hint for zero-page sources suggests citadel ingest --force without indicating it requires a path. Since --force is explicitly refused without paths, this log entry is misleading for users following it later.

            store.append_log(
                f"ingested {key} but the session produced no wiki changes (0 pages); "
                "retry with `citadel ingest --retry` or `citadel ingest --force`"
            )

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

citadel/status.py:304

• The new NO PAGES/uncited marker currently applies to every manifest entry, including sources that are also present in the failures catalog (notably UNREADABLE sources, which ingest records in both places for stat caching). That can produce misleading NO PAGES labels for sources that were never ingested successfully; it also makes the status output inconsistent with the intent that uncited means “paid session succeeded but contributed zero entries”. Consider suppressing the marker for any key that has a non-duplicate, non-curate failures record.

        except Exception:  # noqa: BLE001 - status is read-only and must degrade, never raise
            refs = None
        if refs is not None:
            for row in report.ingested:
                row.uncited = not refs.get(row.key)

citadel/ingest.py:2582

• retry_candidates() can classify a source as both failed and uncited (e.g., UNREADABLE sources are recorded in the failures catalog and are marked done in the manifest). That breaks the docstring claim that the lists are disjoint, and it can lead to duplicated paths in citadel ingest --retry (same key appearing in both buckets). Exclude keys already selected into the failed bucket from the uncited bucket.

            uncited = [
                key for key in sorted(manifest_dict) if not refs.get(key) and config.source_path_for_key(key).exists()
            ]