Surface failed + zero-page sources and add citadel ingest --retry

.github/copilot-instructions.md

@@ -34,7 +34,11 @@ session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the liv
 `--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`),
 `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source,
 `--force <paths>` deliberately re-reads already-ingested sources as a reconcile — it requires
-explicit paths and is refused without them), `refresh [--limit N] [--min-age-days D] [--dry-run] [--jobs N]`
+explicit paths and is refused without them, `--retry` re-runs everything STUCK without naming
+paths: every failed source still on disk plus every ingested source no wiki page cites — the
+zero-entry sources the run report flags as `no_pages` and `status` marks `NO PAGES` — as forced
+reconciles; it refuses paths/`--force` and exits 0 when nothing is stuck),
+`refresh [--limit N] [--min-age-days D] [--dry-run] [--jobs N]`
 (the THIRD lifecycle: re-verify the least-recently-checked sources — ordered by the manifest's
 `ingested_at` stamp, oldest/stampless first — through forced reconcile sessions on an explicit
 per-run budget of N sources; the sustainable alternative to regenerating the wiki after a model
@@ -342,13 +346,15 @@ is scoped to that signature (flags actually passed + an auth-shaped message), so
 problem still fails instead of looping.
 
 **Status is the read-only corpus view** (`status.py`, `citadel status`): the manifest + failures
-catalog + one stat-only walk (never re-hashes) rendered as a per-source state table — ingested
+catalog + one stat-only walk (never re-hashes) + one wiki traversal rendered as a per-source state
+table — ingested
 (model + rules_version, `(stale)` when it predates the current rulebook, `checked YYYY-MM-DD` from
 the `ingested_at` stamp, the last session's cost when recorded, with copilot's AI credits shown
 beside the dollars they converted into — with `Recorded LLM cost` / `Recorded AI credits` corpus
-totals above the table), failed (reason, attempts),
+totals above the table, and a `NO PAGES (nothing cites this source)` marker on every ingested
+source the wiki does not cite — a paid session that produced zero entries), failed (reason, attempts),
 skipped-duplicate, ignored (pattern), oversized (over `CITADEL_MAX_SOURCE_BYTES`, with the size),
-pending.
+pending — closing with a `citadel ingest --retry` hint whenever anything is failed or uncited.
 
 **Other modules:** `okf.py` is the OKF format core (parse/dump, type→folder routing, link math, and
 the non-negotiable `safe_join` path guard — reuse it for any wiki-relative path). `grammar.py` is

CHANGELOG.md

@@ -24,6 +24,18 @@ All notable changes to this project are documented here. The format is based on
 
 ### Added
 
+- **Zero-page and failed sources are visible — and retryable with one command.** A brand-new
+  source whose session "succeeds" with zero page changes used to be marked done and never looked
+  at again, indistinguishable from a healthy ingest. Now: the run report carries a WARNING section
+  (`IngestReport.no_pages`) plus a `log.md` line for each such source, the live progress renders
+  its `no changes` verdict in yellow, and `citadel status` marks every ingested source that NO
+  wiki page cites with `NO PAGES (nothing cites this source)` (also `"uncited"` in `--json`),
+  ending with a retry hint whenever anything is stuck. The new **`citadel ingest --retry`**
+  re-runs the whole stuck set without naming paths: every failed source still on disk (errored /
+  timed-out / unreadable — deliberate `duplicate` skips excluded) plus every ingested-but-uncited
+  source as a forced reconcile. It prints the computed set first, refuses `--force`/explicit
+  paths, and exits 0 cleanly when there is nothing to retry. The failed-run report also points at
+  the retry lane from its Errors section.
 - **`citadel doctor` warns when `CITADEL_INGEST_MODEL` shadows `COPILOT_MODEL`.** The copilot
   backend also reads its own `COPILOT_MODEL` env var, and the `--model` flag citadel passes
   overrides it. A user who points copilot at a BYOK provider (`COPILOT_PROVIDER_BASE_URL`, a

CLAUDE.md

@@ -31,7 +31,11 @@ session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the liv
 `--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`),
 `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source,
 `--force <paths>` deliberately re-reads already-ingested sources as a reconcile — it requires
-explicit paths and is refused without them), `refresh [--limit N] [--min-age-days D] [--dry-run] [--jobs N]`
+explicit paths and is refused without them, `--retry` re-runs everything STUCK without naming
+paths: every failed source still on disk plus every ingested source no wiki page cites — the
+zero-entry sources the run report flags as `no_pages` and `status` marks `NO PAGES` — as forced
+reconciles; it refuses paths/`--force` and exits 0 when nothing is stuck),
+`refresh [--limit N] [--min-age-days D] [--dry-run] [--jobs N]`
 (the THIRD lifecycle: re-verify the least-recently-checked sources — ordered by the manifest's
 `ingested_at` stamp, oldest/stampless first — through forced reconcile sessions on an explicit
 per-run budget of N sources; the sustainable alternative to regenerating the wiki after a model
@@ -339,13 +343,15 @@ is scoped to that signature (flags actually passed + an auth-shaped message), so
 problem still fails instead of looping.
 
 **Status is the read-only corpus view** (`status.py`, `citadel status`): the manifest + failures
-catalog + one stat-only walk (never re-hashes) rendered as a per-source state table — ingested
+catalog + one stat-only walk (never re-hashes) + one wiki traversal rendered as a per-source state
+table — ingested
 (model + rules_version, `(stale)` when it predates the current rulebook, `checked YYYY-MM-DD` from
 the `ingested_at` stamp, the last session's cost when recorded, with copilot's AI credits shown
 beside the dollars they converted into — with `Recorded LLM cost` / `Recorded AI credits` corpus
-totals above the table), failed (reason, attempts),
+totals above the table, and a `NO PAGES (nothing cites this source)` marker on every ingested
+source the wiki does not cite — a paid session that produced zero entries), failed (reason, attempts),
 skipped-duplicate, ignored (pattern), oversized (over `CITADEL_MAX_SOURCE_BYTES`, with the size),
-pending.
+pending — closing with a `citadel ingest --retry` hint whenever anything is failed or uncited.
 
 **Other modules:** `okf.py` is the OKF format core (parse/dump, type→folder routing, link math, and
 the non-negotiable `safe_join` path guard — reuse it for any wiki-relative path). `grammar.py` is

citadel/cli.py

@@ -116,6 +116,15 @@ def build_parser() -> argparse.ArgumentParser:
         "without them, so a whole-corpus re-read (one agent session per source) can never "
         "happen by accident.",
     )
+    p_ingest.add_argument(
+        "--retry",
+        action="store_true",
+        help="Retry everything stuck, without naming paths: re-run every FAILED source "
+        "(errored/timed-out/unreadable, from the failures catalog) that is still on disk, and "
+        "force-reconcile every ingested source that NO wiki page cites (it produced 0 entries — "
+        "the `NO PAGES` marker in `citadel status`). Prints the retry set first; refuses "
+        "explicit paths and --force (use those for a hand-picked re-read).",
+    )
     p_ingest.add_argument(
         "--jobs",
         "-j",
@@ -413,6 +422,12 @@ def cmd_ingest(args: argparse.Namespace) -> int:
     agent session per source, so forcing the ENTIRE corpus must never happen by accident — the
     flag alone is refused with exit 2, before ``ingest.ingest`` is ever called.
 
+    ``--retry`` is the no-typing complement: it computes its own bounded set —
+    ``ingest.retry_candidates()``, the failed sources still on disk plus the ingested ones no
+    wiki page cites — prints it, and runs those as a FORCED re-read. It therefore refuses
+    explicit paths and ``--force`` (exit 2): the point of the flag is that citadel picks the
+    set, and combining the two would blur which one wins. Nothing to retry is a clean exit 0.
+
     ``--jobs N`` is a usage error below 1 (exit 2, like ``--force`` without paths) rather than an
     exception out of the API layer; omitted, the run takes ``CITADEL_JOBS`` (default 1, serial)."""
     from . import config, ingest
@@ -421,6 +436,15 @@ def cmd_ingest(args: argparse.Namespace) -> int:
         print(f"error: --jobs must be at least 1 (got {args.jobs}); 1 means the serial default.", file=sys.stderr)
         return 2
 
+    if args.retry and (args.paths or args.force):
+        print(
+            "error: --retry computes its own retry set (every failed source plus every ingested "
+            "source no wiki page cites) — combine no paths and no --force with it; use "
+            "`citadel ingest --force <paths>` for a hand-picked re-read instead.",
+            file=sys.stderr,
+        )
+        return 2
+
     if args.force and not args.paths:
         print(
             "error: --force requires explicit paths (a forced re-read runs one agent session per "
@@ -429,6 +453,24 @@ def cmd_ingest(args: argparse.Namespace) -> int:
         )
         return 2
 
+    paths = args.paths or None
+    force = args.force
+    if args.retry:
+        failed, uncited = ingest.retry_candidates()
+        if not failed and not uncited:
+            print("Nothing to retry: no failed sources and every ingested source is cited by at least one page.")
+            return 0
+        if failed:
+            print(f"Retrying {len(failed)} failed source(s):")
+            for key in failed:
+                print(f"  - {config.display_key(key)}")
+        if uncited:
+            print(f"Force-reconciling {len(uncited)} ingested source(s) that no wiki page cites:")
+            for key in uncited:
+                print(f"  - {config.display_key(key)}")
+        paths = [str(config.source_path_for_key(key)) for key in failed + uncited]
+        force = True
+
     if args.verbose:
         config.LLM_VERBOSE = True
     # `is not None` (not truthiness) so an explicit `--log-dir ""` is honored as "disable logging"
@@ -444,9 +486,7 @@ def cmd_ingest(args: argparse.Namespace) -> int:
         # CITADEL_LLM_VERBOSE — not just the --verbose flag — also drops the spinner that would
         # otherwise clobber the streamed transcript.
         progress = ConsoleProgress(spinner=not config.LLM_VERBOSE)
-    report = ingest.ingest(
-        args.paths or None, progress=progress, full_rescan=args.full_rescan, force=args.force, jobs=args.jobs
-    )
+    report = ingest.ingest(paths, progress=progress, full_rescan=args.full_rescan, force=force, jobs=args.jobs)
     print(report.render())
     # Non-zero on a per-source error OR a structural problem left behind (a broken
     # cross-link the agent introduced) — so ingest gates the wiki's integrity in CI.

citadel/ingest.py

@@ -125,6 +125,12 @@ class IngestReport:
     # segment 1 ("raw/book.txt (segments 1-3 of 7 restored)") — see citadel/resume.py. Recorded
     # whether or not the resumed source then succeeded: the earlier work was reused either way.
     resumed: list[str] = field(default_factory=list)
+    # rel-keys of FRESH sources (a plain ingest — not a reconcile, not a delete cleanup) whose
+    # session succeeded but changed NOTHING: no page created, updated, or deleted. Suspicious by
+    # construction — a brand-new source that contributes zero facts is usually a session that
+    # under-delivered, yet it is marked done and never revisited on its own. Surfaced as a WARNING
+    # so it is easy to spot and retry (`citadel ingest --retry`, or `--force <path>`).
+    no_pages: list[str] = field(default_factory=list)
     # The wiki-history note from wikigit.autocommit ("wiki git: committed <sha>", or a warning
     # naming what was skipped and why) — empty when the history layer had nothing to say.
     wiki_git: str = ""
@@ -198,12 +204,23 @@ def render(self) -> str:
         if self.skipped:
             lines.append("Skipped (already ingested):")
             lines.extend(f"  - {p}" for p in self.skipped)
+        if self.no_pages:
+            lines.append("WARNING — ingested but produced NO wiki changes (no page created, updated, or deleted):")
+            lines.extend(f"  - {p}" for p in self.no_pages)
+            lines.append(
+                "  These sources are marked done and will not be revisited automatically. "
+                "Retry them with `citadel ingest --retry` (or `citadel ingest --force <path>`)."
+            )
         if self.broken_links:
             lines.append("WARNING — broken cross-links (run `citadel lint`):")
             lines.extend(f"  - {src} -> {tgt}" for src, tgt in self.broken_links)
         if self.errors:
             lines.append("Errors:")
             lines.extend(f"  - {e}" for e in self.errors)
+            lines.append(
+                "  Failed sources stay in the failures catalog (`citadel status` lists them) and are "
+                "retried on the next run — or right away with `citadel ingest --retry`."
+            )
         if self.wiki_git:
             lines.append(self.wiki_git)
         return "\n".join(lines)
@@ -2030,6 +2047,11 @@ class _SourceJob:
       asserts no reference survived and may legitimately empty the wiki).
     - ``sha_stat``: the (sha256, stat) discovery already took for the source, threaded into the
       failures catalog so an unchanged stuck source joins the stat quick check.
+    - ``warn_no_pages``: True for a FRESH source (a plain ingest of a new key, file or repo) —
+      a successful session that then changed NOTHING lands on ``report.no_pages`` as a warning.
+      Deliberately False for reconciles (an unchanged verdict is a legitimate outcome of
+      re-reading a source, and ``citadel refresh`` would otherwise flag its whole slice) and for
+      delete cleanups (empty means nothing cited the source — the expected case).
     """
 
     key: str
@@ -2039,6 +2061,7 @@ class _SourceJob:
     extra_check: Callable[[], list[str]] | None = None
     allow_emptying: bool = False
     sha_stat: tuple[str | None, os.stat_result | None] = (None, None)
+    warn_no_pages: bool = False
 
 
 @dataclass
@@ -2160,6 +2183,10 @@ def _record_source_run(run: _JobRun, emit, report: IngestReport, failures_dict,
     report.pages_updated.extend(outcome.updated)
     report.pages_written.extend(outcome.created + outcome.updated)
     report.pages_deleted.extend(outcome.deleted)
+    if job.warn_no_pages and not (outcome.created or outcome.updated or outcome.deleted):
+        # A fresh source folded in with zero page changes: marked done below, so without this
+        # warning it would silently never contribute anything (see IngestReport.no_pages).
+        report.no_pages.append(job.key)
     # The manifest stamp covers every session whose work this promote landed — this run's plus
     # whatever an earlier run already paid for the segments a checkpoint restored — while
     # ``report.usage`` above stays strictly this run's spend, so nothing is double-counted
@@ -2513,6 +2540,48 @@ def _pending_session(
     return llm.run_ingest_session(rel_key, kind=kind)
 
 
+def retry_candidates() -> tuple[list[str], list[str]]:
+    """The source keys ``citadel ingest --retry`` re-runs, as ``(failed, uncited)``.
+
+    - ``failed``: every source in the failures catalog that is still on disk — errored / timed-out
+      / unreadable records alike (an unchanged unreadable file is re-evaluated for free, and a
+      fixed one — hydrated placeholder, re-exported document — now ingests). Deliberate skips
+      stay skipped: a same-basename ``duplicate`` record is a decision, not a failure, and a
+      ``curate`` record is keyed by a PAGE, not a source (``citadel curate --retry`` owns those).
+    - ``uncited``: every INGESTED source that no wiki page cites (the same
+      ``store.citing_pages_map`` verdict as the ``Referenced by`` column of
+      ``wiki/sources/index.md``, and ``citadel status``'s ``NO PAGES`` marker) — a session was
+      paid for, the source is marked done, and yet nothing in the wiki carries its facts. These
+      are re-run as FORCED reconciles.
+
+    Both lists are sorted and disjoint (a key can only be in one catalog); vanished files are
+    excluded — a retry cannot read what is not there (a vanished INGESTED source is the deletion
+    sweep's job, not a retry's). Read-only: computing candidates changes nothing. Best-effort
+    like ``status``'s marker: a wiki that cannot be traversed degrades to an empty ``uncited``
+    list instead of taking the recovery command down — the failed sources are still retried,
+    which is exactly the situation ``--retry`` exists for."""
+    failed: list[str] = []
+    for key, entry in sorted(failures.load().items()):
+        if not isinstance(entry, dict):
+            continue
+        if entry.get("reason") in (failures.DUPLICATE, failures.CURATE):
+            continue
+        if config.source_path_for_key(key).exists():
+            failed.append(key)
+    manifest_dict = manifest.load()
+    uncited: list[str] = []
+    if manifest_dict:
+        try:
+            refs = store.citing_pages_map(list(manifest_dict))
+        except Exception:  # noqa: BLE001 - recovery must degrade, never crash on a broken wiki
+            refs = None
+        if refs is not None:
+            uncited = [
+                key for key in sorted(manifest_dict) if not refs.get(key) and config.source_path_for_key(key).exists()
+            ]
+    return failed, uncited
+
+
 @pagecache.bypass
 def ingest(
     paths: list[str] | None = None,
@@ -3025,6 +3094,9 @@ def done(usage: llm.SessionUsage | None) -> None:
             on_success=done,
             prepare_error="prepare audio transcript" if is_audio else "write source text",
             sha_stat=sha_stat,
+            # A brand-new key (not a reconcile of changed/forced bytes) that produces zero page
+            # changes is worth a warning — see _SourceJob.warn_no_pages.
+            warn_no_pages=rel_key not in changed_keys,
         )
 
     # Repo sources: each git repository under raw/ is folded in by ONE session reading a
@@ -3065,7 +3137,13 @@ def done(usage: llm.SessionUsage | None) -> None:
             manifest.save(manifest_dict)
             report.processed.append(rjob.key)
 
-        return _SourceJob(key=rjob.key, build_sessions=build, on_success=done, prepare_error="build digest")
+        return _SourceJob(
+            key=rjob.key,
+            build_sessions=build,
+            on_success=done,
+            prepare_error="build digest",
+            warn_no_pages=rjob.kind == "repo",  # a fresh repo digest yielding nothing is suspicious
+        )
 
     # Deleted sources: a tracked source vanished from disk (full run only). If any page still
     # cites it, run a `kind="delete"` cleanup session that strips that provenance, gated by a
@@ -3162,6 +3240,12 @@ def done(_usage: llm.SessionUsage | None) -> None:
                 f"{len(report.pages_updated)} updated, {len(report.pages_deleted)} deleted "
                 f"(model: {model})"
             )
+        for key in report.no_pages:
+            store.append_log(
+                f"ingested {key} but the session produced no wiki changes (no page created, "
+                f"updated, or deleted); retry with `citadel ingest --retry` or "
+                f"`citadel ingest --force {key}`"
+            )
         for old_key, new_key in report.moved:
             store.append_log(
                 f"reorganized {new_key}: same content already ingested as {old_key}; "