Merge pull request #139 from MarkusNeusinger/claude/model-ingest-cache-issue-qoc560

ingest: add --reingest, the delete-then-fresh-import escape hatch

Author: MarkusNeusinger

.github/copilot-instructions.md

@@ -34,7 +34,12 @@ session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the liv
 `--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`),
 `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source,
 `--force <paths>` deliberately re-reads already-ingested sources as a reconcile — it requires
-explicit paths and is refused without them, `--retry` re-runs everything STUCK without naming
+explicit paths and is refused without them, `--reingest <paths>` goes one step further: each named
+tracked source is re-imported FRESH — a `kind="delete"` cleanup session strips its previous facts
+(manifest entry dropped), then the same run ingests it as a brand-new source under the current
+model + rules — the full re-think reconcile's keep-the-existing-treatment rule deliberately avoids
+(requires explicit paths like `--force`; the two flags and `--retry` refuse to combine; a failed
+cleanup blocks that source's fresh session), `--retry` re-runs everything STUCK without naming
 paths: every failed source still on disk plus every ingested source no wiki page cites — the
 zero-entry sources the run report flags as `no_pages` and `status` marks `NO PAGES` — as forced
 reconciles; it refuses paths/`--force` and exits 0 when nothing is stuck),
@@ -211,7 +216,13 @@ addressing them through `ingest`. The flow per source:
   `CITADEL_MAX_SOURCE_CHARS` is folded in over several passes (all against one staging copy — see
   the promote bullet below). `ingest --force <paths>` bypasses the sha short-circuit: the named
   sources land in pending as reconciles (a repo re-digests in full), and the manifest is re-stamped
-  with the current model + rules version.
+  with the current model + rules version. `ingest --reingest <paths>` rides force's partitioning
+  but re-imports fresh instead of reconciling: a delete-cleanup job per named tracked source runs
+  in the deletions group (always first), dropping the manifest key so the pending session plans the
+  plain ingest/image/audio/pdf kind of a new key (`repo` for a tracked repo — safe only because the
+  cleanup already stripped the pages a first-time brief would duplicate); a failed cleanup refuses
+  that source's fresh session, and the cleanup never prunes the transcript/pdftext caches (the
+  bytes are unchanged and about to be re-read).
 - **Discovery is incremental and deletion-safe**: one iterative `os.scandir`
   walk over every `CITADEL_RAW_DIRS` root keeps each file's stat; the **manifest doubles as the
   scan cache** (an entry's `size`/`mtime_ns`/`ctime_ns`/`hashed_at_ns` are a skip-hint — sha256

CHANGELOG.md

@@ -6,6 +6,33 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+### Added
+
+- **`citadel ingest --reingest <paths>` — re-import an already-ingested source from scratch.**
+  `--force` runs a reconcile, which deliberately keeps the source's existing treatment (genre,
+  page structure) and only verifies/updates its facts — so a source first ingested under an old
+  rulebook or a weak model never gets re-thought, and "all ok, nothing new" is the correct
+  reconcile verdict even when a fresh read would organize it completely differently. `--reingest`
+  is the escape hatch: for each named tracked source it first runs a `kind="delete"` cleanup
+  session that strips the source's previous facts from the wiki (dropping its manifest entry),
+  then — in the same run, deletions always run first — ingests it as a brand-new source under the
+  current model, rules, and wiki state (fresh cross-links included). This replaces the manual
+  move-the-file-out / full ingest / move-it-back / ingest dance. A failed cleanup refuses the
+  fresh session for that source (nothing is written on top of the old facts), tracked repos take
+  the first-time `repo` brief over a full digest, and the flag requires explicit paths like
+  `--force` (exit 2 without them; `--force` and `--retry` refuse to combine with it). The
+  `tasks/delete.md` brief now covers the still-on-disk cleanup case.
+
+### Changed
+
+- **Forced/refresh reconciles now re-read with fresh eyes, not just re-verify.** The
+  `tasks/reconcile.md` brief (driving `ingest --force` and every `citadel refresh` session) now
+  explicitly instructs the agent that a re-read is more than verification: mine the unchanged
+  source for wiki-worthy facts the first ingest missed, cross-link against today's wiki (pages
+  that did not exist back then may now be the right home or neighbor), and strip claims the
+  source never actually supported — while still keeping the existing genre treatment (structure
+  churn stays `--reingest`'s job).
+
 ## [0.6.0] - 2026-07-30
 
 ### Changed (breaking)

CLAUDE.md

@@ -31,7 +31,12 @@ session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the liv
 `--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`),
 `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source,
 `--force <paths>` deliberately re-reads already-ingested sources as a reconcile — it requires
-explicit paths and is refused without them, `--retry` re-runs everything STUCK without naming
+explicit paths and is refused without them, `--reingest <paths>` goes one step further: each named
+tracked source is re-imported FRESH — a `kind="delete"` cleanup session strips its previous facts
+(manifest entry dropped), then the same run ingests it as a brand-new source under the current
+model + rules — the full re-think reconcile's keep-the-existing-treatment rule deliberately avoids
+(requires explicit paths like `--force`; the two flags and `--retry` refuse to combine; a failed
+cleanup blocks that source's fresh session), `--retry` re-runs everything STUCK without naming
 paths: every failed source still on disk plus every ingested source no wiki page cites — the
 zero-entry sources the run report flags as `no_pages` and `status` marks `NO PAGES` — as forced
 reconciles; it refuses paths/`--force` and exits 0 when nothing is stuck),
@@ -208,7 +213,13 @@ addressing them through `ingest`. The flow per source:
   `CITADEL_MAX_SOURCE_CHARS` is folded in over several passes (all against one staging copy — see
   the promote bullet below). `ingest --force <paths>` bypasses the sha short-circuit: the named
   sources land in pending as reconciles (a repo re-digests in full), and the manifest is re-stamped
-  with the current model + rules version.
+  with the current model + rules version. `ingest --reingest <paths>` rides force's partitioning
+  but re-imports fresh instead of reconciling: a delete-cleanup job per named tracked source runs
+  in the deletions group (always first), dropping the manifest key so the pending session plans the
+  plain ingest/image/audio/pdf kind of a new key (`repo` for a tracked repo — safe only because the
+  cleanup already stripped the pages a first-time brief would duplicate); a failed cleanup refuses
+  that source's fresh session, and the cleanup never prunes the transcript/pdftext caches (the
+  bytes are unchanged and about to be re-read).
 - **Discovery is incremental and deletion-safe**: one iterative `os.scandir`
   walk over every `CITADEL_RAW_DIRS` root keeps each file's stat; the **manifest doubles as the
   scan cache** (an entry's `size`/`mtime_ns`/`ctime_ns`/`hashed_at_ns` are a skip-hint — sha256

citadel/cli.py

@@ -116,6 +116,15 @@ def build_parser() -> argparse.ArgumentParser:
         "without them, so a whole-corpus re-read (one agent session per source) can never "
         "happen by accident.",
     )
+    p_ingest.add_argument(
+        "--reingest",
+        action="store_true",
+        help="Re-import the given already-ingested sources FRESH: first a delete cleanup session "
+        "strips each source's previous facts from the wiki (its manifest entry is dropped), then "
+        "the same run ingests it as a brand-new source under the current model + rules — the "
+        "full re-think --force's reconcile deliberately avoids (reconcile keeps the existing "
+        "treatment). Costs two sessions per source. Requires explicit paths, like --force.",
+    )
     p_ingest.add_argument(
         "--retry",
         action="store_true",
@@ -422,11 +431,17 @@ def cmd_ingest(args: argparse.Namespace) -> int:
     agent session per source, so forcing the ENTIRE corpus must never happen by accident — the
     flag alone is refused with exit 2, before ``ingest.ingest`` is ever called.
 
+    ``--reingest`` is force's bigger sibling — strip the named sources' previous facts (a delete
+    cleanup session each), then re-import them fresh in the same run. Same explicit-paths rule as
+    ``--force`` (exit 2 without them), and the two flags refuse each other: reconcile-in-place
+    and strip-and-re-import are different intents, and combining them would blur which one wins.
+
     ``--retry`` is the no-typing complement: it computes its own bounded set —
     ``ingest.retry_candidates()``, the failed sources still on disk plus the ingested ones no
     wiki page cites — prints it, and runs those as a FORCED re-read. It therefore refuses
-    explicit paths and ``--force`` (exit 2): the point of the flag is that citadel picks the
-    set, and combining the two would blur which one wins. Nothing to retry is a clean exit 0.
+    explicit paths, ``--force`` and ``--reingest`` (exit 2): the point of the flag is that
+    citadel picks the set, and combining them would blur which one wins. Nothing to retry is a
+    clean exit 0.
 
     ``--jobs N`` is a usage error below 1 (exit 2, like ``--force`` without paths) rather than an
     exception out of the API layer; omitted, the run takes ``CITADEL_JOBS`` (default 1, serial)."""
@@ -436,11 +451,21 @@ def cmd_ingest(args: argparse.Namespace) -> int:
         print(f"error: --jobs must be at least 1 (got {args.jobs}); 1 means the serial default.", file=sys.stderr)
         return 2
 
-    if args.retry and (args.paths or args.force):
+    if args.retry and (args.paths or args.force or args.reingest):
         print(
             "error: --retry computes its own retry set (every failed source plus every ingested "
-            "source no wiki page cites) — combine no paths and no --force with it; use "
-            "`citadel ingest --force <paths>` for a hand-picked re-read instead.",
+            "source no wiki page cites) and cannot be combined with explicit paths, --force, or "
+            "--reingest; use `citadel ingest --force <paths>` (or `--reingest <paths>`) for a "
+            "hand-picked re-read instead.",
+            file=sys.stderr,
+        )
+        return 2
+
+    if args.force and args.reingest:
+        print(
+            "error: --force and --reingest are different intents — --force reconciles the "
+            "source's facts in place (keeping its existing treatment), --reingest strips them "
+            "and re-imports the source fresh. Pick one.",
             file=sys.stderr,
         )
         return 2
@@ -453,6 +478,15 @@ def cmd_ingest(args: argparse.Namespace) -> int:
         )
         return 2
 
+    if args.reingest and not args.paths:
+        print(
+            "error: --reingest requires explicit paths (each source runs a delete cleanup "
+            "session plus a fresh ingest session; name the files or directories to re-import, "
+            "e.g. `citadel ingest --reingest raw/notes.md`).",
+            file=sys.stderr,
+        )
+        return 2
+
     paths = args.paths or None
     force = args.force
     if args.retry:
@@ -486,7 +520,9 @@ def cmd_ingest(args: argparse.Namespace) -> int:
         # CITADEL_LLM_VERBOSE — not just the --verbose flag — also drops the spinner that would
         # otherwise clobber the streamed transcript.
         progress = ConsoleProgress(spinner=not config.LLM_VERBOSE)
-    report = ingest.ingest(paths, progress=progress, full_rescan=args.full_rescan, force=force, jobs=args.jobs)
+    report = ingest.ingest(
+        paths, progress=progress, full_rescan=args.full_rescan, force=force, jobs=args.jobs, reingest=args.reingest
+    )
     print(report.render())
     # Non-zero on a per-source error OR a structural problem left behind (a broken
     # cross-link the agent introduced) — so ingest gates the wiki's integrity in CI.