From 7047ac0412df8036707da53ed87cc1e09c661b39 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 10:20:22 +0000 Subject: [PATCH 1/4] ingest: add --reingest, the delete-then-fresh-import escape hatch --force runs a reconcile, which deliberately keeps a source's existing treatment and only re-verifies its facts - so a source first ingested under an old rulebook or a weak model is never re-thought, and "all ok, nothing new" is the correct reconcile verdict even when a fresh read would organize it completely differently (e.g. into a Registry page). `citadel ingest --reingest ` re-imports the named tracked sources from scratch: a kind="delete" cleanup session strips each source's previous facts (its manifest key is dropped), then the same run - the deletion group always runs first - ingests it as a brand-new source under the current model, rules, and wiki state. A failed cleanup refuses that source's fresh session (nothing is written on top of the old facts), tracked repos take the first-time repo brief over a full digest (safe because the cleanup already removed the pages a first-time brief would duplicate), the transcript/pdftext caches are kept (same bytes, about to be re-read), and the flag requires explicit paths like --force; --force/--retry refuse to combine with it. tasks/delete.md now briefs the still-on-disk cleanup case. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01SPu8qnxePMUS1whYwen11L --- .github/copilot-instructions.md | 15 ++- CHANGELOG.md | 17 +++ CLAUDE.md | 15 ++- citadel/cli.py | 48 ++++++- citadel/ingest.py | 108 ++++++++++++++- citadel/progress.py | 7 +- citadel/rules/tasks/delete.md | 6 +- tests/test_cli.py | 26 ++++ tests/test_ingest_parallel.py | 2 +- tests/test_ingest_progress.py | 13 +- tests/test_reingest.py | 226 ++++++++++++++++++++++++++++++++ 11 files changed, 461 insertions(+), 22 deletions(-) create mode 100644 tests/test_reingest.py diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index ce243b5..7344851 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -34,7 +34,12 @@ session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the liv `--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`), `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source, `--force ` deliberately re-reads already-ingested sources as a reconcile — it requires -explicit paths and is refused without them, `--retry` re-runs everything STUCK without naming +explicit paths and is refused without them, `--reingest ` goes one step further: each named +tracked source is re-imported FRESH — a `kind="delete"` cleanup session strips its previous facts +(manifest entry dropped), then the same run ingests it as a brand-new source under the current +model + rules — the full re-think reconcile's keep-the-existing-treatment rule deliberately avoids +(requires explicit paths like `--force`; the two flags and `--retry` refuse to combine; a failed +cleanup blocks that source's fresh session), `--retry` re-runs everything STUCK without naming paths: every failed source still on disk plus every ingested source no wiki page cites — the zero-entry sources the run report flags as `no_pages` and `status` marks `NO PAGES` — as forced reconciles; it refuses paths/`--force` and exits 0 when nothing is stuck), @@ -211,7 +216,13 @@ addressing them through `ingest`. The flow per source: `CITADEL_MAX_SOURCE_CHARS` is folded in over several passes (all against one staging copy — see the promote bullet below). `ingest --force ` bypasses the sha short-circuit: the named sources land in pending as reconciles (a repo re-digests in full), and the manifest is re-stamped - with the current model + rules version. + with the current model + rules version. `ingest --reingest ` rides force's partitioning + but re-imports fresh instead of reconciling: a delete-cleanup job per named tracked source runs + in the deletions group (always first), dropping the manifest key so the pending session plans the + plain ingest/image/audio/pdf kind of a new key (`repo` for a tracked repo — safe only because the + cleanup already stripped the pages a first-time brief would duplicate); a failed cleanup refuses + that source's fresh session, and the cleanup never prunes the transcript/pdftext caches (the + bytes are unchanged and about to be re-read). - **Discovery is incremental and deletion-safe**: one iterative `os.scandir` walk over every `CITADEL_RAW_DIRS` root keeps each file's stat; the **manifest doubles as the scan cache** (an entry's `size`/`mtime_ns`/`ctime_ns`/`hashed_at_ns` are a skip-hint — sha256 diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ef935b..4c98295 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,23 @@ All notable changes to this project are documented here. The format is based on ## [Unreleased] +### Added + +- **`citadel ingest --reingest ` — re-import an already-ingested source from scratch.** + `--force` runs a reconcile, which deliberately keeps the source's existing treatment (genre, + page structure) and only verifies/updates its facts — so a source first ingested under an old + rulebook or a weak model never gets re-thought, and "all ok, nothing new" is the correct + reconcile verdict even when a fresh read would organize it completely differently. `--reingest` + is the escape hatch: for each named tracked source it first runs a `kind="delete"` cleanup + session that strips the source's previous facts from the wiki (dropping its manifest entry), + then — in the same run, deletions always run first — ingests it as a brand-new source under the + current model, rules, and wiki state (fresh cross-links included). This replaces the manual + move-the-file-out / full ingest / move-it-back / ingest dance. A failed cleanup refuses the + fresh session for that source (nothing is written on top of the old facts), tracked repos take + the first-time `repo` brief over a full digest, and the flag requires explicit paths like + `--force` (exit 2 without them; `--force` and `--retry` refuse to combine with it). The + `tasks/delete.md` brief now covers the still-on-disk cleanup case. + ## [0.6.0] - 2026-07-30 ### Changed (breaking) diff --git a/CLAUDE.md b/CLAUDE.md index 00e638f..fa5adea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,7 +31,12 @@ session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the liv `--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`), `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source, `--force ` deliberately re-reads already-ingested sources as a reconcile — it requires -explicit paths and is refused without them, `--retry` re-runs everything STUCK without naming +explicit paths and is refused without them, `--reingest ` goes one step further: each named +tracked source is re-imported FRESH — a `kind="delete"` cleanup session strips its previous facts +(manifest entry dropped), then the same run ingests it as a brand-new source under the current +model + rules — the full re-think reconcile's keep-the-existing-treatment rule deliberately avoids +(requires explicit paths like `--force`; the two flags and `--retry` refuse to combine; a failed +cleanup blocks that source's fresh session), `--retry` re-runs everything STUCK without naming paths: every failed source still on disk plus every ingested source no wiki page cites — the zero-entry sources the run report flags as `no_pages` and `status` marks `NO PAGES` — as forced reconciles; it refuses paths/`--force` and exits 0 when nothing is stuck), @@ -208,7 +213,13 @@ addressing them through `ingest`. The flow per source: `CITADEL_MAX_SOURCE_CHARS` is folded in over several passes (all against one staging copy — see the promote bullet below). `ingest --force ` bypasses the sha short-circuit: the named sources land in pending as reconciles (a repo re-digests in full), and the manifest is re-stamped - with the current model + rules version. + with the current model + rules version. `ingest --reingest ` rides force's partitioning + but re-imports fresh instead of reconciling: a delete-cleanup job per named tracked source runs + in the deletions group (always first), dropping the manifest key so the pending session plans the + plain ingest/image/audio/pdf kind of a new key (`repo` for a tracked repo — safe only because the + cleanup already stripped the pages a first-time brief would duplicate); a failed cleanup refuses + that source's fresh session, and the cleanup never prunes the transcript/pdftext caches (the + bytes are unchanged and about to be re-read). - **Discovery is incremental and deletion-safe**: one iterative `os.scandir` walk over every `CITADEL_RAW_DIRS` root keeps each file's stat; the **manifest doubles as the scan cache** (an entry's `size`/`mtime_ns`/`ctime_ns`/`hashed_at_ns` are a skip-hint — sha256 diff --git a/citadel/cli.py b/citadel/cli.py index d70729a..a26d4a3 100644 --- a/citadel/cli.py +++ b/citadel/cli.py @@ -116,6 +116,15 @@ def build_parser() -> argparse.ArgumentParser: "without them, so a whole-corpus re-read (one agent session per source) can never " "happen by accident.", ) + p_ingest.add_argument( + "--reingest", + action="store_true", + help="Re-import the given already-ingested sources FRESH: first a delete cleanup session " + "strips each source's previous facts from the wiki (its manifest entry is dropped), then " + "the same run ingests it as a brand-new source under the current model + rules — the " + "full re-think --force's reconcile deliberately avoids (reconcile keeps the existing " + "treatment). Costs two sessions per source. Requires explicit paths, like --force.", + ) p_ingest.add_argument( "--retry", action="store_true", @@ -422,11 +431,17 @@ def cmd_ingest(args: argparse.Namespace) -> int: agent session per source, so forcing the ENTIRE corpus must never happen by accident — the flag alone is refused with exit 2, before ``ingest.ingest`` is ever called. + ``--reingest`` is force's bigger sibling — strip the named sources' previous facts (a delete + cleanup session each), then re-import them fresh in the same run. Same explicit-paths rule as + ``--force`` (exit 2 without them), and the two flags refuse each other: reconcile-in-place + and strip-and-re-import are different intents, and combining them would blur which one wins. + ``--retry`` is the no-typing complement: it computes its own bounded set — ``ingest.retry_candidates()``, the failed sources still on disk plus the ingested ones no wiki page cites — prints it, and runs those as a FORCED re-read. It therefore refuses - explicit paths and ``--force`` (exit 2): the point of the flag is that citadel picks the - set, and combining the two would blur which one wins. Nothing to retry is a clean exit 0. + explicit paths, ``--force`` and ``--reingest`` (exit 2): the point of the flag is that + citadel picks the set, and combining them would blur which one wins. Nothing to retry is a + clean exit 0. ``--jobs N`` is a usage error below 1 (exit 2, like ``--force`` without paths) rather than an exception out of the API layer; omitted, the run takes ``CITADEL_JOBS`` (default 1, serial).""" @@ -436,11 +451,21 @@ def cmd_ingest(args: argparse.Namespace) -> int: print(f"error: --jobs must be at least 1 (got {args.jobs}); 1 means the serial default.", file=sys.stderr) return 2 - if args.retry and (args.paths or args.force): + if args.retry and (args.paths or args.force or args.reingest): print( "error: --retry computes its own retry set (every failed source plus every ingested " - "source no wiki page cites) — combine no paths and no --force with it; use " - "`citadel ingest --force ` for a hand-picked re-read instead.", + "source no wiki page cites) — combine no paths, no --force and no --reingest with it; " + "use `citadel ingest --force ` (or `--reingest `) for a hand-picked " + "re-read instead.", + file=sys.stderr, + ) + return 2 + + if args.force and args.reingest: + print( + "error: --force and --reingest are different intents — --force reconciles the " + "source's facts in place (keeping its existing treatment), --reingest strips them " + "and re-imports the source fresh. Pick one.", file=sys.stderr, ) return 2 @@ -453,6 +478,15 @@ def cmd_ingest(args: argparse.Namespace) -> int: ) return 2 + if args.reingest and not args.paths: + print( + "error: --reingest requires explicit paths (each source runs a delete cleanup " + "session plus a fresh ingest session; name the files or directories to re-import, " + "e.g. `citadel ingest --reingest raw/notes.md`).", + file=sys.stderr, + ) + return 2 + paths = args.paths or None force = args.force if args.retry: @@ -486,7 +520,9 @@ def cmd_ingest(args: argparse.Namespace) -> int: # CITADEL_LLM_VERBOSE — not just the --verbose flag — also drops the spinner that would # otherwise clobber the streamed transcript. progress = ConsoleProgress(spinner=not config.LLM_VERBOSE) - report = ingest.ingest(paths, progress=progress, full_rescan=args.full_rescan, force=force, jobs=args.jobs) + report = ingest.ingest( + paths, progress=progress, full_rescan=args.full_rescan, force=force, jobs=args.jobs, reingest=args.reingest + ) print(report.render()) # Non-zero on a per-source error OR a structural problem left behind (a broken # cross-link the agent introduced) — so ingest gates the wiki's integrity in CI. diff --git a/citadel/ingest.py b/citadel/ingest.py index e310785..c52f8d1 100644 --- a/citadel/ingest.py +++ b/citadel/ingest.py @@ -121,6 +121,11 @@ class IngestReport: # rel-keys of tracked sources that VANISHED from disk (a full run only): their provenance is # reconciled out of the wiki by a cleanup agent session, then the manifest key is dropped. sources_deleted: list[str] = field(default_factory=list) + # `--reingest` only: tracked sources whose previous facts were STRIPPED by a cleanup session + # (manifest key dropped) ahead of the fresh plain-ingest session the same run then ran for + # them — the deliberate re-think of an already-ingested source. The fresh session's own + # outcome lands in `processed`/`errors` like any other source's. + reingest_cleaned: list[str] = field(default_factory=list) # `--jobs N` only: sources whose session raced a CONCURRENT source's promote over the same page # and were therefore re-run serially afterwards (the re-run's own success/failure is reported # like any other source's). Surfaced because it is the one place parallel ingest costs money a @@ -182,6 +187,9 @@ def render(self) -> str: if self.sources_deleted: lines.append("Sources removed (deleted from disk; citations reconciled out):") lines.extend(f" - {s}" for s in self.sources_deleted) + if self.reingest_cleaned: + lines.append("Re-ingested fresh (previous facts stripped first, then imported as new):") + lines.extend(f" - {s}" for s in self.reingest_cleaned) if self.resumed: lines.append("Resumed (continued from an earlier run's checkpoint):") lines.extend(f" - {r}" for r in self.resumed) @@ -606,6 +614,7 @@ def ingest( full_rescan: bool = False, force: bool = False, jobs: int | None = None, + reingest: bool = False, ) -> IngestReport: """Run one ingest. Exactly one source = one all-or-nothing agent job (a chunked source runs several ``llm.run_ingest_session`` passes inside that one job). @@ -638,6 +647,20 @@ def ingest( CLI pre-empts it with the same message and a friendly exit 2), and a path-scoped run never sweeps deletions (``swept_roots=None`` below). + ``reingest`` (the ``--reingest`` flag) goes one deliberate step past ``force``: instead of + reconciling the named tracked sources around their existing treatment, each is re-imported + FRESH — first a ``kind="delete"`` cleanup session strips its previous facts from the wiki and + its manifest key is dropped, then (same run, deletions always run first) the source lands in + pending as a brand-new key and runs a plain ``kind="ingest"`` session (``kind="repo"`` for a + tracked repo — the cleanup already removed the pages a first-time brief would otherwise + duplicate). That is the escape hatch from reconcile's keep-the-existing-genre-treatment rule: + the source is re-thought from scratch against the CURRENT wiki and rules (e.g. after a model + upgrade or a new genre), at the cost of a cleanup session plus a full ingest session per + source. Like ``force`` it requires explicit paths (same ValueError here, same CLI exit 2) and + implies force's partitioning (sha short-circuit and dedup drop bypassed). If the cleanup + session fails, the fresh ingest for that source is refused (a per-source failure, retried by + re-running ``--reingest``) — new pages are never written on top of the old facts. + Deletion detection is guarded (operational safety over thoroughness): candidates come from the walked-seen-set diff, each positively confirmed with ``.exists()``; any walk error aborts the entire sweep for the run; an unreachable root @@ -689,6 +712,14 @@ def ingest( "--force requires explicit paths (a forced re-read runs one agent session per " "source; name the files or directories to force, e.g. `citadel ingest --force raw/notes.md`)." ) + if reingest and not paths: + # Same guard, sharper teeth: a reingest costs a cleanup session PLUS a full ingest + # session per source, so a whole-corpus reingest by accident is twice as expensive. + raise ValueError( + "--reingest requires explicit paths (each source runs a delete cleanup session plus " + "a fresh ingest session; name the files or directories to re-import, e.g. " + "`citadel ingest --reingest raw/notes.md`)." + ) # ONE mutating run per workspace: the staging sweep, promote's prune, and the manifest/ # failures saves are all destructive under concurrency (see runlock's module docstring). @@ -698,12 +729,19 @@ def ingest( # Same place, same reason: under the exclusive lock, leftovers on disk belong to dead runs. # Age-based only — a checkpoint's own guards decide whether it is USABLE (see resume.sweep). resume.sweep() - return _ingest_run(paths, progress, full_rescan=full_rescan, force=force, jobs=workers) + return _ingest_run(paths, progress, full_rescan=full_rescan, force=force, jobs=workers, reingest=reingest) -def _ingest_run(paths: list[str] | None, progress, *, full_rescan: bool, force: bool, jobs: int = 1) -> IngestReport: +def _ingest_run( + paths: list[str] | None, progress, *, full_rescan: bool, force: bool, jobs: int = 1, reingest: bool = False +) -> IngestReport: """The body of :func:`ingest`, running under the exclusive workspace run lock.""" + # A reingest rides force's partitioning wholesale (sha short-circuit and dedup drop bypassed, + # stat cache distrusted for the named paths); what it changes beyond force — the cleanup jobs + # and the fresh session kinds — is keyed off `reingest`/`reingest_keys` below. + force = force or reingest + # `--jobs N` emits from WORKER threads (a source's start/done event fires where the work # happens), so the callback — which is whatever the caller passed — is serialized here. That # keeps "your progress callback is never invoked concurrently" a property of the API rather than @@ -863,6 +901,22 @@ def emit(event: str, **data) -> None: # one not yet tracked is brand new. Captured before the manifest is mutated below. pending_keys = {manifest.rel_key(p) for p in scan.pending} changed_keys = pending_keys & set(manifest_dict) + # --reingest: every named TRACKED source is re-imported fresh instead of reconciled — a + # delete-cleanup job (group 1, always before pending) strips its previous facts and drops its + # manifest key, and emptying changed_keys here makes its pending session plan the plain + # ingest/image/audio/pdf kind of a brand-new key. Tracked repos join reingest_keys below. + reingest_keys: set[str] = set() + if reingest: + reingest_keys = set(changed_keys) + changed_keys = set() + # A tracked repo (force gave it kind="repo-reconcile") is re-imported fresh the same way: + # cleanup first, then the FIRST-TIME brief over a full digest. kind="repo" is safe here + # precisely because the cleanup precedes it — the pages a first-time brief would + # otherwise duplicate are already stripped (contrast _partition_repos' force rule). + repo_pending = [ + _RepoJob(path=r.path, key=r.key, kind="repo", old_commit=None) if r.old_commit else r for r in repo_pending + ] + reingest_keys.update(r.key for r in repo_pending if r.key in manifest_dict) # --- Reorganized sources: a file that only MOVED (or is a byte-for-byte duplicate) is # recognized and NOT re-ingested. For a real move (the old path is gone) repoint the wiki's @@ -1000,6 +1054,9 @@ def emit(event: str, **data) -> None: deleted=len(deleted_sources), repos=len(repo_pending), jobs=jobs, + # Each reingest source runs a cleanup JOB on top of its pending session — counted so the + # overall progress total matches the jobs that will actually run. + reingest=len(reingest_keys), ) # --- The per-source jobs (the SourceJob loop): DELETION cleanups first, then files, then repos, @@ -1034,6 +1091,15 @@ def _file_job(src: Path) -> _SourceJob: office = scan.office_text.get(src) def build() -> tuple[list, list[str], "_Resume | None"]: + # A reingest source may only run its fresh session on a wiki its cleanup actually + # cleaned: the cleanup job (group 1) pops the manifest key on success, so a key still + # tracked here means that cleanup failed — refuse rather than write new pages on top + # of the old facts (a per-source prepare failure; re-running --reingest retries both). + if rel_key in reingest_keys and rel_key in manifest_dict: + raise RuntimeError( + "the delete cleanup for this reingest failed, so its previous facts are " + "still in the wiki; fix that failure and re-run `citadel ingest --reingest`" + ) # Plan the pass(es): an Office source materializes its extracted text to a temp .md # the agent reads; an audio/video source is transcribed HERE through the whisper seam # (content-addressed cache; a raise is a retryable per-source prepare_error, and the @@ -1122,6 +1188,13 @@ def done(usage: llm.SessionUsage | None) -> None: # run-level ``force``), which re-digests in FULL (see _partition_repos). def _repo_job(rjob: _RepoJob) -> _SourceJob: def build() -> tuple[list, list[str], "_Resume | None"]: + # Same cleanup post-condition as _file_job's: a reingested repo whose cleanup failed + # (key still tracked) must not run the first-time brief on top of its old pages. + if rjob.key in reingest_keys and rjob.key in manifest_dict: + raise RuntimeError( + "the delete cleanup for this reingest failed, so its previous facts are " + "still in the wiki; fix that failure and re-run `citadel ingest --reingest`" + ) only: list[str] | None = None change_summary: str | None = None if rjob.kind == "repo-reconcile" and rjob.old_commit and not force: @@ -1166,8 +1239,10 @@ def done(usage: llm.SessionUsage | None) -> None: # cites it, run a `kind="delete"` cleanup session that strips that provenance, gated by a # post-condition that the wiki no longer references it (else the whole cleanup is rolled back # and retried next full run — the manifest key is dropped only on success). A deletion that - # nothing cites plans NO session and just loses its manifest key. - def _delete_job(key: str) -> _SourceJob: + # nothing cites plans NO session and just loses its manifest key. With ``reingest_cleanup`` + # the SAME job strips a still-on-disk source ahead of its fresh re-import (`--reingest`); the + # bookkeeping differs only where the "source is gone" premise does not hold. + def _delete_job(key: str, reingest_cleanup: bool = False) -> _SourceJob: def build() -> tuple[list, list[str], "_Resume | None"]: if not store.find_raw_references(key): return [], [], None # nothing cites it: no cleanup session, just forget it below @@ -1186,7 +1261,14 @@ def done(_usage: llm.SessionUsage | None) -> None: # this must NOT gate on the extension: a PDF routes by %PDF- MAGIC (is_pdf_file), so it # can be cached under any name, and an ext gate would orphan its plaintext extraction. del_sha = manifest.entry_sha(entry) if entry is not None else None - if entry is not None and not _sha_shared_by_other_entry(manifest_dict, del_sha, key): + # ... except on a reingest cleanup: the source still exists with the SAME bytes, and + # the fresh session moments away re-reads exactly the cached transcript/extraction — + # pruning here would throw away work the run is about to re-buy. + if ( + not reingest_cleanup + and entry is not None + and not _sha_shared_by_other_entry(manifest_dict, del_sha, key) + ): transcribe.prune_cached(del_sha) pdftext.prune_cached(del_sha) # A resume checkpoint is KEY-addressed (not content-addressed like those two caches), @@ -1196,7 +1278,10 @@ def done(_usage: llm.SessionUsage | None) -> None: manifest_dict.pop(key, None) failures.clear(failures_dict, key) manifest.save(manifest_dict) - report.sources_deleted.append(key) + if reingest_cleanup: + report.reingest_cleaned.append(key) + else: + report.sources_deleted.append(key) return _SourceJob( key=key, @@ -1216,7 +1301,10 @@ def done(_usage: llm.SessionUsage | None) -> None: # a later run with nothing pending would never rebuild the derived files. pending_interrupt: BaseException | None = None groups = ( - [_delete_job(key) for key in deleted_sources], + # Reingest cleanups ride the deletion group, so the always-first ordering above holds for + # them too: a source's old facts are stripped before ANY pending session runs. + [_delete_job(key) for key in deleted_sources] + + [_delete_job(key, reingest_cleanup=True) for key in sorted(reingest_keys)], [_file_job(src) for src in scan.pending], [_repo_job(r) for r in repo_pending], ) @@ -1238,6 +1326,7 @@ def done(_usage: llm.SessionUsage | None) -> None: or report.moved or report.unreadable or report.sources_deleted + or report.reingest_cleaned or repointed or failures_changed or pruned_ignored @@ -1281,6 +1370,11 @@ def done(_usage: llm.SessionUsage | None) -> None: f"raw source {key} was deleted from disk; reconciled its citations out of the " "wiki and dropped it from the manifest" ) + for key in report.reingest_cleaned: + store.append_log( + f"reingest {key}: stripped its previous facts ahead of the fresh import " + "(deliberate re-read as a new source)" + ) # The wiki-history commit comes LAST, after the log/index/failures writes above, so one # commit captures the run's complete state. Best-effort by contract: the wiki is already # promoted, so a git problem is a report note, never a failed run. diff --git a/citadel/progress.py b/citadel/progress.py index 82136fc..25f4d9b 100644 --- a/citadel/progress.py +++ b/citadel/progress.py @@ -171,6 +171,7 @@ def on_start( deleted: int = 0, repos: int = 0, jobs: int = 1, + reingest: int = 0, ) -> None: bits = [] if skipped: @@ -179,6 +180,10 @@ def on_start( bits.append(f"{moved} reorganized") if unreadable: bits.append(f"{unreadable} unreadable") + if reingest: + # --reingest: these sources run a cleanup job on top of their pending session; the + # extra jobs are folded into the overall total below. + bits.append(f"{reingest} re-imported fresh (old facts stripped first)") if pending == 0 and repos == 0 and deleted == 0: extra = f" ({', '.join(bits)})" if bits else "" self._print(Text(f"Nothing to ingest{extra}.", style="dim")) @@ -201,7 +206,7 @@ def on_start( headline = Text(f"Ingesting {' + '.join(counts)}{extra}", style="bold") if jobs > 1: headline.append(f" [{jobs} at a time]", style="dim") - self._begin(pending + repos + deleted, headline) + self._begin(pending + repos + deleted + reingest, headline) def on_source_start(self, index: int, total: int, source: str) -> None: label = f"[{index}/{total}] {config.display_key(source)}" diff --git a/citadel/rules/tasks/delete.md b/citadel/rules/tasks/delete.md index 48ba02e..a5e9cce 100644 --- a/citadel/rules/tasks/delete.md +++ b/citadel/rules/tasks/delete.md @@ -1,7 +1,9 @@ # delete — strip a removed source's provenance -The source was **removed from disk** and no longer exists. Do **NOT** try to open it. Remove the -provenance that depended on it: +The source was **removed from disk** and no longer exists — or is being **deliberately +re-imported from scratch** (a reingest: the file may still sit on disk, but its previous facts +must go so the fresh import that follows starts clean). Either way, do **NOT** try to open the +source. Remove the provenance that depended on it: 1. Search the wiki (Grep/Glob/Read) for every page that cites it: a `resource:` frontmatter field naming it, or a `[^sN]` footnote whose `## Sources` definition links to it (e.g. diff --git a/tests/test_cli.py b/tests/test_cli.py index 98467e7..fd3e489 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -201,6 +201,32 @@ def test_ingest_force_without_paths_is_rejected(ingest_spy): assert not ingest_spy.called +def test_ingest_reingest_flag_reaches_ingest(ingest_spy): + """``--reingest`` hands ``reingest=True`` through to ``ingest.ingest`` alongside the explicit + paths; without the flag the kwarg defaults to False.""" + assert cli.main(["ingest", "--quiet", "raw/a.md"]) == 0 + assert ingest_spy.kwargs["reingest"] is False + assert cli.main(["ingest", "--quiet", "--reingest", "raw/a.md"]) == 0 + assert ingest_spy.kwargs["reingest"] is True + assert ingest_spy.paths == ["raw/a.md"] + + +def test_ingest_reingest_without_paths_is_rejected(ingest_spy, capsys): + """Same explicit-paths rule as ``--force``, with sharper teeth (two sessions per source): + ``--reingest`` alone is refused with exit 2 before ``ingest.ingest`` is reached.""" + assert cli.main(["ingest", "--quiet", "--reingest"]) == 2 + assert not ingest_spy.called + assert "--reingest requires explicit paths" in capsys.readouterr().err + + +def test_ingest_reingest_refuses_force_and_retry(ingest_spy, capsys): + """``--reingest`` and ``--force`` are different intents (strip-and-re-import vs reconcile in + place) and refuse each other; ``--retry`` computes its own set and refuses both.""" + assert cli.main(["ingest", "--quiet", "--force", "--reingest", "raw/a.md"]) == 2 + assert cli.main(["ingest", "--quiet", "--retry", "--reingest"]) == 2 + assert not ingest_spy.called + + def test_ingest_source_error_exits_1(ingest_spy): ingest_spy.report = _report(errors=["raw/a.md: agent failed"]) assert cli.main(["ingest", "--quiet"]) == 1 diff --git a/tests/test_ingest_parallel.py b/tests/test_ingest_parallel.py index cf350fb..e243dc0 100644 --- a/tests/test_ingest_parallel.py +++ b/tests/test_ingest_parallel.py @@ -393,7 +393,7 @@ def test_jobs_below_one_is_refused(tmp_citadel, fake_agent): def test_cli_jobs_is_threaded_through(tmp_citadel, monkeypatch, capsys): seen: dict = {} - def fake_ingest(paths=None, progress=None, full_rescan=False, force=False, jobs=None): + def fake_ingest(paths=None, progress=None, full_rescan=False, force=False, jobs=None, reingest=False): seen["jobs"] = jobs return ingest.IngestReport([], [], [], []) diff --git a/tests/test_ingest_progress.py b/tests/test_ingest_progress.py index f68a8ac..d508f66 100644 --- a/tests/test_ingest_progress.py +++ b/tests/test_ingest_progress.py @@ -23,7 +23,16 @@ def test_ingest_emits_progress_events(tmp_citadel, fake_agent, transformer_page) for expected in ("source_start", "source_done", "finalize", "done"): assert expected in names, f"missing event: {expected}" start = next(d for e, d in events if e == "start") - assert start == {"pending": 1, "skipped": 0, "moved": 0, "unreadable": 0, "deleted": 0, "repos": 0, "jobs": 1} + assert start == { + "pending": 1, + "skipped": 0, + "moved": 0, + "unreadable": 0, + "deleted": 0, + "repos": 0, + "jobs": 1, + "reingest": 0, + } done = next(d for e, d in events if e == "source_done") assert done["source"] == "raw/notes.md" assert done["index"] == 1 and done["total"] == 1 @@ -83,6 +92,8 @@ def fake(rel_key, kind="ingest", read_path=None, segment=None): # `--jobs N`: the worker count the run was given, so the console can show how many sources # are in flight at once (rich gives each its own spinner row). 1 is the serial default. "jobs": 1, + # `--reingest` cleanup jobs on top of the pending sessions — 0 on every non-reingest run. + "reingest": 0, } # Deletions first, then files, then repos — and per-GROUP counters restarting at 1/1. assert [d["source"] for e, d in events if e == "source_start"] == ["raw/gone.md", "raw/note.md", "raw/svc"] diff --git a/tests/test_reingest.py b/tests/test_reingest.py new file mode 100644 index 0000000..11370c4 --- /dev/null +++ b/tests/test_reingest.py @@ -0,0 +1,226 @@ +"""``ingest --reingest`` (offline): strip a tracked source's previous facts, then re-import it +FRESH in the same run — the deliberate full re-think that ``--force``'s reconcile (which keeps the +existing treatment) cannot deliver. The decided semantics, pinned tests-first: + +- a reingested tracked FILE runs exactly two jobs in order: a ``kind="delete"`` cleanup (its + manifest key dropped on success) and then a plain ``kind="ingest"`` session — never a + reconcile — with the manifest re-stamped fresh under the CURRENT model + rules_version; +- a reingested tracked source that NO page cites plans no cleanup session (the key is just + forgotten) and goes straight to the fresh ingest; +- a FAILED cleanup blocks the fresh session for that source: nothing is written on top of the + old facts, the live wiki and manifest entry stay as they were, and the failure is recorded; +- a reingested REPO runs the cleanup and then the FIRST-TIME ``kind="repo"`` brief over a FULL + digest (safe exactly because the cleanup removed the pages a first-time brief would duplicate); +- ``reingest`` requires explicit paths (API ValueError; the CLI pre-empts with exit 2) and a + path-scoped reingest run never sweeps deletions; +- the transcript/extraction caches are NOT pruned by a reingest cleanup (the source still exists + with the same bytes, and the fresh session is about to re-read exactly that cached text). + +``llm.run_ingest_session`` is replaced by ``fake_agent``. +""" + +from __future__ import annotations + +import pytest +from conftest import REAL_RULES_DIR, delete_citing_pages + +from citadel import config, failures, ingest, manifest, pdftext, repo + + +def _reingest_side_effect(cite_page, rel_path="misc/note-v2.md", fact="A rethought fact."): + """The canonical two-kind fake session: the ``delete`` pass strips every citing page from the + staging copy, the fresh ``ingest`` pass writes the new page.""" + + def _run(rel_key, kind="ingest", **kw): + if kind == "delete": + delete_citing_pages(rel_key) + else: + cite_page(rel_path, rel_key, fact) + + return _run + + +# -------------------------------------------------------------------------------------------- +# reingest on FILE sources +# -------------------------------------------------------------------------------------------- + + +def test_reingest_runs_cleanup_then_plain_ingest(tmp_citadel, fake_agent, cite_page, monkeypatch): + """The core semantics: cleanup first (``kind="delete"``), then a plain ``kind="ingest"`` — + NEVER the ``reconcile`` a plain ``--force`` runs — and the manifest entry is a FRESH stamp + under the current model. Control on the same corpus first: ``--force`` still reconciles.""" + wiki, raw = tmp_citadel.wiki, tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + monkeypatch.setattr(config, "ingest_model_label", lambda: "fake:model-a") + agent = fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/note.md", rel_key, "A weak-model fact.")) + ingest.ingest() + assert agent.calls == [("raw/notes.md", "ingest")] + + agent.reset() + ingest.ingest([str(raw / "notes.md")], force=True) + assert agent.calls == [("raw/notes.md", "reconcile")] # control: force stays a reconcile + + monkeypatch.setattr(config, "ingest_model_label", lambda: "fake:model-b") + agent = fake_agent(side_effect=_reingest_side_effect(cite_page)) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert agent.calls == [("raw/notes.md", "delete"), ("raw/notes.md", "ingest")] + assert report.reingest_cleaned == ["raw/notes.md"] + assert report.sources_deleted == [] # a reingest is not a deletion + assert report.processed == ["raw/notes.md"] + assert not report.errors + assert not (wiki / "misc" / "note.md").exists() # the old treatment is gone... + assert (wiki / "misc" / "note-v2.md").exists() # ...replaced by the fresh import + entry = tmp_citadel.read_manifest()["raw/notes.md"] + assert entry["model"] == "fake:model-b" # a brand-new stamp, not a carried one + + +def test_reingest_without_paths_refused_at_the_api_layer(tmp_citadel, fake_agent): + """Like force, but twice as expensive per source (cleanup + fresh ingest): ``reingest=True`` + with no paths raises ValueError before any work, no matter the caller.""" + agent = fake_agent() + with pytest.raises(ValueError, match="--reingest requires explicit paths"): + ingest.ingest(reingest=True) + assert agent.count == 0 + + +def test_reingest_uncited_source_skips_cleanup_session(tmp_citadel, fake_agent, cite_page): + """A tracked source no page cites has nothing to strip: the cleanup job plans ZERO sessions + (the key is just forgotten) and the fresh ingest runs directly — one paid session, not two.""" + raw = tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + agent = fake_agent() # writes nothing: the source ends up ingested but uncited (no_pages) + assert ingest.ingest().no_pages == ["raw/notes.md"] + + agent = fake_agent(side_effect=_reingest_side_effect(cite_page)) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert agent.calls == [("raw/notes.md", "ingest")] # no delete session was paid for + assert report.reingest_cleaned == ["raw/notes.md"] # the key was still dropped + re-imported + assert report.processed == ["raw/notes.md"] + assert not report.errors + + +def test_reingest_cleanup_failure_blocks_the_fresh_ingest(tmp_citadel, fake_agent, cite_page): + """All-or-nothing per source, across BOTH jobs: a failed cleanup leaves the live wiki and the + manifest entry untouched, and the fresh session is REFUSED (never run on top of the old + facts) — recorded as a failure, so re-running ``--reingest`` retries the pair.""" + wiki, raw = tmp_citadel.wiki, tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + agent = fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/note.md", rel_key, "A fact.")) + ingest.ingest() + page = wiki / "misc" / "note.md" + before_bytes = page.read_bytes() + entry_before = dict(tmp_citadel.read_manifest()["raw/notes.md"]) + + def failing(rel_key, kind="ingest", **kw): + if kind == "delete": + raise RuntimeError("cleanup boom") + cite_page("misc/note-v2.md", rel_key, "Must never land.") + + agent = fake_agent(side_effect=failing) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert agent.calls == [("raw/notes.md", "delete")] # the fresh session never ran + assert report.reingest_cleaned == [] + assert report.processed == [] + assert any("cleanup boom" in e for e in report.errors) + assert any("reingest" in e for e in report.errors) # the refusal names the why + assert page.read_bytes() == before_bytes # live wiki untouched + assert not (wiki / "misc" / "note-v2.md").exists() + assert tmp_citadel.read_manifest()["raw/notes.md"] == entry_before # entry kept, not dropped + assert "raw/notes.md" in failures.load() + + +def test_reingest_path_run_never_sweeps_deletions(tmp_citadel, fake_agent, cite_page, seed_cited_deleted_source): + """A reingest run is path-scoped by construction — the rest of the manifest must never be + read as deletion candidates (the same ``swept_roots=None`` rule as ``--force``).""" + wiki, raw = tmp_citadel.wiki, tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + agent = fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/note.md", rel_key, "A fact.")) + ingest.ingest() + seed_cited_deleted_source() + + agent = fake_agent(side_effect=_reingest_side_effect(cite_page)) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert agent.calls == [("raw/notes.md", "delete"), ("raw/notes.md", "ingest")] + assert report.sources_deleted == [] + assert (wiki / "concepts" / "topic.md").exists() # the vanished source's page untouched + assert "raw/gone.md" in tmp_citadel.read_manifest() # still tracked; swept only by a full run + + +def test_reingest_cleanup_keeps_the_extraction_cache(tmp_citadel, fake_agent, cite_page, monkeypatch): + """The cleanup must NOT prune the content-addressed transcript/extraction caches: the source + still exists with the same bytes, and the fresh session moments later re-reads exactly that + cached text (contrast a real deletion, which prunes).""" + raw = tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/note.md", rel_key, "A fact.")) + ingest.ingest() + + pruned: list[str] = [] + monkeypatch.setattr(pdftext, "prune_cached", lambda sha: pruned.append(sha)) + fake_agent(side_effect=_reingest_side_effect(cite_page)) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert report.processed == ["raw/notes.md"] + assert pruned == [] # the cache the fresh session would re-buy stays + + +# -------------------------------------------------------------------------------------------- +# reingest on REPO sources +# -------------------------------------------------------------------------------------------- + + +def test_reingest_repo_runs_cleanup_then_first_time_brief_over_full_digest( + repo_wiki, fake_agent, make_repo, cite_page, monkeypatch +): + """A reingested tracked repo runs the cleanup and then ``kind="repo"`` — the FIRST-TIME brief, + which force deliberately never uses because it would duplicate pages; here the cleanup + removed them first — over a FULL digest (``only=None``, no change summary).""" + raw = repo_wiki.raw + agent = fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/svc.md", rel_key, "A repo fact.")) + make_repo(raw, "svc", {"README.md": "# Svc\n", "app.py": "x\n"}) + ingest.ingest() + assert agent.calls == [("raw/svc", "repo")] + + captured: dict = {} + real_build = repo.build_digest + + def capturing_build(path, key, *, only=None, change_summary=None, **kw): + captured["only"] = only + captured["change_summary"] = change_summary + return real_build(path, key, only=only, change_summary=change_summary, **kw) + + monkeypatch.setattr(repo, "build_digest", capturing_build) + monkeypatch.setattr(repo, "changed_files", lambda *a, **k: ["README.md"]) # a diff must not be consulted + + def rethink(rel_key, kind="repo", **kw): + if kind == "delete": + delete_citing_pages(rel_key) + else: + cite_page("misc/svc-v2.md", rel_key, "A rethought repo fact.") + + agent = fake_agent(side_effect=rethink) + report = ingest.ingest([str(raw / "svc")], reingest=True) + + assert agent.calls == [("raw/svc", "delete"), ("raw/svc", "repo")] # never repo-reconcile + assert report.reingest_cleaned == ["raw/svc"] + assert report.processed == ["raw/svc"] + assert captured == {"only": None, "change_summary": None} # full digest, no change summary + assert manifest.is_repo_entry(repo_wiki.read_manifest()["raw/svc"]) # re-tracked as a repo + + +# -------------------------------------------------------------------------------------------- +# the rules layer: the delete brief must cover the still-on-disk reingest cleanup +# -------------------------------------------------------------------------------------------- + + +def test_delete_brief_carries_the_reingest_note(): + """``tasks/delete.md`` briefs the reingest cleanup too: the file may still exist on disk, and + the agent must strip the provenance without opening it — otherwise a cleanup session that + finds the file present second-guesses its task.""" + text = (REAL_RULES_DIR / "tasks" / "delete.md").read_text(encoding="utf-8").lower() + assert "reingest" in text + assert "still" in text and "disk" in text From aab0fe1882f65bb8ccf1b08bb18df0e089f38292 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 10:25:40 +0000 Subject: [PATCH 2/4] cli: clarify the --retry combination refusal message Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01SPu8qnxePMUS1whYwen11L --- citadel/cli.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/citadel/cli.py b/citadel/cli.py index a26d4a3..fce4241 100644 --- a/citadel/cli.py +++ b/citadel/cli.py @@ -454,9 +454,9 @@ def cmd_ingest(args: argparse.Namespace) -> int: if args.retry and (args.paths or args.force or args.reingest): print( "error: --retry computes its own retry set (every failed source plus every ingested " - "source no wiki page cites) — combine no paths, no --force and no --reingest with it; " - "use `citadel ingest --force ` (or `--reingest `) for a hand-picked " - "re-read instead.", + "source no wiki page cites) and cannot be combined with explicit paths, --force, or " + "--reingest; use `citadel ingest --force ` (or `--reingest `) for a " + "hand-picked re-read instead.", file=sys.stderr, ) return 2 From 5a1705aef8db62a9bca428dd1cf489756ca6183d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 11:51:49 +0000 Subject: [PATCH 3/4] rules: reconcile re-reads with fresh eyes, not just re-verification The reconcile brief drives both `ingest --force` and every `citadel refresh` session, and it only asked the agent to verify/update the facts the wiki already carries - so a re-read after a model or rules upgrade correctly answered "all ok, nothing new" even when the first pass had missed facts, or the wiki had since grown pages this source should link to. Now the brief says a re-read is more than verification: mine the unchanged source for facts the first ingest missed, cross-link against today's wiki, and strip claims the source never supported. The existing-genre-treatment rule stays - structural re-thinks remain --reingest's job. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01SPu8qnxePMUS1whYwen11L --- CHANGELOG.md | 10 ++++++++++ citadel/rules/tasks/reconcile.md | 19 ++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c98295..6bf5fcf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,16 @@ All notable changes to this project are documented here. The format is based on `--force` (exit 2 without them; `--force` and `--retry` refuse to combine with it). The `tasks/delete.md` brief now covers the still-on-disk cleanup case. +### Changed + +- **Forced/refresh reconciles now re-read with fresh eyes, not just re-verify.** The + `tasks/reconcile.md` brief (driving `ingest --force` and every `citadel refresh` session) now + explicitly instructs the agent that a re-read is more than verification: mine the unchanged + source for wiki-worthy facts the first ingest missed, cross-link against today's wiki (pages + that did not exist back then may now be the right home or neighbor), and strip claims the + source never actually supported — while still keeping the existing genre treatment (structure + churn stays `--reingest`'s job). + ## [0.6.0] - 2026-07-30 ### Changed (breaking) diff --git a/citadel/rules/tasks/reconcile.md b/citadel/rules/tasks/reconcile.md index a13a738..04fdfd4 100644 --- a/citadel/rules/tasks/reconcile.md +++ b/citadel/rules/tasks/reconcile.md @@ -1,9 +1,22 @@ # reconcile — re-fold a source the wiki already cites The wiki already holds facts derived from this source, and the source **changed since it was -last ingested — or is being deliberately re-read** (a forced run: the source may be unchanged — -re-verify the wiki's facts against it and apply the **current** rules, which may themselves have -changed since the first ingest). +last ingested — or is being deliberately re-read** (a forced run or a `citadel refresh`: the +source may be unchanged — re-verify the wiki's facts against it and apply the **current** rules, +which may themselves have changed since the first ingest). + +**A re-read is more than verification.** The wiki, the rules, and the model have all moved since +this source was first ingested, so read the WHOLE source again with fresh eyes: + +- **Mine it for wiki-worthy facts the first ingest missed.** A fact the wiki does not carry is + genuinely new even when the source is unchanged — the first pass may simply have overlooked or + under-valued it. Add it, cited as usual. +- **Cross-link against today's wiki.** Pages that did not exist at first ingest may now be the + right home or neighbor for this source's facts — search first (`core.md`), then link or merge + instead of leaving the old pages isolated. +- **Remove what the source never supported.** The same rule as a fact the changed file no longer + supports: a claim cited to this source that the source does not actually state (a first-pass + misreading) loses this source's marker — and the sentence, if no other source remains. **Reconcile — do not merely append**, otherwise a corrected number leaves the stale one standing next to the new one: From 0e16828a7961b0728bd28e211b5c97a12b6581c4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 12:45:27 +0000 Subject: [PATCH 4/4] docs: ledger the 2026-07-31 leuchtfeuer verify-corpus run (PR #139) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01SPu8qnxePMUS1whYwen11L --- docs/verify-corpus-backlog.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/verify-corpus-backlog.md b/docs/verify-corpus-backlog.md index 3f591c0..4c213f7 100644 --- a/docs/verify-corpus-backlog.md +++ b/docs/verify-corpus-backlog.md @@ -46,6 +46,8 @@ Row format: | VCB-002 | 2026-07-25 | 2026-07-25 | beverages | every factual sentence carries a footnote | wiki-generation | **haiku-only.** One uncited sentence survived the agent's own `citadel check` because it is inside a sentence the parser reads as cited: `concepts/coffee.md` — "The plant and its cultivation are dominated by two species: [Arabica and Robusta…". Lint's advisory `missing citations` caught it; `check` did not. Route: the `tasks/ingest.md` self-check step (re-read every paragraph for a trailing footnote), and worth asking whether this shape should be a `check` error rather than a lint advisory. | open | | VCB-003 | 2026-07-25 | 2026-07-25 | beverages | `[^llmN]` is for model-supplied facts only, never a shortcut | wiki-generation | **haiku-only.** 4 pages carry model-supplied facts (`concepts/caffeine.md`, `concepts/tea.md`, `objects/aurora-midnight.md`, `organizations/thornbury-lin.md`) where the sonnet showcase of the same corpus has none — the corpus is fully covered by its raw sources, so an `[^llmN]` here is a weaker model reaching for the LLM lane instead of the source. Not a provenance violation (the lane is honest and lint surfaces it), so a soft miss. Route: `citadel/rules/core.md`'s `[^llmN]` section — make "prefer dropping the claim over sourcing it to yourself" explicit. | open | | VCB-004 | 2026-07-25 | 2026-07-25 | beverages | dense cross-linking between related pages | wiki-generation | **Unattributed between model and `--jobs`.** 36 un-linked mentions where the committed sonnet showcase lints 0 (e.g. `concepts/tea.md → concepts/coffee.md`, `organizations/caffe-aurora.md → concepts/cold-brew.md`). Two candidate causes and this run cannot separate them: haiku links less densely, and `--jobs 4` means concurrent sessions cannot see each other's new pages (the documented, accepted cost of parallelism — `curate` is its designed cleanup). Route: re-run beverages on haiku with `--jobs 1` and diff this counter; only if serial haiku is also high is this a rules-lane miss rather than a parallelism trade-off. | open | +| VCB-006 | 2026-07-31 | 2026-07-31 | leuchtfeuer | T4 intra-wave temporal supersession (decommission 30 Sep → 31 Jul 2026) | wiki-generation | The 8 Apr 2026 portal-minutes session recorded the brought-forward KOMET decommission (31 Jul 2026) on `projects/seagull-customer-portal.md` but left `projects/projekt-leuchtfeuer.md` still stating "switched off on 30 September 2026" as the live plan (cited to the 20 Mar go-live mail, ingested one session earlier in the same wave). Cross-PAGE supersession: the new source's session updated the page it landed on but did not hunt other pages asserting the now-old value. Route: `tasks/ingest.md` — when a new source supersedes a dated value, grep the whole wiki for the superseded value and update every page presenting it as current. | open | +| VCB-007 | 2026-07-31 | 2026-07-31 | leuchtfeuer | style-profile quirks (§I, `CITADEL_STYLE_PROFILES=1`) | wiki-generation | Opinions all correctly attributed (O1/O2/O3 pass), but the persons pages carry no style-quirk entries — Vogelsang's nautical metaphors / "Fair winds" sign-off and Duszek's "—MD" are absent; the POD abbreviation (portal minutes) also has no expansion/page where WMS and MDE got both. Route: `genres/first-person.md` + the style-profile brief — spell out that sign-offs and recurring metaphors belong on the persons page when profiling is ON. | open | ## Resolved @@ -59,6 +61,32 @@ Row format: verdict, and the `VCB-` ids the run touched, or `misses: none`. This `## Runs` heading stays singular; runs nest under it.)* +### 2026-07-31 leuchtfeuer (grading PR #139 — `--reingest`, the reconcile fresh-eyes brief, the delete-brief reingest note) + +- **Mode:** Mode A in a scratch sandbox, full wave protocol (`stages/initial` → wave2 → wave3 with + the memo delete), then the NOOP re-run, a `--force` probe, and a live `--reingest` probe of the + PR's own feature on the heavily-cited kickoff minutes. Retrieval-first grade afterwards. +- **Model:** `claude:claude-sonnet-5` (`CITADEL_INGEST_MODEL=sonnet`) · serial · wave 1 paid the + per-source hermetic-auth retry (container auth shape), waves 2-3 ran with `CITADEL_HERMETIC=0`. +- **Result:** `check` + `lint` exit 0 after every wave AND after both probes. Wave kinds exactly as + scripted (6 ingest / 1 reconcile + 3 ingest + 5 NOOP / 1 delete + 3 ingest + 8 NOOP); idempotency + re-run zero sessions. **D1 delete propagation clean** (18k/02:00/memo-ref ∅ on all pages; €310k + survives attributed to Brandt; retraction recorded). C1/C2 planted values cited with honest + `[^llm]` correction notes; M1+M2 both flagged as callouts (2/2 stretch); Q1/Q2 attributed to the + original authors; O1/O2 attributed, O2 never retro-written as "was right"; S1 pilot/portal kept + apart and cross-linked; G1–G4 in English, cited to the German files; German-function-word grep + clean; TCO honestly bare; AP-1 a single `op-` thread. Retrieval battery **9/9 correct+cited**, + findability 8/9 in band (`rb-golive` rank 2 behind the pilot page — texture, above floor). +- **PR #139 probes:** the `--force` probe under the new fresh-eyes reconcile brief found no missing + facts and no churn but **fixed three imprecise first-pass locators** (wrapped attendee lines) — + exactly the intended "more than verification" behavior with a faithful wiki still converging. + The `--reingest` probe ran delete-cleanup + fresh import in one run (report's "Re-ingested fresh" + section, manifest re-stamped, gates green) and the full battery passed on the post-reingest wiki. +- **Misses:** VCB-006 (T4 cross-page supersession — the one temporal miss, on plain wave-3 ingest + sessions untouched by the PR's diff), VCB-007 (style-quirk capture). Verdict: **PASS** on the + delete/reconcile/reingest machinery under test; T4 recorded as the run's temporal miss with its + rules-lane route. + ### 2026-07-29 werkhof (first run — the registry corpus, grading PR #134's Registry feature) - **Mode:** Mode A into the corpus workspace itself (`CITADEL_WORKSPACE=corpora/werkhof`), building