diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index ce243b5..7344851 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -34,7 +34,12 @@ session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the liv `--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`), `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source, `--force ` deliberately re-reads already-ingested sources as a reconcile — it requires -explicit paths and is refused without them, `--retry` re-runs everything STUCK without naming +explicit paths and is refused without them, `--reingest ` goes one step further: each named +tracked source is re-imported FRESH — a `kind="delete"` cleanup session strips its previous facts +(manifest entry dropped), then the same run ingests it as a brand-new source under the current +model + rules — the full re-think reconcile's keep-the-existing-treatment rule deliberately avoids +(requires explicit paths like `--force`; the two flags and `--retry` refuse to combine; a failed +cleanup blocks that source's fresh session), `--retry` re-runs everything STUCK without naming paths: every failed source still on disk plus every ingested source no wiki page cites — the zero-entry sources the run report flags as `no_pages` and `status` marks `NO PAGES` — as forced reconciles; it refuses paths/`--force` and exits 0 when nothing is stuck), @@ -211,7 +216,13 @@ addressing them through `ingest`. The flow per source: `CITADEL_MAX_SOURCE_CHARS` is folded in over several passes (all against one staging copy — see the promote bullet below). `ingest --force ` bypasses the sha short-circuit: the named sources land in pending as reconciles (a repo re-digests in full), and the manifest is re-stamped - with the current model + rules version. + with the current model + rules version. `ingest --reingest ` rides force's partitioning + but re-imports fresh instead of reconciling: a delete-cleanup job per named tracked source runs + in the deletions group (always first), dropping the manifest key so the pending session plans the + plain ingest/image/audio/pdf kind of a new key (`repo` for a tracked repo — safe only because the + cleanup already stripped the pages a first-time brief would duplicate); a failed cleanup refuses + that source's fresh session, and the cleanup never prunes the transcript/pdftext caches (the + bytes are unchanged and about to be re-read). - **Discovery is incremental and deletion-safe**: one iterative `os.scandir` walk over every `CITADEL_RAW_DIRS` root keeps each file's stat; the **manifest doubles as the scan cache** (an entry's `size`/`mtime_ns`/`ctime_ns`/`hashed_at_ns` are a skip-hint — sha256 diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ef935b..6bf5fcf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,33 @@ All notable changes to this project are documented here. The format is based on ## [Unreleased] +### Added + +- **`citadel ingest --reingest ` — re-import an already-ingested source from scratch.** + `--force` runs a reconcile, which deliberately keeps the source's existing treatment (genre, + page structure) and only verifies/updates its facts — so a source first ingested under an old + rulebook or a weak model never gets re-thought, and "all ok, nothing new" is the correct + reconcile verdict even when a fresh read would organize it completely differently. `--reingest` + is the escape hatch: for each named tracked source it first runs a `kind="delete"` cleanup + session that strips the source's previous facts from the wiki (dropping its manifest entry), + then — in the same run, deletions always run first — ingests it as a brand-new source under the + current model, rules, and wiki state (fresh cross-links included). This replaces the manual + move-the-file-out / full ingest / move-it-back / ingest dance. A failed cleanup refuses the + fresh session for that source (nothing is written on top of the old facts), tracked repos take + the first-time `repo` brief over a full digest, and the flag requires explicit paths like + `--force` (exit 2 without them; `--force` and `--retry` refuse to combine with it). The + `tasks/delete.md` brief now covers the still-on-disk cleanup case. + +### Changed + +- **Forced/refresh reconciles now re-read with fresh eyes, not just re-verify.** The + `tasks/reconcile.md` brief (driving `ingest --force` and every `citadel refresh` session) now + explicitly instructs the agent that a re-read is more than verification: mine the unchanged + source for wiki-worthy facts the first ingest missed, cross-link against today's wiki (pages + that did not exist back then may now be the right home or neighbor), and strip claims the + source never actually supported — while still keeping the existing genre treatment (structure + churn stays `--reingest`'s job). + ## [0.6.0] - 2026-07-30 ### Changed (breaking) diff --git a/CLAUDE.md b/CLAUDE.md index 00e638f..fa5adea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,7 +31,12 @@ session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the liv `--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`), `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source, `--force ` deliberately re-reads already-ingested sources as a reconcile — it requires -explicit paths and is refused without them, `--retry` re-runs everything STUCK without naming +explicit paths and is refused without them, `--reingest ` goes one step further: each named +tracked source is re-imported FRESH — a `kind="delete"` cleanup session strips its previous facts +(manifest entry dropped), then the same run ingests it as a brand-new source under the current +model + rules — the full re-think reconcile's keep-the-existing-treatment rule deliberately avoids +(requires explicit paths like `--force`; the two flags and `--retry` refuse to combine; a failed +cleanup blocks that source's fresh session), `--retry` re-runs everything STUCK without naming paths: every failed source still on disk plus every ingested source no wiki page cites — the zero-entry sources the run report flags as `no_pages` and `status` marks `NO PAGES` — as forced reconciles; it refuses paths/`--force` and exits 0 when nothing is stuck), @@ -208,7 +213,13 @@ addressing them through `ingest`. The flow per source: `CITADEL_MAX_SOURCE_CHARS` is folded in over several passes (all against one staging copy — see the promote bullet below). `ingest --force ` bypasses the sha short-circuit: the named sources land in pending as reconciles (a repo re-digests in full), and the manifest is re-stamped - with the current model + rules version. + with the current model + rules version. `ingest --reingest ` rides force's partitioning + but re-imports fresh instead of reconciling: a delete-cleanup job per named tracked source runs + in the deletions group (always first), dropping the manifest key so the pending session plans the + plain ingest/image/audio/pdf kind of a new key (`repo` for a tracked repo — safe only because the + cleanup already stripped the pages a first-time brief would duplicate); a failed cleanup refuses + that source's fresh session, and the cleanup never prunes the transcript/pdftext caches (the + bytes are unchanged and about to be re-read). - **Discovery is incremental and deletion-safe**: one iterative `os.scandir` walk over every `CITADEL_RAW_DIRS` root keeps each file's stat; the **manifest doubles as the scan cache** (an entry's `size`/`mtime_ns`/`ctime_ns`/`hashed_at_ns` are a skip-hint — sha256 diff --git a/citadel/cli.py b/citadel/cli.py index d70729a..fce4241 100644 --- a/citadel/cli.py +++ b/citadel/cli.py @@ -116,6 +116,15 @@ def build_parser() -> argparse.ArgumentParser: "without them, so a whole-corpus re-read (one agent session per source) can never " "happen by accident.", ) + p_ingest.add_argument( + "--reingest", + action="store_true", + help="Re-import the given already-ingested sources FRESH: first a delete cleanup session " + "strips each source's previous facts from the wiki (its manifest entry is dropped), then " + "the same run ingests it as a brand-new source under the current model + rules — the " + "full re-think --force's reconcile deliberately avoids (reconcile keeps the existing " + "treatment). Costs two sessions per source. Requires explicit paths, like --force.", + ) p_ingest.add_argument( "--retry", action="store_true", @@ -422,11 +431,17 @@ def cmd_ingest(args: argparse.Namespace) -> int: agent session per source, so forcing the ENTIRE corpus must never happen by accident — the flag alone is refused with exit 2, before ``ingest.ingest`` is ever called. + ``--reingest`` is force's bigger sibling — strip the named sources' previous facts (a delete + cleanup session each), then re-import them fresh in the same run. Same explicit-paths rule as + ``--force`` (exit 2 without them), and the two flags refuse each other: reconcile-in-place + and strip-and-re-import are different intents, and combining them would blur which one wins. + ``--retry`` is the no-typing complement: it computes its own bounded set — ``ingest.retry_candidates()``, the failed sources still on disk plus the ingested ones no wiki page cites — prints it, and runs those as a FORCED re-read. It therefore refuses - explicit paths and ``--force`` (exit 2): the point of the flag is that citadel picks the - set, and combining the two would blur which one wins. Nothing to retry is a clean exit 0. + explicit paths, ``--force`` and ``--reingest`` (exit 2): the point of the flag is that + citadel picks the set, and combining them would blur which one wins. Nothing to retry is a + clean exit 0. ``--jobs N`` is a usage error below 1 (exit 2, like ``--force`` without paths) rather than an exception out of the API layer; omitted, the run takes ``CITADEL_JOBS`` (default 1, serial).""" @@ -436,11 +451,21 @@ def cmd_ingest(args: argparse.Namespace) -> int: print(f"error: --jobs must be at least 1 (got {args.jobs}); 1 means the serial default.", file=sys.stderr) return 2 - if args.retry and (args.paths or args.force): + if args.retry and (args.paths or args.force or args.reingest): print( "error: --retry computes its own retry set (every failed source plus every ingested " - "source no wiki page cites) — combine no paths and no --force with it; use " - "`citadel ingest --force ` for a hand-picked re-read instead.", + "source no wiki page cites) and cannot be combined with explicit paths, --force, or " + "--reingest; use `citadel ingest --force ` (or `--reingest `) for a " + "hand-picked re-read instead.", + file=sys.stderr, + ) + return 2 + + if args.force and args.reingest: + print( + "error: --force and --reingest are different intents — --force reconciles the " + "source's facts in place (keeping its existing treatment), --reingest strips them " + "and re-imports the source fresh. Pick one.", file=sys.stderr, ) return 2 @@ -453,6 +478,15 @@ def cmd_ingest(args: argparse.Namespace) -> int: ) return 2 + if args.reingest and not args.paths: + print( + "error: --reingest requires explicit paths (each source runs a delete cleanup " + "session plus a fresh ingest session; name the files or directories to re-import, " + "e.g. `citadel ingest --reingest raw/notes.md`).", + file=sys.stderr, + ) + return 2 + paths = args.paths or None force = args.force if args.retry: @@ -486,7 +520,9 @@ def cmd_ingest(args: argparse.Namespace) -> int: # CITADEL_LLM_VERBOSE — not just the --verbose flag — also drops the spinner that would # otherwise clobber the streamed transcript. progress = ConsoleProgress(spinner=not config.LLM_VERBOSE) - report = ingest.ingest(paths, progress=progress, full_rescan=args.full_rescan, force=force, jobs=args.jobs) + report = ingest.ingest( + paths, progress=progress, full_rescan=args.full_rescan, force=force, jobs=args.jobs, reingest=args.reingest + ) print(report.render()) # Non-zero on a per-source error OR a structural problem left behind (a broken # cross-link the agent introduced) — so ingest gates the wiki's integrity in CI. diff --git a/citadel/ingest.py b/citadel/ingest.py index e310785..c52f8d1 100644 --- a/citadel/ingest.py +++ b/citadel/ingest.py @@ -121,6 +121,11 @@ class IngestReport: # rel-keys of tracked sources that VANISHED from disk (a full run only): their provenance is # reconciled out of the wiki by a cleanup agent session, then the manifest key is dropped. sources_deleted: list[str] = field(default_factory=list) + # `--reingest` only: tracked sources whose previous facts were STRIPPED by a cleanup session + # (manifest key dropped) ahead of the fresh plain-ingest session the same run then ran for + # them — the deliberate re-think of an already-ingested source. The fresh session's own + # outcome lands in `processed`/`errors` like any other source's. + reingest_cleaned: list[str] = field(default_factory=list) # `--jobs N` only: sources whose session raced a CONCURRENT source's promote over the same page # and were therefore re-run serially afterwards (the re-run's own success/failure is reported # like any other source's). Surfaced because it is the one place parallel ingest costs money a @@ -182,6 +187,9 @@ def render(self) -> str: if self.sources_deleted: lines.append("Sources removed (deleted from disk; citations reconciled out):") lines.extend(f" - {s}" for s in self.sources_deleted) + if self.reingest_cleaned: + lines.append("Re-ingested fresh (previous facts stripped first, then imported as new):") + lines.extend(f" - {s}" for s in self.reingest_cleaned) if self.resumed: lines.append("Resumed (continued from an earlier run's checkpoint):") lines.extend(f" - {r}" for r in self.resumed) @@ -606,6 +614,7 @@ def ingest( full_rescan: bool = False, force: bool = False, jobs: int | None = None, + reingest: bool = False, ) -> IngestReport: """Run one ingest. Exactly one source = one all-or-nothing agent job (a chunked source runs several ``llm.run_ingest_session`` passes inside that one job). @@ -638,6 +647,20 @@ def ingest( CLI pre-empts it with the same message and a friendly exit 2), and a path-scoped run never sweeps deletions (``swept_roots=None`` below). + ``reingest`` (the ``--reingest`` flag) goes one deliberate step past ``force``: instead of + reconciling the named tracked sources around their existing treatment, each is re-imported + FRESH — first a ``kind="delete"`` cleanup session strips its previous facts from the wiki and + its manifest key is dropped, then (same run, deletions always run first) the source lands in + pending as a brand-new key and runs a plain ``kind="ingest"`` session (``kind="repo"`` for a + tracked repo — the cleanup already removed the pages a first-time brief would otherwise + duplicate). That is the escape hatch from reconcile's keep-the-existing-genre-treatment rule: + the source is re-thought from scratch against the CURRENT wiki and rules (e.g. after a model + upgrade or a new genre), at the cost of a cleanup session plus a full ingest session per + source. Like ``force`` it requires explicit paths (same ValueError here, same CLI exit 2) and + implies force's partitioning (sha short-circuit and dedup drop bypassed). If the cleanup + session fails, the fresh ingest for that source is refused (a per-source failure, retried by + re-running ``--reingest``) — new pages are never written on top of the old facts. + Deletion detection is guarded (operational safety over thoroughness): candidates come from the walked-seen-set diff, each positively confirmed with ``.exists()``; any walk error aborts the entire sweep for the run; an unreachable root @@ -689,6 +712,14 @@ def ingest( "--force requires explicit paths (a forced re-read runs one agent session per " "source; name the files or directories to force, e.g. `citadel ingest --force raw/notes.md`)." ) + if reingest and not paths: + # Same guard, sharper teeth: a reingest costs a cleanup session PLUS a full ingest + # session per source, so a whole-corpus reingest by accident is twice as expensive. + raise ValueError( + "--reingest requires explicit paths (each source runs a delete cleanup session plus " + "a fresh ingest session; name the files or directories to re-import, e.g. " + "`citadel ingest --reingest raw/notes.md`)." + ) # ONE mutating run per workspace: the staging sweep, promote's prune, and the manifest/ # failures saves are all destructive under concurrency (see runlock's module docstring). @@ -698,12 +729,19 @@ def ingest( # Same place, same reason: under the exclusive lock, leftovers on disk belong to dead runs. # Age-based only — a checkpoint's own guards decide whether it is USABLE (see resume.sweep). resume.sweep() - return _ingest_run(paths, progress, full_rescan=full_rescan, force=force, jobs=workers) + return _ingest_run(paths, progress, full_rescan=full_rescan, force=force, jobs=workers, reingest=reingest) -def _ingest_run(paths: list[str] | None, progress, *, full_rescan: bool, force: bool, jobs: int = 1) -> IngestReport: +def _ingest_run( + paths: list[str] | None, progress, *, full_rescan: bool, force: bool, jobs: int = 1, reingest: bool = False +) -> IngestReport: """The body of :func:`ingest`, running under the exclusive workspace run lock.""" + # A reingest rides force's partitioning wholesale (sha short-circuit and dedup drop bypassed, + # stat cache distrusted for the named paths); what it changes beyond force — the cleanup jobs + # and the fresh session kinds — is keyed off `reingest`/`reingest_keys` below. + force = force or reingest + # `--jobs N` emits from WORKER threads (a source's start/done event fires where the work # happens), so the callback — which is whatever the caller passed — is serialized here. That # keeps "your progress callback is never invoked concurrently" a property of the API rather than @@ -863,6 +901,22 @@ def emit(event: str, **data) -> None: # one not yet tracked is brand new. Captured before the manifest is mutated below. pending_keys = {manifest.rel_key(p) for p in scan.pending} changed_keys = pending_keys & set(manifest_dict) + # --reingest: every named TRACKED source is re-imported fresh instead of reconciled — a + # delete-cleanup job (group 1, always before pending) strips its previous facts and drops its + # manifest key, and emptying changed_keys here makes its pending session plan the plain + # ingest/image/audio/pdf kind of a brand-new key. Tracked repos join reingest_keys below. + reingest_keys: set[str] = set() + if reingest: + reingest_keys = set(changed_keys) + changed_keys = set() + # A tracked repo (force gave it kind="repo-reconcile") is re-imported fresh the same way: + # cleanup first, then the FIRST-TIME brief over a full digest. kind="repo" is safe here + # precisely because the cleanup precedes it — the pages a first-time brief would + # otherwise duplicate are already stripped (contrast _partition_repos' force rule). + repo_pending = [ + _RepoJob(path=r.path, key=r.key, kind="repo", old_commit=None) if r.old_commit else r for r in repo_pending + ] + reingest_keys.update(r.key for r in repo_pending if r.key in manifest_dict) # --- Reorganized sources: a file that only MOVED (or is a byte-for-byte duplicate) is # recognized and NOT re-ingested. For a real move (the old path is gone) repoint the wiki's @@ -1000,6 +1054,9 @@ def emit(event: str, **data) -> None: deleted=len(deleted_sources), repos=len(repo_pending), jobs=jobs, + # Each reingest source runs a cleanup JOB on top of its pending session — counted so the + # overall progress total matches the jobs that will actually run. + reingest=len(reingest_keys), ) # --- The per-source jobs (the SourceJob loop): DELETION cleanups first, then files, then repos, @@ -1034,6 +1091,15 @@ def _file_job(src: Path) -> _SourceJob: office = scan.office_text.get(src) def build() -> tuple[list, list[str], "_Resume | None"]: + # A reingest source may only run its fresh session on a wiki its cleanup actually + # cleaned: the cleanup job (group 1) pops the manifest key on success, so a key still + # tracked here means that cleanup failed — refuse rather than write new pages on top + # of the old facts (a per-source prepare failure; re-running --reingest retries both). + if rel_key in reingest_keys and rel_key in manifest_dict: + raise RuntimeError( + "the delete cleanup for this reingest failed, so its previous facts are " + "still in the wiki; fix that failure and re-run `citadel ingest --reingest`" + ) # Plan the pass(es): an Office source materializes its extracted text to a temp .md # the agent reads; an audio/video source is transcribed HERE through the whisper seam # (content-addressed cache; a raise is a retryable per-source prepare_error, and the @@ -1122,6 +1188,13 @@ def done(usage: llm.SessionUsage | None) -> None: # run-level ``force``), which re-digests in FULL (see _partition_repos). def _repo_job(rjob: _RepoJob) -> _SourceJob: def build() -> tuple[list, list[str], "_Resume | None"]: + # Same cleanup post-condition as _file_job's: a reingested repo whose cleanup failed + # (key still tracked) must not run the first-time brief on top of its old pages. + if rjob.key in reingest_keys and rjob.key in manifest_dict: + raise RuntimeError( + "the delete cleanup for this reingest failed, so its previous facts are " + "still in the wiki; fix that failure and re-run `citadel ingest --reingest`" + ) only: list[str] | None = None change_summary: str | None = None if rjob.kind == "repo-reconcile" and rjob.old_commit and not force: @@ -1166,8 +1239,10 @@ def done(usage: llm.SessionUsage | None) -> None: # cites it, run a `kind="delete"` cleanup session that strips that provenance, gated by a # post-condition that the wiki no longer references it (else the whole cleanup is rolled back # and retried next full run — the manifest key is dropped only on success). A deletion that - # nothing cites plans NO session and just loses its manifest key. - def _delete_job(key: str) -> _SourceJob: + # nothing cites plans NO session and just loses its manifest key. With ``reingest_cleanup`` + # the SAME job strips a still-on-disk source ahead of its fresh re-import (`--reingest`); the + # bookkeeping differs only where the "source is gone" premise does not hold. + def _delete_job(key: str, reingest_cleanup: bool = False) -> _SourceJob: def build() -> tuple[list, list[str], "_Resume | None"]: if not store.find_raw_references(key): return [], [], None # nothing cites it: no cleanup session, just forget it below @@ -1186,7 +1261,14 @@ def done(_usage: llm.SessionUsage | None) -> None: # this must NOT gate on the extension: a PDF routes by %PDF- MAGIC (is_pdf_file), so it # can be cached under any name, and an ext gate would orphan its plaintext extraction. del_sha = manifest.entry_sha(entry) if entry is not None else None - if entry is not None and not _sha_shared_by_other_entry(manifest_dict, del_sha, key): + # ... except on a reingest cleanup: the source still exists with the SAME bytes, and + # the fresh session moments away re-reads exactly the cached transcript/extraction — + # pruning here would throw away work the run is about to re-buy. + if ( + not reingest_cleanup + and entry is not None + and not _sha_shared_by_other_entry(manifest_dict, del_sha, key) + ): transcribe.prune_cached(del_sha) pdftext.prune_cached(del_sha) # A resume checkpoint is KEY-addressed (not content-addressed like those two caches), @@ -1196,7 +1278,10 @@ def done(_usage: llm.SessionUsage | None) -> None: manifest_dict.pop(key, None) failures.clear(failures_dict, key) manifest.save(manifest_dict) - report.sources_deleted.append(key) + if reingest_cleanup: + report.reingest_cleaned.append(key) + else: + report.sources_deleted.append(key) return _SourceJob( key=key, @@ -1216,7 +1301,10 @@ def done(_usage: llm.SessionUsage | None) -> None: # a later run with nothing pending would never rebuild the derived files. pending_interrupt: BaseException | None = None groups = ( - [_delete_job(key) for key in deleted_sources], + # Reingest cleanups ride the deletion group, so the always-first ordering above holds for + # them too: a source's old facts are stripped before ANY pending session runs. + [_delete_job(key) for key in deleted_sources] + + [_delete_job(key, reingest_cleanup=True) for key in sorted(reingest_keys)], [_file_job(src) for src in scan.pending], [_repo_job(r) for r in repo_pending], ) @@ -1238,6 +1326,7 @@ def done(_usage: llm.SessionUsage | None) -> None: or report.moved or report.unreadable or report.sources_deleted + or report.reingest_cleaned or repointed or failures_changed or pruned_ignored @@ -1281,6 +1370,11 @@ def done(_usage: llm.SessionUsage | None) -> None: f"raw source {key} was deleted from disk; reconciled its citations out of the " "wiki and dropped it from the manifest" ) + for key in report.reingest_cleaned: + store.append_log( + f"reingest {key}: stripped its previous facts ahead of the fresh import " + "(deliberate re-read as a new source)" + ) # The wiki-history commit comes LAST, after the log/index/failures writes above, so one # commit captures the run's complete state. Best-effort by contract: the wiki is already # promoted, so a git problem is a report note, never a failed run. diff --git a/citadel/progress.py b/citadel/progress.py index 82136fc..25f4d9b 100644 --- a/citadel/progress.py +++ b/citadel/progress.py @@ -171,6 +171,7 @@ def on_start( deleted: int = 0, repos: int = 0, jobs: int = 1, + reingest: int = 0, ) -> None: bits = [] if skipped: @@ -179,6 +180,10 @@ def on_start( bits.append(f"{moved} reorganized") if unreadable: bits.append(f"{unreadable} unreadable") + if reingest: + # --reingest: these sources run a cleanup job on top of their pending session; the + # extra jobs are folded into the overall total below. + bits.append(f"{reingest} re-imported fresh (old facts stripped first)") if pending == 0 and repos == 0 and deleted == 0: extra = f" ({', '.join(bits)})" if bits else "" self._print(Text(f"Nothing to ingest{extra}.", style="dim")) @@ -201,7 +206,7 @@ def on_start( headline = Text(f"Ingesting {' + '.join(counts)}{extra}", style="bold") if jobs > 1: headline.append(f" [{jobs} at a time]", style="dim") - self._begin(pending + repos + deleted, headline) + self._begin(pending + repos + deleted + reingest, headline) def on_source_start(self, index: int, total: int, source: str) -> None: label = f"[{index}/{total}] {config.display_key(source)}" diff --git a/citadel/rules/tasks/delete.md b/citadel/rules/tasks/delete.md index 48ba02e..a5e9cce 100644 --- a/citadel/rules/tasks/delete.md +++ b/citadel/rules/tasks/delete.md @@ -1,7 +1,9 @@ # delete — strip a removed source's provenance -The source was **removed from disk** and no longer exists. Do **NOT** try to open it. Remove the -provenance that depended on it: +The source was **removed from disk** and no longer exists — or is being **deliberately +re-imported from scratch** (a reingest: the file may still sit on disk, but its previous facts +must go so the fresh import that follows starts clean). Either way, do **NOT** try to open the +source. Remove the provenance that depended on it: 1. Search the wiki (Grep/Glob/Read) for every page that cites it: a `resource:` frontmatter field naming it, or a `[^sN]` footnote whose `## Sources` definition links to it (e.g. diff --git a/citadel/rules/tasks/reconcile.md b/citadel/rules/tasks/reconcile.md index a13a738..04fdfd4 100644 --- a/citadel/rules/tasks/reconcile.md +++ b/citadel/rules/tasks/reconcile.md @@ -1,9 +1,22 @@ # reconcile — re-fold a source the wiki already cites The wiki already holds facts derived from this source, and the source **changed since it was -last ingested — or is being deliberately re-read** (a forced run: the source may be unchanged — -re-verify the wiki's facts against it and apply the **current** rules, which may themselves have -changed since the first ingest). +last ingested — or is being deliberately re-read** (a forced run or a `citadel refresh`: the +source may be unchanged — re-verify the wiki's facts against it and apply the **current** rules, +which may themselves have changed since the first ingest). + +**A re-read is more than verification.** The wiki, the rules, and the model have all moved since +this source was first ingested, so read the WHOLE source again with fresh eyes: + +- **Mine it for wiki-worthy facts the first ingest missed.** A fact the wiki does not carry is + genuinely new even when the source is unchanged — the first pass may simply have overlooked or + under-valued it. Add it, cited as usual. +- **Cross-link against today's wiki.** Pages that did not exist at first ingest may now be the + right home or neighbor for this source's facts — search first (`core.md`), then link or merge + instead of leaving the old pages isolated. +- **Remove what the source never supported.** The same rule as a fact the changed file no longer + supports: a claim cited to this source that the source does not actually state (a first-pass + misreading) loses this source's marker — and the sentence, if no other source remains. **Reconcile — do not merely append**, otherwise a corrected number leaves the stale one standing next to the new one: diff --git a/docs/verify-corpus-backlog.md b/docs/verify-corpus-backlog.md index 3f591c0..4c213f7 100644 --- a/docs/verify-corpus-backlog.md +++ b/docs/verify-corpus-backlog.md @@ -46,6 +46,8 @@ Row format: | VCB-002 | 2026-07-25 | 2026-07-25 | beverages | every factual sentence carries a footnote | wiki-generation | **haiku-only.** One uncited sentence survived the agent's own `citadel check` because it is inside a sentence the parser reads as cited: `concepts/coffee.md` — "The plant and its cultivation are dominated by two species: [Arabica and Robusta…". Lint's advisory `missing citations` caught it; `check` did not. Route: the `tasks/ingest.md` self-check step (re-read every paragraph for a trailing footnote), and worth asking whether this shape should be a `check` error rather than a lint advisory. | open | | VCB-003 | 2026-07-25 | 2026-07-25 | beverages | `[^llmN]` is for model-supplied facts only, never a shortcut | wiki-generation | **haiku-only.** 4 pages carry model-supplied facts (`concepts/caffeine.md`, `concepts/tea.md`, `objects/aurora-midnight.md`, `organizations/thornbury-lin.md`) where the sonnet showcase of the same corpus has none — the corpus is fully covered by its raw sources, so an `[^llmN]` here is a weaker model reaching for the LLM lane instead of the source. Not a provenance violation (the lane is honest and lint surfaces it), so a soft miss. Route: `citadel/rules/core.md`'s `[^llmN]` section — make "prefer dropping the claim over sourcing it to yourself" explicit. | open | | VCB-004 | 2026-07-25 | 2026-07-25 | beverages | dense cross-linking between related pages | wiki-generation | **Unattributed between model and `--jobs`.** 36 un-linked mentions where the committed sonnet showcase lints 0 (e.g. `concepts/tea.md → concepts/coffee.md`, `organizations/caffe-aurora.md → concepts/cold-brew.md`). Two candidate causes and this run cannot separate them: haiku links less densely, and `--jobs 4` means concurrent sessions cannot see each other's new pages (the documented, accepted cost of parallelism — `curate` is its designed cleanup). Route: re-run beverages on haiku with `--jobs 1` and diff this counter; only if serial haiku is also high is this a rules-lane miss rather than a parallelism trade-off. | open | +| VCB-006 | 2026-07-31 | 2026-07-31 | leuchtfeuer | T4 intra-wave temporal supersession (decommission 30 Sep → 31 Jul 2026) | wiki-generation | The 8 Apr 2026 portal-minutes session recorded the brought-forward KOMET decommission (31 Jul 2026) on `projects/seagull-customer-portal.md` but left `projects/projekt-leuchtfeuer.md` still stating "switched off on 30 September 2026" as the live plan (cited to the 20 Mar go-live mail, ingested one session earlier in the same wave). Cross-PAGE supersession: the new source's session updated the page it landed on but did not hunt other pages asserting the now-old value. Route: `tasks/ingest.md` — when a new source supersedes a dated value, grep the whole wiki for the superseded value and update every page presenting it as current. | open | +| VCB-007 | 2026-07-31 | 2026-07-31 | leuchtfeuer | style-profile quirks (§I, `CITADEL_STYLE_PROFILES=1`) | wiki-generation | Opinions all correctly attributed (O1/O2/O3 pass), but the persons pages carry no style-quirk entries — Vogelsang's nautical metaphors / "Fair winds" sign-off and Duszek's "—MD" are absent; the POD abbreviation (portal minutes) also has no expansion/page where WMS and MDE got both. Route: `genres/first-person.md` + the style-profile brief — spell out that sign-offs and recurring metaphors belong on the persons page when profiling is ON. | open | ## Resolved @@ -59,6 +61,32 @@ Row format: verdict, and the `VCB-` ids the run touched, or `misses: none`. This `## Runs` heading stays singular; runs nest under it.)* +### 2026-07-31 leuchtfeuer (grading PR #139 — `--reingest`, the reconcile fresh-eyes brief, the delete-brief reingest note) + +- **Mode:** Mode A in a scratch sandbox, full wave protocol (`stages/initial` → wave2 → wave3 with + the memo delete), then the NOOP re-run, a `--force` probe, and a live `--reingest` probe of the + PR's own feature on the heavily-cited kickoff minutes. Retrieval-first grade afterwards. +- **Model:** `claude:claude-sonnet-5` (`CITADEL_INGEST_MODEL=sonnet`) · serial · wave 1 paid the + per-source hermetic-auth retry (container auth shape), waves 2-3 ran with `CITADEL_HERMETIC=0`. +- **Result:** `check` + `lint` exit 0 after every wave AND after both probes. Wave kinds exactly as + scripted (6 ingest / 1 reconcile + 3 ingest + 5 NOOP / 1 delete + 3 ingest + 8 NOOP); idempotency + re-run zero sessions. **D1 delete propagation clean** (18k/02:00/memo-ref ∅ on all pages; €310k + survives attributed to Brandt; retraction recorded). C1/C2 planted values cited with honest + `[^llm]` correction notes; M1+M2 both flagged as callouts (2/2 stretch); Q1/Q2 attributed to the + original authors; O1/O2 attributed, O2 never retro-written as "was right"; S1 pilot/portal kept + apart and cross-linked; G1–G4 in English, cited to the German files; German-function-word grep + clean; TCO honestly bare; AP-1 a single `op-` thread. Retrieval battery **9/9 correct+cited**, + findability 8/9 in band (`rb-golive` rank 2 behind the pilot page — texture, above floor). +- **PR #139 probes:** the `--force` probe under the new fresh-eyes reconcile brief found no missing + facts and no churn but **fixed three imprecise first-pass locators** (wrapped attendee lines) — + exactly the intended "more than verification" behavior with a faithful wiki still converging. + The `--reingest` probe ran delete-cleanup + fresh import in one run (report's "Re-ingested fresh" + section, manifest re-stamped, gates green) and the full battery passed on the post-reingest wiki. +- **Misses:** VCB-006 (T4 cross-page supersession — the one temporal miss, on plain wave-3 ingest + sessions untouched by the PR's diff), VCB-007 (style-quirk capture). Verdict: **PASS** on the + delete/reconcile/reingest machinery under test; T4 recorded as the run's temporal miss with its + rules-lane route. + ### 2026-07-29 werkhof (first run — the registry corpus, grading PR #134's Registry feature) - **Mode:** Mode A into the corpus workspace itself (`CITADEL_WORKSPACE=corpora/werkhof`), building diff --git a/tests/test_cli.py b/tests/test_cli.py index 98467e7..fd3e489 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -201,6 +201,32 @@ def test_ingest_force_without_paths_is_rejected(ingest_spy): assert not ingest_spy.called +def test_ingest_reingest_flag_reaches_ingest(ingest_spy): + """``--reingest`` hands ``reingest=True`` through to ``ingest.ingest`` alongside the explicit + paths; without the flag the kwarg defaults to False.""" + assert cli.main(["ingest", "--quiet", "raw/a.md"]) == 0 + assert ingest_spy.kwargs["reingest"] is False + assert cli.main(["ingest", "--quiet", "--reingest", "raw/a.md"]) == 0 + assert ingest_spy.kwargs["reingest"] is True + assert ingest_spy.paths == ["raw/a.md"] + + +def test_ingest_reingest_without_paths_is_rejected(ingest_spy, capsys): + """Same explicit-paths rule as ``--force``, with sharper teeth (two sessions per source): + ``--reingest`` alone is refused with exit 2 before ``ingest.ingest`` is reached.""" + assert cli.main(["ingest", "--quiet", "--reingest"]) == 2 + assert not ingest_spy.called + assert "--reingest requires explicit paths" in capsys.readouterr().err + + +def test_ingest_reingest_refuses_force_and_retry(ingest_spy, capsys): + """``--reingest`` and ``--force`` are different intents (strip-and-re-import vs reconcile in + place) and refuse each other; ``--retry`` computes its own set and refuses both.""" + assert cli.main(["ingest", "--quiet", "--force", "--reingest", "raw/a.md"]) == 2 + assert cli.main(["ingest", "--quiet", "--retry", "--reingest"]) == 2 + assert not ingest_spy.called + + def test_ingest_source_error_exits_1(ingest_spy): ingest_spy.report = _report(errors=["raw/a.md: agent failed"]) assert cli.main(["ingest", "--quiet"]) == 1 diff --git a/tests/test_ingest_parallel.py b/tests/test_ingest_parallel.py index cf350fb..e243dc0 100644 --- a/tests/test_ingest_parallel.py +++ b/tests/test_ingest_parallel.py @@ -393,7 +393,7 @@ def test_jobs_below_one_is_refused(tmp_citadel, fake_agent): def test_cli_jobs_is_threaded_through(tmp_citadel, monkeypatch, capsys): seen: dict = {} - def fake_ingest(paths=None, progress=None, full_rescan=False, force=False, jobs=None): + def fake_ingest(paths=None, progress=None, full_rescan=False, force=False, jobs=None, reingest=False): seen["jobs"] = jobs return ingest.IngestReport([], [], [], []) diff --git a/tests/test_ingest_progress.py b/tests/test_ingest_progress.py index f68a8ac..d508f66 100644 --- a/tests/test_ingest_progress.py +++ b/tests/test_ingest_progress.py @@ -23,7 +23,16 @@ def test_ingest_emits_progress_events(tmp_citadel, fake_agent, transformer_page) for expected in ("source_start", "source_done", "finalize", "done"): assert expected in names, f"missing event: {expected}" start = next(d for e, d in events if e == "start") - assert start == {"pending": 1, "skipped": 0, "moved": 0, "unreadable": 0, "deleted": 0, "repos": 0, "jobs": 1} + assert start == { + "pending": 1, + "skipped": 0, + "moved": 0, + "unreadable": 0, + "deleted": 0, + "repos": 0, + "jobs": 1, + "reingest": 0, + } done = next(d for e, d in events if e == "source_done") assert done["source"] == "raw/notes.md" assert done["index"] == 1 and done["total"] == 1 @@ -83,6 +92,8 @@ def fake(rel_key, kind="ingest", read_path=None, segment=None): # `--jobs N`: the worker count the run was given, so the console can show how many sources # are in flight at once (rich gives each its own spinner row). 1 is the serial default. "jobs": 1, + # `--reingest` cleanup jobs on top of the pending sessions — 0 on every non-reingest run. + "reingest": 0, } # Deletions first, then files, then repos — and per-GROUP counters restarting at 1/1. assert [d["source"] for e, d in events if e == "source_start"] == ["raw/gone.md", "raw/note.md", "raw/svc"] diff --git a/tests/test_reingest.py b/tests/test_reingest.py new file mode 100644 index 0000000..11370c4 --- /dev/null +++ b/tests/test_reingest.py @@ -0,0 +1,226 @@ +"""``ingest --reingest`` (offline): strip a tracked source's previous facts, then re-import it +FRESH in the same run — the deliberate full re-think that ``--force``'s reconcile (which keeps the +existing treatment) cannot deliver. The decided semantics, pinned tests-first: + +- a reingested tracked FILE runs exactly two jobs in order: a ``kind="delete"`` cleanup (its + manifest key dropped on success) and then a plain ``kind="ingest"`` session — never a + reconcile — with the manifest re-stamped fresh under the CURRENT model + rules_version; +- a reingested tracked source that NO page cites plans no cleanup session (the key is just + forgotten) and goes straight to the fresh ingest; +- a FAILED cleanup blocks the fresh session for that source: nothing is written on top of the + old facts, the live wiki and manifest entry stay as they were, and the failure is recorded; +- a reingested REPO runs the cleanup and then the FIRST-TIME ``kind="repo"`` brief over a FULL + digest (safe exactly because the cleanup removed the pages a first-time brief would duplicate); +- ``reingest`` requires explicit paths (API ValueError; the CLI pre-empts with exit 2) and a + path-scoped reingest run never sweeps deletions; +- the transcript/extraction caches are NOT pruned by a reingest cleanup (the source still exists + with the same bytes, and the fresh session is about to re-read exactly that cached text). + +``llm.run_ingest_session`` is replaced by ``fake_agent``. +""" + +from __future__ import annotations + +import pytest +from conftest import REAL_RULES_DIR, delete_citing_pages + +from citadel import config, failures, ingest, manifest, pdftext, repo + + +def _reingest_side_effect(cite_page, rel_path="misc/note-v2.md", fact="A rethought fact."): + """The canonical two-kind fake session: the ``delete`` pass strips every citing page from the + staging copy, the fresh ``ingest`` pass writes the new page.""" + + def _run(rel_key, kind="ingest", **kw): + if kind == "delete": + delete_citing_pages(rel_key) + else: + cite_page(rel_path, rel_key, fact) + + return _run + + +# -------------------------------------------------------------------------------------------- +# reingest on FILE sources +# -------------------------------------------------------------------------------------------- + + +def test_reingest_runs_cleanup_then_plain_ingest(tmp_citadel, fake_agent, cite_page, monkeypatch): + """The core semantics: cleanup first (``kind="delete"``), then a plain ``kind="ingest"`` — + NEVER the ``reconcile`` a plain ``--force`` runs — and the manifest entry is a FRESH stamp + under the current model. Control on the same corpus first: ``--force`` still reconciles.""" + wiki, raw = tmp_citadel.wiki, tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + monkeypatch.setattr(config, "ingest_model_label", lambda: "fake:model-a") + agent = fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/note.md", rel_key, "A weak-model fact.")) + ingest.ingest() + assert agent.calls == [("raw/notes.md", "ingest")] + + agent.reset() + ingest.ingest([str(raw / "notes.md")], force=True) + assert agent.calls == [("raw/notes.md", "reconcile")] # control: force stays a reconcile + + monkeypatch.setattr(config, "ingest_model_label", lambda: "fake:model-b") + agent = fake_agent(side_effect=_reingest_side_effect(cite_page)) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert agent.calls == [("raw/notes.md", "delete"), ("raw/notes.md", "ingest")] + assert report.reingest_cleaned == ["raw/notes.md"] + assert report.sources_deleted == [] # a reingest is not a deletion + assert report.processed == ["raw/notes.md"] + assert not report.errors + assert not (wiki / "misc" / "note.md").exists() # the old treatment is gone... + assert (wiki / "misc" / "note-v2.md").exists() # ...replaced by the fresh import + entry = tmp_citadel.read_manifest()["raw/notes.md"] + assert entry["model"] == "fake:model-b" # a brand-new stamp, not a carried one + + +def test_reingest_without_paths_refused_at_the_api_layer(tmp_citadel, fake_agent): + """Like force, but twice as expensive per source (cleanup + fresh ingest): ``reingest=True`` + with no paths raises ValueError before any work, no matter the caller.""" + agent = fake_agent() + with pytest.raises(ValueError, match="--reingest requires explicit paths"): + ingest.ingest(reingest=True) + assert agent.count == 0 + + +def test_reingest_uncited_source_skips_cleanup_session(tmp_citadel, fake_agent, cite_page): + """A tracked source no page cites has nothing to strip: the cleanup job plans ZERO sessions + (the key is just forgotten) and the fresh ingest runs directly — one paid session, not two.""" + raw = tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + agent = fake_agent() # writes nothing: the source ends up ingested but uncited (no_pages) + assert ingest.ingest().no_pages == ["raw/notes.md"] + + agent = fake_agent(side_effect=_reingest_side_effect(cite_page)) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert agent.calls == [("raw/notes.md", "ingest")] # no delete session was paid for + assert report.reingest_cleaned == ["raw/notes.md"] # the key was still dropped + re-imported + assert report.processed == ["raw/notes.md"] + assert not report.errors + + +def test_reingest_cleanup_failure_blocks_the_fresh_ingest(tmp_citadel, fake_agent, cite_page): + """All-or-nothing per source, across BOTH jobs: a failed cleanup leaves the live wiki and the + manifest entry untouched, and the fresh session is REFUSED (never run on top of the old + facts) — recorded as a failure, so re-running ``--reingest`` retries the pair.""" + wiki, raw = tmp_citadel.wiki, tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + agent = fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/note.md", rel_key, "A fact.")) + ingest.ingest() + page = wiki / "misc" / "note.md" + before_bytes = page.read_bytes() + entry_before = dict(tmp_citadel.read_manifest()["raw/notes.md"]) + + def failing(rel_key, kind="ingest", **kw): + if kind == "delete": + raise RuntimeError("cleanup boom") + cite_page("misc/note-v2.md", rel_key, "Must never land.") + + agent = fake_agent(side_effect=failing) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert agent.calls == [("raw/notes.md", "delete")] # the fresh session never ran + assert report.reingest_cleaned == [] + assert report.processed == [] + assert any("cleanup boom" in e for e in report.errors) + assert any("reingest" in e for e in report.errors) # the refusal names the why + assert page.read_bytes() == before_bytes # live wiki untouched + assert not (wiki / "misc" / "note-v2.md").exists() + assert tmp_citadel.read_manifest()["raw/notes.md"] == entry_before # entry kept, not dropped + assert "raw/notes.md" in failures.load() + + +def test_reingest_path_run_never_sweeps_deletions(tmp_citadel, fake_agent, cite_page, seed_cited_deleted_source): + """A reingest run is path-scoped by construction — the rest of the manifest must never be + read as deletion candidates (the same ``swept_roots=None`` rule as ``--force``).""" + wiki, raw = tmp_citadel.wiki, tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + agent = fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/note.md", rel_key, "A fact.")) + ingest.ingest() + seed_cited_deleted_source() + + agent = fake_agent(side_effect=_reingest_side_effect(cite_page)) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert agent.calls == [("raw/notes.md", "delete"), ("raw/notes.md", "ingest")] + assert report.sources_deleted == [] + assert (wiki / "concepts" / "topic.md").exists() # the vanished source's page untouched + assert "raw/gone.md" in tmp_citadel.read_manifest() # still tracked; swept only by a full run + + +def test_reingest_cleanup_keeps_the_extraction_cache(tmp_citadel, fake_agent, cite_page, monkeypatch): + """The cleanup must NOT prune the content-addressed transcript/extraction caches: the source + still exists with the same bytes, and the fresh session moments later re-reads exactly that + cached text (contrast a real deletion, which prunes).""" + raw = tmp_citadel.raw + (raw / "notes.md").write_text("stable content\n", encoding="utf-8") + fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/note.md", rel_key, "A fact.")) + ingest.ingest() + + pruned: list[str] = [] + monkeypatch.setattr(pdftext, "prune_cached", lambda sha: pruned.append(sha)) + fake_agent(side_effect=_reingest_side_effect(cite_page)) + report = ingest.ingest([str(raw / "notes.md")], reingest=True) + + assert report.processed == ["raw/notes.md"] + assert pruned == [] # the cache the fresh session would re-buy stays + + +# -------------------------------------------------------------------------------------------- +# reingest on REPO sources +# -------------------------------------------------------------------------------------------- + + +def test_reingest_repo_runs_cleanup_then_first_time_brief_over_full_digest( + repo_wiki, fake_agent, make_repo, cite_page, monkeypatch +): + """A reingested tracked repo runs the cleanup and then ``kind="repo"`` — the FIRST-TIME brief, + which force deliberately never uses because it would duplicate pages; here the cleanup + removed them first — over a FULL digest (``only=None``, no change summary).""" + raw = repo_wiki.raw + agent = fake_agent(side_effect=lambda rel_key, **kw: cite_page("misc/svc.md", rel_key, "A repo fact.")) + make_repo(raw, "svc", {"README.md": "# Svc\n", "app.py": "x\n"}) + ingest.ingest() + assert agent.calls == [("raw/svc", "repo")] + + captured: dict = {} + real_build = repo.build_digest + + def capturing_build(path, key, *, only=None, change_summary=None, **kw): + captured["only"] = only + captured["change_summary"] = change_summary + return real_build(path, key, only=only, change_summary=change_summary, **kw) + + monkeypatch.setattr(repo, "build_digest", capturing_build) + monkeypatch.setattr(repo, "changed_files", lambda *a, **k: ["README.md"]) # a diff must not be consulted + + def rethink(rel_key, kind="repo", **kw): + if kind == "delete": + delete_citing_pages(rel_key) + else: + cite_page("misc/svc-v2.md", rel_key, "A rethought repo fact.") + + agent = fake_agent(side_effect=rethink) + report = ingest.ingest([str(raw / "svc")], reingest=True) + + assert agent.calls == [("raw/svc", "delete"), ("raw/svc", "repo")] # never repo-reconcile + assert report.reingest_cleaned == ["raw/svc"] + assert report.processed == ["raw/svc"] + assert captured == {"only": None, "change_summary": None} # full digest, no change summary + assert manifest.is_repo_entry(repo_wiki.read_manifest()["raw/svc"]) # re-tracked as a repo + + +# -------------------------------------------------------------------------------------------- +# the rules layer: the delete brief must cover the still-on-disk reingest cleanup +# -------------------------------------------------------------------------------------------- + + +def test_delete_brief_carries_the_reingest_note(): + """``tasks/delete.md`` briefs the reingest cleanup too: the file may still exist on disk, and + the agent must strip the provenance without opening it — otherwise a cleanup session that + finds the file present second-guesses its task.""" + text = (REAL_RULES_DIR / "tasks" / "delete.md").read_text(encoding="utf-8").lower() + assert "reingest" in text + assert "still" in text and "disk" in text