ingest: add --reingest, the delete-then-fresh-import escape hatch

[MarkusNeusinger]

What

--force runs a reconcile, and the reconcile brief deliberately keeps a source's existing treatment (genre, page structure) — it only re-verifies/updates facts. So a source first ingested under an old rulebook or a weak model is never re-thought: "all ok, nothing new" is the correct reconcile verdict even when a fresh read would organize it completely differently (e.g. into a Registry page that didn't exist when it was first ingested). The only way to get a true fresh import was the manual move-the-file-out / full ingest / move-it-back / ingest dance.

citadel ingest --reingest <paths> does that in one run: for each named tracked source, a kind="delete" cleanup session strips its previous facts from the wiki (manifest key dropped), then — deletions always run first — the same run ingests it as a brand-new source under the current model, rules, and wiki state (fresh cross-links included).

Semantics, pinned tests-first in tests/test_reingest.py:

• cleanup then plain kind="ingest" — never a reconcile; the manifest gets a fresh stamp
• an uncited tracked source plans no cleanup session (key just forgotten), straight to the fresh ingest
• a failed cleanup refuses that source's fresh session — nothing is written on top of the old facts
• a tracked repo takes the first-time repo brief over a full digest (safe only because the cleanup removed the pages a first-time brief would duplicate)
• the content-addressed transcript/pdftext caches are kept (same bytes, about to be re-read)
• requires explicit paths like --force (exit 2 without); --force/--retry refuse to combine with it
• path-scoped, so it never arms the deletion sweep
• tasks/delete.md now briefs the still-on-disk cleanup case, and the progress reporter counts the extra cleanup jobs in the overall total

Testing

• uv run pytest -q — 1313 passed, 1 skipped
• uv run ruff check . / ruff format --check . — clean
• CITADEL_WORKSPACE=corpora/beverages uv run python -m citadel lint — OK
• verify-corpus leuchtfeuer (the corpus stressing reconcile/delete/force) is running in a sandbox now, since the diff touches ingest.py + rules/; results will be posted on this PR (with the backlog-ledger entry committed) when the waves finish

🤖 Generated with Claude Code

https://claude.ai/code/session_01SPu8qnxePMUS1whYwen11L

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR introduces a new citadel ingest --reingest <paths> escape hatch that re-imports already-tracked sources from scratch by first running a kind="delete" cleanup (to strip prior facts and drop the manifest key) and then running a fresh first-time ingest in the same run.

Changes:

• Add reingest plumbing to the ingest engine: plan cleanup jobs ahead of pending sessions, enforce “cleanup must succeed before fresh ingest,” and record reingest_cleaned in the run report.
• Expose --reingest in the CLI with explicit-paths guarding and mutual exclusions with --force / --retry.
• Update progress reporting, rules brief (tasks/delete.md), changelog/docs, and add targeted tests pinning the semantics.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
citadel/ingest.py	Implements reingest planning (cleanup + fresh import), refusal logic, cache-prune exceptions, and reporting/logging updates.
citadel/cli.py	Adds --reingest flag and CLI-level validation/refusal rules; threads the flag into ingest.ingest(...).
citadel/progress.py	Extends start/progress totals to account for extra reingest cleanup jobs.
citadel/rules/tasks/delete.md	Updates the delete task brief to cover the “still on disk during reingest cleanup” case.
tests/test_reingest.py	New tests specifying and validating reingest semantics (order, refusal on cleanup failure, repo handling, cache behavior).
tests/test_cli.py	Adds CLI plumbing tests for --reingest, including refusal without paths and mutual exclusions.
tests/test_ingest_progress.py	Updates expected progress start payload to include reingest.
tests/test_ingest_parallel.py	Updates a local ingest stub signature to accept the new reingest kwarg.
CLAUDE.md	Documents --reingest behavior and constraints.
.github/copilot-instructions.md	Regenerated documentation reflecting the --reingest addition.
CHANGELOG.md	Adds an Unreleased entry documenting --reingest.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (3)

citadel/ingest.py:1197

• Same as the file path: when a reingest cleanup fails for a repo, the refusal error can overwrite the failures catalog entry and hide the original cleanup failure detail. Carrying the prior failure detail forward makes the persisted failure record more diagnosable.

            if rjob.key in reingest_keys and rjob.key in manifest_dict:
                raise RuntimeError(
                    "the delete cleanup for this reingest failed, so its previous facts are "
                    "still in the wiki; fix that failure and re-run `citadel ingest --reingest`"
                )

citadel/ingest.py:1102

• If a reingest cleanup fails, the subsequent fresh-session refusal raises a new error that overwrites the failures catalog entry, losing the original cleanup failure detail. Including the prior failure detail in the refusal message keeps citadel status actionable even after the run output has scrolled away.

This issue also appears on line 1193 of the same file.

            if rel_key in reingest_keys and rel_key in manifest_dict:
                raise RuntimeError(
                    "the delete cleanup for this reingest failed, so its previous facts are "
                    "still in the wiki; fix that failure and re-run `citadel ingest --reingest`"
                )

citadel/ingest.py:719

• The public API currently allows ingest.ingest(paths, force=True, reingest=True) even though the CLI (and the docstring) treat these flags as mutually exclusive intents. This can lead to ambiguous programmatic usage and makes the API contract diverge from the CLI contract.

    if reingest and not paths:
        # Same guard, sharper teeth: a reingest costs a cleanup session PLUS a full ingest
        # session per source, so a whole-corpus reingest by accident is twice as expensive.
        raise ValueError(
            "--reingest requires explicit paths (each source runs a delete cleanup session plus "

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

citadel/ingest.py:127

• The reingest_cleaned field comment says previous facts were stripped “by a cleanup session”, but --reingest also marks uncited sources as cleaned without running any delete session (the key is just forgotten). This comment is misleading about the behavior and invariants.

    # `--reingest` only: tracked sources whose previous facts were STRIPPED by a cleanup session
    # (manifest key dropped) ahead of the fresh plain-ingest session the same run then ran for
    # them — the deliberate re-think of an already-ingested source. The fresh session's own
    # outcome lands in `processed`/`errors` like any other source's.

[MarkusNeusinger]

verify-corpus leuchtfeuer — PASS (full wave protocol + live probes of this PR's features)

Mode A, scratch sandbox, claude:claude-sonnet-5, full run ledgered in docs/verify-corpus-backlog.md (commit 0e16828).

Waves — session kinds exactly as scripted, citadel check + lint exit 0 after every wave:

stage	sessions	result
wave 1	6× ingest	6 processed, 0 errors
wave 2	1× reconcile (charter Rev B) + 3× ingest, 5 NOOP	budget/go-live/DB superseded, not appended
wave 3	1× delete (retracted memo) + 3× ingest, 8 NOOP	D1 clean: 18,000/02:00/memo-ref all ∅; €310k survives attributed to Brandt; retraction recorded
idempotency	0 sessions	NOOP holds

Hard gates: D1 delete propagation clean, C1/C2 planted counterfactuals cited with honest [^llm] correction notes, M1+M2 both flagged as [!CONTRADICTION] callouts (2/2 stretch), Q1/Q2 attributed to the original authors, O1/O2 attributed opinions only, S1 pilot/portal kept apart, G1–G4 in English cited to the German files. Retrieval battery 9/9 correct+cited, findability 8/9 in band (rb-golive rank 2, above floor).

Probes of this PR's changes, on the live sandbox wiki:

• --force under the new fresh-eyes reconcile brief: found no missing facts, produced no churn, but fixed three imprecise first-pass locators (attendee names wrapped across two source lines) — the intended "more than verification" behavior, still converging on a faithful wiki.
• --reingest on the heavily-cited kickoff minutes: delete cleanup + fresh import in one run, report's new "Re-ingested fresh" section, manifest re-stamped fresh, gates green — and the full retrieval battery passed on the post-reingest wiki.

Misses (ledgered, neither caused by this diff): VCB-006 — T4's intra-wave decommission supersession (30 Sep → 31 Jul 2026) recorded on the portal page but the programme page still words the old date as the live plan; a cross-page supersession gap in plain wave-3 ingest sessions, routed to tasks/ingest.md. VCB-007 — style-quirk capture partial.

---

Generated by Claude Code

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.