ingest: bounded parallel ingest (--jobs N), on a context-local wiki redirect

[MarkusNeusinger]

What

The 2026-07 audit's backlog #11 — the last open item in its ranked table — shipped in the two steps the audit itself scoped: first the unblock (finding 1.2.5), then the feature.

Unblock. The per-source staging redirect assigned config.WIKI_DIR/INDEX_PATH/LOG_PATH/MANIFEST_PATH and os.environ["CITADEL_WIKI_DIR"] in place, so one process could only ever be inside one redirect. The wiki path is now read through accessors (config.wiki_dir(), index_path(), sources_index_path(), log_path(), manifest_path(), failures_path()) backed by a ContextVar — per-thread by construction, so N workers each hold their own redirect while the main thread still sees the live wiki. Unset, the accessors return the module attributes verbatim, so every read path, every CLI command, the MCP server, and every test that monkeypatches the layout behave exactly as before. The session's child processes get their staging wiki through an explicit per-spawn env (config.child_env()) instead of a global mutation.

Feature. citadel ingest --jobs N / -j (env CITADEL_JOBS, default 1 = the serial behavior, line for line) folds N sources in concurrently, each on its own staging copy — the isolation primitive was already there. Every guarantee is unmoved: one promote per source, all-or-nothing, nothing partial on the live wiki, the manifest still saved per completed source. What is shared is serialized rather than raced:

• One lock guards the only two moments that touch the live wiki — the clone and the promote. The minutes-long sessions run entirely outside it. (The clone's base hashes are taken off the fresh staging tree, so the lock covers the copy alone and the wiki is read once, not twice.)
• The promote applies only its own delta, decided against its clone: a staging copy also holds untouched copies of every page the source did not write, so judging by "differs from live" would let a promote revert a page a concurrent source had just rewritten.
• The prune is base-aware: only what this source's own base had and its staging lacks, so a concurrent source's new page is never deleted as "the agent removed it".
• A promote whose pages have moved since the clone is refused before it writes a byte (_ConcurrentChange), and that source is re-run serially at the end of its group — where it sees the winner's page and merges into it, i.e. the result a serial run would have produced. Those re-runs are surfaced on the report (Re-run serially), because they are the one place parallelism costs a session serial ingest would not have spent.
• A chunked source's resume checkpoint is measured against its own clone too — see the review section below; this is the one place where the new concurrency could actually damage a wiki, and it did until it was found.
• Report / manifest / failures bookkeeping stays on the main thread (workers only plan, stage, run, promote), so those writes are as single-threaded as before, and the progress callback is serialized so a caller's callback is never invoked concurrently. An interrupt cancels queued sources, lets in-flight ones roll themselves back, and still records work already promoted.

The default stays 1 on purpose: the real cost is cross-linking, not safety — concurrent sessions cannot see each other's new pages, so they link less richly and can duplicate a topic (curate is the designed cleanup). curate itself remains serial by construction (its clusters share pages by definition, so nearly every pair would race); refresh inherits CITADEL_JOBS, since it drives ingest.

Three defects the parallel path exposed are fixed at the root: the promote's leftover-temp sweep deleted hidden *.citadeltmp files — another writer's in-flight atomic manifest/failures save — config.atomic_write_text's temp name was unique per PID but not per thread, and the transcript log's sequence counter was bumped without a lock while its own comment promised concurrent files never collide.

Docs: CITADEL_JOBS in docs/configuration.md + the .env template, a "working off a big backlog" recipe in docs/recipes.md, CHANGELOG, CLAUDE.md (architecture + the read-the-wiki-through-the-accessor convention), and the audit itself — #11 ticked off with what shipped and what moved during implementation, finding 1.2.5 / § 1.5 "strictly serial" / § 3.3 annotated, plus a state-of-the-backlog note (all 15 ranked items done; where the still-open work lives).

What review found (worth reading before merging)

An adversarial cross-review by a second model, run specifically against the concurrency, found the one path where this could corrupt a wiki — not in the conflict detection, but where the new parallelism meets the existing resume checkpoints of a chunked source. Both manifestations were reproduced offline before the fix and are now pinned (tests/test_ingest_parallel_resume.py):

• _checkpoint_delta computed "what promoting staging onto live would do". Under --jobs the live wiki drifts between a source's clone and its checkpoint, so a page a concurrent source had created read as a deletion this source made, and one it had rewritten read as this source's change carrying stale bytes. A checkpoint is durable: a later run — even a strictly serial one — replayed that delta and pruned a fully-ingested source's page off the live wiki, with no conflict, no error and no delete session.
• resume.save compounded it by recording the base state at save time (after the concurrent promote), so replay's guard verified "live unchanged since I saved" while the delta embodied "live as of my clone" — the wrong invariant, which a concurrent promote passes straight through. A raced chunked source then re-ran with zero sessions and promoted its stale copy over the winner's page, silently violating the documented merge contract.

Both now derive from the source's own clone snapshot — exactly the rule the base-aware promote uses — so a page another source touched fails the guard and the checkpoint is dropped for a full restart.

Copilot's rounds contributed the lock-window narrowing (hash the clone, not the live wiki), the serialized progress callback, the missing resumed_note on the conflict path (now with one owner, _record_spend, so a future branch cannot forget it), and two hygiene fixes (a dropped word in the --jobs help text, a leaked file handle in a test). One Copilot finding was a false positive — future.result() on a cancelled future cannot abort the run, since a future is only cancelled after an interrupt is recorded and the first interrupt wins — but the loop now skips cancelled futures explicitly rather than relying on that reasoning.

Testing

• uv run pytest -q — 1216 passed, 1 skipped
• uv run ruff check . / uv run ruff format --check . — clean
• CITADEL_WORKSPACE=corpora/beverages uv run python -m citadel lint — OK
• 20 new offline tests (tests/test_ingest_parallel.py, tests/test_ingest_parallel_resume.py) drive the concurrency for real: the fake sessions synchronize on a threading.Barrier, so a test requiring two sources in flight at once cannot pass on a serial run. Counter-checked by forcing the driver serial — the concurrency tests fail exactly as intended. Each safety property is pinned to the failure it prevents (a concurrent source's pages surviving a promote, only-my-own-delta, the serial re-run and its failure path, the checkpoint that must not bank another source's page as a deletion, the raced chunked source that must really re-run, the interrupt keeping promoted work, the hidden-temp sweep, the never-concurrent progress callback).
• verify-corpus was not run: the claude CLI in this container is not authenticated (a live 4-source --jobs 4 run failed every session with Authentication error). That run did still exercise the parallel driver end-to-end with real subprocesses — four sessions in flight simultaneously, per-source failure recording, finalize. The rules tree is untouched (no rules_version change) and the serial path is unchanged, so wiki quality for the default configuration cannot have moved; a corpus grade of a --jobs > 1 run is worth doing on a machine with a logged-in CLI.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GrYKfDw5w6qsFZJR7ssSMt

[Copilot]

Pull request overview

This PR adds bounded parallelism to citadel ingest via --jobs N / CITADEL_JOBS, enabled by refactoring the per-source staging redirect from process-global mutation (config.WIKI_DIR / os.environ) to a context-local (ContextVar) accessor model (config.wiki_dir() and related path accessors). This preserves ingest’s staging/promote safety guarantees while allowing multiple sources to be folded concurrently in one process.

Changes:

• Introduces citadel ingest --jobs N (and CITADEL_JOBS) to run multiple source sessions concurrently, with serialized, base-aware promotion and serial re-runs on detected conflicts.
• Reworks wiki-path consumption across the codebase to go through redirect-aware accessors (config.wiki_dir(), index_path(), etc.) and passes the staging wiki path to child processes via config.child_env().
• Adds a comprehensive offline concurrency test suite and updates docs/config/templates/changelog to describe the new knob and its trade-offs.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_usage_accounting.py	Updates test to use config.wiki_dir() for redirect-aware wiki path access.
tests/test_runlock.py	Updates test to use config.wiki_dir() when referencing the live wiki.
tests/test_pagecache.py	Updates tests to use config.wiki_dir() and adjusts docstrings to match redirect-aware semantics.
tests/test_netdrive.py	Uses config.wiki_dir() in netdrive layout tests to respect redirects/out-of-repo layouts.
tests/test_ingest_staging.py	Updates staging assertions to use config.wiki_dir() and reflect redirect behavior.
tests/test_ingest_resume.py	Updates resume tests to write/remove pages under config.wiki_dir().
tests/test_ingest_reconcile_delete.py	Updates deletion reconciliation test to use config.wiki_dir().
tests/test_ingest_progress.py	Extends progress event expectations to include the jobs field.
tests/test_ingest_parallel.py	New: offline tests that prove true concurrency (Barrier) and pin safety properties under --jobs.
tests/test_ingest_core.py	Updates core ingest tests to use redirect-aware wiki path access.
tests/test_ingest_chunking.py	Updates chunking tests to assert against config.wiki_dir() in staging.
tests/test_curate.py	Updates curate test to read log via config.log_path().
tests/conftest.py	Updates fixtures and helpers to read/write wiki paths via accessors at call time.
docs/recipes.md	Documents a “working off a big backlog” recipe using citadel ingest --jobs N.
docs/configuration.md	Adds CITADEL_JOBS documentation and explains safety vs. cross-linking trade-off.
docs/audit-2026-07.md	Marks audit backlog #11 complete and records shipped design details.
CLAUDE.md	Updates architecture/conventions to require wiki access via accessors and documents bounded parallelism.
citadel/wikigit.py	Switches from config.WIKI_DIR to config.wiki_dir() for redirect-aware wiki location.
citadel/viewer/init.py	Uses config.wiki_dir() for viewer naming, source-link calculation, and output path defaults.
citadel/transcribe.py	Derives transcript cache sibling directory from config.wiki_dir().
citadel/templates/env.example	Adds commented CITADEL_JOBS section and throughput guidance.
citadel/store_core.py	Makes read/write paths redirect-aware (config.wiki_dir(), config.index_path(), etc.).
citadel/server.py	Uses config.wiki_dir() for safe on-disk presence checks.
citadel/runlock.py	Derives lock file location from config.wiki_dir() parent.
citadel/resume.py	Derives resume cache dir from config.wiki_dir() parent.
citadel/progress.py	Adds jobs to on_start and introduces on_source_retry for conflict-triggered serial re-runs.
citadel/pdftext.py	Derives PDF text cache dir from config.wiki_dir() parent.
citadel/pagecache.py	Updates documentation references to match redirect-aware wiki semantics.
citadel/manifest.py	Uses config.manifest_path() for redirect-aware manifest IO and warnings.
citadel/llm.py	Ensures LLM subprocesses receive the staging wiki via env=config.child_env() and uses config.wiki_dir() in prompt building.
citadel/linkgraph.py	Makes link rewrite and raw-reference rewrite write targets redirect-aware (config.wiki_dir()).
citadel/ingest.py	Implements --jobs concurrency, base-aware promotion, conflict detection + serial re-run, and thread-safe staging naming.
citadel/grammar.py	Uses config.wiki_dir() when resolving/normalizing internal link paths.
citadel/failures.py	Uses config.failures_path() for redirect-aware failures catalog IO.
citadel/doctor.py	Uses redirect-aware manifest/wiki paths (e.g., config.manifest_path(), config.wiki_dir()).
citadel/curate.py	Uses config.wiki_dir() when walking/reading wiki pages and selecting pages.
citadel/config.py	Introduces wiki accessors + ContextVar redirect, child_env(), CITADEL_JOBS, and thread-unique temp naming for atomic writes.
citadel/cli.py	Adds --jobs/-j flag, validates usage, and threads the value through to ingest.ingest().
citadel/catalogs.py	Writes generated catalogs/indexes using redirect-aware wiki paths.
CHANGELOG.md	Records the new bounded parallel ingest feature and the context-local redirect change.

[Copilot]

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

citadel/ingest.py:2459

• emit() can be called concurrently from worker threads under --jobs > 1 (e.g. source_start is emitted inside _attempt_source). If the progress callback has any internal state (including ConsoleProgress’s _label/spinner fields), concurrent calls can race and produce incorrect output or intermittent errors (which are then silently swallowed). Serializing progress-callback invocations with a simple lock keeps behavior deterministic while still allowing the sessions themselves to run in parallel.

    def emit(event: str, **data) -> None:
        if progress is not None:
            try:
                progress(event, data)
            except Exception:  # noqa: BLE001 - progress must never break ingest
                pass

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/test_ingest_parallel.py:413

• This test opens a file handle for ConsoleProgress(stream=...) without closing it. Using a context manager avoids leaking descriptors (and keeps pytest warning configurations from turning this into a failure).

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.