ingest: exclude the wiki from discovery, add a size ceiling, fix the UNC subprocess cwd

[MarkusNeusinger]

Three field-reported problems, all in the discovery / child-process seam that a raw root on a Windows network share exercises.

1. The wiki could become one of its own raw sources

With a raw root above the wiki — CITADEL_RAW_DIRS=T:\ and the wiki at T:\llmWiki\data-science\wiki, a normal way to say "scan this whole drive" — discovery walked the wiki's generated pages back in as sources, run after run, each pass citing the last. The wiki dir is not hidden and matches no ignore glob, so the only defense was a hand-written CITADEL_IGNORE_PATTERNS entry.

• ingest._is_wiki_internal prunes the wiki out of every walk. Containment is grammar.is_within — purely lexical, no resolve() and no abspath() — so it costs nothing per directory entry and can never block or syscall on a dead mount.
• A raw root that is the wiki is refused without being counted as entered, so the deletion sweep is not armed for a root nothing scanned.
• An explicitly named path inside the wiki is refused too — including in _discover_repos, since a CITADEL_WIKI_GIT wiki holds a .git and would otherwise be digested as one repo source.
• The exclusion is announced once per run (silence would read as "there was nothing there").
• The run-start migration sweep now also clears keys inside the wiki from the manifest and the failures catalog, so a workspace that already self-ingested cleans itself up on the next plain citadel ingest (those files still exist on disk, so deletion detection would never have removed them).

2. No size-based discovery filter

Ignore patterns match names; nothing matched size. Every untracked candidate is stream-hashed in full before anything can classify it, so a raw root that also holds machine data (the reported case: 617 .tdms sensor dumps, 10.6 GB in one folder) paid a full sha256 per file just to end up recorded as unreadable binary.

CITADEL_MAX_SOURCE_BYTES skips anything larger straight from the walk's own stat — never opened, never hashed, never written to the manifest or the failures catalog. Skips are reported, not silent: a stderr NOTE, an Oversized section on the run report, and an Oversized bucket in citadel status (--json included, as {"key", "size_bytes"}).

Design calls:

• Off by default (0 = no limit). Silently dropping a large-but-legitimate source — a 2 GB lecture recording under CITADEL_AUDIO_SUPPORT, a scanned archive PDF — would be worse than a slow scan, and citadel's contract is that everything under a raw root is either ingested or reported.
• Explicit wins: citadel ingest big.tdms bypasses the ceiling, exactly as it does for hidden names and ignore globs.
• An already-ingested source that later crosses the ceiling stays in the wiki and is not swept as deleted — the sweep's positive .exists() confirmation covers it; it simply stops being re-checked.
• A negative value clamps to 0 with a CONFIG_WARNINGS entry (mirroring CITADEL_JOBS), so citadel doctor names the typo.

3. Agent sessions ran with a UNC working directory

Path.resolve() rewrites a Windows mapped drive into its UNC form — T:\team-wiki → \\fileserver\share\team-wiki — whichever spelling the .env used, and that resolved path was what citadel passed as the agent CLI's cwd. The copilot backend then refuses to work at all ("environment blocks UNC/network paths"), and git treats the spelling as a different repository (safe.directory / core.filemode). The same workspace behaves perfectly when the user cds into T:\... themselves.

The resolved form stays the one identity everywhere (manifest keys, root containment, staging vs. live). What changes is only what a child process is handed:

• config.NATIVE_FORMS / native_form() / child_cwd() — a tiny registry of non-resolved spellings, populated from what citadel already holds: the CITADEL_WORKSPACE value or the launching CWD (walked upward, since the marker walk resolves upward), and each CITADEL_*_DIR / CITADEL_RAW_DIRS value on its way through _resolve_dir_entry.
• Consumers: llm._run_session / _stream_subprocess (cwd), llm._external_dirs (the --add-dir / --include-directories grants have to name paths the agent can reach from that cwd), and wikigit._git (-C).
• Nothing is guessed: an alias is recorded only when both spellings of the same directory are known and resolution turned a non-UNC path into a UNC one. On POSIX and for ordinary Windows paths the registry stays empty and native_form is the identity function, so those invocations are byte-for-byte unchanged.

WSL is documented as a non-workaround: a DrvFs mount of the same share fails differently (Operation not permitted on the staging copy, core.filemode on git init).

Also

• Two citadel doctor checks: wiki placement (WARN when the wiki dir sits inside a walked raw root — discovery handles it, but the nesting still costs clarity: a stray citation into the wiki looks like legal provenance) and child paths (names the working directory sessions will actually run in; WARNs when only a UNC spelling is known, with the fix).
• config._path_id extracted as the one spelling-normalizer (source_roots now uses it).
• Docs: docs/configuration.md (the knob, plus new sections on wiki-under-raw and Windows mapped drives), docs/troubleshooting.md (three new symptom-first entries), docs/maintenance.md, citadel/templates/env.example, CLAUDE.md + the generated .github/copilot-instructions.md, CHANGELOG.

Tests

30 new offline tests across test_ingest_discovery.py, test_netdrive.py, test_doctor.py, test_status.py, test_config_env.py — the UNC behavior is testable from Linux because the spelling predicates (_is_unc_path / _prefer_native) are pure string functions and the registry is a monkeypatchable config attribute.

Gates run locally, all green:

• uv run pytest -q — 1251 passed, 1 skipped
• uv run ruff check . — clean
• uv run ruff format --check . — clean
• CITADEL_WORKSPACE=corpora/beverages uv run python -m citadel lint — exit 0

verify-corpus was not run: it spawns real agent CLI sessions, and this change alters no prompt, rule, or session content — the agent-facing prompt is byte-identical, the size ceiling is off by default, and the native-form registry is empty on Linux, so a corpus run would exercise nothing new. Worth a run on a Windows share if you want the end-to-end confirmation of #3.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U2dxmXzQpQmeYospgrjSi3

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR hardens ingest discovery and child-process spawning for Windows/network-share workspaces by preventing self-ingest of the wiki, introducing an opt-in discovery size ceiling, and avoiding UNC working directories/grants for agent and git subprocesses.

Changes:

• Exclude the live wiki directory from raw discovery (including explicit paths) and prune any previously self-ingested wiki keys from manifest/failures.
• Add CITADEL_MAX_SOURCE_BYTES to skip oversized files at discovery (reported via ingest report + status bucket + JSON).
• Add a “native path” registry to prefer drive-letter spellings for subprocess cwd, agent directory grants, and wiki-git -C; add corresponding doctor checks and documentation/tests.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
citadel/config.py	Adds native-path registry + CITADEL_MAX_SOURCE_BYTES parsing; path identity normalization via _path_id.
citadel/ingest.py	Excludes wiki from discovery, adds size ceiling support, reports exclusions/oversized, prunes untrackable keys.
citadel/llm.py	Spawns agent sessions using config.child_cwd() and grants external dirs in child-friendly spelling.
citadel/wikigit.py	Runs git with -C set to config.native_form(cwd) to avoid UNC issues on Windows.
citadel/status.py	Adds oversized bucket (render + --json) based on ingest’s stat-only walk.
citadel/doctor.py	Adds wiki placement and child paths checks for self-ingest clarity + UNC advisory.
tests/test_ingest_discovery.py	Adds coverage for wiki-exclusion behavior, pruning self-ingested keys, and the size ceiling behavior.
tests/test_netdrive.py	Adds unit tests for UNC detection, aliasing logic, subprocess cwd/grants, and wikigit -C usage.
tests/test_doctor.py	Adds tests for the new doctor checks and inventory listing.
tests/test_status.py	Updates status headers/JSON schema and adds oversized-bucket behavior test.
tests/test_config_env.py	Adds tests for CITADEL_MAX_SOURCE_BYTES parsing and negative-value clamping/warning.
docs/configuration.md	Documents the new size ceiling and clarifies wiki-under-raw and mapped-drive/UNC behavior.
docs/troubleshooting.md	Adds symptom-first entries for slow scans/huge files, UNC failures, and prior self-ingest.
docs/maintenance.md	Updates status semantics to include oversized bucket.
citadel/templates/env.example	Documents CITADEL_MAX_SOURCE_BYTES in the template.
CHANGELOG.md	Changelog entries for size ceiling, wiki self-ingest guard, and UNC child-cwd fix.
CLAUDE.md	Updates contributor guidance for new status bucket and UNC child-cwd behavior.
.github/copilot-instructions.md	Regenerated to match CLAUDE.md updates.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

citadel/ingest.py:505

• The explicit repo-discovery wiki guard has the same ~ expansion gap as _discover_walk: os.path.abspath(p) does not expand user home, so ~/.../wiki paths could slip past the wiki exclusion and be treated as repo sources (e.g., when the wiki has a .git under CITADEL_WIKI_GIT).

        for raw in paths:
            p = Path(raw)
            if _is_wiki_internal(Path(os.path.abspath(p))):
                continue

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

citadel/status.py:271

• StatusReport.oversized currently includes every oversized file seen on disk, even if that key is already tracked in the manifest/failures. That can make a source show up as both Ingested and Oversized, which contradicts this module’s own definition of oversized as “never tracked” and makes status --json ambiguous for scripts.

    tracked = set(manifest_dict) | set(failures_dict)
    present, oversized = _walk_state()
    report.pending = sorted(key for key in present if key not in tracked)
    report.oversized = oversized
    report.ignored = _ignored_names()

citadel/ingest.py:432

• For explicitly requested directories, _scan_tree() is called with p as provided. If the user passes a relative directory like . (or any relative path that contains the wiki), the walk will build relative child paths and _is_wiki_internal() (which assumes absolute paths) will fail to recognize and prune the wiki subtree. Making the scanned root absolute (without resolve()) preserves the no-syscall property and keeps the wiki exclusion reliable for explicit directory ingest too.

            if p.is_dir():
                if not _is_repo_source(p):
                    _scan_tree(p, walk)
                continue