ingest: exclude the wiki from discovery, add a size ceiling, fix the UNC subprocess cwd by MarkusNeusinger · Pull Request #131 · MarkusNeusinger/cite-citadel

MarkusNeusinger · 2026-07-27T14:01:35Z

Three field-reported problems, all in the discovery / child-process seam that a raw root on a Windows network share exercises.

1. The wiki could become one of its own raw sources

With a raw root above the wiki — CITADEL_RAW_DIRS=T:\ and the wiki at T:\llmWiki\data-science\wiki, a normal way to say "scan this whole drive" — discovery walked the wiki's generated pages back in as sources, run after run, each pass citing the last. The wiki dir is not hidden and matches no ignore glob, so the only defense was a hand-written CITADEL_IGNORE_PATTERNS entry.

ingest._is_wiki_internal prunes the wiki out of every walk. Containment is grammar.is_within — purely lexical, no resolve() and no abspath() — so it costs nothing per directory entry and can never block or syscall on a dead mount.
A raw root that is the wiki is refused without being counted as entered, so the deletion sweep is not armed for a root nothing scanned.
An explicitly named path inside the wiki is refused too — including in _discover_repos, since a CITADEL_WIKI_GIT wiki holds a .git and would otherwise be digested as one repo source.
The exclusion is announced once per run (silence would read as "there was nothing there").
The run-start migration sweep now also clears keys inside the wiki from the manifest and the failures catalog, so a workspace that already self-ingested cleans itself up on the next plain citadel ingest (those files still exist on disk, so deletion detection would never have removed them).

2. No size-based discovery filter

Ignore patterns match names; nothing matched size. Every untracked candidate is stream-hashed in full before anything can classify it, so a raw root that also holds machine data (the reported case: 617 .tdms sensor dumps, 10.6 GB in one folder) paid a full sha256 per file just to end up recorded as unreadable binary.

CITADEL_MAX_SOURCE_BYTES skips anything larger straight from the walk's own stat — never opened, never hashed, never written to the manifest or the failures catalog. Skips are reported, not silent: a stderr NOTE, an Oversized section on the run report, and an Oversized bucket in citadel status (--json included, as {"key", "size_bytes"}).

Design calls:

Off by default (0 = no limit). Silently dropping a large-but-legitimate source — a 2 GB lecture recording under CITADEL_AUDIO_SUPPORT, a scanned archive PDF — would be worse than a slow scan, and citadel's contract is that everything under a raw root is either ingested or reported.
Explicit wins: citadel ingest big.tdms bypasses the ceiling, exactly as it does for hidden names and ignore globs.
An already-ingested source that later crosses the ceiling stays in the wiki and is not swept as deleted — the sweep's positive .exists() confirmation covers it; it simply stops being re-checked.
A negative value clamps to 0 with a CONFIG_WARNINGS entry (mirroring CITADEL_JOBS), so citadel doctor names the typo.

3. Agent sessions ran with a UNC working directory

Path.resolve() rewrites a Windows mapped drive into its UNC form — T:\team-wiki → \\fileserver\share\team-wiki — whichever spelling the .env used, and that resolved path was what citadel passed as the agent CLI's cwd. The copilot backend then refuses to work at all ("environment blocks UNC/network paths"), and git treats the spelling as a different repository (safe.directory / core.filemode). The same workspace behaves perfectly when the user cds into T:\... themselves.

The resolved form stays the one identity everywhere (manifest keys, root containment, staging vs. live). What changes is only what a child process is handed:

config.NATIVE_FORMS / native_form() / child_cwd() — a tiny registry of non-resolved spellings, populated from what citadel already holds: the CITADEL_WORKSPACE value or the launching CWD (walked upward, since the marker walk resolves upward), and each CITADEL_*_DIR / CITADEL_RAW_DIRS value on its way through _resolve_dir_entry.
Consumers: llm._run_session / _stream_subprocess (cwd), llm._external_dirs (the --add-dir / --include-directories grants have to name paths the agent can reach from that cwd), and wikigit._git (-C).
Nothing is guessed: an alias is recorded only when both spellings of the same directory are known and resolution turned a non-UNC path into a UNC one. On POSIX and for ordinary Windows paths the registry stays empty and native_form is the identity function, so those invocations are byte-for-byte unchanged.

WSL is documented as a non-workaround: a DrvFs mount of the same share fails differently (Operation not permitted on the staging copy, core.filemode on git init).

Also

Two citadel doctor checks: wiki placement (WARN when the wiki dir sits inside a walked raw root — discovery handles it, but the nesting still costs clarity: a stray citation into the wiki looks like legal provenance) and child paths (names the working directory sessions will actually run in; WARNs when only a UNC spelling is known, with the fix).
config._path_id extracted as the one spelling-normalizer (source_roots now uses it).
Docs: docs/configuration.md (the knob, plus new sections on wiki-under-raw and Windows mapped drives), docs/troubleshooting.md (three new symptom-first entries), docs/maintenance.md, citadel/templates/env.example, CLAUDE.md + the generated .github/copilot-instructions.md, CHANGELOG.

Tests

30 new offline tests across test_ingest_discovery.py, test_netdrive.py, test_doctor.py, test_status.py, test_config_env.py — the UNC behavior is testable from Linux because the spelling predicates (_is_unc_path / _prefer_native) are pure string functions and the registry is a monkeypatchable config attribute.

Gates run locally, all green:

uv run pytest -q — 1251 passed, 1 skipped
uv run ruff check . — clean
uv run ruff format --check . — clean
CITADEL_WORKSPACE=corpora/beverages uv run python -m citadel lint — exit 0

verify-corpus was not run: it spawns real agent CLI sessions, and this change alters no prompt, rule, or session content — the agent-facing prompt is byte-identical, the size ceiling is off by default, and the native-form registry is empty on Linux, so a corpus run would exercise nothing new. Worth a run on a Windows share if you want the end-to-end confirmation of #3.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U2dxmXzQpQmeYospgrjSi3

Generated by Claude Code

Three field-reported problems with a raw root on a Windows network share, all in the discovery/child-process seam: 1. The wiki could become one of its own sources. With a raw root ABOVE the wiki (CITADEL_RAW_DIRS=T:\ and the wiki at T:\llmWiki\ds\wiki), discovery walked the generated pages back in run after run; only a hand-written CITADEL_IGNORE_PATTERNS entry stopped it. Discovery now prunes the wiki dir from every walk (lexically — no resolve(), no syscall per entry), refuses a raw root that IS the wiki without arming the deletion sweep, refuses an explicit path inside the wiki (including a git-backed wiki that would otherwise digest as a repo source), announces the exclusion once per run, and sweeps already-self-ingested keys out of the manifest + failures catalog at run start. 2. No size-based discovery filter. Ignore patterns match names; nothing matched size, so a folder of multi-GB sensor dumps was stream-hashed in full just to end up recorded as unreadable binary. CITADEL_MAX_SOURCE_BYTES skips those from the walk's own stat — never opened, never hashed, never tracked — while reporting them (stderr NOTE, an Oversized report section, an Oversized status bucket). Off by default; explicit paths bypass it. 3. Agent sessions ran with a UNC working directory. Path.resolve() rewrites a mapped drive (T:\team-wiki) into \\server\share\team-wiki whatever the .env said, and that was passed as the CLI's cwd — where copilot refuses to run ("environment blocks UNC/network paths") and git reads a different repo. The resolved form stays the one identity; child processes (agent cwd + directory grants, git -C) now get the non-resolved spelling citadel already held. An alias is recorded only when both spellings of one directory are known and resolution produced a UNC path, so POSIX and ordinary Windows paths are unchanged. Plus two doctor checks (wiki placement, child paths), docs, and tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2dxmXzQpQmeYospgrjSi3

The note claimed "lies under a configured raw root", which is only true for the walk case — an explicitly named path inside the wiki is refused whether or not a raw root sits above it. Say what actually happened, and keep the raw-root advice as the conditional half. Also key the external-dirs grant test off the RESOLVED raw dir, exactly as _external_dirs looks it up, so the lookup cannot depend on tmp_path already being resolved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2dxmXzQpQmeYospgrjSi3

Copilot

Pull request overview

This PR hardens ingest discovery and child-process spawning for Windows/network-share workspaces by preventing self-ingest of the wiki, introducing an opt-in discovery size ceiling, and avoiding UNC working directories/grants for agent and git subprocesses.

Changes:

Exclude the live wiki directory from raw discovery (including explicit paths) and prune any previously self-ingested wiki keys from manifest/failures.
Add CITADEL_MAX_SOURCE_BYTES to skip oversized files at discovery (reported via ingest report + status bucket + JSON).
Add a “native path” registry to prefer drive-letter spellings for subprocess cwd, agent directory grants, and wiki-git -C; add corresponding doctor checks and documentation/tests.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`citadel/config.py`	Adds native-path registry + `CITADEL_MAX_SOURCE_BYTES` parsing; path identity normalization via `_path_id`.
`citadel/ingest.py`	Excludes wiki from discovery, adds size ceiling support, reports exclusions/oversized, prunes untrackable keys.
`citadel/llm.py`	Spawns agent sessions using `config.child_cwd()` and grants external dirs in child-friendly spelling.
`citadel/wikigit.py`	Runs git with `-C` set to `config.native_form(cwd)` to avoid UNC issues on Windows.
`citadel/status.py`	Adds `oversized` bucket (render + `--json`) based on ingest’s stat-only walk.
`citadel/doctor.py`	Adds `wiki placement` and `child paths` checks for self-ingest clarity + UNC advisory.
`tests/test_ingest_discovery.py`	Adds coverage for wiki-exclusion behavior, pruning self-ingested keys, and the size ceiling behavior.
`tests/test_netdrive.py`	Adds unit tests for UNC detection, aliasing logic, subprocess cwd/grants, and wikigit `-C` usage.
`tests/test_doctor.py`	Adds tests for the new doctor checks and inventory listing.
`tests/test_status.py`	Updates status headers/JSON schema and adds oversized-bucket behavior test.
`tests/test_config_env.py`	Adds tests for `CITADEL_MAX_SOURCE_BYTES` parsing and negative-value clamping/warning.
`docs/configuration.md`	Documents the new size ceiling and clarifies wiki-under-raw and mapped-drive/UNC behavior.
`docs/troubleshooting.md`	Adds symptom-first entries for slow scans/huge files, UNC failures, and prior self-ingest.
`docs/maintenance.md`	Updates `status` semantics to include oversized bucket.
`citadel/templates/env.example`	Documents `CITADEL_MAX_SOURCE_BYTES` in the template.
`CHANGELOG.md`	Changelog entries for size ceiling, wiki self-ingest guard, and UNC child-cwd fix.
`CLAUDE.md`	Updates contributor guidance for new status bucket and UNC child-cwd behavior.
`.github/copilot-instructions.md`	Regenerated to match `CLAUDE.md` updates.

Copilot review, config.py:332 — correct and load-bearing. The registry was only populated by _resolve_dir_entry, i.e. only for an ABSOLUTE CITADEL_*_DIR override. A default wiki (WORKSPACE_ROOT / "wiki", no override at all) and a relative override (CITADEL_WIKI_DIR=wiki, joined onto an already-resolved UNC root) both recorded nothing — so wikigit still ran `git -C \\server\share\...` on exactly the layout the reported git failures came from, even while the agent cwd was drive-letter friendly. native_form now falls back to the nearest ANCESTOR with an alias and rebases onto it. Sound because a drive mapping covers the whole subtree: if \\srv\share\team-wiki and T:\team-wiki are the same directory, so are their identically-named children. Still purely lexical, and an empty registry (every POSIX layout) short-circuits before any of it. The explicit record in _resolve_dir_entry stays for the one spelling nothing can derive: an absolute override pointing outside the workspace. Copilot's second comment (the wiki-exclusion NOTE wording) was already fixed in 1646cd9 — that review ran against the previous head. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2dxmXzQpQmeYospgrjSi3

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

citadel/ingest.py:505

The explicit repo-discovery wiki guard has the same ~ expansion gap as _discover_walk: os.path.abspath(p) does not expand user home, so ~/.../wiki paths could slip past the wiki exclusion and be treated as repo sources (e.g., when the wiki has a .git under CITADEL_WIKI_GIT).

        for raw in paths:
            p = Path(raw)
            if _is_wiki_internal(Path(os.path.abspath(p))):
                continue

Copilot review, ingest.py:408/502. The wiki guard for explicit paths used os.path.abspath without expanding ~, so an unexpanded path could not be recognized as wiki-internal. Reachable impact today was limited — such a path also fails os.stat and is dropped rather than ingested — but the guard should hold by construction, not by a downstream accident. Fixed at the root instead of in the guard: _explicit_path() expands ~ for every explicitly requested path in both _discover_walk and _discover_repos. That matches what every other configured path already gets (_resolve_dir_entry, workspace.init, CITADEL_WORKSPACE) and fixes a second, more concrete gap on the platform this PR is about: Windows cmd.exe (and PowerShell, for a native binary's arguments) does not expand ~, so `citadel ingest ~/ws/raw/notes.md` silently ingested nothing. os.path.expanduser, not Path.expanduser: the latter raises on an unresolvable home (~nosuchuser), and discovery must never raise on user input. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2dxmXzQpQmeYospgrjSi3

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

    tracked = set(manifest_dict) | set(failures_dict)
-    report.pending = sorted(key for key in _present_source_keys() if key not in tracked)
+    present, oversized = _walk_state()
+    report.pending = sorted(key for key in present if key not in tracked)
+    report.oversized = oversized
    report.ignored = _ignored_names()


Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

citadel/status.py:271

StatusReport.oversized currently includes every oversized file seen on disk, even if that key is already tracked in the manifest/failures. That can make a source show up as both Ingested and Oversized, which contradicts this module’s own definition of oversized as “never tracked” and makes status --json ambiguous for scripts.

    tracked = set(manifest_dict) | set(failures_dict)
    present, oversized = _walk_state()
    report.pending = sorted(key for key in present if key not in tracked)
    report.oversized = oversized
    report.ignored = _ignored_names()

citadel/ingest.py:432

For explicitly requested directories, _scan_tree() is called with p as provided. If the user passes a relative directory like . (or any relative path that contains the wiki), the walk will build relative child paths and _is_wiki_internal() (which assumes absolute paths) will fail to recognize and prune the wiki subtree. Making the scanned root absolute (without resolve()) preserves the no-syscall property and keeps the wiki exclusion reliable for explicit directory ingest too.

            if p.is_dir():
                if not _is_repo_source(p):
                    _scan_tree(p, walk)
                continue

Copilot AI review requested due to automatic review settings July 27, 2026 14:01

Copilot started reviewing on behalf of MarkusNeusinger July 27, 2026 14:02 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Comment thread citadel/config.py

Comment thread citadel/ingest.py

Copilot AI review requested due to automatic review settings July 27, 2026 14:09

Copilot started reviewing on behalf of MarkusNeusinger July 27, 2026 14:10 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Comment thread citadel/ingest.py Outdated

Copilot AI review requested due to automatic review settings July 27, 2026 14:13

Copilot started reviewing on behalf of MarkusNeusinger July 27, 2026 14:14 View session

MarkusNeusinger merged commit d3054d6 into main Jul 27, 2026
11 checks passed

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 27, 2026 14:18

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ingest: exclude the wiki from discovery, add a size ceiling, fix the UNC subprocess cwd - #131

ingest: exclude the wiki from discovery, add a size ceiling, fix the UNC subprocess cwd#131
MarkusNeusinger merged 4 commits into
mainfrom
claude/wiki-discovery-unc-paths-ouk1zn

MarkusNeusinger commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

MarkusNeusinger commented Jul 27, 2026

1. The wiki could become one of its own raw sources

2. No size-based discovery filter

3. Agent sessions ran with a UNC working directory

Also

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants