diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 2ed4166..2a6f60b 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,7 +1,9 @@ # GitHub Copilot instructions — cite-citadel -Repository guidance for GitHub Copilot. This mirrors [`CLAUDE.md`](../CLAUDE.md); keep the two in -sync when either changes. +Repository guidance for GitHub Copilot. **Generated from [`CLAUDE.md`](../CLAUDE.md)** — do not edit +this file by hand: change `CLAUDE.md` and regenerate with +`CITADEL_WRITE_COPILOT_DOC=1 uv run pytest tests/test_packaging.py -k copilot -q`. The drift guard in +`tests/test_packaging.py` fails whenever the two disagree. ## What this is @@ -29,17 +31,27 @@ uv run python -m citadel Subcommands: `init [DIR]` (scaffold a workspace: `citadel.toml` marker, `.env`, `raw/`, `wiki/`; idempotent), `ingest [paths…]` (fold raw/ into the wiki; `--verbose`/`-v` streams the agent session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the progress spinner, +`--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`), `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source, `--force ` deliberately re-reads already-ingested sources as a reconcile — it requires -explicit paths and is refused without them), `curate [--dry-run] [--limit N] [--stale-rules] +explicit paths and is refused without them), `refresh [--limit N] [--min-age-days D] [--dry-run] [--jobs N]` +(the THIRD lifecycle: re-verify the least-recently-checked sources — ordered by the manifest's +`ingested_at` stamp, oldest/stampless first — through forced reconcile sessions on an explicit +per-run budget of N sources; the sustainable alternative to regenerating the wiki after a model +upgrade), `curate [--dry-run] [--limit N] [--stale-rules] [--diff PATH] [--retry]` (the SECOND lifecycle: improve EXISTING pages — re-sort/split/re-ground/resolve contradictions/fix locators — against a recomputed findings checklist), `status` (read-only per-source state table: ingested / failed / skipped-duplicate / ignored / pending; MCP twin `wiki_status`), `doctor` (read-only setup health check — OK/WARN/FAIL lines for workspace / rules / config-parse fallbacks / -agent CLI / raw roots / -manifest / billing / wiki-git state / a best-effort PyPI update check / workspace coherence; needs no workspace, exits 1 only on a FAIL), `serve` (MCP -stdio server), `search [--tag T] [--limit N]`, `define ` / `read ` / +agent CLI / the inert-`CITADEL_INGEST_MODEL`-on-copilot/gemini advisory / raw roots / +manifest / billing / the HTTP-serve posture / wiki-git state / a best-effort PyPI update check / workspace coherence; needs no workspace, exits 1 only on a FAIL), +`serve [--http [--host H] [--port P] [--path /mcp] [--read-only]]` (the MCP server — stdio by +default; `--http` serves the SAME surface over MCP's Streamable HTTP transport for a client that is +not on this machine, mandatory bearer token, loopback default, optional read-only), `capture [--from WHO] [--topic T]` (append one attributed note from a +conversation to the raw/ capture log `raw/captures/YYYY-MM.md`; `-` reads stdin; the next ingest +folds it in — the conversational-capture bridge, MCP twin `wiki_capture`), +`search [--tag T] [--limit N]`, `define ` / `read ` / `raw [--locator L]` / `neighbors ` / `index` / `sources` (CLI twins of the `wiki_define`/`wiki_read`/`wiki_raw`/`wiki_neighbors`/`wiki_index`/`wiki_sources` MCP tools — full CLI↔MCP parity), @@ -104,7 +116,8 @@ the wiki: `citadel check` + `lint` exit 0 (structural eligibility), then a **ret grade — driving citadel's own read tools (`search`/`read`/`index`/`tags`) to prove each answer-key guarantee is both correct+cited and easily findable, dropping to a file-level grep only to separate a wiki-creation defect from a retrieval one (its misses feed two optimization lanes: the ingest/rules -generator and the search tools). Corpora live +generator and the search tools — persisted per run in the committed ledger +`docs/verify-corpus-backlog.md`). Corpora live **outside** `citadel/`, so they never ship in the wheel. The repo-root `raw/` + `wiki/` are a gitignored developer workspace (the checkout's `citadel.toml` marker still makes it a workspace). @@ -177,25 +190,25 @@ it is itself a workspace. deleted (vanished from disk, full runs only) / same-basename document duplicates (skipped in favor of one preferred format). A pending Office source is extracted to text first; a pending image is read visually; a pending audio/video - recording (`CITADEL_AUDIO_SUPPORT`, opt-in) is transcribed through the whisper seam - (`transcribe.py`, cached in `.citadel_transcripts/` beside the wiki) and the agent reads the - `[HH:MM:SS]`-stamped transcript; a pending PDF (pypdf is a bundled dep — `CITADEL_PDF_TEXT`, - default auto) gets its text layer extracted (`pdftext.py`, cached in `.citadel_pdftext/`) and - the agent reads the `[p. N]`-marked extraction, falling back to the direct agent read when - there is no usable text layer; a pending source larger than + recording (`CITADEL_AUDIO_SUPPORT`, opt-in) is transcribed through a local whisper-class CLI + (`transcribe.py`, content-addressed cache `.citadel_transcripts/` beside the wiki) and the agent + reads the `[HH:MM:SS]`-stamped transcript; a pending PDF (pypdf is a bundled dep — + `CITADEL_PDF_TEXT`, default auto) gets its text layer extracted (`pdftext.py`, content-addressed + cache `.citadel_pdftext/`) and the agent reads the `[p. N]`-marked extraction, falling back to + the direct agent read when there is no usable text layer; a pending source larger than `CITADEL_MAX_SOURCE_CHARS` is folded in over several passes (all against one staging copy — see the promote bullet below). `ingest --force ` bypasses the sha short-circuit: the named sources land in pending as reconciles (a repo re-digests in full), and the manifest is re-stamped with the current model + rules version. -- **Discovery is incremental and deletion-safe**: one `os.scandir` walk over every - `CITADEL_RAW_DIRS` root; the manifest doubles as the scan cache (an entry's - `size`/`mtime_ns`/`ctime_ns`/`hashed_at_ns` are a skip-hint — sha256 stays the sole arbiter of - "changed"; `--full-rescan` distrusts the cache). Deletion candidates come from the - walked-seen-set diff and each is positively confirmed with `.exists()`; any walk error aborts - the whole sweep, an unreachable root contributes no candidates, keys under no configured root - are logged and never swept, and a workspace-identity mismatch whose keys do not resolve refuses - the sweep. A flaky share or unmounted root must NEVER read as mass deletion — don't weaken - these guards. +- **Discovery is incremental and deletion-safe**: one iterative `os.scandir` + walk over every `CITADEL_RAW_DIRS` root keeps each file's stat; the **manifest doubles as the + scan cache** (an entry's `size`/`mtime_ns`/`ctime_ns`/`hashed_at_ns` are a skip-hint — sha256 + stays the sole arbiter of "changed"; `--full-rescan` distrusts the cache). Deletion candidates + come from the walked-seen-set diff and each is positively **confirmed with `.exists()`**; any + walk error aborts the whole sweep, an unreachable root contributes no candidates, keys under no + configured root are logged and never swept, and a workspace-identity mismatch whose keys do not + resolve refuses the sweep. A flaky share or unmounted root must NEVER read as mass deletion — + don't weaken these guards. - For each pending source it runs the agent against a **per-source staging copy** of the wiki (a sibling dir, never the live wiki), then snapshots before/after and **diffs by content hash** to learn what the agent created/updated/deleted — the agent's file edits *are* the result (the @@ -220,7 +233,20 @@ it is itself a workspace. vanished source's stale provenance before any pending session touches a page that still cites it (else that pre-existing bad citation would fail the pending session's validation and roll it back). This all-or-nothing + network-share-hardened machinery (`_robust_*`, `robust_mkdir`) is - load-bearing — don't simplify it away. **One mutating run per workspace**: ingest and curate take + load-bearing — don't simplify it away. **Bounded parallelism** (`--jobs N` / `CITADEL_JOBS`, default 1 = serial): the per-source staging + copy IS the isolation primitive, so N sources run at once through the same `_SourceJob` loop — + worker threads only plan/stage/run/promote, while every SHARED write (report, manifest, failures) + stays on the main thread. One lock (`_LIVE_WIKI_LOCK`) guards the only two moments that touch the + live wiki: the clone (taken together with a hash snapshot of what was cloned) and the promote. + Under it the promote is **base-aware** — it prunes only what its own clone had and its staging + lacks (else it would delete a concurrent source's new pages), and it REFUSES, before writing a + byte, if a path it would touch has moved since the clone; that source is then re-run SERIALLY at + the end of its group, where it merges into the winner's page (reported as `raced`). What + parallelism costs is cross-linking, not safety — concurrent sessions cannot see each other's new + pages — which is why the default is 1. The redirect this rides on is context-local + (`config.wiki_redirect` → a ContextVar behind `config.wiki_dir()`; children get their wiki via + `config.child_env()`), never a `config.WIKI_DIR`/`os.environ` assignment. + **One mutating run per workspace**: ingest and curate take an exclusive run lock (`runlock.py`, a dotfile sibling of the wiki; stale locks reclaimed via dead-pid/mtime, refreshed per source) so a second concurrent run fails loud instead of silently destroying the first one's staging/promotes; manifest + failures saves are atomic @@ -234,37 +260,67 @@ the source and rules by path, never embeds file content — which keeps argv tin file, whether it reads a source, and its format policy; an unknown kind fails loud. `kind` selects the propagation: `ingest` (new), `reconcile` (changed source — update/remove stale facts, don't just append), `delete` (source removed — strip its provenance), `repo`/`repo-reconcile` (a whole -git repo folded as one digest), `image`/`image-reconcile` (an image read visually), +git repo folded as one digest), `image`/`image-reconcile` (an image source read visually), `audio`/`audio-reconcile` (an audio/video source read via its whisper transcript), -`pdf`/`pdf-reconcile` (a PDF read via its pypdf text-layer extraction), and `curate` -(improve an existing page cluster against a findings file read by path, not a raw source). -`run_ingest_session` is the single seam tests monkeypatch; it returns the session's best-effort `SessionUsage` (the backend's OWN cost/usage report: claude's result envelope, gemini's `--session-summary` behind a cached `--help` feature probe; None when nothing was reported — accounting is strictly passive and can never fail a session), which ingest sums per source into the manifest stamp and per run onto the reports. +`pdf`/`pdf-reconcile` (a PDF read via its pypdf text-layer extraction), and +`curate` (improve an existing page cluster against a findings file — reads that file by path, not a +raw source). A large source is split into segments and folded in over several passes +(`segment=(part, total)` on `run_ingest_session`, telling later passes to MERGE into earlier ones). +`run_ingest_session` is the single seam tests monkeypatch; it returns the session's best-effort +`SessionUsage` (the backend's OWN cost/usage report: claude's result envelope, gemini's +`--session-summary` behind a cached `--help` feature probe; None when nothing was reported — +accounting is strictly passive and can never fail a session), which ingest sums per source into +the manifest stamp and per run onto the reports. **Two checking layers, one implementation** (`validate.py`): - `citadel check` / `wiki_validate` — the **strict per-page gate** (required fields, honest/defined citations, relative non-broken links, no `[[wikilinks]]`). The ingest agent self-runs it; ingest re-runs it and fails the source on any error. - `citadel lint` (`lint.py`) — a **pure offline health check** (contradictions, orphans, missing - cites, broken links, stale, fabricated sources, undefined abbreviations, and **locator issues** - — an out-of-range `lines A-B` range or a missing `§ Heading`, via `lint.check_locators`, shared - with curate). Only *structural* problems (missing type, broken links, bad sources, wikilinks) flip - its non-zero exit (code 3 — lint's own, distinct from the usage/no-workspace exit 2); the rest — - locator issues included — are advisory. Both layers parse - citations/links/fences through `grammar.py`, so lint and `citadel check` agree by construction: a - citation into `raw/` or `docs/` is legal provenance (never a broken link), and a link inside a - ``` code fence is literal text. - -**Curate is the second wiki lifecycle** (`curate.py`, `citadel curate`) — **no persisted queue**: -the plan is recomputed from offline detectors (`rules_version_drift`, `page_length_hard`, -`contradiction`, `orphan`, `llm_drift`, `resort` via `okf.folder_for_type`, `locator`) each run. -Each page CLUSTER runs ONE staged `kind="curate"` session over ingest's staging machinery; **the -staging diff-by-hash is the single result arbiter** (empty = NOOP, clean = applied, fail = -revert-and-stop → attempt-capped failures-catalog record). `--dry-run`/`--limit`/`--stale-rules`/ -`--diff` shape the run; `--retry` re-includes attempt-capped clusters (the explicit retry that -bypasses the cap); sessions use `CITADEL_CURATE_MODEL` (falls back to the ingest model). -`status.py` (`citadel status`) is the read-only per-source state view (manifest + failures + one -stat-only walk; never re-hashes), incl. each source's last-session cost and a `Recorded LLM cost` -corpus total. + cites, broken links, stale, fabricated sources, undefined abbreviations, near-duplicate/malformed + open points, and **locator issues** — a `lines A-B` range past a text source's end or a + `§ Heading` naming a heading the source lacks, via `lint.check_locators`, shared with curate). + Only *structural* problems (missing type, broken links, bad sources, wikilinks) flip its non-zero + exit (code 3 — lint's own, so CI can tell "wiki has problems" from the usage/no-workspace exit 2); + the rest — locator issues included — are advisory. Both layers parse citations/links/fences + through `grammar.py`, so lint and `citadel check` agree by construction: a citation into `raw/` or + `docs/` is legal provenance (never a broken link), and a link inside a ``` code fence is literal text. + +**Curate is the second wiki lifecycle** (`curate.py`, `citadel curate`). It has **no persisted +queue — the plan is recomputed from offline detectors every run** (the wiki IS the database): +`rules_version_drift`, `page_length_hard`, `contradiction`, `orphan`, `llm_drift`, `resort` +(type↔folder mismatch via `okf.folder_for_type`), and `locator` (from `lint.check_locators`); +fact re-verification is pre-filtered offline through manifest shas (`reverify_candidates` — changed += reconcile's job, gone = delete's job). Each planned page CLUSTER (page + cited raw files + link +neighbors) runs ONE staged `kind="curate"` session over ingest's existing staging machinery, its +findings written to a temp file referenced by path. **The staging diff-by-hash is the single result +arbiter** (empty = NOOP, clean promoted = applied, exception/check-fail = failed → revert-and-stop). +A failed cluster lands in the failures catalog keyed by page rel_path with an additive `attempts` +counter (default cap 2, never auto-retried until an explicit retry). `--dry-run` prints the plan +with zero sessions; `--limit`/`--stale-rules` shape it; `--diff PATH` writes a per-page change +report; `--retry` re-includes attempt-capped clusters (the explicit retry that bypasses the cap); +curate sessions run under `CITADEL_CURATE_MODEL` (falling back to the ingest model). + +**Refresh is the third lifecycle** (`refresh.py`, `citadel refresh`): budget-controlled +re-verification of existing SOURCES, so an aging wiki is brought up to the current model + rules a +slice at a time instead of ever being regenerated. Every successful session stamps its source's +manifest entry with an `ingested_at` last-checked time (`manifest.now_iso`; stamped ONLY in +`mark_done`/the repo done-hook — moves and cache re-stamps CARRY the old stamp, so "last checked" +never lies). `refresh.plan()` orders the manifest by that stamp (oldest first, a stampless +pre-refresh entry counting as oldest; only model-imported, still-on-disk sources qualify; +`--min-age-days` drops fresh ones so scheduled runs self-limit) and `refresh.refresh(limit=N)` +hands the queue head to `ingest.ingest(paths, force=True)` — one `kind="reconcile"` session per +source through the existing staging machinery, the success re-stamp rotating it to the back, so +repeated runs walk the corpus round-robin with NO persisted queue (the manifest IS the queue). +The budget is always explicit (`limit >= 1` enforced, default 1), mirroring `--force`'s +no-accidental-corpus-wide-run refusal. CLI-only, like curate. + +**Status is the read-only corpus view** (`status.py`, `citadel status`): the manifest + failures +catalog + one stat-only walk (never re-hashes) rendered as a per-source state table — ingested +(model + rules_version, `(stale)` when it predates the current rulebook, `checked YYYY-MM-DD` from +the `ingested_at` stamp, the last session's cost when recorded — with a `Recorded LLM cost` corpus +total above the table), failed (reason, attempts), +skipped-duplicate, ignored (pattern), pending. **Other modules:** `okf.py` is the OKF format core (parse/dump, type→folder routing, link math, and the non-negotiable `safe_join` path guard — reuse it for any wiki-relative path). `grammar.py` is @@ -272,69 +328,120 @@ the **single home of the markdown grammar** (link/footnote/fence/Sources-heading source-citation predicates, and the `[^sN]` **locator** parser — `parse_locator`/`source_headings`, which `lint.check_locators` and curate consume) that `store`, `validate`, `lint`, and the viewer all parse through; never re-define any of it locally. `store.py` is a thin **facade** re-exporting the "database" -API, split by responsibility into four sibling modules (import them through `store`): `store_core.py` -(`load()`, the single swappable `search()` seam, `read/write/delete_page` — both mutators share the -reserved-name guard refusing `index.md`/`*/index.md`/`log.md`/dotfiles — plus the CLI/MCP text -providers and the `log.md` writer); `linkgraph.py` (the deterministic link-rewrite safety nets -`rewrite_links`, `rewrite_raw_references`, `find_raw_references`, `find_broken_links`, and the -`inbound_map` backlink graph, all fence-aware via `grammar.py`); `catalogs.py` (`rebuild_indexes()`, -regenerating `index.md`, per-folder `index.md`, `sources/index.md`, and `open-points/index.md` -mechanically from frontmatter + manifest); and `open_points.py` (parsing `## Open Points` threads -and deriving each point's status). `pagecache.py` is the read-path snapshot cache behind -`store_core.load()` (`CITADEL_PAGE_CACHE`, `auto` = on only in `citadel serve`, which opts in): the -last load is kept in memory and re-validated per call by a stat-only scandir fingerprint over -exactly the files `load()` parses (~4 ms vs ~700 ms at 1000 pages), with search's per-page -term-frequency tables memoized on the same snapshot (a 1000-page `wiki_search`: ~1.4 s → ~50 ms). -Nothing is persisted, and staleness is designed out: the fingerprint is taken before AND after the -load and must match, a snapshot younger than the settle window is never stored, the single slot is -keyed by wiki dir, `write_page`/`delete_page` invalidate, and `ingest()`/`curate()` wear -`@pagecache.bypass` so the diff-by-hash machinery always reads disk. `manifest.py` tracks idempotency in +API, split by responsibility into four sibling modules (import them through `store`, not directly): +`store_core.py` (`load()`, the single swappable `search()` seam, `read/write/delete_page` — both +mutators share the reserved-name guard that refuses `index.md`/`*/index.md`/`log.md`/dotfiles — +plus the CLI/MCP text providers and the `log.md` writer); `linkgraph.py` (the deterministic +link-rewrite safety nets `rewrite_links`, `rewrite_raw_references`, `find_raw_references`, +`find_broken_links`, and the `inbound_map` backlink graph, all fence-aware via `grammar.py`); +`catalogs.py` (`rebuild_indexes()`, which regenerates `index.md`, per-folder `index.md`, +`sources/index.md`, and `open-points/index.md` mechanically from frontmatter + manifest); and +`open_points.py` (parsing `## Open Points` threads and deriving each point's status). +`pagecache.py` is the read-path snapshot cache behind `store_core.load()` (`CITADEL_PAGE_CACHE`, +`auto` = on only in `citadel serve`, which opts in): a long-lived MCP server otherwise re-walks and +re-parses the WHOLE wiki on every tool call, so the last load is kept in memory and re-validated +per call by a **stat-only** scandir fingerprint (`(rel_path, size, mtime_ns, ctime_ns)` over exactly +the files `load()` parses) — ~4 ms vs ~700 ms at 1000 pages, with search's per-page term-frequency +tables memoized on the same snapshot (a 1000-page `wiki_search`: ~1.4 s → ~50 ms). Nothing is +persisted (the wiki stays the database) and staleness is designed out: the fingerprint is taken +BEFORE and AFTER the load and must match, a snapshot whose newest stamp is younger than the settle +window is never stored (coarse-mtime filesystems), the single slot is keyed by wiki dir (ingest's +staging redirect can never be served the live wiki), `write_page`/`delete_page` invalidate, and +`ingest()`/`curate()` wear `@pagecache.bypass` so the diff-by-hash machinery always reads disk. +`manifest.py` tracks idempotency in `wiki/.citadel_ingested.json` (per source: sha256 or git commit + importing model + the last session's backend-reported `cost_usd`/`tokens_in`/`tokens_out`, carried across moves/re-stamps -like `ingested_at`). `repo.py` builds -the digest for git-repo sources. `extract.py` pulls text from Office files (stdlib-only); the legacy -OLE/CFBF salvage lives in `extract_ole.py`, imported lazily only when a legacy `.ppt`/`.doc`/`.xls` -is dispatched. `transcribe.py` is the whisper-CLI seam for audio/video sources -(`CITADEL_AUDIO_SUPPORT`, opt-in): one shell-out per content, the `[HH:MM:SS]`-per-line transcript -cached content-addressed in `.citadel_transcripts/` beside the wiki — the same cached text -`lint`/`wiki_raw`/the viewer verify and serve audio citations against; `transcript_for` is the -ingest seam tests monkeypatch. `pdftext.py` is the same idea for PDFs (bundled pypdf, -`CITADEL_PDF_TEXT`): the extracted `[p. N]`-marked text layer, cached in `.citadel_pdftext/`, -read under the `pdf`/`pdf-reconcile` kinds and verified/served by `lint`/`wiki_raw`/the viewer; -`text_for` is its (never-raising, best-effort) ingest seam. `resume.py` is the chunked-source resume-checkpoint store (`CITADEL_RESUME`): after each completed segment ingest banks the promote-shaped delta in `.citadel_resume/` beside the wiki, guarded by identity (source sha, model, rules version, segment content, prompt knobs), blob integrity, per-page live base state, re-validation of the replay and an attempt cap — so an interrupted large source continues at its failed segment instead of re-buying the earlier ones, and every guard failure degrades to a full restart. `curate.py` is the second lifecycle and `status.py` the read-only per-source state -view (both above); `doctor.py` (`citadel doctor`) is the read-only setup health check (OK/WARN/FAIL -lines over workspace resolution, the rules tree, the agent CLI on PATH, raw-root reachability, -manifest parse + stamp, failures summary, the API-key/PDF/audio advisories, the wiki-git state, a -best-effort PyPI update check naming the right upgrade command per install method, and workspace -coherence). +like `ingested_at`). `failures.py` +persists the sources that could NOT be ingested (`wiki/.citadel_failures.json`: unreadable / +errored / timed-out, with a reason), surfaced by `store` under a "Could not ingest" section of +`sources/index.md`. `repo.py` builds the digest for git-repo sources. `extract.py` pulls text from +Office files (stdlib-only): OOXML `.pptx`/`.docx`/`.xlsx` (+ macro-enabled) via zipfile+ElementTree, +and legacy OLE `.ppt`/`.doc`/`.xls` via the CFBF reader + best-effort text salvage in +`extract_ole.py` (imported lazily, only when a legacy OLE file is dispatched); its +`extract_media` also pulls embedded raster images out of OOXML files so the agent can view them. +`transcribe.py` is the whisper-CLI seam for audio/video sources (`CITADEL_AUDIO_SUPPORT`, opt-in): +detection by extension+magic, one shell-out per content (openai-whisper flag convention), the +`[HH:MM:SS]`-per-line transcript cached content-addressed in `.citadel_transcripts/` beside the +wiki — the same cached text `lint`/`wiki_raw`/the viewer verify and serve audio citations against; +`transcript_for` is the ingest seam tests monkeypatch (whisper itself is never an LLM concern, so +this lives beside `extract.py`, not in `llm.py`). +`pdftext.py` is the same idea for PDFs (the audio pattern applied to the PDF class): pypdf is a +bundled runtime dep (PDFs are a common raw/ class), so with `CITADEL_PDF_TEXT` (default auto = on +when pypdf imports, which it always does unless deliberately uninstalled) a +genuine PDF's (`%PDF-` magic) text layer is extracted once per content into a `[p. N]`-page-marked +line-stable text, cached content-addressed in `.citadel_pdftext/` beside the wiki — the agent +reads it under the `pdf`/`pdf-reconcile` kinds while citing the original `.pdf` with `lines A-B` +locators the same cache lets `lint`/`wiki_raw`/the viewer verify offline; `text_for` is the ingest +seam, strictly best-effort (scanned / encrypted / corrupt / pypdf force-removed → None → the +agent-native read with agent-verified `p. N` locators — never a failed source). +`resume.py` is the chunked-source resume-checkpoint store (`CITADEL_RESUME`): after each completed segment ingest banks the promote-shaped delta in `.citadel_resume/` beside the wiki, guarded by identity (source sha, model, rules version, segment content, prompt knobs), blob integrity, per-page live base state, re-validation of the replay and an attempt cap — so an interrupted large source continues at its failed segment instead of re-buying the earlier ones, and every guard failure degrades to a full restart. `curate.py` is the second lifecycle (offline detectors + staged cluster sessions; see above). +`status.py` is the read-only per-source state view; `doctor.py` (`citadel doctor`) is the read-only +setup health check (OK/WARN/FAIL lines over workspace resolution, the rules tree, the agent CLI on +PATH, raw-root reachability, manifest parse + stamp, failures summary, the API-key/PDF/audio +advisories, the wiki-git state, a best-effort PyPI update check naming the right upgrade command +per install method, and workspace coherence). `wikigit.py` is the best-effort wiki-HISTORY layer: after every run that changed the wiki (ingest or curate) it commits the whole wiki dir as ONE commit (and pushes to `CITADEL_WIKI_GIT_REMOTE` when set), so every change is a reviewable diff; `auto` (default) only acts when the wiki dir is already its own git repo, `CITADEL_WIKI_GIT=1` also `git init`s it on first use (refusing an embedded repo -inside another working tree), and any git problem is a report note, never a failed run. `server.py` is the -FastMCP stdio server (12 tools — 11 read-only incl. `wiki_raw` (the cited-source reader, backed by -`rawsource.py`), `wiki_neighbors` (a page's links-out/backlinks/cited-sources graph), `wiki_lint` -(tunable `stale_days`) and `wiki_status` (the per-source state view), -only `wiki_ingest` mutates; every tool carries MCP behavior -annotations, never raises (error strings instead), and sets `initialize.instructions`). The -`viewer/` subpackage builds the -self-contained offline HTML viewer (`template.html`/`app.css`/`app.js` are package-data assets loaded -via `importlib.resources`). `config.py` resolves all paths/settings. `cli.py` mirrors the MCP tools as -subcommands with full parity (`define`/`read`/`raw`/`neighbors`/`index`/`sources` twin the readers; `view` stays -CLI-only, `wiki_lint`/`wiki_status` close the `lint`/`status` gaps from the MCP side). `rawsource.py` backs `wiki_raw`/`citadel raw` -— the provenance-gated, locator-aware reader for the raw source behind a `[^sN]` citation (verify-only). +inside another working tree), and any git problem is a report note, never a failed run. `server.py` is the FastMCP stdio server (13 +tools — 11 read-only incl. `wiki_raw` (the cited-source reader, backed by `rawsource.py`), +`wiki_neighbors` (a page's links-out/backlinks/cited-sources graph), `wiki_lint` (with a tunable +`stale_days`) and `wiki_status` (the per-source state view), plus two mutating: +`wiki_capture` (append-only conversational note capture into `raw/captures/`, backed by +`capture.py` — it never touches the wiki) and `wiki_ingest` (the only wiki-writer); every tool carries MCP behavior +annotations — `readOnlyHint`/`destructiveHint`/`idempotentHint`/`openWorldHint` — never raises, +returning error strings instead, and hands the recommended tool flow up through +`initialize.instructions`; the server also publishes four workflow **prompts** — +`wiki_answer`/`wiki_verify`/`wiki_capture_note`/`wiki_health`, slash-command-like packaged flows +— and the wiki's documents as `wiki://` **resources** (`wiki://index`/`wiki://sources`/ +`wiki://tags` + the `wiki://page/{folder}/{name}` per-page template), byte-identical to their +tool twins and sharing the tools' never-raise contract). `httpserve.py` is the OPT-IN +**Streamable HTTP** transport for that same surface (`citadel serve --http`) — the only network +surface citadel has, so it is strict by construction: a mandatory bearer token +(`CITADEL_HTTP_TOKEN`, ≥16 chars — `serve` refuses to start without one) checked in a ~30-line ASGI +wrapper BEFORE the MCP session layer (constant-time compare, 401 + `WWW-Authenticate`, no +"unauthenticated for a minute" mode), a loopback bind by default (a public bind warns — the +transport is plain HTTP, the intended remote path is a TLS-terminating tunnel), a Host/Origin +admission policy owned by that same wrapper (the SDK's own middleware couples the two checks to +one flag and can express neither "any host" nor "one browser origin") — `Host` derived from the +bind address or named in `CITADEL_HTTP_ALLOWED_HOSTS` (a tunnel forwards its own hostname; a +wildcard `0.0.0.0` bind derives nothing, so it REFUSES to start instead of 421-ing every request), +browser `Origin`s refused unless `CITADEL_HTTP_ALLOWED_ORIGINS` admits them — and an optional read-only mode +(`--read-only`/`CITADEL_HTTP_READ_ONLY`) that has the two mutating tools refuse while leaving the +11 readers and the advertised tool list untouched (`server.set_read_only`). No new dependency — +starlette/uvicorn already ship with `mcp`. The `viewer/` subpackage builds the self-contained offline HTML +viewer (build logic in `__init__.py`; `template.html`/`app.css`/`app.js` are real package-data +assets loaded via `importlib.resources`). `config.py` resolves all paths/settings. `cli.py` mirrors +the MCP tools as subcommands (full parity: `define`/`read`/`raw`/`neighbors`/`index`/`sources`/`capture` twin their tools; +`view` stays CLI-only and `wiki_lint`/`wiki_status` close the `lint`/`status` gaps from the MCP side). `capture.py` +is the conversational-capture bridge behind `wiki_capture`/`citadel capture`: an append-only, +dated, attributed note into the monthly `raw/captures/YYYY-MM.md` log under the primary raw root +— an ordinary raw source the normal lifecycle ingests/reconciles, so captured statements get real +`[^sN]` line locators and the wiki is never written directly (docs/capture.md also documents the +save-the-transcript-as-a-file lane for whole conversations). `rawsource.py` backs +`wiki_raw`/`citadel raw`: the provenance-gated, locator-aware reader for the raw source behind a +`[^sN]` citation (verify-only — the wiki stays the synthesized layer for retrieval). ## Conventions specific to this codebase -- **`config.*` is read at call time** (`from . import config` then `config.WIKI_DIR`), never imported +- **`config.*` is read at call time** (`from . import config` then `config.RAW_DIR`), never imported by value — so tests can monkeypatch the whole filesystem layout. Honor this when adding code. + The WIKI path is read through the ACCESSORS (`config.wiki_dir()`, `index_path()`, + `sources_index_path()`, `log_path()`, `manifest_path()`, `failures_path()`), never as + `config.WIKI_DIR`: ingest redirects it per source through a ContextVar, so only the accessors see + the staging copy. The module attributes stay the process-wide base tests monkeypatch. - **Tests redirect everything to `tmp_path`** by monkeypatching `config.*` (including `WORKSPACE_ROOT`, which the agent's `cwd` reads) and replace `llm.run_ingest_session` with a fake that writes files into the temp wiki. No test spawns a real LLM CLI. Follow that pattern; keep tests offline. - **Never hand-edit generated files** — `index.md`, `log.md`, any `*/index.md`, `sources/index.md`, `.citadel_viewer.html`, and `.citadel_ingested.json` are regenerated. The ingest agent prompt and - `store.delete_page` both refuse to touch them. + `store.delete_page` both refuse to touch them. In the REPO, `.github/copilot-instructions.md` is + generated too: it is THIS file with a swapped header (both agents get one instruction set, so a + feature can't be documented for one and not the other). Edit `CLAUDE.md`, then regenerate with + `CITADEL_WRITE_COPILOT_DOC=1 uv run pytest tests/test_packaging.py -k copilot -q`; the drift guard + in `tests/test_packaging.py` fails the suite when the two disagree. - **Provenance grammar is load-bearing:** raw facts cite `[^sN]` → a real `raw/` file; model-supplied facts use `[^llmN]` (source: `LLM`) and must never be disguised as raw citations. A `[^sN]` to a missing file fails lint/check. @@ -347,12 +454,16 @@ CLI-only, `wiki_lint`/`wiki_status` close the `lint`/`status` gaps from the MCP - Config knobs live in the workspace-root `.env` (auto-loaded, gitignored; template: `citadel/templates/env.example`): `CITADEL_LLM_CLI`, `CITADEL_INGEST_MODEL`, `CITADEL_CURATE_MODEL` (model for `citadel curate` sessions; falls back to - `CITADEL_INGEST_MODEL`), `CITADEL_LLM_TIMEOUT`, `CITADEL_PAGE_CACHE` (the serve-side page - snapshot cache: `auto` = on in `citadel serve` only, `1` = everywhere, `0` = never), - `CITADEL_LLM_VERBOSE`, `CITADEL_LLM_LOG_DIR`, + `CITADEL_INGEST_MODEL`), `CITADEL_LLM_TIMEOUT`, `CITADEL_HERMETIC` (session isolation — append claude's `--bare` when the + installed binary advertises it, so personal `~/.claude` config never leaks into ingest; default + on, probe-gated), `CITADEL_PAGE_CACHE` (the serve-side page snapshot cache: `auto` = on in + `citadel serve` only, `1` = everywhere, `0` = never), the `CITADEL_HTTP_*` serving knobs + (`TOKEN` — mandatory for `serve --http`, no default; `HOST`/`PORT`/`PATH` — loopback:8765/mcp; + `READ_ONLY`), `CITADEL_LLM_VERBOSE`, `CITADEL_LLM_LOG_DIR`, `CITADEL_REPO_SUPPORT`, `CITADEL_IMAGE_SUPPORT` (read images visually), `CITADEL_AUDIO_SUPPORT` (opt-in whisper transcript ingest for audio/video, with `CITADEL_WHISPER_CLI`/ - `CITADEL_WHISPER_MODEL`/`CITADEL_WHISPER_TIMEOUT` tuning the seam), `CITADEL_MAX_SOURCE_CHARS` + `CITADEL_WHISPER_MODEL`/`CITADEL_WHISPER_TIMEOUT` tuning the seam), `CITADEL_JOBS` (how many sources ingest folds in + concurrently; 1 = serial), `CITADEL_MAX_SOURCE_CHARS` (large-source chunking threshold), `CITADEL_RESUME` (resume checkpoints for those chunked sources: continue at the segment an interrupted run died on instead of re-paying for the earlier ones; default on), `CITADEL_DEDUP_BY_BASENAME` (skip same-basename document @@ -360,7 +471,8 @@ CLI-only, `wiki_lint`/`wiki_status` close the `lint`/`status` gaps from the MCP `desktop.ini`, `~$` locks, …; a `+` prefix extends the built-in defaults), `CITADEL_WIKI_LANG` (target language of all wiki prose, default `en`; verbatim quotes stay original), `CITADEL_PDF_MODE` (`text` | `images` — whether the agent also reads a PDF's figures), - `CITADEL_PDF_TEXT` (`auto` | `1` | `0` — the bundled-pypdf text-layer pre-pass), + `CITADEL_PDF_TEXT` (`auto` | `1` | `0` — the pypdf text-layer pre-pass; auto = on when pypdf + imports, which it does by default; `0` forces agent-native reading), `CITADEL_STYLE_PROFILES` (opt-in persona/style capture on `persons/` pages, default `0`), `CITADEL_WIKI_GIT` (wiki-history auto-commit after ingest/curate: `auto` acts only when the wiki dir is its own git repo, `1` also `git init`s it, `0` off) + `CITADEL_WIKI_GIT_REMOTE` (optional diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a30342..0c277e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ All notable changes to this project are documented here. The format is based on ## [Unreleased] +## [0.5.0] - 2026-07-25 + ### Added - **Bounded parallel ingest — `citadel ingest --jobs N`** (the 2026-07 audit's backlog #11, and @@ -65,45 +67,6 @@ All notable changes to this project are documented here. The format is based on listen, whether the writers are exposed, and warns about a too-short token or a public bind before you ever start it. -### Changed - -- **`citadel serve` stops re-reading the whole wiki on every call** (the 2026-07 audit's backlog - #15, closing its finding 1.2.6 and the remainder of the § 1.3 retrieval assessment). An MCP - server lives for a whole client session, but every read tool re-walked and re-parsed the entire - corpus per call — at 1000 pages that is ~0.7 s before a single result is scored, and a - `wiki_search` cost ~1.4 s. A new `citadel/pagecache.py` keeps the last load in memory and - re-validates it on **every** consult with a stat-only `os.scandir` walk (~4 ms at 1000 pages) - over exactly the files `load()` parses; search additionally memoizes that snapshot's per-page - term-frequency tables. Measured at 1000 pages: `load()` ~700 ms → ~9 ms, `wiki_search` ~1.4 s → - ~50 ms, `tag:`/`type:` queries ~630 ms → ~11 ms. **Nothing is persisted** — the wiki stays the - database, there is no index file to go stale, and the filesystem still answers "did anything - change?" on every call. Staleness is designed out rather than hoped away: the fingerprint is - taken before AND after the load and must match (a page rewritten mid-load is never cached), a - snapshot whose newest stamp is younger than a 2 s settle window is not stored at all (a - coarse-timestamp filesystem could hide a same-length same-tick rewrite), the single slot is keyed - by wiki directory (ingest's staging redirect can never be served the live wiki, and the endless - stream of staging dirs cannot accumulate), `write_page`/`delete_page` invalidate directly, and - `ingest()`/`curate()` wear `@pagecache.bypass` so the staged diff-by-hash always reads the truth - from disk. Off by default: `citadel serve` opts in (`CITADEL_PAGE_CACHE=auto`), `1` enables it - wherever citadel reads the wiki, `0` restores the pre-cache behavior everywhere. -- **Ranked BM25 search behind the unchanged `search()` seam** (the 2026-07 audit's backlog #1). - Queries now share the offline viewer's grammar — bare terms are AND-matched (English stopwords - exempt, so "how do you brew coffee" matches on *brew coffee*; a query no page fully matches is - retried once as OR so the closest pages still surface), and `tag:x` / `type:y` tokens filter - instead of match (tag by prefix, type exactly; an operator-only query like `type:person` lists - the filtered pages) — closing the audit's "two divergent search implementations" finding. - Ranking is real BM25 (term-frequency saturation, prose-field length normalization, - Lucene-smoothed IDF) over the title 3.0 / aliases 2.5 / tags 2.0 / description 1.5 / body 1.0 - field ladder, plus the exact-phrase bonus, computed in memory per call — no persisted index, - the wiki stays the database, zero new dependencies. The audit-scoped SQLite FTS5 route was - built first and rejected on measurement: FTS5's `bm25()` clamps the IDF of any term appearing - in more than half the corpus to ~0, so in a topical wiki the topic word ("coffee" in a coffee - wiki) degenerated every score to noise; the Python scorer keeps IDF strictly positive, and - "how do you brew coffee" now ranks the brewing page first instead of losing it entirely. - Signature, MCP surface, and the `pages=` tag-filter seam are unchanged. - -### Added - - **Resumable chunked ingest** (the 2026-07 audit's backlog #9): a large source folded in over several segments no longer throws away the earlier segments' paid agent work when a run dies at segment N. Each completed segment banks the delta it produced — the *promote that would have @@ -294,6 +257,59 @@ All notable changes to this project are documented here. The format is based on automatically once made available offline (previously the stat quick check could skip the hydrated file forever). +### Changed + +- **`citadel serve` stops re-reading the whole wiki on every call** (the 2026-07 audit's backlog + #15, closing its finding 1.2.6 and the remainder of the § 1.3 retrieval assessment). An MCP + server lives for a whole client session, but every read tool re-walked and re-parsed the entire + corpus per call — at 1000 pages that is ~0.7 s before a single result is scored, and a + `wiki_search` cost ~1.4 s. A new `citadel/pagecache.py` keeps the last load in memory and + re-validates it on **every** consult with a stat-only `os.scandir` walk (~4 ms at 1000 pages) + over exactly the files `load()` parses; search additionally memoizes that snapshot's per-page + term-frequency tables. Measured at 1000 pages: `load()` ~700 ms → ~9 ms, `wiki_search` ~1.4 s → + ~50 ms, `tag:`/`type:` queries ~630 ms → ~11 ms. **Nothing is persisted** — the wiki stays the + database, there is no index file to go stale, and the filesystem still answers "did anything + change?" on every call. Staleness is designed out rather than hoped away: the fingerprint is + taken before AND after the load and must match (a page rewritten mid-load is never cached), a + snapshot whose newest stamp is younger than a 2 s settle window is not stored at all (a + coarse-timestamp filesystem could hide a same-length same-tick rewrite), the single slot is keyed + by wiki directory (ingest's staging redirect can never be served the live wiki, and the endless + stream of staging dirs cannot accumulate), `write_page`/`delete_page` invalidate directly, and + `ingest()`/`curate()` wear `@pagecache.bypass` so the staged diff-by-hash always reads the truth + from disk. Off by default: `citadel serve` opts in (`CITADEL_PAGE_CACHE=auto`), `1` enables it + wherever citadel reads the wiki, `0` restores the pre-cache behavior everywhere. +- **Ranked BM25 search behind the unchanged `search()` seam** (the 2026-07 audit's backlog #1). + Queries now share the offline viewer's grammar — bare terms are AND-matched (English stopwords + exempt, so "how do you brew coffee" matches on *brew coffee*; a query no page fully matches is + retried once as OR so the closest pages still surface), and `tag:x` / `type:y` tokens filter + instead of match (tag by prefix, type exactly; an operator-only query like `type:person` lists + the filtered pages) — closing the audit's "two divergent search implementations" finding. + Ranking is real BM25 (term-frequency saturation, prose-field length normalization, + Lucene-smoothed IDF) over the title 3.0 / aliases 2.5 / tags 2.0 / description 1.5 / body 1.0 + field ladder, plus the exact-phrase bonus, computed in memory per call — no persisted index, + the wiki stays the database, zero new dependencies. The audit-scoped SQLite FTS5 route was + built first and rejected on measurement: FTS5's `bm25()` clamps the IDF of any term appearing + in more than half the corpus to ~0, so in a topical wiki the topic word ("coffee" in a coffee + wiki) degenerated every score to noise; the Python scorer keeps IDF strictly positive, and + "how do you brew coffee" now ranks the brewing page first instead of losing it entirely. + Signature, MCP surface, and the `pages=` tag-filter seam are unchanged. + +### Fixed + +- **An auth-shaped session failure on a hermetic run now names the knob that caused it.** Hermetic + sessions (`CITADEL_HERMETIC=1`, the default) append claude's `--bare`, which deliberately skips + the user's personal agent configuration — and on machines where that configuration is also where + the CLI keeps its **credentials** (a managed container, a devcontainer, an `apiKeyHelper` in + `~/.claude/settings.json`), every session then failed on authentication while the same CLI worked + perfectly when run by hand. The backend reports that as *"Authentication error · This may be a + temporary network issue, please try again"*, which sends you hunting a network problem that does + not exist; nothing in the error, the report, or the failures catalog mentioned hermetic mode. + Auth-shaped failures raised from a run that really passed an isolation flag (read off the argv, so + the probe-gated case where nothing was passed is untouched) now carry a one-line hint naming + `CITADEL_HERMETIC=0`, and `docs/troubleshooting.md` gained the symptom with the fix. Found while + live-testing parallel ingest on this release: all four sources failed identically, and the + message pointed at the wrong layer. + ## [0.4.0] - 2026-07-16 ### Changed @@ -833,7 +849,8 @@ First public, pip-installable release (`pip install cite-citadel`), and the PyPI - Shared citation/link/fence parsing consolidated into `grammar.py`; viewer moved to a subpackage with a golden bundle test; Office/OLE extraction isolated. -[Unreleased]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.4.0...HEAD +[Unreleased]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.5.0...HEAD +[0.5.0]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.4.0...v0.5.0 [0.4.0]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.3.0...v0.4.0 [0.3.0]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.2.0...v0.3.0 [0.2.0]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.1.0...v0.2.0 diff --git a/CLAUDE.md b/CLAUDE.md index 1611069..3c2a867 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -434,7 +434,11 @@ save-the-transcript-as-a-file lane for whole conversations). `rawsource.py` back tests offline. - **Never hand-edit generated files** — `index.md`, `log.md`, any `*/index.md`, `sources/index.md`, `.citadel_viewer.html`, and `.citadel_ingested.json` are regenerated. The ingest agent prompt and - `store.delete_page` both refuse to touch them. + `store.delete_page` both refuse to touch them. In the REPO, `.github/copilot-instructions.md` is + generated too: it is THIS file with a swapped header (both agents get one instruction set, so a + feature can't be documented for one and not the other). Edit `CLAUDE.md`, then regenerate with + `CITADEL_WRITE_COPILOT_DOC=1 uv run pytest tests/test_packaging.py -k copilot -q`; the drift guard + in `tests/test_packaging.py` fails the suite when the two disagree. - **Provenance grammar is load-bearing:** raw facts cite `[^sN]` → a real `raw/` file; model-supplied facts use `[^llmN]` (source: `LLM`) and must never be disguised as raw citations. A `[^sN]` to a missing file fails lint/check. diff --git a/README.md b/README.md index 42e3f8f..6deb000 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,13 @@ doctor` warns when a newer release is on PyPI and prints the update command matc use `uv run python -m citadel` — the `uv run citadel` shorthand can be antivirus-blocked (see the contributor note below). +**A large backlog?** `citadel ingest --jobs N` folds N sources in at once, each on its own staging +copy, with every guarantee unmoved (one promote per source, all-or-nothing, nothing partial in the +wiki). The default is **1** because the cost of concurrency here is not safety but cross-linking — +concurrent sessions cannot see each other's new pages, so they link less richly, and `citadel +curate` is the designed cleanup. See +[docs/recipes.md](https://github.com/MarkusNeusinger/cite-citadel/blob/main/docs/recipes.md). + **Local models.** For a fully private wiki, point the same agent CLI at a local model (Ollama) so nothing you ingest ever leaves your machine or LAN — see [Local models (Ollama)](https://github.com/MarkusNeusinger/cite-citadel/blob/main/docs/configuration.md#local-models-ollama). diff --git a/citadel/__init__.py b/citadel/__init__.py index f02d6fd..6bab089 100644 --- a/citadel/__init__.py +++ b/citadel/__init__.py @@ -3,4 +3,4 @@ # The ONE source of the version: pyproject.toml reads it via hatch's dynamic version # ([tool.hatch.version] path = "citadel/__init__.py"), and `citadel --version` prints it. # No `: str` annotation — hatchling's default version regex would not match it. -__version__ = "0.4.0" +__version__ = "0.5.0" diff --git a/citadel/llm.py b/citadel/llm.py index 23bc1b5..09fb408 100644 --- a/citadel/llm.py +++ b/citadel/llm.py @@ -622,6 +622,30 @@ def _hermetic_flags(cli: str, cli_path: str) -> list[str]: return [flag for flag in flags if re.search(re.escape(flag) + r"(?![\w-])", help_text)] +# Auth-shaped failure signatures in a backend's own error text. +_AUTH_ERROR_RE = re.compile(r"authenticat|unauthorized|not logged in|credential|api[ _-]?key|\b401\b", re.I) + + +def _hermetic_auth_hint(cli: str, argv: list[str], message: str) -> str: + """The one hint that explains an auth-shaped session failure while hermetic mode is on, or ``""``. + + ``--bare`` deliberately skips the user's own agent configuration — and on a machine whose CLI + credentials live in exactly that configuration (a managed container, an ``apiKeyHelper`` setup), + EVERY citadel session then fails on authentication while the same CLI works fine interactively. + The backend blames the network for it (*"This may be a temporary network issue, please try + again"*), which sends you looking in the wrong place, so name the knob that actually explains + it. Derived from ``argv`` rather than the config, so the hint appears only when a flag really + was passed (hermetic mode is probe-gated: an older binary is handed nothing).""" + flags = [flag for flag in _HERMETIC_FLAGS.get(cli, ()) if flag in argv] + if not flags or not _AUTH_ERROR_RE.search(message): + return "" + return ( + f" [hermetic session isolation is on, so {' '.join(flags)} was passed: it skips your personal" + f" {cli} configuration, which on some machines is where the CLI keeps its credentials." + " If that CLI works interactively but every session fails here, set CITADEL_HERMETIC=0.]" + ) + + def _gemini_summary_file(cli: str, cli_path: str) -> Path | None: """A fresh temp file for gemini's ``--session-summary`` stats JSON, or None when the backend is not gemini or its binary does not ADVERTISE the flag in ``--help`` (probed once per @@ -922,11 +946,12 @@ def _run_session( env = _last_result_envelope(out) if isinstance(env, dict) and env.get("is_error"): status = env.get("api_error_status") - error = RuntimeError( + message = ( "claude CLI error" + (f" ({status})" if status else "") + f": {env.get('result') or err or 'unknown error'}" ) + error = RuntimeError(message + _hermetic_auth_hint(cli, argv, message)) # A failure envelope still reports what the session COST (error_max_turns, API # errors) — carry it on the exception so the run total counts the failed spend # (the documented "failed sessions included" contract; the manifest stamp stays @@ -934,14 +959,16 @@ def _run_session( error.session_usage = _usage_from_claude_envelope(env) raise error if returncode != 0: - error = RuntimeError(f"the claude CLI failed (exit {returncode}): {(err or out)[:500]}") + message = f"the claude CLI failed (exit {returncode}): {(err or out)[:500]}" + error = RuntimeError(message + _hermetic_auth_hint(cli, argv, message)) error.session_usage = _usage_from_claude_envelope(env) raise error return _usage_from_claude_envelope(env) # copilot / gemini (and any unknown CLI): the exit code is the success signal. if returncode != 0: - raise RuntimeError(f"the {cli!r} CLI failed (exit {returncode}): {(err or out)[:500]}") + message = f"the {cli!r} CLI failed (exit {returncode}): {(err or out)[:500]}" + raise RuntimeError(message + _hermetic_auth_hint(cli, argv, message)) return None diff --git a/docs/audit-2026-07.md b/docs/audit-2026-07.md index a09b4c0..57c5e27 100644 --- a/docs/audit-2026-07.md +++ b/docs/audit-2026-07.md @@ -30,6 +30,7 @@ comparison — and each has a design that closes it without compromising the pro Opus/Sonnet runs. Each filled cell is a single non-deterministic ingest (self-flagged). - **`refresh` is unreleased** — in CHANGELOG `[Unreleased]` while `__version__` is 0.4.0; a release PR is a pending act. + *✅ Resolved 2026-07-25 — released in 0.5.0 together with backlog #1–#15 (see § 4).* - **The verify-corpus "two optimization lanes" backlog is ephemeral** — every grading miss is routed into a creation lane (rules/prompts) or a retrieval lane (search), but the lane findings live only in per-run grade reports; nothing is persisted in the repo, so misses that didn't @@ -40,7 +41,9 @@ comparison — and each has a design that closes it without compromising the pro all-or-nothing (a failure at segment N discards N−1 segments' work — *the WORK half was closed 2026-07-24 by backlog #9's resume checkpoints; the promotion half stays all-or-nothing by design*), no self-update command, - `curate`/`refresh`/`view` CLI-only, no offline PDF/image reader, best-effort OLE salvage, + `curate`/`refresh`/`view` CLI-only, no offline image reader (*the PDF half of this line closed + 2026-07-24 with backlog #10 — a PDF's text layer now extracts offline through the bundled + pypdf; images stay agent-read*), best-effort OLE salvage, best-effort wiki-git, provider-side rate limiting. ### 1.2 Code-level findings @@ -124,12 +127,12 @@ Coverage is broad (44 test files, ~1:1 with modules, codecov patch gate 80%). Th |---|---| | `citadel/extract_ole.py` (222-line hand-rolled CFBF binary parser) | 2 offline tests; real coverage is the kontor corpus, which needs a live LLM run | | `viewer/app.js` (~2k lines: search, graph layout, popovers, keyboard nav) | zero automated tests — Python build/bundle only | +| `citadel/lint.py` | no dedicated test file; functionally covered but scattered across four files | +| `citadel/store_core.py` `neighbors_text`, `append_log`, `citadel/catalogs.py`, `citadel/linkgraph.py` | indirect/thin only | *✅ Rows 1–2 resolved 2026-07-24 — backlog #8 shipped (see § 4): a CFBF-writer fixture pins the container machinery (mini-FAT, DIFAT, multi-sector chains, cycle/truncation guards) offline, and a headless-Chromium smoke test drives the built viewer's JS in CI.* -| `citadel/lint.py` | no dedicated test file; functionally covered but scattered across four files | -| `citadel/store_core.py` `neighbors_text`, `append_log`, `citadel/catalogs.py`, `citadel/linkgraph.py` | indirect/thin only | ### 1.5 Operational gaps diff --git a/docs/configuration.md b/docs/configuration.md index ba41800..ac8b63d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -92,7 +92,7 @@ $env:CITADEL_LLM_CLI = "copilot" | Variable | Default | What it does | |----------|---------|--------------| | `CITADEL_LLM_TIMEOUT` | `1200` | Per-call CLI timeout in seconds. Raise it for opus or large raw files. | -| `CITADEL_HERMETIC` | `1` | Hermetic agent sessions: append the backend's session-isolation flag (claude `--bare` — skips user hooks/`CLAUDE.md`/MCP discovery) so your personal agent config never leaks into ingest. Only passed when the installed binary advertises the flag in `--help` (older CLIs run unchanged); `0` deliberately runs sessions with your personal config. copilot/gemini have no such flag today. | +| `CITADEL_HERMETIC` | `1` | Hermetic agent sessions: append the backend's session-isolation flag (claude `--bare` — skips user hooks/`CLAUDE.md`/MCP discovery) so your personal agent config never leaks into ingest. Only passed when the installed binary advertises the flag in `--help` (older CLIs run unchanged); `0` deliberately runs sessions with your personal config. copilot/gemini have no such flag today. **Set it to `0` if every session fails on authentication while the CLI works interactively** — on some setups (managed containers, an `apiKeyHelper`) the skipped personal config is where the credentials live; see [troubleshooting](troubleshooting.md#every-session-fails-with-an-authentication-error-but-the-cli-works-when-i-run-it-myself). | | `CITADEL_LLM_LOG_DIR` | (off) | Write one transcript per source (prompt + full CLI stdout/stderr + exit code + duration). Relative paths resolve under the workspace root. **Local-only — keep out of VCS** (transcripts can contain source content). CLI flag: `--log-dir`. | | `CITADEL_LLM_VERBOSE` | `0` | `1`/`true` streams each session's output live. CLI flag: `-v`. | diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 80f1c12..2388fc8 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -22,6 +22,25 @@ log in (each CLI's own auth flow), then set `CITADEL_LLM_CLI` in the workspace ` the binary isn't on `PATH`, point at it with `CLAUDE_CODE_PATH` / `COPILOT_CLI_PATH` / `GEMINI_CLI_PATH`. +### Every session fails with an authentication error, but the CLI works when I run it myself + +Hermetic sessions (`CITADEL_HERMETIC=1`, the default) append the backend's session-isolation flag — +claude's `--bare` — so your personal agent configuration never leaks into ingest. On some machines +that same configuration is where the CLI keeps its **credentials** (a managed container, a +devcontainer, an `apiKeyHelper` in `~/.claude/settings.json`), and skipping it leaves the session +unauthenticated. The backend reports this as *"Authentication error · This may be a temporary network +issue, please try again"*, which points at the network rather than the cause — so citadel appends the +hint naming this knob whenever an auth-shaped failure happens on a hermetic run. + +The fix is one line in the workspace `.env`: + +```bash +CITADEL_HERMETIC=0 # run sessions with your personal agent config (and its credentials) +``` + +You lose only the isolation (your hooks / `CLAUDE.md` / MCP servers are visible to ingest sessions +again). If instead the CLI fails interactively too, it is a plain login problem — see above. + ### Rate limits, or a session that runs too long and times out Each source gets one agent session, capped by `CITADEL_LLM_TIMEOUT` (default 1200s) — raise it for diff --git a/docs/verify-corpus-backlog.md b/docs/verify-corpus-backlog.md index 1b5c317..e8c68c2 100644 --- a/docs/verify-corpus-backlog.md +++ b/docs/verify-corpus-backlog.md @@ -42,10 +42,10 @@ Row format: | id | first seen | last seen | corpus | guarantee | lane | defect + route | status | |----|------------|-----------|--------|-----------|------|----------------|--------| - -*(No open entries yet — this ledger starts empty on 2026-07-24; the next grading run seeds it. -Historical per-run reports predating this file were not retro-imported: their sandboxes are gone, -so their misses cannot be re-verified.)* +| VCB-001 | 2026-07-25 | 2026-07-25 | beverages | offline-verifiable locators (schema.md § Sources) | wiki-generation | **haiku-only.** Haiku cites whole sections (`§ Caffeine — …`) where sonnet cites `line 9`, and it appends its own gloss to the heading, so 5 locators name a heading the source does not contain and `lint.check_locators` cannot verify any of them (`concepts/caffeine.md` ×2, `concepts/coffee.md`, `objects/aurora-midnight.md`, `organizations/caffe-aurora.md`; one concatenates two headings: `§ It Started With Lina, What We Actually Mean By "Sourcing" — company history…`). Route: `citadel/rules/schema.md`'s locator grammar — state that `§` takes the heading text VERBATIM and nothing else, and that `lines A-B` is preferred for a text source; the reference model needs no such spelling-out, a weaker one does. | open | +| VCB-002 | 2026-07-25 | 2026-07-25 | beverages | every factual sentence carries a footnote | wiki-generation | **haiku-only.** One uncited sentence survived the agent's own `citadel check` because it is inside a sentence the parser reads as cited: `concepts/coffee.md` — "The plant and its cultivation are dominated by two species: [Arabica and Robusta…". Lint's advisory `missing citations` caught it; `check` did not. Route: the `tasks/ingest.md` self-check step (re-read every paragraph for a trailing footnote), and worth asking whether this shape should be a `check` error rather than a lint advisory. | open | +| VCB-003 | 2026-07-25 | 2026-07-25 | beverages | `[^llmN]` is for model-supplied facts only, never a shortcut | wiki-generation | **haiku-only.** 4 pages carry model-supplied facts (`concepts/caffeine.md`, `concepts/tea.md`, `objects/aurora-midnight.md`, `organizations/thornbury-lin.md`) where the sonnet showcase of the same corpus has none — the corpus is fully covered by its raw sources, so an `[^llmN]` here is a weaker model reaching for the LLM lane instead of the source. Not a provenance violation (the lane is honest and lint surfaces it), so a soft miss. Route: `citadel/rules/core.md`'s `[^llmN]` section — make "prefer dropping the claim over sourcing it to yourself" explicit. | open | +| VCB-004 | 2026-07-25 | 2026-07-25 | beverages | dense cross-linking between related pages | wiki-generation | **Unattributed between model and `--jobs`.** 36 un-linked mentions where the committed sonnet showcase lints 0 (e.g. `concepts/tea.md → concepts/coffee.md`, `organizations/caffe-aurora.md → concepts/cold-brew.md`). Two candidate causes and this run cannot separate them: haiku links less densely, and `--jobs 4` means concurrent sessions cannot see each other's new pages (the documented, accepted cost of parallelism — `curate` is its designed cleanup). Route: re-run beverages on haiku with `--jobs 1` and diff this counter; only if serial haiku is also high is this a rules-lane miss rather than a parallelism trade-off. | open | ## Resolved @@ -57,3 +57,31 @@ so their misses cannot be re-verified.)* *(Newest first. One `### ` sub-block per grading run — mode, model, `rules_version`, verdict, and the `VCB-` ids the run touched, or `misses: none`. This `## Runs` heading stays singular; runs nest under it.)* + +### 2026-07-25 beverages + a purpose-built overlap corpus + +- **Mode:** Mode A, but scoped as a **parallelism + model** test rather than a full answer-key grade: + the question was whether `citadel ingest --jobs N` (shipped in 0.5.0) holds its guarantees under + real page contention, and how a weaker model behaves on the same corpus. Structural gates ran in + full; content grading was a comparison against the committed sonnet showcase of the same corpus + plus per-fact checks on the overlap corpus, not a walk of `beverages/ground-truth.md`. A full + answer-key grade is still owed. +- **Model:** `claude:haiku` · **rules_version:** `c99fc290864f` · `--jobs 4` · `CITADEL_HERMETIC=0` + (the container authenticates the CLI through user-level settings that `--bare` skips — the + hermetic-auth hint shipped in the same release came out of this). +- **Overlap corpus** (4 sources all describing one object, written for this run): forced **3 of 4 + sources to race** the same page. `check` and `lint` both exit 0 with every counter zero, the one + merged page cites all four sources, and every source's unique facts survived three consecutive + serial re-runs. Promote-once, all-or-nothing, and merge-into-the-winner hold under a 3-way + collision. 7 sessions / 277 s / $0.93. +- **beverages** (14 sources): 14/14 ingested, 0 failed, **3 raced**, 35 pages vs the showcase's 36, + `check` exit 0, `lint` exit 0 (0 broken links, 0 orphans, 0 fabricated sources, 0 wikilinks), + $3.55 recorded. The runner was killed mid-run by the harness, which incidentally verified the + crash contract: the live wiki stayed valid, 7 sources were durably recorded and 7 returned to + `pending`, and the resumed run re-ran exactly those 7 — nothing half-ingested, nothing paid for + twice. +- **Verdict:** parallel ingest PASSES on structure and provenance integrity. The deltas are model + quality, not concurrency: **VCB-001** (unverifiable `§` locators — the sharpest haiku-vs-sonnet + gap), **VCB-002** (one uncited sentence), **VCB-003** (unnecessary `[^llmN]` pages), and + **VCB-004** (cross-linking, unattributed between haiku and `--jobs` until a serial-haiku baseline + runs). diff --git a/tests/test_llm.py b/tests/test_llm.py index 1ff5e8c..8d2ada5 100644 --- a/tests/test_llm.py +++ b/tests/test_llm.py @@ -777,6 +777,54 @@ def fake_run_session(cli, argv, stdin_text, *, log_label=None): assert "--bare" not in seen["argv"] +def test_hermetic_auth_hint_only_when_the_flag_was_passed_and_the_failure_is_auth_shaped(): + """The diagnosis shortcut for the one failure hermetic mode can CAUSE: `--bare` skips the user's + own agent config, which on some machines (a managed container, an `apiKeyHelper` setup) is where + the CLI's credentials live — so every session fails on auth while the CLI works interactively, + and the backend blames the network for it. The hint names CITADEL_HERMETIC=0, and appears only + when the flag really was in the argv AND the message is auth-shaped (never on quota, timeouts, + or a non-hermetic run — a run whose argv lacks the flag must read exactly as before).""" + auth = "claude CLI error: Authentication error - This may be a temporary network issue" + hinted = llm._hermetic_auth_hint("claude", ["claude", "-p", "--bare"], auth) + assert "CITADEL_HERMETIC=0" in hinted and "--bare" in hinted + # Same failure, but hermetic mode was off (or the binary did not advertise the flag): no hint. + assert llm._hermetic_auth_hint("claude", ["claude", "-p"], auth) == "" + # Hermetic run, but the failure has nothing to do with credentials: no hint. + assert llm._hermetic_auth_hint("claude", ["claude", "-p", "--bare"], "claude CLI error (429): quota") == "" + # A CLI with no isolation flag registered can never trip it. + assert llm._hermetic_auth_hint("copilot", ["copilot", "-p", "--bare"], auth) == "" + + +def test_run_session_appends_the_hermetic_auth_hint_to_the_raised_error(monkeypatch): + """The wiring, on the envelope path AND the exit-code path: the hint reaches the message the run + report and the failures catalog show, so the operator sees it without reading a transcript.""" + + def fake_run(*a, **k): + return _FakeProc( + returncode=0, + stdout='{"type":"result","is_error":true,"result":"Authentication error - This may be a temporary network issue"}', + ) + + monkeypatch.setattr(subprocess, "run", fake_run) + with pytest.raises(RuntimeError) as exc: + llm._run_session("claude", ["claude", "-p", "--bare"], "PROMPT") + assert "CITADEL_HERMETIC=0" in str(exc.value) + + def fake_run_exit(*a, **k): + return _FakeProc(returncode=1, stdout="", stderr="Invalid API key") + + monkeypatch.setattr(subprocess, "run", fake_run_exit) + with pytest.raises(RuntimeError) as exc: + llm._run_session("claude", ["claude", "-p", "--bare"], "PROMPT") + assert "CITADEL_HERMETIC=0" in str(exc.value) + + # And the same auth failure WITHOUT the flag keeps the bare backend message. + monkeypatch.setattr(subprocess, "run", fake_run) + with pytest.raises(RuntimeError) as exc: + llm._run_session("claude", ["claude", "-p"], "PROMPT") + assert "CITADEL_HERMETIC" not in str(exc.value) + + def test_run_session_claude_is_error_raises(monkeypatch): """A claude result envelope with is_error=true raises (e.g. quota/auth).""" diff --git a/tests/test_packaging.py b/tests/test_packaging.py index 1e7bded..ab3792d 100644 --- a/tests/test_packaging.py +++ b/tests/test_packaging.py @@ -9,6 +9,7 @@ from __future__ import annotations +import os import re import tomllib from pathlib import Path @@ -18,11 +19,36 @@ ROOT = Path(__file__).resolve().parents[1] +CLAUDE_MD = ROOT / "CLAUDE.md" +COPILOT_MD = ROOT / ".github" / "copilot-instructions.md" + +# The one hand-written part of the generated Copilot file: its own title + provenance note. Everything +# below it is CLAUDE.md verbatim, so the two can never say different things about the same code. +COPILOT_HEADER = """# GitHub Copilot instructions — cite-citadel + +Repository guidance for GitHub Copilot. **Generated from [`CLAUDE.md`](../CLAUDE.md)** — do not edit +this file by hand: change `CLAUDE.md` and regenerate with +`CITADEL_WRITE_COPILOT_DOC=1 uv run pytest tests/test_packaging.py -k copilot -q`. The drift guard in +`tests/test_packaging.py` fails whenever the two disagree. + +""" + def _pyproject() -> dict: return tomllib.loads((ROOT / "pyproject.toml").read_text(encoding="utf-8")) +def _copilot_from_claude(claude_text: str) -> str: + """Derive `.github/copilot-instructions.md` from CLAUDE.md: swap the title/intro for the + generated-file header, and re-root repo-relative links one directory deeper (the Copilot file + lives in `.github/`).""" + first_section = re.search(r"^## ", claude_text, re.MULTILINE) + assert first_section, "CLAUDE.md has no `## ` section heading to generate from" + body = claude_text[first_section.start() :] + body = re.sub(r"\]\((?!https?://|#|\.\./|/)([^)]+)\)", r"](../\1)", body) + return COPILOT_HEADER + body + + def test_version_is_dynamic_and_hatch_reads_it_from_the_package_init(): data = _pyproject() assert "version" not in data["project"], "version must not be duplicated statically in [project]" @@ -99,6 +125,23 @@ def test_configuration_doc_covers_every_env_knob(): } +def test_copilot_instructions_mirror_claude_md(): + """Mechanical drift guard replacing the prose-only 'keep the two in sync' promise: the Copilot + instruction file is a pure derivation of CLAUDE.md (header swap + relative-link re-rooting), so a + feature documented in one is documented in both. Drift was real — the file shipped 105 lines + behind CLAUDE.md, still claiming 12 MCP tools and knowing nothing of `refresh`, `capture`, + `--jobs`, or `serve --http`. Set `CITADEL_WRITE_COPILOT_DOC=1` to regenerate instead of compare.""" + expected = _copilot_from_claude(CLAUDE_MD.read_text(encoding="utf-8")) + if os.environ.get("CITADEL_WRITE_COPILOT_DOC") == "1": + COPILOT_MD.write_text(expected, encoding="utf-8") + actual = COPILOT_MD.read_text(encoding="utf-8") + assert actual == expected, ( + "`.github/copilot-instructions.md` has drifted from CLAUDE.md — regenerate it with " + "`CITADEL_WRITE_COPILOT_DOC=1 uv run pytest tests/test_packaging.py -k copilot -q` " + "(edit CLAUDE.md, never the generated file)." + ) + + def test_readme_links_are_absolute_for_pypi(): """README.md ships as the PyPI long-description, where relative repo links 404 (owner report on the v0.1.0 release page). Every markdown link outside fenced code blocks must be absolute