release: 0.5.0 — release-prep audit, generated Copilot instructions, hermetic-auth hint

CHANGELOG.md

@@ -6,6 +6,8 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-07-25
+
 ### Added
 
 - **Bounded parallel ingest — `citadel ingest --jobs N`** (the 2026-07 audit's backlog #11, and
@@ -65,45 +67,6 @@ All notable changes to this project are documented here. The format is based on
   listen, whether the writers are exposed, and warns about a too-short token or a public bind before
   you ever start it.
 
-### Changed
-
-- **`citadel serve` stops re-reading the whole wiki on every call** (the 2026-07 audit's backlog
-  #15, closing its finding 1.2.6 and the remainder of the § 1.3 retrieval assessment). An MCP
-  server lives for a whole client session, but every read tool re-walked and re-parsed the entire
-  corpus per call — at 1000 pages that is ~0.7 s before a single result is scored, and a
-  `wiki_search` cost ~1.4 s. A new `citadel/pagecache.py` keeps the last load in memory and
-  re-validates it on **every** consult with a stat-only `os.scandir` walk (~4 ms at 1000 pages)
-  over exactly the files `load()` parses; search additionally memoizes that snapshot's per-page
-  term-frequency tables. Measured at 1000 pages: `load()` ~700 ms → ~9 ms, `wiki_search` ~1.4 s →
-  ~50 ms, `tag:`/`type:` queries ~630 ms → ~11 ms. **Nothing is persisted** — the wiki stays the
-  database, there is no index file to go stale, and the filesystem still answers "did anything
-  change?" on every call. Staleness is designed out rather than hoped away: the fingerprint is
-  taken before AND after the load and must match (a page rewritten mid-load is never cached), a
-  snapshot whose newest stamp is younger than a 2 s settle window is not stored at all (a
-  coarse-timestamp filesystem could hide a same-length same-tick rewrite), the single slot is keyed
-  by wiki directory (ingest's staging redirect can never be served the live wiki, and the endless
-  stream of staging dirs cannot accumulate), `write_page`/`delete_page` invalidate directly, and
-  `ingest()`/`curate()` wear `@pagecache.bypass` so the staged diff-by-hash always reads the truth
-  from disk. Off by default: `citadel serve` opts in (`CITADEL_PAGE_CACHE=auto`), `1` enables it
-  wherever citadel reads the wiki, `0` restores the pre-cache behavior everywhere.
-- **Ranked BM25 search behind the unchanged `search()` seam** (the 2026-07 audit's backlog #1).
-  Queries now share the offline viewer's grammar — bare terms are AND-matched (English stopwords
-  exempt, so "how do you brew coffee" matches on *brew coffee*; a query no page fully matches is
-  retried once as OR so the closest pages still surface), and `tag:x` / `type:y` tokens filter
-  instead of match (tag by prefix, type exactly; an operator-only query like `type:person` lists
-  the filtered pages) — closing the audit's "two divergent search implementations" finding.
-  Ranking is real BM25 (term-frequency saturation, prose-field length normalization,
-  Lucene-smoothed IDF) over the title 3.0 / aliases 2.5 / tags 2.0 / description 1.5 / body 1.0
-  field ladder, plus the exact-phrase bonus, computed in memory per call — no persisted index,
-  the wiki stays the database, zero new dependencies. The audit-scoped SQLite FTS5 route was
-  built first and rejected on measurement: FTS5's `bm25()` clamps the IDF of any term appearing
-  in more than half the corpus to ~0, so in a topical wiki the topic word ("coffee" in a coffee
-  wiki) degenerated every score to noise; the Python scorer keeps IDF strictly positive, and
-  "how do you brew coffee" now ranks the brewing page first instead of losing it entirely.
-  Signature, MCP surface, and the `pages=` tag-filter seam are unchanged.
-
-### Added
-
 - **Resumable chunked ingest** (the 2026-07 audit's backlog #9): a large source folded in over
   several segments no longer throws away the earlier segments' paid agent work when a run dies at
   segment N. Each completed segment banks the delta it produced — the *promote that would have
@@ -294,6 +257,59 @@ All notable changes to this project are documented here. The format is based on
   automatically once made available offline (previously the stat quick check could skip the
   hydrated file forever).
 
+### Changed
+
+- **`citadel serve` stops re-reading the whole wiki on every call** (the 2026-07 audit's backlog
+  #15, closing its finding 1.2.6 and the remainder of the § 1.3 retrieval assessment). An MCP
+  server lives for a whole client session, but every read tool re-walked and re-parsed the entire
+  corpus per call — at 1000 pages that is ~0.7 s before a single result is scored, and a
+  `wiki_search` cost ~1.4 s. A new `citadel/pagecache.py` keeps the last load in memory and
+  re-validates it on **every** consult with a stat-only `os.scandir` walk (~4 ms at 1000 pages)
+  over exactly the files `load()` parses; search additionally memoizes that snapshot's per-page
+  term-frequency tables. Measured at 1000 pages: `load()` ~700 ms → ~9 ms, `wiki_search` ~1.4 s →
+  ~50 ms, `tag:`/`type:` queries ~630 ms → ~11 ms. **Nothing is persisted** — the wiki stays the
+  database, there is no index file to go stale, and the filesystem still answers "did anything
+  change?" on every call. Staleness is designed out rather than hoped away: the fingerprint is
+  taken before AND after the load and must match (a page rewritten mid-load is never cached), a
+  snapshot whose newest stamp is younger than a 2 s settle window is not stored at all (a
+  coarse-timestamp filesystem could hide a same-length same-tick rewrite), the single slot is keyed
+  by wiki directory (ingest's staging redirect can never be served the live wiki, and the endless
+  stream of staging dirs cannot accumulate), `write_page`/`delete_page` invalidate directly, and
+  `ingest()`/`curate()` wear `@pagecache.bypass` so the staged diff-by-hash always reads the truth
+  from disk. Off by default: `citadel serve` opts in (`CITADEL_PAGE_CACHE=auto`), `1` enables it
+  wherever citadel reads the wiki, `0` restores the pre-cache behavior everywhere.
+- **Ranked BM25 search behind the unchanged `search()` seam** (the 2026-07 audit's backlog #1).
+  Queries now share the offline viewer's grammar — bare terms are AND-matched (English stopwords
+  exempt, so "how do you brew coffee" matches on *brew coffee*; a query no page fully matches is
+  retried once as OR so the closest pages still surface), and `tag:x` / `type:y` tokens filter
+  instead of match (tag by prefix, type exactly; an operator-only query like `type:person` lists
+  the filtered pages) — closing the audit's "two divergent search implementations" finding.
+  Ranking is real BM25 (term-frequency saturation, prose-field length normalization,
+  Lucene-smoothed IDF) over the title 3.0 / aliases 2.5 / tags 2.0 / description 1.5 / body 1.0
+  field ladder, plus the exact-phrase bonus, computed in memory per call — no persisted index,
+  the wiki stays the database, zero new dependencies. The audit-scoped SQLite FTS5 route was
+  built first and rejected on measurement: FTS5's `bm25()` clamps the IDF of any term appearing
+  in more than half the corpus to ~0, so in a topical wiki the topic word ("coffee" in a coffee
+  wiki) degenerated every score to noise; the Python scorer keeps IDF strictly positive, and
+  "how do you brew coffee" now ranks the brewing page first instead of losing it entirely.
+  Signature, MCP surface, and the `pages=` tag-filter seam are unchanged.
+
+### Fixed
+
+- **An auth-shaped session failure on a hermetic run now names the knob that caused it.** Hermetic
+  sessions (`CITADEL_HERMETIC=1`, the default) append claude's `--bare`, which deliberately skips
+  the user's personal agent configuration — and on machines where that configuration is also where
+  the CLI keeps its **credentials** (a managed container, a devcontainer, an `apiKeyHelper` in
+  `~/.claude/settings.json`), every session then failed on authentication while the same CLI worked
+  perfectly when run by hand. The backend reports that as *"Authentication error · This may be a
+  temporary network issue, please try again"*, which sends you hunting a network problem that does
+  not exist; nothing in the error, the report, or the failures catalog mentioned hermetic mode.
+  Auth-shaped failures raised from a run that really passed an isolation flag (read off the argv, so
+  the probe-gated case where nothing was passed is untouched) now carry a one-line hint naming
+  `CITADEL_HERMETIC=0`, and `docs/troubleshooting.md` gained the symptom with the fix. Found while
+  live-testing parallel ingest on this release: all four sources failed identically, and the
+  message pointed at the wrong layer.
+
 ## [0.4.0] - 2026-07-16
 
 ### Changed
@@ -833,7 +849,8 @@ First public, pip-installable release (`pip install cite-citadel`), and the PyPI
 - Shared citation/link/fence parsing consolidated into `grammar.py`; viewer moved to a subpackage
   with a golden bundle test; Office/OLE extraction isolated.
 
-[Unreleased]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.4.0...HEAD
+[Unreleased]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.5.0...HEAD
+[0.5.0]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.4.0...v0.5.0
 [0.4.0]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.2.0...v0.3.0
 [0.2.0]: https://github.com/MarkusNeusinger/cite-citadel/compare/v0.1.0...v0.2.0

CLAUDE.md

@@ -434,7 +434,11 @@ save-the-transcript-as-a-file lane for whole conversations). `rawsource.py` back
   tests offline.
 - **Never hand-edit generated files** — `index.md`, `log.md`, any `*/index.md`, `sources/index.md`,
   `.citadel_viewer.html`, and `.citadel_ingested.json` are regenerated. The ingest agent prompt and
-  `store.delete_page` both refuse to touch them.
+  `store.delete_page` both refuse to touch them. In the REPO, `.github/copilot-instructions.md` is
+  generated too: it is THIS file with a swapped header (both agents get one instruction set, so a
+  feature can't be documented for one and not the other). Edit `CLAUDE.md`, then regenerate with
+  `CITADEL_WRITE_COPILOT_DOC=1 uv run pytest tests/test_packaging.py -k copilot -q`; the drift guard
+  in `tests/test_packaging.py` fails the suite when the two disagree.
 - **Provenance grammar is load-bearing:** raw facts cite `[^sN]` → a real `raw/` file; model-supplied
   facts use `[^llmN]` (source: `LLM`) and must never be disguised as raw citations. A `[^sN]` to a
   missing file fails lint/check.

README.md

@@ -62,6 +62,13 @@ doctor` warns when a newer release is on PyPI and prints the update command matc
 use `uv run python -m citadel` — the `uv run citadel` shorthand can be antivirus-blocked (see the
 contributor note below).
 
+**A large backlog?** `citadel ingest --jobs N` folds N sources in at once, each on its own staging
+copy, with every guarantee unmoved (one promote per source, all-or-nothing, nothing partial in the
+wiki). The default is **1** because the cost of concurrency here is not safety but cross-linking —
+concurrent sessions cannot see each other's new pages, so they link less richly, and `citadel
+curate` is the designed cleanup. See
+[docs/recipes.md](https://github.com/MarkusNeusinger/cite-citadel/blob/main/docs/recipes.md).
+
 **Local models.** For a fully private wiki, point the same agent CLI at a local model (Ollama) so
 nothing you ingest ever leaves your machine or LAN — see
 [Local models (Ollama)](https://github.com/MarkusNeusinger/cite-citadel/blob/main/docs/configuration.md#local-models-ollama).

citadel/__init__.py

@@ -3,4 +3,4 @@
 # The ONE source of the version: pyproject.toml reads it via hatch's dynamic version
 # ([tool.hatch.version] path = "citadel/__init__.py"), and `citadel --version` prints it.
 # No `: str` annotation — hatchling's default version regex would not match it.
-__version__ = "0.4.0"
+__version__ = "0.5.0"

citadel/llm.py

@@ -622,6 +622,30 @@ def _hermetic_flags(cli: str, cli_path: str) -> list[str]:
     return [flag for flag in flags if re.search(re.escape(flag) + r"(?![\w-])", help_text)]
 
 
+# Auth-shaped failure signatures in a backend's own error text.
+_AUTH_ERROR_RE = re.compile(r"authenticat|unauthorized|not logged in|credential|api[ _-]?key|\b401\b", re.I)
+
+
+def _hermetic_auth_hint(cli: str, argv: list[str], message: str) -> str:
+    """The one hint that explains an auth-shaped session failure while hermetic mode is on, or ``""``.
+
+    ``--bare`` deliberately skips the user's own agent configuration — and on a machine whose CLI
+    credentials live in exactly that configuration (a managed container, an ``apiKeyHelper`` setup),
+    EVERY citadel session then fails on authentication while the same CLI works fine interactively.
+    The backend blames the network for it (*"This may be a temporary network issue, please try
+    again"*), which sends you looking in the wrong place, so name the knob that actually explains
+    it. Derived from ``argv`` rather than the config, so the hint appears only when a flag really
+    was passed (hermetic mode is probe-gated: an older binary is handed nothing)."""
+    flags = [flag for flag in _HERMETIC_FLAGS.get(cli, ()) if flag in argv]
+    if not flags or not _AUTH_ERROR_RE.search(message):
+        return ""
+    return (
+        f" [hermetic session isolation is on, so {' '.join(flags)} was passed: it skips your personal"
+        f" {cli} configuration, which on some machines is where the CLI keeps its credentials."
+        " If that CLI works interactively but every session fails here, set CITADEL_HERMETIC=0.]"
+    )
+
+
 def _gemini_summary_file(cli: str, cli_path: str) -> Path | None:
     """A fresh temp file for gemini's ``--session-summary`` stats JSON, or None when the backend
     is not gemini or its binary does not ADVERTISE the flag in ``--help`` (probed once per
@@ -922,26 +946,29 @@ def _run_session(
                 env = _last_result_envelope(out)
         if isinstance(env, dict) and env.get("is_error"):
             status = env.get("api_error_status")
-            error = RuntimeError(
+            message = (
                 "claude CLI error"
                 + (f" ({status})" if status else "")
                 + f": {env.get('result') or err or 'unknown error'}"
             )
+            error = RuntimeError(message + _hermetic_auth_hint(cli, argv, message))
             # A failure envelope still reports what the session COST (error_max_turns, API
             # errors) — carry it on the exception so the run total counts the failed spend
             # (the documented "failed sessions included" contract; the manifest stamp stays
             # success-only regardless).
             error.session_usage = _usage_from_claude_envelope(env)
             raise error
         if returncode != 0:
-            error = RuntimeError(f"the claude CLI failed (exit {returncode}): {(err or out)[:500]}")
+            message = f"the claude CLI failed (exit {returncode}): {(err or out)[:500]}"
+            error = RuntimeError(message + _hermetic_auth_hint(cli, argv, message))
             error.session_usage = _usage_from_claude_envelope(env)
             raise error
         return _usage_from_claude_envelope(env)
 
     # copilot / gemini (and any unknown CLI): the exit code is the success signal.
     if returncode != 0:
-        raise RuntimeError(f"the {cli!r} CLI failed (exit {returncode}): {(err or out)[:500]}")
+        message = f"the {cli!r} CLI failed (exit {returncode}): {(err or out)[:500]}"
+        raise RuntimeError(message + _hermetic_auth_hint(cli, argv, message))
     return None
 
 

docs/audit-2026-07.md

@@ -30,6 +30,7 @@ comparison — and each has a design that closes it without compromising the pro
   Opus/Sonnet runs. Each filled cell is a single non-deterministic ingest (self-flagged).
 - **`refresh` is unreleased** — in CHANGELOG `[Unreleased]` while `__version__` is 0.4.0; a
   release PR is a pending act.
+  *✅ Resolved 2026-07-25 — released in 0.5.0 together with backlog #1–#15 (see § 4).*
 - **The verify-corpus "two optimization lanes" backlog is ephemeral** — every grading miss is
   routed into a creation lane (rules/prompts) or a retrieval lane (search), but the lane findings
   live only in per-run grade reports; nothing is persisted in the repo, so misses that didn't
@@ -40,7 +41,9 @@ comparison — and each has a design that closes it without compromising the pro
   all-or-nothing (a failure at segment N discards N−1 segments' work — *the WORK half was closed
   2026-07-24 by backlog #9's resume checkpoints; the promotion half stays all-or-nothing by
   design*), no self-update command,
-  `curate`/`refresh`/`view` CLI-only, no offline PDF/image reader, best-effort OLE salvage,
+  `curate`/`refresh`/`view` CLI-only, no offline image reader (*the PDF half of this line closed
+  2026-07-24 with backlog #10 — a PDF's text layer now extracts offline through the bundled
+  pypdf; images stay agent-read*), best-effort OLE salvage,
   best-effort wiki-git, provider-side rate limiting.
 
 ### 1.2 Code-level findings
@@ -124,12 +127,12 @@ Coverage is broad (44 test files, ~1:1 with modules, codecov patch gate 80%). Th
 |---|---|
 | `citadel/extract_ole.py` (222-line hand-rolled CFBF binary parser) | 2 offline tests; real coverage is the kontor corpus, which needs a live LLM run |
 | `viewer/app.js` (~2k lines: search, graph layout, popovers, keyboard nav) | zero automated tests — Python build/bundle only |
+| `citadel/lint.py` | no dedicated test file; functionally covered but scattered across four files |
+| `citadel/store_core.py` `neighbors_text`, `append_log`, `citadel/catalogs.py`, `citadel/linkgraph.py` | indirect/thin only |
 
 *✅ Rows 1–2 resolved 2026-07-24 — backlog #8 shipped (see § 4): a CFBF-writer fixture pins the
 container machinery (mini-FAT, DIFAT, multi-sector chains, cycle/truncation guards) offline, and
 a headless-Chromium smoke test drives the built viewer's JS in CI.*
-| `citadel/lint.py` | no dedicated test file; functionally covered but scattered across four files |
-| `citadel/store_core.py` `neighbors_text`, `append_log`, `citadel/catalogs.py`, `citadel/linkgraph.py` | indirect/thin only |
 
 ### 1.5 Operational gaps
 

docs/configuration.md

@@ -92,7 +92,7 @@ $env:CITADEL_LLM_CLI = "copilot"
 | Variable | Default | What it does |
 |----------|---------|--------------|
 | `CITADEL_LLM_TIMEOUT` | `1200` | Per-call CLI timeout in seconds. Raise it for opus or large raw files. |
-| `CITADEL_HERMETIC` | `1` | Hermetic agent sessions: append the backend's session-isolation flag (claude `--bare` — skips user hooks/`CLAUDE.md`/MCP discovery) so your personal agent config never leaks into ingest. Only passed when the installed binary advertises the flag in `--help` (older CLIs run unchanged); `0` deliberately runs sessions with your personal config. copilot/gemini have no such flag today. |
+| `CITADEL_HERMETIC` | `1` | Hermetic agent sessions: append the backend's session-isolation flag (claude `--bare` — skips user hooks/`CLAUDE.md`/MCP discovery) so your personal agent config never leaks into ingest. Only passed when the installed binary advertises the flag in `--help` (older CLIs run unchanged); `0` deliberately runs sessions with your personal config. copilot/gemini have no such flag today. **Set it to `0` if every session fails on authentication while the CLI works interactively** — on some setups (managed containers, an `apiKeyHelper`) the skipped personal config is where the credentials live; see [troubleshooting](troubleshooting.md#every-session-fails-with-an-authentication-error-but-the-cli-works-when-i-run-it-myself). |
 | `CITADEL_LLM_LOG_DIR` | (off) | Write one transcript per source (prompt + full CLI stdout/stderr + exit code + duration). Relative paths resolve under the workspace root. **Local-only — keep out of VCS** (transcripts can contain source content). CLI flag: `--log-dir`. |
 | `CITADEL_LLM_VERBOSE` | `0` | `1`/`true` streams each session's output live. CLI flag: `-v`. |