Merge pull request #133 from MarkusNeusinger/claude/llm-backend-model-and-usage-reporting

llm: pass --model to every backend, replace gemini with agy, report real spend

Author: MarkusNeusinger

.claude/skills/bench-model/SKILL.md

@@ -1,11 +1,11 @@
 ---
 name: bench-model
-description: Benchmark an LLM (model and/or agent CLI) on citadel's wiki-building quality — the model-focused twin of verify-corpus (which tests the PIPELINE with a fixed model, while bench-model tests a MODEL with the fixed pipeline). Mode A ingests a corpus into a throwaway sandbox with the chosen CITADEL_INGEST_MODEL / CITADEL_LLM_CLI, grades it with verify-corpus's retrieval-first method, then applies a DISCRIMINATIVE tier (locator precision, oblique-query retrieval, merge quality, redundancy/cross-links, judgment delta on contradictions + planted-false claims) so runs by models of different strength never tie at the top — if two models both ace the grade, the test was too easy, which is itself a finding. Ends with a side-by-side metrics table and a verdict (is the cheaper model's wiki acceptable, where does it degrade first, what rule changes would close the gap). Use whenever the user wants to compare models on wiki creation (sonnet vs haiku, a new Claude model, gemini/copilot, or open/local models via the CITADEL_LLM_CLI seam), asks "does model X suffice for ingest?", wants a cost/quality trade-off measured, or wants to re-run the model bench from the 2026-07 audit — even if they do not say the word "skill". Takes a corpus name, a model id, and optionally a CLI and a baseline sandbox to compare against.
+description: Benchmark an LLM (model and/or agent CLI) on citadel's wiki-building quality — the model-focused twin of verify-corpus (which tests the PIPELINE with a fixed model, while bench-model tests a MODEL with the fixed pipeline). Mode A ingests a corpus into a throwaway sandbox with the chosen CITADEL_INGEST_MODEL / CITADEL_LLM_CLI, grades it with verify-corpus's retrieval-first method, then applies a DISCRIMINATIVE tier (locator precision, oblique-query retrieval, merge quality, redundancy/cross-links, judgment delta on contradictions + planted-false claims) so runs by models of different strength never tie at the top — if two models both ace the grade, the test was too easy, which is itself a finding. Ends with a side-by-side metrics table and a verdict (is the cheaper model's wiki acceptable, where does it degrade first, what rule changes would close the gap). Use whenever the user wants to compare models on wiki creation (sonnet vs haiku, a new Claude model, agy/copilot, or open/local models via the CITADEL_LLM_CLI seam), asks "does model X suffice for ingest?", wants a cost/quality trade-off measured, or wants to re-run the model bench from the 2026-07 audit — even if they do not say the word "skill". Takes a corpus name, a model id, and optionally a CLI and a baseline sandbox to compare against.
 ---
 
 # Bench a model on wiki creation
 
-**Usage:** `bench-model <corpus> <model> [--cli claude|copilot|gemini] [--baseline SANDBOX] [--grade-only SANDBOX]`
+**Usage:** `bench-model <corpus> <model> [--cli claude|copilot|agy] [--baseline SANDBOX] [--grade-only SANDBOX]`
 
 Everything sandbox/grading-related follows **verify-corpus** (read its SKILL.md first — sandbox
 recipe, phase 1 structural gates, phase 2 retrieval-first grading, the creation-vs-retrieval miss
@@ -36,7 +36,7 @@ REPO="$(git rev-parse --show-toplevel)"
 SANDBOX="$(mktemp -d)/bench-<corpus>-<model>"
 uv run python -m citadel init "$SANDBOX"
 cat > "$SANDBOX/.env" <<EOF
-CITADEL_LLM_CLI=claude                 # or copilot / gemini — the seam for open/local models
+CITADEL_LLM_CLI=claude                 # or copilot / agy — the seam for open/local models
 CITADEL_INGEST_MODEL=<model>
 CITADEL_LLM_LOG_DIR=$SANDBOX/logs
 CITADEL_RAW_DIR=$REPO/corpora/<corpus>/raw

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -9,7 +9,7 @@ body:
         version and how to reproduce it are the most helpful parts.
 
         Reminder: cite-citadel ships no LLM SDK and no API key. Ingest shells out to a
-        coding-agent CLI (`claude` / `copilot` / `gemini`) that *you* have installed and
+        coding-agent CLI (`claude` / `copilot` / `agy`) that *you* have installed and
         logged in. If the problem is really about that CLI's own auth or billing, it may
         belong upstream — but file here if you are unsure.
 
@@ -48,7 +48,7 @@ body:
       options:
         - claude
         - copilot
-        - gemini
+        - agy (Antigravity CLI)
         - other / local model
         - not applicable (read-only command)
     validations:

.github/copilot-instructions.md

@@ -11,10 +11,10 @@ this file by hand: change `CLAUDE.md` and regenerate with
 personal wiki in Google's [Open Knowledge Format](../docs/okf-reference.md), with an MCP server so an
 AI can search and read it. It implements Karpathy's LLM-Wiki pattern: drop arbitrary text-bearing
 files into `raw/`, and one agentic CLI session per source folds each into a cross-linked OKF wiki
-under `wiki/`. Pure Python 3.12, KISS. Runtime deps are only `mcp`, `pyyaml`, and `pypdf` (all
-pure-Python, no native/transitive weight) — **there is no LLM SDK and no API key**: ingest shells
+under `wiki/`. Pure Python 3.12, KISS. Runtime deps are only `mcp`, `pyyaml`, `pypdf`, and `rich`
+(all pure-Python, no native weight) — **there is no LLM SDK and no API key**: ingest shells
 out to a coding-agent CLI you already have logged in
-(`claude`/`copilot`/`gemini`).
+(`claude`/`copilot`/`agy`).
 
 ## Commands
 
@@ -30,7 +30,7 @@ uv run python -m citadel <subcommand>
 
 Subcommands: `init [DIR]` (scaffold a workspace: `citadel.toml` marker, `.env`, `raw/`, `wiki/`;
 idempotent), `ingest [paths…]` (fold raw/ into the wiki; `--verbose`/`-v` streams the agent
-session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the progress spinner,
+session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the live progress display,
 `--jobs N`/`-j` folds N sources in CONCURRENTLY (default 1 = serial; `CITADEL_JOBS`),
 `--full-rescan` distrusts the manifest's stat cache and re-hashes every tracked source,
 `--force <paths>` deliberately re-reads already-ingested sources as a reconcile — it requires
@@ -44,7 +44,7 @@ contradictions/fix locators — against a recomputed findings checklist), `statu
 per-source state table: ingested / failed / skipped-duplicate / ignored / oversized / pending; MCP twin
 `wiki_status`), `doctor`
 (read-only setup health check — OK/WARN/FAIL lines for workspace / rules / config-parse fallbacks /
-agent CLI / the inert-`CITADEL_INGEST_MODEL`-on-copilot/gemini advisory / raw roots / wiki placement
+agent CLI / the configured ingest model / raw roots / wiki placement
 (the wiki nested inside a raw root) / child paths (the UNC-vs-drive-letter cwd) /
 manifest / billing / the HTTP-serve posture / wiki-git state / a best-effort PyPI update check / workspace coherence; needs no workspace, exits 1 only on a FAIL),
 `serve [--http [--host H] [--port P] [--path /mcp] [--read-only]]` (the MCP server — stdio by
@@ -276,10 +276,21 @@ git repo folded as one digest), `image`/`image-reconcile` (an image source read
 raw source). A large source is split into segments and folded in over several passes
 (`segment=(part, total)` on `run_ingest_session`, telling later passes to MERGE into earlier ones).
 `run_ingest_session` is the single seam tests monkeypatch; it returns the session's best-effort
-`SessionUsage` (the backend's OWN cost/usage report: claude's result envelope, gemini's
-`--session-summary` behind a cached `--help` feature probe; None when nothing was reported —
-accounting is strictly passive and can never fail a session), which ingest sums per source into
-the manifest stamp and per run onto the reports.
+`SessionUsage` (the backend's OWN report of what it spent AND which model actually served the
+session: claude's result envelope - `total_cost_usd` plus the `modelUsage` map, whose PRIMARY
+entry is the one carrying the token volume, since claude routes cheap side work to a smaller
+model; copilot's `--output-format json` JSONL - no dollars, so its own billing unit is recorded
+instead: `totalNanoAiu`, the counter behind the `N AIC used` session footer (1 AIC = 1e9 nanoAiu),
+converted to `cost_usd` at GitHub's fixed published $0.01/credit so a mixed corpus keeps ONE
+comparable total while the un-derived credits stay stamped beside it (the retired
+`totalPremiumRequests` is deliberately ignored); agy's `--output-format stream-json` - the
+opening `init.model` plus the
+closing `result.usage` token totals; None when nothing was reported - accounting is strictly
+passive and can never fail a session), which ingest sums per source into the manifest stamp and
+per run onto the reports. **The REPORTED model wins**: `config.model_label_for(reported)` stamps
+the backend plus what actually ran, and falls back to the configured label only when the backend
+named nothing (an Ollama/proxy backend, or agy left on its own default), so the manifest never
+claims a model that never ran.
 
 **Two checking layers, one implementation** (`validate.py`):
 - `citadel check` / `wiki_validate` — the **strict per-page gate** (required fields, honest/defined
@@ -324,11 +335,18 @@ repeated runs walk the corpus round-robin with NO persisted queue (the manifest
 The budget is always explicit (`limit >= 1` enforced, default 1), mirroring `--force`'s
 no-accidental-corpus-wide-run refusal. CLI-only, like curate.
 
+An auth-shaped session failure under hermetic isolation is **retried once without the isolation
+flags**: on a machine whose CLI credentials live in exactly the personal config `--bare` skips,
+every session used to die on authentication until the user found `CITADEL_HERMETIC=0`. The retry
+is scoped to that signature (flags actually passed + an auth-shaped message), so a real credential
+problem still fails instead of looping.
+
 **Status is the read-only corpus view** (`status.py`, `citadel status`): the manifest + failures
 catalog + one stat-only walk (never re-hashes) rendered as a per-source state table — ingested
 (model + rules_version, `(stale)` when it predates the current rulebook, `checked YYYY-MM-DD` from
-the `ingested_at` stamp, the last session's cost when recorded — with a `Recorded LLM cost` corpus
-total above the table), failed (reason, attempts),
+the `ingested_at` stamp, the last session's cost when recorded, with copilot's AI credits shown
+beside the dollars they converted into — with `Recorded LLM cost` / `Recorded AI credits` corpus
+totals above the table), failed (reason, attempts),
 skipped-duplicate, ignored (pattern), oversized (over `CITADEL_MAX_SOURCE_BYTES`, with the size),
 pending.
 
@@ -390,6 +408,21 @@ setup health check (OK/WARN/FAIL lines over workspace resolution, the rules tree
 PATH, raw-root reachability, manifest parse + stamp, failures summary, the API-key/PDF/audio
 advisories, the wiki-git state, a best-effort PyPI update check naming the right upgrade command
 per install method, and workspace coherence).
+`progress.py` is the CLI's live console reporter (`rich`), wired in only by `cmd_ingest`/`cmd_refresh`
+— the MCP server passes no progress, so its stdio stays clean. On a terminal it renders a live
+region: one spinner row per IN-FLIGHT source (which is what makes `--jobs N` legible) plus an
+overall bar, while finished sources scroll away above it as permanent one-line verdicts carrying
+what the session actually spent (`[2/3] OK raw/notes.md 18.4s 2 created $0.0123 1.2k in / 456 out
+claude-opus-5` — only fields the backend genuinely reported). Off a TTY or under `--verbose` it
+degrades to a START line plus a verdict line per source. Two invariants are load-bearing: every
+composed string is ASCII-only and the spinner is pinned to rich's ASCII `line` frames (a cp1252
+Windows console must be able to encode it), and every write goes through a swallow-everything guard
+— console output must never be able to fail a run that already spent money. Source keys are
+shortened by `config.display_key`, which drops the whole prefix before the raw folder in three
+tiers: an exact prefix match, a cut at the last path segment NAMED like a configured root (this is
+what rescues a Windows drive letter mapped to a share, where `T:\proj\raw` and the key's
+`//fileserver.../proj/raw/...` are one folder but share no text), and finally a `.../`-marked tail
+clip, so no absolute key can ever flood the console.
 `wikigit.py` is the best-effort wiki-HISTORY layer: after every run that changed the wiki (ingest or
 curate) it commits the whole wiki dir as ONE commit (and pushes to `CITADEL_WIKI_GIT_REMOTE` when
 set), so every change is a reviewable diff; `auto` (default) only acts when the wiki dir is already
@@ -473,7 +506,9 @@ save-the-transcript-as-a-file lane for whole conversations). `rawsource.py` back
   Windows/SMB failures.
 - Config knobs live in the workspace-root `.env` (auto-loaded, gitignored; template:
   `citadel/templates/env.example`): `CITADEL_LLM_CLI`,
-  `CITADEL_INGEST_MODEL`, `CITADEL_CURATE_MODEL` (model for `citadel curate` sessions; falls back to
+  `CITADEL_INGEST_MODEL` (passed as `--model` to EVERY backend - claude, copilot and agy all
+  honor it; unset (the default) means "run the CLI's own default model"),
+  `CITADEL_CURATE_MODEL` (model for `citadel curate` sessions; falls back to
   `CITADEL_INGEST_MODEL`), `CITADEL_LLM_TIMEOUT`, `CITADEL_HERMETIC` (session isolation — append claude's `--bare` when the
   installed binary advertises it, so personal `~/.claude` config never leaks into ingest; default
   on, probe-gated), `CITADEL_PAGE_CACHE` (the serve-side page snapshot cache: `auto` = on in

CHANGELOG.md

@@ -6,8 +6,65 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+### Changed (breaking)
+
+- **The `gemini` backend is replaced by `agy` (Google's Antigravity CLI).** The Gemini CLI it
+  wrapped no longer exists, so `CITADEL_LLM_CLI=gemini` now fails immediately with a migration
+  hint instead of shelling out to a missing binary (`citadel doctor` FAILs on it too). The new
+  backend runs `agy -p <prompt> --output-format stream-json --dangerously-skip-permissions`, grants
+  out-of-workspace directories with a repeatable `--add-dir`, and reports its token usage through
+  the closing `result` event — the `--session-summary` stats-file machinery is gone with the CLI it
+  probed for. Rename `GEMINI_CLI_PATH` to `AGY_CLI_PATH` if you set it.
+- **`CITADEL_INGEST_MODEL` now applies to every backend, and defaults to unset.** It used to be a
+  claude-only knob hard-defaulted to `sonnet`, inert on the others (`citadel doctor` actively warned
+  about it). `claude`, `copilot` and `agy` all honor `--model`, so it is passed through to all
+  three; leaving it unset runs the CLI's own default model, and the per-backend `COPILOT_MODEL` /
+  `GEMINI_MODEL` guesswork is gone (the model recorded per source is now the one the backend
+  *reported*). Set it explicitly if you were relying on the old implicit `sonnet`.
+
 ### Added
 
+- **The model a source was imported with is now the model that actually ran.** Each backend's own
+  session envelope is parsed for the serving model — claude's `modelUsage` map (the PRIMARY entry
+  by token volume, so the small model claude routes side work to never wins), copilot's JSONL
+  `assistant.message`/`model.call_start` events, agy's opening `init` event — and that id is what
+  gets stamped into the manifest as `<cli>:<model>`. The configured label is only a fallback for a
+  backend that named nothing (an Ollama/proxy setup, or agy left on its own default), so the wiki
+  can no longer claim a model that never served it.
+- **copilot spend is recorded in AI credits — and priced.** copilot quotes no dollars, so its
+  `totalNanoAiu` counter (the `N AIC used` figure its interactive session footer shows;
+  1 AIC = 1e9 nanoAiu) is stamped as a new `aic` field, and converted to `cost_usd` at GitHub's
+  fixed published $0.01/credit — so a mixed claude+copilot corpus finally has ONE comparable cost
+  total, with the un-derived credits always shown beside the dollars (`$0.0251 (2.5083 AIC)`).
+  `citadel status` gained a `Recorded AI credits` total and a `--json` `aic_total`. The retired
+  `premiumRequests` counter is deliberately not recorded — GitHub replaced it with credits, and it
+  was far too coarse to describe a session (every ingest is "1"). copilot also runs under
+  `--output-format json` now — `-s` trimmed away the very envelope that names the model and counts
+  the spend.
+- **An authentication failure under hermetic isolation now retries once without it.** On machines
+  where the CLI's credentials live in exactly the personal configuration `--bare` skips (managed
+  containers, an `apiKeyHelper`, some WSL setups), *every* session failed with "Not logged in"
+  until you found `CITADEL_HERMETIC=0` — while the same CLI worked fine interactively. Isolation is
+  a hardening nicety; ingesting at all is the product. The retry is scoped to that exact signature
+  (an isolation flag really was passed **and** the message is auth-shaped), so a genuinely
+  logged-out CLI still fails instead of looping.
+- **Full provenance under every source in `citadel view` and `sources/index.md`.** The viewer's
+  source reader and hover popover show the model, what the import cost (dollars or premium
+  requests), the token split and the last-checked date; the generated sources catalog grew matching
+  `Cost` / `Tokens` / `Checked` columns (`—` where the backend reported nothing — an unknown figure
+  is never rendered as a measured zero).
+- **The ingest console is now a live `rich` display, and it shows what each source cost.** On a
+  terminal the run renders a live region — one spinner row per IN-FLIGHT source plus an overall
+  bar — while finished sources scroll away above it as permanent verdict lines that now carry the
+  spend and the model that actually ran (`[2/3] OK raw/notes.md 18.4s 2 created $0.0123
+  1.2k in / 456 out claude-opus-5`; only fields the backend genuinely reported appear). This also
+  repairs `--jobs N`, which used to switch the animation OFF entirely: a single
+  carriage-return-rewritten line could name only one source, so several in flight had nothing to
+  show. `rich` joins `mcp`/`pyyaml`/`pypdf` as a runtime dependency (pure-Python, two small
+  transitive packages). ASCII-only output and the ASCII spinner frames are preserved deliberately,
+  so a legacy Windows code page can still encode every line, and console failures are still
+  swallowed — output must never break a run that already spent money.
+
 - **`CITADEL_MAX_SOURCE_BYTES` — a discovery size ceiling.** Ignore patterns matched *names*;
   nothing matched **size**, so a raw root that also holds machine data (the reported case: 617
   `.tdms` sensor dumps, 10.6 GB in one folder) was stream-hashed in full on every first scan just to
@@ -25,6 +82,16 @@ All notable changes to this project are documented here. The format is based on
 
 ### Fixed
 
+- **A long absolute source key no longer floods the ingest console.** `config.display_key` collapsed
+  a path to `raw/<below>` only when it matched a configured root as a string — so a Windows drive
+  letter mapped to a share (root configured as `T:\proj\raw`, key arriving as
+  `//fileserver.corp.example/proj/raw/...`) matched nothing and printed the whole UNC path, FQDN
+  host and all, on every line. It now falls back to cutting at the last path segment *named* like a
+  configured root, and finally to a `.../`-marked tail clip for a key under no known root at all.
+  Verdict lines additionally clip the key from the LEFT (keeping the identifying filename) rather
+  than clipping the line from the right, so a narrow terminal never drops the cost and model at the
+  end of the line.
+
 - **Explicitly requested ingest paths now expand `~`.** Every other configured path already did
   (`config._resolve_dir_entry`, `workspace.init`, `CITADEL_WORKSPACE`); the `citadel ingest
   <paths…>` / `wiki_ingest` arguments were the outlier. A POSIX shell expands `~` before citadel