Skip to content

llm/doctor: surface a failed copilot session's real error, warn on model shadowing - #135

Merged
MarkusNeusinger merged 1 commit into
mainfrom
claude/citadel-pdf-ingest-camh6f
Jul 30, 2026
Merged

llm/doctor: surface a failed copilot session's real error, warn on model shadowing#135
MarkusNeusinger merged 1 commit into
mainfrom
claude/citadel-pdf-ingest-camh6f

Conversation

@MarkusNeusinger

Copy link
Copy Markdown
Owner

What

A copilot session pointed at a BYOK provider (an Ollama-style local endpoint configured via COPILOT_PROVIDER_BASE_URL + COPILOT_MODEL) died instantly with HTTP 404 when a stale CITADEL_INGEST_MODEL in .env named a model that provider does not serve — and the recorded failure reason showed none of that. Two fixes:

  1. Surface the real error. A failed copilot session exits non-zero with an empty stderr; the message that names the fix ("Model '…' not found on provider at http://… (HTTP 404)") travels as a session.error event near the end of its --output-format json JSONL stdout, while the stream opens with ephemeral MCP/skills status events. The failure reason citadel records (run report, failures catalog, citadel status) was (err or out)[:500] — exactly that opening noise — so the real error only surfaced in a --log-dir transcript. _error_from_copilot_jsonl now extracts the session.error message(s) (collapsed onto one line, duplicates folded) and prefers them as the failure detail. This also lets an auth-shaped copilot failure reach the hermetic-retry detection, which scans the message. A stream naming no session.error keeps the old stderr-then-stdout truncation, so no failure shape got less diagnostic.

  2. Name the trap in doctor. citadel's --model flag overrides copilot's own COPILOT_MODEL env var, so the check_ingest_model check now WARNs when the copilot backend has both set and disagreeing — naming the shadowing and the BYOK provider (when configured) that must actually serve the requested model.

Test fixtures are trimmed, anonymized, shape-faithful copies of the real failed session's stream (placeholder hostnames/model ids only).

Testing

  • uv run pytest -q — 1287 passed, 1 skipped
  • uv run ruff check . / uv run ruff format --check . — clean
  • CITADEL_WORKSPACE=corpora/beverages uv run python -m citadel lint — OK
  • verify-corpus not run: the diff is confined to the failure-message path of an already-failed session plus a read-only doctor advisory — successful-ingest behavior (prompt, argv, staging, usage parsing) is byte-identical, and this environment has no authenticated agent CLI to spend a real ingest.

🤖 Generated with Claude Code


Generated by Claude Code

…del shadowing

A failed copilot session exits non-zero with an empty stderr; the message
that names the fix (e.g. a BYOK Ollama-style provider that does not serve
the configured model, HTTP 404) travels as a session.error event near the
END of its JSONL stdout, while the stream opens with ephemeral MCP/skills
status events. The recorded failure reason was the first 500 chars of raw
stdout - exactly that noise - so the real error only surfaced in a
--log-dir transcript. Extract and prefer the session.error message(s),
which also lets an auth-shaped copilot failure reach the hermetic-retry
detection; a stream naming no session.error keeps the old truncation.

doctor's ingest-model check now WARNs when CITADEL_INGEST_MODEL shadows a
set-but-different COPILOT_MODEL on the copilot backend - the --model flag
overrides the env var, so a stale .env model breaks a BYOK provider setup
with an instant per-source failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mu9JX3cFvypoJbhY2RdcUK
Copilot AI review requested due to automatic review settings July 30, 2026 08:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves diagnostics and configuration guidance for the copilot backend, ensuring failures expose the real underlying error and citadel doctor warns about a common model-selection “shadowing” trap when using BYOK providers.

Changes:

  • Prefer session.error messages from copilot’s JSONL stream as the recorded failure detail (instead of truncating early-stream noise).
  • Add a citadel doctor WARN when CITADEL_INGEST_MODEL (passed via --model) overrides a different COPILOT_MODEL, optionally naming the configured BYOK provider URL.
  • Extend test coverage to lock in both behaviors and keep the previous fallback path when session.error is absent.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
citadel/llm.py Extracts/uses session.error from copilot JSONL on non-zero exit to surface actionable failure reasons.
citadel/doctor.py Adds an ingest-model warning specific to copilot when CITADEL_INGEST_MODEL shadows COPILOT_MODEL (and notes BYOK provider when set).
tests/test_llm.py Adds tests validating session.error extraction, dedup/whitespace collapsing, and fallback behavior when no session.error exists.
tests/test_doctor.py Adds tests asserting the new doctor warning triggers only for the copilot backend with disagreeing model knobs.
CHANGELOG.md Documents the new doctor warning and improved copilot failure reporting.

@MarkusNeusinger
MarkusNeusinger merged commit b11c4ab into main Jul 30, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants