llm/doctor: surface a failed copilot session's real error, warn on model shadowing - #135
Merged
Merged
Conversation
…del shadowing A failed copilot session exits non-zero with an empty stderr; the message that names the fix (e.g. a BYOK Ollama-style provider that does not serve the configured model, HTTP 404) travels as a session.error event near the END of its JSONL stdout, while the stream opens with ephemeral MCP/skills status events. The recorded failure reason was the first 500 chars of raw stdout - exactly that noise - so the real error only surfaced in a --log-dir transcript. Extract and prefer the session.error message(s), which also lets an auth-shaped copilot failure reach the hermetic-retry detection; a stream naming no session.error keeps the old truncation. doctor's ingest-model check now WARNs when CITADEL_INGEST_MODEL shadows a set-but-different COPILOT_MODEL on the copilot backend - the --model flag overrides the env var, so a stale .env model breaks a BYOK provider setup with an instant per-source failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mu9JX3cFvypoJbhY2RdcUK
There was a problem hiding this comment.
Pull request overview
This PR improves diagnostics and configuration guidance for the copilot backend, ensuring failures expose the real underlying error and citadel doctor warns about a common model-selection “shadowing” trap when using BYOK providers.
Changes:
- Prefer
session.errormessages from copilot’s JSONL stream as the recorded failure detail (instead of truncating early-stream noise). - Add a
citadel doctorWARN whenCITADEL_INGEST_MODEL(passed via--model) overrides a differentCOPILOT_MODEL, optionally naming the configured BYOK provider URL. - Extend test coverage to lock in both behaviors and keep the previous fallback path when
session.erroris absent.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
citadel/llm.py |
Extracts/uses session.error from copilot JSONL on non-zero exit to surface actionable failure reasons. |
citadel/doctor.py |
Adds an ingest-model warning specific to copilot when CITADEL_INGEST_MODEL shadows COPILOT_MODEL (and notes BYOK provider when set). |
tests/test_llm.py |
Adds tests validating session.error extraction, dedup/whitespace collapsing, and fallback behavior when no session.error exists. |
tests/test_doctor.py |
Adds tests asserting the new doctor warning triggers only for the copilot backend with disagreeing model knobs. |
CHANGELOG.md |
Documents the new doctor warning and improved copilot failure reporting. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A copilot session pointed at a BYOK provider (an Ollama-style local endpoint configured via
COPILOT_PROVIDER_BASE_URL+COPILOT_MODEL) died instantly with HTTP 404 when a staleCITADEL_INGEST_MODELin.envnamed a model that provider does not serve — and the recorded failure reason showed none of that. Two fixes:Surface the real error. A failed copilot session exits non-zero with an empty stderr; the message that names the fix ("Model '…' not found on provider at http://… (HTTP 404)") travels as a
session.errorevent near the end of its--output-format jsonJSONL stdout, while the stream opens with ephemeral MCP/skills status events. The failure reason citadel records (run report, failures catalog,citadel status) was(err or out)[:500]— exactly that opening noise — so the real error only surfaced in a--log-dirtranscript._error_from_copilot_jsonlnow extracts thesession.errormessage(s) (collapsed onto one line, duplicates folded) and prefers them as the failure detail. This also lets an auth-shaped copilot failure reach the hermetic-retry detection, which scans the message. A stream naming nosession.errorkeeps the old stderr-then-stdout truncation, so no failure shape got less diagnostic.Name the trap in
doctor. citadel's--modelflag overrides copilot's ownCOPILOT_MODELenv var, so thecheck_ingest_modelcheck now WARNs when the copilot backend has both set and disagreeing — naming the shadowing and the BYOK provider (when configured) that must actually serve the requested model.Test fixtures are trimmed, anonymized, shape-faithful copies of the real failed session's stream (placeholder hostnames/model ids only).
Testing
uv run pytest -q— 1287 passed, 1 skippeduv run ruff check ./uv run ruff format --check .— cleanCITADEL_WORKSPACE=corpora/beverages uv run python -m citadel lint— OK🤖 Generated with Claude Code
Generated by Claude Code