Skip to content

feat(mcp): ground get_answer synthesis in the body it serves; slim low-confidence misses#778

Merged
RaghavChamadiya merged 1 commit into
mainfrom
feat/mcp-answer-quality
Jul 11, 2026
Merged

feat(mcp): ground get_answer synthesis in the body it serves; slim low-confidence misses#778
RaghavChamadiya merged 1 commit into
mainfrom
feat/mcp-answer-quality

Conversation

@RaghavChamadiya

Copy link
Copy Markdown
Member

What

Two get_answer quality fixes surfaced by dogfooding the MCP tools.

1. Synthesis now sees the same body it serves

For a question-matched symbol, the hydrator fed the synthesis LLM a 40-line source excerpt while the response inlined the symbol's full 120-line body in symbol_bodies. A docstring-heavy definition (its logic past line 40) was truncated before its answer-bearing code reached the model, so synthesis hedged with a low-confidence "not in the excerpts" answer on the very symbol whose full body the agent then received.

The top matched symbols are now read at the inline-body depth during hydration, so the model reasons over the same body the response serves. Bounded so a class-name flood (every sibling method matching through the parent's qualified name) cannot balloon the synthesis prompt.

2. Cheaper low-confidence misses

A low-confidence miss carried a full per-hit key_symbols dump that duplicated best_guesses and went unused. It is dropped (best_guesses + code_rationale carry the choosing signal), along with the page-excerpt DB round-trip that only fed it. The low-confidence answer branch keeps a grounding block but lean: hits with snippets, symbols pipeable, docstrings stripped.

Also extracts the symbol_bodies ranking into a tested _gather_body_candidates helper.

Testing

  • New calibration test asserts the synthesis excerpt reaches the inline-body depth for a docstring-heavy matched symbol, and that a class-name flood does not upgrade every sibling.
  • Unit tests for the body-candidate ranking and the leaner payload.
  • Retrieval battery green; full mcp unit suite passing.

…w-confidence misses

Synthesis fed the LLM a 40-line source excerpt for a question-matched
symbol while the response inlined the symbol's full 120-line body. A
docstring-heavy definition was therefore truncated before its
answer-bearing logic ever reached the model, so synthesis hedged
("not in the excerpts") on the very symbol whose full body the response
then handed to the agent. Read the top matched symbols at the
inline-body depth during hydration so the model reasons over the same
body the response serves, and add a calibration test that locks the two
depths together.

Also slim the low-confidence miss payload. The gated response carried a
full per-hit key_symbols dump that duplicated best_guesses and went
unused; drop it (best_guesses + code_rationale carry the choosing
signal) along with the excerpt-enrichment DB round-trip that only fed
it. The low-confidence answer branch keeps a grounding block but lean,
with symbols pipeable and docstrings stripped.

Extracts the symbol_bodies ranking into a tested _gather_body_candidates
helper. Battery green; unit tests added.
@repowise-bot

repowise-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown

✅ Health: 7.6 (unchanged)

📋 At a glance
1 file changed health · 4 hotspots touched · 4 new findings introduced · 5 co-change pairs left out · 2 files with recent fix history · 1 dead-code finding.

Files & modules (2)
  • packages (3 files)
    • .../tool_answer/config.py
    • .../tool_answer/answer.py
    • .../tool_answer/symbols.py
  • tests (1 file)
    • .../mcp/test_answer_payload.py

⚠️ Change risk: moderate (riskier than 55% of this repo's commits · raw 9.1/10)
This change's risk is driven by:

  • more lines added than baseline
  • more scattered than baseline

🩹 Review priority (files here with the most recent bug-fix history — defects cluster, so review these first)

File Score Δ Why
.../mcp/test_answer_payload.py 8.8 → 8.9 ▲ +0.1 🔻 introduced primitive obsession
🔎 More signals (3)

🔥 Hotspots touched (4)

  • .../tool_answer/config.py — 5 commits/90d, 5 dependents · primary owner: Swati Ahuja (91%)
  • .../tool_answer/answer.py — 10 commits/90d, 2 dependents · primary owner: Swati Ahuja (61%)
  • .../mcp/test_answer_payload.py — 2 commits/90d, 1 dependents · primary owner: Swati Ahuja (88%)
1 more
  • .../tool_answer/symbols.py — 5 commits/90d, 3 dependents · primary owner: Raghav Chamadiya (53%)

🔗 Hidden coupling (3 files)

  • .../tool_answer/config.py co-changes with .../mcp/test_answer_calibration.py (3× — 🟢 routine) — not in this PR.
  • .../tool_answer/answer.py co-changes with these files (not in this PR):
    • .../mcp_server/tool_symbol.py (3× — 🟢 routine)
    • .../mcp/test_answer_calibration.py (3× — 🟢 routine)
    • .../mcp_server/tool_search.py (3× — 🟢 routine)
  • .../tool_answer/symbols.py co-changes with .../mcp/test_answer_calibration.py (3× — 🟢 routine) — not in this PR.

💀 Dead code (1 finding)

  • 💀 .../tool_answer/answer.py _json_default (confidence 0.65)

📊 Full report · ⭐ Star Repowise · 📥 Install bot · Last updated 2026-07-11 11:19 UTC
Silence on a single PR with [skip repowise] in the title · Per-repo toggle on repowise.dev/settings?tab=bot

@RaghavChamadiya RaghavChamadiya merged commit 2dc41d0 into main Jul 11, 2026
4 of 7 checks passed
@RaghavChamadiya RaghavChamadiya deleted the feat/mcp-answer-quality branch July 11, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants