fix: pass agent reasoning to suggestion extractor#577
Open
beng360 wants to merge 2 commits into
Open
Conversation
…teps The SuggestionExtractor was generating irrelevant suggestions because it only received the summary and truncated citations (500 chars), never the agent's reasoning stream. This caused it to produce generic infra commands even when the correct action was "merge existing PR" or "no action needed." Changes: - Pass agent_reasoning from summarization pipeline to extract_suggestions() - Rewrite prompt: allow 0 suggestions, recognize existing PRs, no padding - Increase citation output caps (600/1200 chars) so PR context isn't lost Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Walkthrough
ChangesSuggestion extraction reasoning input
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Summarization as generate_incident_summary_from_chat
participant Extractor as SuggestionExtractor
participant Prompt as _build_suggestions_prompt
participant LLM
Summarization->>Extractor: extract_suggestions(agent_reasoning)
Extractor->>Prompt: _build_suggestions_prompt(agent_reasoning, citations)
Prompt->>Prompt: build INVESTIGATOR REASONING block
Prompt->>Prompt: apply recency-based citation truncation
Prompt-->>Extractor: prompt text
Extractor->>LLM: send prompt
LLM-->>Extractor: JSON suggestions (0-5)
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
|
Validated against staging incident Before (old extractor, no reasoning):
All generic, none actionable. The extractor had no idea PR #67 existed or that the incident was transient. After (with agent reasoning + new prompt): [
{
"title": "Merge PR #67 and run terraform import to fix alert noise",
"description": "This is the 4th identical firing of this alert. PR #67 changes the alert duration from 0s to 300s, eliminating false positives caused by pod restart windows during rolling deploys. A terraform import is required before merge to avoid state conflicts.",
"type": "remediate",
"risk": "low",
"command": "gh pr merge 67 --repo arvo-ai/aurora-saas-prod --squash"
},
{
"title": "Monitor Weaviate integration after second deploy completes",
"description": "Weaviate is a new stateful component introduced in sha-259b4f7. Verify it is healthy and correctly integrated with aurora-oss-t2v-transformers once the rolling update finishes, as StatefulSet issues may not surface immediately.",
"type": "diagnostic",
"risk": "safe",
"command": "kubectl get pods -n aurora -l app.kubernetes.io/name=aurora-oss-weaviate -o wide && kubectl logs aurora-oss-weaviate-0 -n aurora --tail=50"
}
]Root cause was simple: |
![arvo-ai-staging[bot]](https://avatars.githubusercontent.com/in/4088260?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@server/chat/background/suggestion_extractor.py`:
- Around line 157-166: The recent-citation cap logic in suggestion_extractor.py
is using the wrong base index for the sliced citations, so items in
citations[-15:] can be misclassified when there are fewer than 15 total
citations. Update the index calculation in the citation loop so it is based on
the actual slice start (for example, the enumerated position within the recent
subset) rather than len(citations) - 15, and use that corrected index to decide
between the 1200 and 600 character caps in the suggestion formatting logic.
- Line 200: The parsed suggestions list in _parse_suggestions_response() is not
enforcing the documented 0–5 limit, so all returned items are still being turned
into Suggestion objects. Update the parsing flow in SuggestionExtractor to clamp
the post-parse list to at most five entries before constructing Suggestion
instances, and keep the existing contract in sync with the return handling so
extra suggestions are discarded consistently.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: ffa82a57-34b0-4baf-a274-2bbadb4efc69
📒 Files selected for processing (2)
server/chat/background/suggestion_extractor.pyserver/chat/background/summarization.py
Resolve conflicts: suggestion_extractor.py and summarization.py. The old SuggestionExtractor class is superseded by generate_recommendations() in recommender.py (merged from main). Keep main's version which already passes agent_reasoning through the new recommender pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent_reasoningfrom the summarization pipeline intoSuggestionExtractor.extract_suggestions()— previously only the summary and 500-char truncated citations were sent, so the extractor had no idea what the agent actually concludedTested against staging incident
4127c547(the 4th firing of the error log spike alert). Before: 4 irrelevant generic infra commands. After: 2 suggestions — "merge PR #67" (correct) and "monitor Weaviate" (reasonable).Closes DEV-1256
Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit