feat: structured output for Social Media Analyst (#796)#802
feat: structured output for Social Media Analyst (#796)#802HrushiYadav wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces structured sentiment reporting for the Social Media Analyst, including new Pydantic schemas, rendering utilities, and comprehensive unit tests. The analyst now performs a secondary extraction step to pin deterministic fields like sentiment score and confidence while preserving the full research narrative. Feedback focuses on removing redundant type checks in the analyst node, clarifying overlapping numeric ranges and enum distinctions in the schema descriptions, and improving Markdown formatting in the rendered output.
| # invoke_structured_or_freetext falls back to plain llm.invoke when | ||
| # structured output fails; in that case it returns an AIMessage, not a str. | ||
| if hasattr(report, "content"): | ||
| report = report.content |
There was a problem hiding this comment.
The check for hasattr(report, "content") and the accompanying comment are redundant. The invoke_structured_or_freetext utility function (defined in tradingagents/agents/utils/structured.py) is already typed to return a str and explicitly returns the .content attribute when falling back to the plain LLM. Thus, report will always be a string at this point, and the comment about it returning an AIMessage is incorrect.
| # invoke_structured_or_freetext falls back to plain llm.invoke when | |
| # structured output fails; in that case it returns an AIMessage, not a str. | |
| if hasattr(report, "content"): | |
| report = report.content | |
| # invoke_structured_or_freetext already returns the string content | |
| # from either the rendered structured output or the fallback response. |
There was a problem hiding this comment.
Fixed in 1ec8997 - removed the redundant check and comment.
| overall_band: SentimentBand = Field( | ||
| description=( | ||
| "Categorical sentiment label. Pick the band that best matches the " | ||
| "overall_score: Bullish (7-10), Mildly Bullish (5.5-7), Neutral/Mixed " | ||
| "(4.5-5.5), Mildly Bearish (3-4.5), Bearish (0-3)." | ||
| ), | ||
| ) |
There was a problem hiding this comment.
The description for overall_band contains overlapping numeric ranges (e.g., 7.0 is in both Bullish and Mildly Bullish) and groups Neutral/Mixed together even though they are separate members in the SentimentBand enum. Providing non-overlapping ranges and a clear distinction between 'Neutral' and 'Mixed' will help the LLM select the correct enum value more reliably.
'Neutral' typically implies a lack of strong signal or a flat outlook, while 'Mixed' implies high-volume but conflicting signals.
overall_band: SentimentBand = Field(
description=(
"Categorical sentiment label. Pick the band that best matches the "
"overall_score: Bullish (7.1-10), Mildly Bullish (5.6-7.0), Neutral "
"(4.5-5.5, flat outlook), Mixed (4.5-5.5, conflicting signals), "
"Mildly Bearish (3.0-4.4), Bearish (0-2.9)."
),
)There was a problem hiding this comment.
Fixed in 1ec8997 - non-overlapping ranges applied and Neutral vs Mixed distinction clarified.
| header = "\n".join([ | ||
| f"**Overall Sentiment:** **{report.overall_band.value}** " | ||
| f"(Score: {report.overall_score:.1f}/10, Confidence: {report.confidence})", | ||
| "", | ||
| ]) |
There was a problem hiding this comment.
The current implementation of render_sentiment_report results in only a single newline between the structured header and the narrative text. In Markdown, a blank line (two newlines) is standard for separating blocks of text. Adding an additional empty string to the join list will ensure proper spacing for downstream display.
| header = "\n".join([ | |
| f"**Overall Sentiment:** **{report.overall_band.value}** " | |
| f"(Score: {report.overall_score:.1f}/10, Confidence: {report.confidence})", | |
| "", | |
| ]) | |
| header = "\n".join([ | |
| f"**Overall Sentiment:** **{report.overall_band.value}** " | |
| f"(Score: {report.overall_score:.1f}/10, Confidence: {report.confidence})", | |
| "", | |
| "", | |
| ]) |
There was a problem hiding this comment.
Fixed in 1ec8997 - added the extra blank line for proper markdown block separation.
1ec8997 to
3ff279f
Compare
Add SentimentReport schema with overall_score (0-10), overall_band (SentimentBand enum), confidence, and narrative fields to schemas.py. The analyst uses tools for the research phase so with_structured_output cannot be applied to the tool-calling chain directly. Instead, bind_structured is called once at factory time and used for a lightweight extraction call after the tool loop produces the free-text report. Falls back gracefully to free-text when the provider does not support structured output. Mirrors the pattern from bba1477/0fda245/7c37249 (Trader, Portfolio Manager, Research Manager). Closes TauricResearch#796.
- Remove redundant hasattr(report, 'content') check; invoke_structured_or_freetext already returns str - Fix overlapping numeric ranges in overall_band description; clarify Neutral vs Mixed distinction - Add extra blank line in render_sentiment_report header for proper markdown block separation
…am refactor Upstream commit 0fcf136 renamed social_media_analyst to sentiment_analyst and redesigned it (no tools, pre-fetches data into prompt, single LLM call). Ported structured output to match new design: format_messages then invoke_structured_or_freetext, return AIMessage for messages state. Updated tests to import create_sentiment_analyst and match new single-call design with simplified mocks.
3ff279f to
a00201e
Compare
Fixes #796.
social_media_analystoutputs vary wildly between runs - some include a numeric score, some only a band, some neither. Downstream agents and dashboards either degrade silently or maintain growing piles of fallback regexes.Added
SentimentReportschema (score, band, confidence, narrative) andrender_sentiment_reporttoschemas.py, following the same pattern used for the Trader, Portfolio Manager, and Research Manager in #434. Since the analyst uses tool calls for its research phase, structured output is applied in a separate extraction call after the tool loop finishes rather than on the tool-calling chain directly. Falls back to free-text if the provider does not support structured output.8 new unit tests covering schema validation, render output, structured path, and fallback. All 19 tests pass.