fix: enforce confidence-range language and data-sufficiency gate in report generation

[drewstarr0408]

Summary

Report generation could assert specific real-world outcomes (e.g. an exact match score) as settled fact even when the underlying simulation graph had effectively zero observed agent interactions (0 edges, 0 successful interviews). The recommendation/synthesis step compounded this by treating that fabricated conclusion as a premise, recommending generation of more synthetic data specifically to manufacture supporting evidence for it — conclusion-first, evidence-manufactured-after.

Repro case: a simulation seeded with only factual/structural data (no outcome data) still produced a report stating a specific final score as fact, while the same report elsewhere admitted 0 edges and 0 interviews existed in the graph.

Root cause

• PLAN_SYSTEM_PROMPT / SECTION_SYSTEM_PROMPT_TEMPLATE told the model simulation output "ARE the predicted future," with no code-level check gating generation on how much real simulation data actually existed (total_nodes/total_edges were computed but never used as a gate).
• SECTION_USER_PROMPT_TEMPLATE fed each prior section's generated text into the next section's prompt as "Completed section content," with no instruction to treat it as provisional — so a later section could build directly on an earlier section's unverified/fabricated claim.

Fix

• Reworded PLAN_SYSTEM_PROMPT, PLAN_USER_PROMPT_TEMPLATE, SECTION_SYSTEM_PROMPT_TEMPLATE, SECTION_USER_PROMPT_TEMPLATE to forbid asserting specific outcomes/scores/winners and require confidence-range language throughout.
• Added a real data-sufficiency gate: when total_nodes == 0 or total_edges == 0, plan_outline() sets self.data_insufficient and injects an explicit "DATA SUFFICIENCY WARNING" into the outline prompt and every section's system prompt, forcing "insufficient data, no outcome projection possible" framing instead of a guess.
• Section prompts now explicitly flag prior sections as unverified LLM-generated content, not fact to build recommendations on.
• Fixed the fallback outline text (used on planning failure), which had the same hard-prediction framing.

Testing

Live end-to-end re-verification was blocked by unrelated API credit exhaustion on my end, so I built an offline mock test (mocks GraphToolsService/LLMClient, no network calls) exercising the actual ReportAgent.plan_outline() gate logic and prompt construction directly:

• Confirms data_insufficient correctly triggers at 0 edges / 0 nodes (including the exact repro shape: nodes present, 0 edges) and correctly stays off on a healthy graph.
• Confirms the DATA SUFFICIENCY WARNING is injected into both outline and section prompts when triggered.
• Confirms none of the hard-outcome phrasing ("the simulation results ARE the predicted future", "Future Prediction Report", bare "are predictions of...") appears in any prompt variant post-fix.

22/22 checks pass. Happy to share the test script if useful, or add it to the repo's test suite if there's a place for it.

🤖 Generated with Claude Code