improve claude response handling, prepare for fable-5 (#2595)

* fix: harden narrative report generation against max_tokens truncation

claude-sonnet-5 has adaptive thinking on by default, and max_tokens is a
hard cap on thinking + response text combined. Existing max_tokens values
(3000-4000) were sized for text-only output on older models, causing a
minority of narrative/collective-statement responses to truncate mid-JSON.

- Raise max_tokens to 8000 across all Anthropic narrative-generation call
  sites (delphi model_provider.py, 801_narrative_report_batch.py,
  server collectiveStatement.ts, reportNarrative.ts, sonnet.ts)
- Add output_config: {effort: "medium"} to bound thinking depth
- Log a warning whenever stop_reason == "max_tokens" so truncation is
  visible in logs instead of silently producing bad data
- Prompt hardening: instruct the model to prioritize completing the JSON
  structure over exhaustive detail if space is limited
- Client-side defense in depth: CommentsReport.jsx no longer requires
  report_data to end with "}" before attempting jsonrepair, which can
  already recover a valid partial object from truncated responses
- Bump @anthropic-ai/sdk in server/node_modules to the 0.110.0 already
  pinned in package.json (needed for output_config typings)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* chore: remove dead per-job model/stage config for Delphi FULL_PIPELINE and batch jobs

jobs.ts built a job_config.stages structure (PCA/UMAP/REPORT configs,
model, include_topics, visualizations) that job_poller.py's FULL_PIPELINE
branch and run_delphi.py never read — confirmed by tracing the full
invocation path. The Anthropic model used for narrative generation is
controlled exclusively by the ANTHROPIC_MODEL environment variable
(job_poller.py reads it directly for both FULL_PIPELINE and
CREATE_NARRATIVE_BATCH jobs, ignoring any per-request model).

Removed the same dead fields from the client since the server never
used them: jobFormData's max_votes/batch_size/model/include_topics, and
the model field on the batchReports request (which was similarly
recorded in job_config but ignored by job_poller.py at generation time).

This keeps a single source of truth for model selection (the
ANTHROPIC_MODEL secret) instead of a misleading per-request parameter
that looked configurable but never was.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(delphi): add claude-fable-5 support with refusal/fallback handling

Claude Fable 5 runs safety classifiers that can decline a request
(stop_reason: "refusal", HTTP 200 - not an error), which none of our
narrative-generation code accounted for. Without handling, a refusal
would silently produce an empty/garbage report.

- Add "claude-fable-5" to model_provider.py's hardcoded model list
- New helpers in model_provider.py: _anthropic_request_extras() (adds
  the server-side fallbacks param + required beta header, but only when
  the active model is claude-fable-5 - a no-op for every other model),
  _narrative_error_json(), and _check_response_for_issues()
- get_response()/get_completion() (the two non-batch HTTP call paths)
  now opt into fallback to claude-opus-4-8 on Fable 5, and turn a
  refusal into a clear "declined by safety classifier" report instead
  of empty content
- get_batch_responses() (unused) explicitly does not add fallbacks,
  since the Batch API rejects that param
- 803_check_batch_status.py (the live batch-results parser) detects
  stop_reason == "refusal" on batch results - which report
  result.type == "succeeded" per Anthropic's docs, so this needed an
  explicit check - and stores the same clear error message instead of
  silently writing "{}"

Server-side fallback isn't available on the Batch API, so a refused
batch item can't be auto-recovered in place; it now at least fails
visibly instead of looking like a successful empty report.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Author: ballPointPenguin

client-report/src/components/commentsReport/CommentsReport.jsx

@@ -34,10 +34,6 @@ const CommentsReport = ({ math, comments, conversation, ptptCount, formatTid, vo
   const jobFormData = {
     job_type: "FULL_PIPELINE",
     priority: 50,
-    max_votes: "",
-    batch_size: "",
-    model: "claude-opus-4-8",
-    include_topics: true,
     include_moderation: reportModLevel !== -2,
   };
   const [isSubmitting, setIsSubmitting] = useState(false);
@@ -283,7 +279,6 @@ const CommentsReport = ({ math, comments, conversation, ptptCount, formatTid, vo
     net
       .polisPost("/api/v3/delphi/batchReports", {
         report_id: report_id,
-        model: "claude-opus-4-8",
         no_cache: false,
         include_moderation: reportModLevel !== -2,
       }, authToken)
@@ -781,10 +776,13 @@ const CommentsReport = ({ math, comments, conversation, ptptCount, formatTid, vo
                 <p>Not enough data has been provided for analysis, please check back later</p>
               );
 
+            // Only bail out early on data that isn't JSON-ish at all. Don't require a
+            // trailing "}" here — responses truncated by max_tokens (e.g. from adaptive
+            // thinking eating into the token budget) are missing their closing brackets,
+            // and jsonrepair below can usually recover a valid partial object from those.
             if (
               typeof report.report_data !== "string" ||
-              !report.report_data.trim().startsWith("{") ||
-              !report.report_data.trim().endsWith("}")
+              !report.report_data.trim().startsWith("{")
             ) {
               return (
                 <article style={{ maxWidth: "600px" }}>

delphi/umap_narrative/801_narrative_report_batch.py

@@ -1087,15 +1087,21 @@ async def prepare_batch_requests(self):
                     - Do not provide explanations, only the JSON
                     - Use the exact structure shown above with "id", "title", "paragraphs", etc.
                     - Include relevant citations to comment IDs in the data
+                    - Keep prose concise (2-4 paragraphs). The full JSON object, including
+                      closing brackets, must fit within the available output length —
+                      prioritize finishing the JSON structure over exhaustive detail
                 """
 
                 # Add to batch requests
+                # max_tokens is a hard cap on thinking + response text combined
+                # (adaptive thinking is on by default on Sonnet 5 / Opus 4.8+),
+                # so this needs real headroom beyond the visible JSON text length.
                 batch_request = {
                     "system": system_lore,
                     "messages": [
                         {"role": "user", "content": model_prompt}
                     ],
-                    "max_tokens": 4000,
+                    "max_tokens": 8000,
                     "metadata": {
                         "topic_name": topic_name,
                         "topic_key": topic_key,
@@ -1310,7 +1316,8 @@ async def submit_batch(self):
                         "custom_id": safe_custom_id,
                         "params": {
                             "model": self.model,
-                            "max_tokens": request.get('max_tokens', 4000),
+                            "max_tokens": request.get('max_tokens', 8000),
+                            "output_config": {"effort": "medium"},
                             "system": system_content,
                             "messages": [user_message]
                         }

delphi/umap_narrative/803_check_batch_status.py

@@ -15,6 +15,9 @@
 from datetime import datetime, timedelta, timezone
 from botocore.exceptions import ClientError
 
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+from umap_narrative.llm_factory_constructor.model_provider import _narrative_error_json
+
 # Configure logging
 logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
 logger = logging.getLogger(__name__)
@@ -135,8 +138,31 @@ async def process_batch_results(self, job_item: Dict) -> bool:
                     custom_id = entry.custom_id
                     response_message = entry.result.message
                     model = response_message.model
-                    text_block = next((b for b in response_message.content if b.type == "text"), None)
-                    content = text_block.text if text_block else "{}"
+                    if response_message.stop_reason == "max_tokens":
+                        logger.warning(
+                            f"Job {job_id}: response for {custom_id} was truncated by max_tokens; "
+                            "output may be incomplete/invalid JSON."
+                        )
+
+                    if response_message.stop_reason == "refusal":
+                        # Batch API refusals still report result.type == "succeeded" — stop_details
+                        # may be null on batch results, so branch on stop_reason alone.
+                        # Server-side fallback isn't available on the Batch API, so this can't be
+                        # auto-recovered here; it needs a separate non-batch retry (which does
+                        # support fallback for claude-fable-5).
+                        logger.warning(
+                            f"Job {job_id}: model {model} declined request for {custom_id} "
+                            "(stop_reason=refusal)."
+                        )
+                        content = _narrative_error_json(
+                            "Model Declined Request",
+                            "This section could not be generated because the request was declined "
+                            "by the model's safety classifier. Try regenerating, or switch to a "
+                            "different model."
+                        )
+                    else:
+                        text_block = next((b for b in response_message.content if b.type == "text"), None)
+                        content = text_block.text if text_block else "{}"
 
                     # Reconstruct the section name from the custom_id
                     parts = custom_id.split('_', 1)

delphi/umap_narrative/llm_factory_constructor/model_provider.py

@@ -16,6 +16,86 @@
 logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
 logger = logging.getLogger(__name__)
 
+# Claude Fable 5 runs safety classifiers that can decline a request
+# (stop_reason: "refusal", HTTP 200 - not an error). When it's the active
+# model, we opt into server-side fallback so a refusal is transparently
+# retried on another model within the same call, rather than silently
+# producing an empty/missing report. Not supported on the Batch API, so
+# this only applies to the direct (non-batch) HTTP calls below.
+# See: https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback
+FABLE_FALLBACK_MODEL = "claude-opus-4-8"
+FABLE_FALLBACK_BETA_HEADER = "server-side-fallback-2026-06-01"
+
+
+def _anthropic_request_extras(model_name: Optional[str]) -> Dict[str, Any]:
+    """
+    Build the extra headers/body fields needed for a direct (non-batch)
+    Anthropic request, given the active model.
+
+    Returns a dict with "headers" and "body" sub-dicts to merge into the
+    request.
+    """
+    extras: Dict[str, Any] = {"headers": {}, "body": {}}
+    if model_name == "claude-fable-5":
+        extras["headers"]["anthropic-beta"] = FABLE_FALLBACK_BETA_HEADER
+        extras["body"]["fallbacks"] = [{"model": FABLE_FALLBACK_MODEL}]
+    return extras
+
+
+def _narrative_error_json(title: str, text: str) -> str:
+    """Build a report_data JSON blob matching the schema the client UI expects."""
+    return json.dumps({
+        "id": "polis_narrative_error_message",
+        "title": title,
+        "paragraphs": [
+            {
+                "id": "polis_narrative_error_message",
+                "title": title,
+                "sentences": [
+                    {
+                        "clauses": [
+                            {
+                                "text": text,
+                                "citations": []
+                            }
+                        ]
+                    }
+                ]
+            }
+        ]
+    })
+
+
+def _check_response_for_issues(response_json: dict, model_name: Optional[str]) -> Optional[str]:
+    """
+    Inspect a completed (non-refused-via-fallback) Anthropic response for
+    known problem stop reasons. Logs a warning either way, and returns a
+    user-facing error message string if the response should not be treated
+    as usable content (e.g. every model in the fallback chain refused).
+    """
+    stop_reason = response_json.get("stop_reason")
+    if stop_reason == "max_tokens":
+        logger.warning(
+            f"Anthropic response for model {model_name} was truncated by max_tokens; "
+            "output may be incomplete/invalid JSON."
+        )
+    elif stop_reason == "refusal":
+        stop_details = response_json.get("stop_details") or {}
+        category = stop_details.get("category")
+        explanation = stop_details.get("explanation")
+        logger.warning(
+            f"Anthropic model {model_name} declined the request (stop_reason=refusal, "
+            f"category={category}): {explanation}"
+        )
+        return (
+            "This section could not be generated because the request was declined "
+            "by the model's safety classifier"
+            + (f" (category: {category})" if category else "")
+            + ". Try regenerating, or switch to a different model."
+        )
+    return None
+
+
 class ModelProvider:
     """Base class for model providers."""
 
@@ -234,21 +314,32 @@ def get_response(self, system_message: str, user_message: str) -> str:
             }
             logger.info(f"Using Anthropic model '{self.model_name}' via direct HTTP request")
             logger.info(f"API key starts with: {self.api_key[:8]}...")
+            extras = _anthropic_request_extras(self.model_name)
+            headers.update(extras["headers"])
             data = {
                 "model": self.model_name,
                 "system": system_message,
                 "messages": [
                     {"role": "user", "content": user_message}
                 ],
-                "max_tokens": 4000
+                # max_tokens is a hard cap on thinking + response text combined
+                # (adaptive thinking is on by default on Sonnet 5 / Opus 4.8+),
+                # so this needs real headroom beyond the visible text length.
+                "max_tokens": 8000,
+                "output_config": {"effort": "medium"},
+                **extras["body"]
             }
             response = requests.post(
                 "https://api.anthropic.com/v1/messages",
                 headers=headers,
                 json=data
             )
             response.raise_for_status()
-            content = response.json()["content"]
+            response_json = response.json()
+            error_message = _check_response_for_issues(response_json, self.model_name)
+            if error_message:
+                return _narrative_error_json("Model Declined Request", error_message)
+            content = response_json["content"]
             text_blocks = [b for b in content if b.get("type") == "text"]
             result = text_blocks[0]["text"] if text_blocks else ""
 
@@ -307,13 +398,19 @@ def get_batch_responses(self, batch_requests: List[Dict[str, Any]]) -> Dict[str,
             }
 
             # Format requests for Batch API
+            # Note: the server-side "fallbacks" param (used for claude-fable-5
+            # refusal recovery elsewhere in this file) is rejected by the Batch
+            # API and must not be added here — a refused batch item just comes
+            # back with stop_reason "refusal" and needs to be resubmitted
+            # separately via the non-batch path.
             formatted_requests = []
             for i, request in enumerate(batch_requests):
                 req = {
                     "model": self.model_name,
                     "system": request.get("system", ""),
                     "messages": request.get("messages", []),
-                    "max_tokens": request.get("max_tokens", 4000)
+                    "max_tokens": request.get("max_tokens", 8000),
+                    "output_config": {"effort": "medium"}
                 }
 
                 # Add request ID (for correlation on response)
@@ -376,14 +473,15 @@ def list_available_models(self) -> List[str]:
         """
         # Anthropic doesn't have a list models endpoint, so we hardcode the known models
         available_models = [
+            "claude-fable-5",
             "claude-sonnet-5",
             "claude-opus-4-8",
             "claude-sonnet-4-6",
         ]
         logger.info(f"Available Anthropic models: {available_models}")
         return available_models
 
-    async def get_completion(self, system: str, prompt: str, max_tokens: int = 4000) -> Dict[str, Any]:
+    async def get_completion(self, system: str, prompt: str, max_tokens: int = 8000) -> Dict[str, Any]:
         """
         Get a completion from the Anthropic API with the new completion format.
         This method is specifically for the batch report generator.
@@ -429,13 +527,19 @@ async def get_completion(self, system: str, prompt: str, max_tokens: int = 4000)
                 "content-type": "application/json"
             }
 
+            extras = _anthropic_request_extras(self.model_name)
+            headers.update(extras["headers"])
             data = {
                 "model": self.model_name,
                 "system": system,
                 "messages": [
                     {"role": "user", "content": prompt}
                 ],
-                "max_tokens": max_tokens
+                # max_tokens is a hard cap on thinking + response text combined
+                # (adaptive thinking is on by default on Sonnet 5 / Opus 4.8+).
+                "max_tokens": max_tokens,
+                "output_config": {"effort": "medium"},
+                **extras["body"]
             }
 
             response = requests.post(
@@ -449,6 +553,9 @@ async def get_completion(self, system: str, prompt: str, max_tokens: int = 4000)
 
             # Parse response — filter by type to skip thinking blocks (Sonnet 5+)
             response_data = response.json()
+            error_message = _check_response_for_issues(response_data, self.model_name)
+            if error_message:
+                return {"content": _narrative_error_json("Model Declined Request", error_message)}
             content = response_data["content"]
             text_blocks = [b for b in content if b.get("type") == "text"]
             result = text_blocks[0]["text"] if text_blocks else ""

server/src/prompts/report_experimental/scripts/sonnet.ts

@@ -136,7 +136,10 @@ async function main() {
   logger.debug(prompt_xml);
   const msg = await anthropic.messages.create({
     model: "claude-sonnet-5",
-    max_tokens: 1000,
+    // max_tokens is a hard cap on thinking + response text combined
+    // (adaptive thinking is on by default on Sonnet 5).
+    max_tokens: 8000,
+    output_config: { effort: "medium" },
     system: system_lore,
     messages: [
       {

server/src/routes/collectiveStatement.ts

@@ -140,12 +140,16 @@ ${JSON.stringify(formattedComments, null, 2)}
 </condensedJSONSchema>
 </responseFormat>
 
-You MUST respond with valid JSON that follows the exact schema above. Each clause must have at least one citation.`;
+You MUST respond with valid JSON that follows the exact schema above. Each clause must have at least one citation. The full JSON object, including closing brackets, must fit within the available output length — prioritize finishing the JSON structure over exhaustive detail.`;
 
   try {
     const response = await anthropic.messages.create({
       model: "claude-opus-4-8",
-      max_tokens: 3000,
+      // max_tokens is a hard cap on thinking + response text combined
+      // (adaptive thinking is on by default on Opus 4.8+), so this needs
+      // real headroom beyond the visible JSON text length.
+      max_tokens: 8000,
+      output_config: { effort: "medium" },
       system: systemPrompt,
       messages: [
         {
@@ -155,6 +159,12 @@ You MUST respond with valid JSON that follows the exact schema above. Each claus
       ],
     });
 
+    if (response.stop_reason === "max_tokens") {
+      logger.warn(
+        "Anthropic collective statement response was truncated by max_tokens; output may be incomplete/invalid JSON."
+      );
+    }
+
     // Parse the JSON response — find text block (adaptive thinking may add thinking blocks)
     const textBlock = response.content.find((b) => b.type === "text");
     let responseText = textBlock?.type === "text" ? textBlock.text : "";