runtime: thinking-off on force-verdict retry (frees token budget)

When the model emits a prose-only first turn (no XML blocks), the
backend injects FORCE_VERDICT_DIRECTIVE as the next user message
and re-rolls. Until now, both turns ran with enable_thinking=self
._enable_thinking (True for Qwen 3). On RK3588 with Qwen3-1.7B at
max_new_tokens=1500, the retry burns its entire budget inside
<think>...</think> just like the first turn — never reaches the
structured tail. The bridge then emits the synthetic
"Model did not produce a structured verdict" fallback and the
user sees a yellow non-answer.

Lab evidence (anonymized transcript from a 'cannot-join-pool'
session): two `thought` events ("You're running diag/summary..." +
"Analyzing diagnostics...") followed by the synthetic verdict at
T+41s — no tool_call, no real verdict.

Fix: track a per-turn `next_thinking` flag. Default to the
backend-level enable_thinking. When we set
`next_content = FORCE_VERDICT_DIRECTIVE` (both the MAX_TURNS-1
preemptive branch AND the prose-only mid-loop branch), also set
`next_thinking = False`. The retry then runs without thinking, so
all 1500 tokens are available for the ~80-token <verdict> block —
diagnostic context is already in KV cache from prior turns so the
model doesn't need to re-reason, just format the conclusion.

Also gates `output_for_parsing = _strip_think(...)` and the
"Analyzing diagnostics..." marker on `next_thinking` instead of
`self._enable_thinking` so the strip + marker correctly skip when
the retry ran without `<think>` blocks.

New test verifies turn 0 runs thinking ON, turn 1 (force-verdict
retry) runs thinking OFF. All 56 existing tests still pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Author: ehsan6sha

src/runtime/rkllm_runtime.py

@@ -1140,6 +1140,16 @@ async def run_troubleshoot(
         next_role: str = "user"
         next_content: str = first_turn_content
         next_keep_history: int = 0   # 0 on first turn; 1 thereafter
+        # next_thinking gates the per-call enable_thinking flag. Default
+        # follows the backend-level filename gate; force-verdict retries
+        # flip this off so the entire max_new_tokens budget can land on
+        # the structured <verdict> block (lab 2026-05-27: with thinking
+        # on, Qwen3-1.7B exhausts the budget inside <think>…</think>
+        # and never reaches the structured tail — the bridge then emits
+        # the synthetic "Model did not produce a structured verdict"
+        # fallback). Diagnostic data is already in KV cache from prior
+        # turns, so the retry only needs to format the conclusion.
+        next_thinking: bool = self._enable_thinking
 
         for turn in range(MAX_TURNS):
             # Last-chance: at MAX_TURNS-1 without a verdict, send the
@@ -1154,6 +1164,7 @@ async def run_troubleshoot(
                 next_role = "user"
                 next_content = FORCE_VERDICT_DIRECTIVE
                 next_keep_history = 1  # keep prior KV
+                next_thinking = False  # see next_thinking init comment
                 force_verdict_attempted = True
 
             try:
@@ -1162,11 +1173,11 @@ async def run_troubleshoot(
                 # routes the message to the right slot in the template.
                 output = await loop.run_in_executor(
                     None,
-                    lambda r=next_role, c=next_content, kh=next_keep_history: (
+                    lambda r=next_role, c=next_content, kh=next_keep_history, t=next_thinking: (
                         self._runtime.generate(
                             c,
                             role=r,
-                            enable_thinking=self._enable_thinking,
+                            enable_thinking=t,
                             keep_history=kh,
                             timeout_s=PER_TURN_TIMEOUT_S,
                         )
@@ -1200,9 +1211,11 @@ async def run_troubleshoot(
             # structured response (tool calls, verdict, recommendations)
             # only exists AFTER `</think>`. Pre-strip so the parsers
             # can't be tripped by stray XML mentions inside the model's
-            # reasoning prose ("I should call <tool_call>").
+            # reasoning prose ("I should call <tool_call>"). Gate on
+            # the per-turn flag (next_thinking) — force-verdict retries
+            # run with thinking off so no `<think>` block exists to strip.
             output_for_parsing = (
-                _strip_think(output) if self._enable_thinking else output
+                _strip_think(output) if next_thinking else output
             )
 
             # NOTE: history list is no longer rebuilt-per-turn — v1.2.3
@@ -1223,7 +1236,7 @@ async def run_troubleshoot(
             if thought_text:
                 # SSE thought schema: minLength 1, maxLength 4000
                 yield {"type": "thought", "payload": thought_text[:4000]}
-            elif self._enable_thinking:
+            elif next_thinking:
                 # Qwen 3 turn was 100% structured output after <think>;
                 # avoid a silent stretch by emitting a tiny marker.
                 yield {"type": "thought", "payload": "Analyzing diagnostics..."}
@@ -1349,6 +1362,12 @@ async def run_troubleshoot(
                     next_role = "user"
                     next_content = FORCE_VERDICT_DIRECTIVE
                     # keep_history=1 already set after first turn.
+                    # Turn off thinking for the retry so the entire
+                    # max_new_tokens budget is available for the
+                    # structured <verdict>/<recommendation> output —
+                    # diagnostic context is already in KV cache, no
+                    # need to re-reason.
+                    next_thinking = False
                     history.append({
                         "role": "user",
                         "content": FORCE_VERDICT_DIRECTIVE,

tests/test_rkllm_runtime.py

@@ -626,6 +626,59 @@ async def collect():
     assert any(FORCE_VERDICT_DIRECTIVE in p for p in prompts_seen)
 
 
+def test_force_verdict_retry_runs_with_thinking_off():
+    """Tier 1 verdict-recovery fix: when the model produces prose-only
+    output on turn 0 (no XML, no tool_call), the backend injects the
+    force-verdict directive AND turns thinking off for the retry so the
+    entire max_new_tokens budget lands on the structured <verdict> block
+    instead of the model exhausting it inside another <think>…</think>.
+
+    Verified by recording the enable_thinking flag on each generate()
+    call: turn 0 runs with thinking ON (backend-level enable_thinking
+    flag), turn 1 (the force-verdict retry) runs with thinking OFF."""
+    import asyncio
+    from src.runtime.rkllm_runtime import RKLLMBackend, FORCE_VERDICT_DIRECTIVE
+
+    turn_outputs = [
+        "I am rambling about your question.",  # prose only -> force-verdict
+        '<verdict>{"summary":"green","severity":"green","root_cause":"ok"}</verdict>',
+    ]
+    turn_idx = {"i": 0}
+    thinking_per_turn: list[bool] = []
+
+    class ThinkingCaptureRuntime:
+        def generate(self, prompt, role="user", enable_thinking=False, keep_history=0, timeout_s=90.0):
+            thinking_per_turn.append(enable_thinking)
+            i = turn_idx["i"]
+            turn_idx["i"] = i + 1
+            return turn_outputs[i] if i < len(turn_outputs) else "(end)"
+
+    # Simulate a Qwen 3 backend where the default IS thinking-on so we
+    # can prove the retry overrides it. Setting _enable_thinking=True
+    # bypasses the filename check.
+    backend = RKLLMBackend(
+        loaded=True,
+        _runtime=ThinkingCaptureRuntime(),
+        _enable_thinking=True,
+    )
+    backend.wire_runtime_deps(tool_executor=None, action_signer=lambda x: "f" * 64)
+
+    async def collect():
+        return [ev async for ev in backend.run_troubleshoot("x")]
+
+    events = asyncio.run(collect())
+    # Both turns ran
+    assert len(thinking_per_turn) == 2, thinking_per_turn
+    # Turn 0 used backend's thinking default (True for Qwen 3)
+    assert thinking_per_turn[0] is True, "turn 0 should use backend's thinking default"
+    # Turn 1 (force-verdict retry) MUST run with thinking off so the
+    # full token budget is available for the structured block
+    assert thinking_per_turn[1] is False, "force-verdict retry must run thinking=False"
+    # Sanity: real verdict came through, not the synthetic fallback
+    verdicts = [e for e in events if e["type"] == "verdict"]
+    assert verdicts[0]["payload"]["root_cause"] == "ok"
+
+
 # ---------------------------------------------------------------------------
 # Qwen 3 thinking-mode swap (2026-05-26)
 # ---------------------------------------------------------------------------