runtime: suppress hallucinated verdicts + require root_cause + de-leak system prompt

Three fixes for the "wrong verdict + no recovery affordance" pattern
observed in the lab 2026-05-28 not-earning transcript:

  - Model emitted severity:red "Internet slow + clock unsynced" while
    both tool_results in the transcript had payload:null (ok=False).
    Pattern-copied directly from the system prompt's GOOD example
    ("Discovery unreachable + clock unsynced") with words rearranged.
  - The verdict had no root_cause, but the schema allowed it, so the
    app's synthetic-detector never triggered the Try-Again button.
  - The thought event leaked the entire system prompt back as model
    output, including the "93ms" number from the BAD example which
    was then stuffed into a fake tool-response shape.

Fixes:

1. system prompt — delete the BAD/GOOD example block AND the FULL
   EXAMPLE three-turn flow. Per advisor: a 1.7B model pattern-copies
   structures, not just numbers; swapping 93ms for a placeholder
   would just copy the placeholder. Inverse rules ("NEVER use
   markdown headings", "Read tool_response field by field") are
   already explicit elsewhere; the 174-example fine-tune teaches
   the format. Add a stronger FIRST-ACTION directive: "Your VERY
   FIRST output token MUST start a <tool_call> block. Do NOT repeat
   these instructions back." Add explicit VERDICT RULES section
   requiring root_cause AND specifying the insufficient_data
   fallback for no-data sessions.

2. parse_verdict — require root_cause (was optional). Rejects when
   missing or empty; the run_troubleshoot loop's no-verdict path
   then synthesizes a proper insufficient_data verdict the app can
   detect.

3. run_troubleshoot — no-evidence guardrail. Track
   successful_tool_count across turns (increments only when
   tool_result.ok is True AND payload is not None). Before
   yielding a verdict, if successful_tool_count is 0 AND the
   model's root_cause is not already insufficient_data, replace
   the verdict with the insufficient_data synthetic verdict + drop
   any recommendations (no evidence == no actions).

Also tighten the fula-ota SSE schema to require root_cause on the
wire (was schema-optional) so the contract matches the parser.

Two new tests pin the guardrail behaviour (overrides on no data;
passes through on healthy session). The existing force-verdict
test was updated to include a successful tool call so its intent
(directive triggers a model verdict) is preserved without
tripping the guardrail. Net 60 tests, all passing.

Companion change in apps/box (separate commit) adds
insufficient_data to SYNTHETIC_VERDICT_CODES + a friendlier copy
for SCHEMA_VIOLATION_RECOVERED so the Try-Again CTA surfaces in
both newly-covered paths.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Author: ehsan6sha

src/runtime/rkllm_runtime.py

@@ -604,28 +604,6 @@ def destroy(self) -> None:
      - `fula.service`, `uniondrive.service` — host systemd services
      If `diag/containers` returns the above names, that is the COMPLETE list — there is no kubelet, no kube-proxy, no etcd, no apiserver. If the user reports "not earning", look at the actual diag/heartbeat + diag/relay + diag/wireguard signals you see in tool results, NOT at hypothetical Kubernetes components.
 
-# BAD vs GOOD examples
-
-❌ BAD — prose recommendations get NO Approve button, user can take no action:
-
-  ### Tier 2 Actions:
-  1. **ntp.resync** - Resync the clock.
-  2. **docker.restart container=ipfs_host** - Restart the container.
-
-✅ GOOD — each recommendation is its own XML block:
-
-  <recommendation>{{"action_name":"ntp.resync","args":{{}},"reasoning":"Clock is unsynced.","confidence":0.85,"tier":2}}</recommendation>
-  <recommendation>{{"action_name":"docker.restart","args":{{"container":"ipfs_host"}},"reasoning":"ipfs_host restart-looping.","confidence":0.75,"tier":2}}</recommendation>
-
-❌ BAD — making up a field:
-
-  "Time Status: Clock offset is significant (93 ms)"
-  (when tool_response actually said `time.status: green, synced: true` and the 93 was `internet.latency_ms_avg`)
-
-✅ GOOD — quote what you actually read:
-
-  "time.status is green (synced=true). internet.latency_ms_avg is 93ms — that's network latency, not a clock offset."
-
 # AVAILABLE TOOLS (read-only)
 
 {tool_list}
@@ -639,26 +617,27 @@ def destroy(self) -> None:
 - restart_fula — no args (tier 2)
 - reset — no args (tier 3, destructive — only after everything else)
 
-# FULL EXAMPLE — three-turn flow
+# RUNBOOK EXCERPTS
 
-Turn 1 (user said "device feels slow"):
-I'll start with a system summary.
-<tool_call>{{"name":"diag/summary","arguments":{{}}}}</tool_call>
+{runbook_excerpt}
 
-Turn 2 (after <tool_response> showing overall=red, internet=red, time=red):
-Internet and time are both red. Drilling in on internet.
-<tool_call>{{"name":"diag/internet","arguments":{{}}}}</tool_call>
+# FIRST ACTION
 
-Turn 3 (after <tool_response> showing dns_ok=true, https_discovery_ok=false):
-Discovery is unreachable; the clock being unsynced makes it worse. Re-sync NTP first.
-<verdict>{{"summary":"Discovery unreachable + clock unsynced.","severity":"red","root_cause":"discovery_https_unreachable"}}</verdict>
-<recommendation>{{"action_name":"ntp.resync","args":{{}},"reasoning":"Many discovery checks rely on accurate timestamps; resync first.","confidence":0.8,"tier":2}}</recommendation>
+Your VERY FIRST output token MUST start a <tool_call> block. Do NOT
+repeat these instructions back. Do NOT acknowledge. Do NOT explain
+what you are about to do. Start with `<tool_call>{{"name":"diag/`.
 
-# RUNBOOK EXCERPTS
+# VERDICT RULES
 
-{runbook_excerpt}
+After tool calls, emit ONE <verdict> with ALL of summary, severity,
+AND root_cause. NEVER omit root_cause. NEVER restate an example
+verdict from your training — base every verdict on the actual
+<tool_response> data you saw in THIS conversation.
 
-Be terse. Start with diag/summary unless the user named a specific symptom. Two or three tool calls, then finalize with a <verdict>."""
+If your tool calls all returned `ok:false` or no diagnostic data,
+your verdict MUST be:
+  <verdict>{{"summary":"Could not gather diagnostic data; please try again.","severity":"yellow","root_cause":"insufficient_data"}}</verdict>
+"""
 
 
 # Directive injected as a user message when the model has stalled
@@ -856,12 +835,22 @@ def parse_verdict(raw_text: str) -> Optional[dict]:
         return None
     summary = obj.get("summary")
     severity = obj.get("severity")
+    root_cause = obj.get("root_cause")
     if not isinstance(summary, str) or severity not in ("green", "yellow", "red"):
         return None
-    out = {"summary": summary[:500], "severity": severity}
-    if isinstance(obj.get("root_cause"), str):
-        out["root_cause"] = obj["root_cause"][:200]
-    return out
+    # root_cause is now REQUIRED — lab 2026-05-28: Qwen3-1.7B was emitting
+    # verdicts without root_cause, the SSE schema accepted them (root_cause
+    # was schema-optional), and the app's synthetic-verdict detector
+    # therefore never offered the Try-Again button. Rejecting here forces
+    # the run_troubleshoot loop into its no-verdict fallback path which
+    # synthesizes a proper verdict with a root_cause the app can detect.
+    if not isinstance(root_cause, str) or not root_cause.strip():
+        return None
+    return {
+        "summary": summary[:500],
+        "severity": severity,
+        "root_cause": root_cause[:200],
+    }
 
 
 def parse_recommendations(raw_text: str) -> list[dict]:
@@ -1150,6 +1139,16 @@ async def run_troubleshoot(
         # fallback). Diagnostic data is already in KV cache from prior
         # turns, so the retry only needs to format the conclusion.
         next_thinking: bool = self._enable_thinking
+        # successful_tool_count: incremented per tool_result with ok=True
+        # AND payload not None. Used by the no-data guardrail below the
+        # tool-call loop to suppress speculative verdicts (lab transcript
+        # 2026-05-28 not-earning scenario: model emitted
+        # "Internet slow + clock unsynced" verdict despite both tool
+        # calls returning ok=False payload=None — pattern-copied the
+        # GOOD example from the system prompt). Zero successful tools
+        # means we have no actual evidence for any severity-red claim;
+        # override with insufficient_data.
+        successful_tool_count: int = 0
 
         for turn in range(MAX_TURNS):
             # Last-chance: at MAX_TURNS-1 without a verdict, send the
@@ -1276,6 +1275,13 @@ async def run_troubleshoot(
                         tool_responses_for_context.append(
                             f"<tool_response>{json.dumps({'name': tc['tool'], 'result': result}, separators=(',', ':'))}</tool_response>"
                         )
+                        # Track successful evidence count for the no-data
+                        # verdict guardrail (see successful_tool_count
+                        # initialization comment + the guardrail block
+                        # before the `if verdict and not emitted_verdict`
+                        # yield below).
+                        if result is not None:
+                            successful_tool_count += 1
                     else:
                         tr_event = {
                             "type": "tool_result",
@@ -1309,6 +1315,37 @@ async def run_troubleshoot(
                     "content": next_content,
                 })
 
+            # No-evidence guardrail: if the model produced a verdict
+            # but NO tool call has ever returned actual data (all ok=False
+            # or all payload=None), suppress the speculative claim and
+            # replace with insufficient_data. Lab 2026-05-28: the
+            # not-earning scenario hit exactly this — model emitted
+            # `severity:red, summary:"Internet slow + clock unsynced"`
+            # while both tool_results in the transcript had payload:null
+            # (ok=False). Without this guardrail the user gets a
+            # confident-but-fabricated diagnosis. The override leaves the
+            # door open for the Try-Again button via the
+            # `insufficient_data` root_cause (apps/box's
+            # SYNTHETIC_VERDICT_CODES). Skip the override when the model
+            # ALREADY chose insufficient_data — that's the honest case.
+            if (
+                verdict
+                and not emitted_verdict
+                and successful_tool_count == 0
+                and verdict.get("root_cause") != "insufficient_data"
+            ):
+                logger.warning(
+                    "verdict_no_evidence_override original_severity=%s original_root_cause=%s",
+                    verdict.get("severity"),
+                    verdict.get("root_cause"),
+                )
+                verdict = {
+                    "summary": "Could not gather diagnostic data; please try again.",
+                    "severity": "yellow",
+                    "root_cause": "insufficient_data",
+                }
+                recommendations = []  # never propose actions without evidence
+
             # Emit verdict (once)
             if verdict and not emitted_verdict:
                 emitted_verdict = True