WolframContext crashes with AgentTools::Internal::Path when Chatbook returns Failure[...] — Context.wl:264 ConfirmBy[..., StringQ] needs graceful fallback

[jakobtfaber]

Summary

Context.wl:264 (relatedWolframAlphaResults) wraps the upstream Chatbook function cbRelatedWolframAlphaResultsinConfirmBy[..., StringQ, "Prompt"]. When the Chatbook function returns Failure[...]— which it does any time the Wolfram|Alpha knowledgebase API returns 500/503 — theConfirmByrejects it and throwsAgentTools::Internal::Path@@Kernel/Tools/Context.wl:264,18-273,10` to the MCP client. There's no user-readable surface for the underlying upstream failure; the user sees only an opaque internal error.

The same ConfirmBy[..., StringQ] shape appears elsewhere in the same file (and in WolframLanguageEvaluator.wl:112), so the same upstream-failure → internal-error conversion happens for related codepaths. Context.wl:241 (relatedWolframAlphaPrompt) shows how to do this gracefully — it explicitly branches on connected/missing-LLMKit and applies the appropriate template before ConfirmBy. Context.wl:264 has no such fallback.

Reproduction

Local environment: macOS 25.4.0 (arm64), Wolfram Engine 14.3.0 for Mac OS X ARM, Chatbook 2.5.0, AgentTools 2.0.0 (the running MCP-server kernel — see "Note on paclet version" below; the antipattern persists identically in the installed 2.1.0 source).

Step 1 — call mcp__WolframAll__WolframContext from any MCP client with an input long enough to trigger a heavy Chatbook lookup. ~250 words of context referencing DSolve/NDSolve will do it. Example payload:

Solve the second-order ODE y''[t] + 2 y'[t] + y[t] == 0 with initial conditions
y[0]==1 and y'[0]==0 using DSolve and NDSolve. [repeat 8 times]

Step 2 — at some point during a session in which the W|A knowledgebase API is intermittently returning 500 (e.g., the <error status="500" message="Unable to verify Ip"> payload I observed today), the call surfaces as:

[Error] An unexpected error occurred: AgentTools::Internal::Path@@Kernel/Tools/Context.wl:264,18-273,10.
Full details of the error have been logged to: ~/Library/Wolfram/Logs/AgentTools/InternalFailures/<timestamp>.mx
Report this issue at https://github.com/WolframResearch/AgentTools/issues/new

Step 3 — load the persisted .mx and inspect the failure structure. The relevant fields are:

"Evaluation" :> Wolfram`AgentTools`Common`relatedWolframAlphaResults["<the long context>", ...]
"Information" -> "Prompt@@Kernel/Tools/Context.wl:264,18-273,10"
"ConfirmationType" -> "ConfirmBy"
"Function" -> StringQ
"Expression" :> Failure["General::ChatbookInternal", <|
    "MessageParameters" :> {Hyperlink["Report this issue »", "https://github.com/WolframResearch/Chatbook/issues/new?..."]},
    ...
|>]

The triggering inner failure was Chatbook's setXMLResult[..., <|"StatusCode" -> 500, "BodyByteArray" -> ByteArray["..."]|>] where the body decodes to <?xml version="1.0" encoding="UTF-8"?><error status="500" message="Unable to verify Ip"></error>.

Why this is a paclet bug, not just an upstream W|A flake

A 500 from the upstream W|A knowledgebase is something the paclet has no control over and cannot prevent — that's fine. But the paclet's response to it is to throw an opaque internal-error message at the user, with no information about what actually went wrong. The user can't distinguish "the W|A backend is having a bad few minutes" from "the paclet is broken" from "my prompt was malformed" — they're indistinguishable surfaces.

Context.wl:241 already has the right shape for graceful degradation:

template  = If[ connected, $wolframAlphaMissingLLMKitTemplate, $wolframAlphaNoCloudTemplate ];
ConfirmBy[ TemplateApply[ template, <| "URL" -> url, "Level" -> level |> ], StringQ, "Result" ]

The W|A-results path at line 264 should do something analogous: catch Failure[...] returns from cbRelatedWolframAlphaResults` explicitly, and surface a short user-readable diagnostic ("Wolfram|Alpha knowledgebase is unavailable; try again, or use the WolframLanguageEvaluator tool for a direct kernel computation") instead of throwing as an internal error.

Affected callsites with the same antipattern

• Kernel/Tools/Context.wl:264 (relatedWolframAlphaResults) — produced the crash above.
• Kernel/Tools/Context.wl:295 (relatedDocumentation) — same shape; will fail identically when the Chatbook documentation lookup returns Failure[...].
• Kernel/Tools/WolframLanguageEvaluator.wl:112 (evaluateWolframLanguage0) — separate failure surface but same antipattern; surfaces when user code happens to return a Failure[...].

Note on paclet version

My running MCP-server kernel was reporting Version → "2.0.0" in every crash log today (11 logs from ~/Library/Wolfram/Logs/AgentTools/InternalFailures/2026-04-28_*.mx) despite my installed paclet at ~/Library/Wolfram/Paclets/Repository/Wolfram__AgentTools-2.1.0/ being v2.1.0 — the kernel was launched against v2.0.0 and stayed running through the in-place upgrade. That's a separate UX issue (no warning to the user that the running kernel is stale; only Claude Code restart fixes it). I'm reading 2.1.0's Kernel/Tools/Context.wl as I file this and the antipattern is structurally identical at the same line number, so the bug is current, not just historical to 2.0.0.

Suggested fix

Replace ConfirmBy[..., StringQ, "Prompt"] at lines 264 and 295 with an explicit pattern-match that lets Failure[...] returns through to a user-readable fallback path. Concretely:

prompt = cb`RelatedWolframAlphaResults[ context, "Prompt", "MaxItems" -> maxItems, "IncludeWLResults" -> includeWLResults ];
prompt = Replace[ prompt, {
    s_String                 :> StringTrim @ s,
    f_Failure                :> wolframAlphaBackendUnavailableTemplate[ f ],
    _                        :> ConfirmBy[ prompt, StringQ, "Prompt" ]  (* falls back to existing throw for truly unexpected types *)
} ];

Where wolframAlphaBackendUnavailableTemplate produces a short user-facing string like:

"The Wolfram|Alpha context lookup is currently unavailable (the upstream knowledgebase returned an error). The result of WolframAlpha[...] calls in this session may be incomplete. Try again in a few minutes, or use the WolframLanguageEvaluator tool for a direct kernel computation."

Same shape applies to relatedDocumentation at line 295.

[jakobtfaber]

Maintainer diagnostic supplement (April 2026 — ConfirmBy / Failure[...] only)

These notes intentionally do not cover wolframscript exit codes or license teardown (139) — orthogonal to Context.wl:264.

---

1. Exact repro input

Use the Ctx-long-proxy long-context probe (same StringRepeat spine as ~/bin/wolfram-mcp-smoke.wls):

Needs["Wolfram`Chatbook`"];
longCtx = StringRepeat[
  "Solve the second-order ODE y''[t] + 2 y'[t] + y[t] == 0 with y[0]==1, y'[0]==0 using DSolve and NDSolve. ",
  8];
Wolfram`Chatbook`RelatedWolframAlphaResults[longCtx, "Prompt"]

Upstream Wolfram\AgentTools`Tools` path funnels cbRelatedWolframAlphaResults through relatedWolframAlphaResults in Kernel/Tools/Context.wl (see Context.wl:264 today).

---

2. Decoded crash-path chain

• Wolfram|Alpha gateway may return HTTP 500 with body Unable to verify Ip (<error message="Unable to verify Ip">) during IP-verification flake (see mcp_wolfram_servers.md Bug 1 narrative in user memory ~/.claude/projects/-Users-jakobfaber/memory/mcp_wolfram_servers.md).
• cbRelatedWolframAlphaResults surfaces Failure[..., XML body ...], not String.
• prompt = Confirm[..., StringQ, "Prompt"] at ~/Library/Wolfram/Paclets/Repository/Wolfram__AgentTools-2.1.0/Kernel/Tools/Context.wl:264 rejects Failure and throws AgentTools::Internal::Path.

Inline .mx sample: archival ~/Library/Wolfram/Logs/AgentTools/InternalFailures/2026-04-28_*.mx from the original filing may have rotated — the structure (HTTP 500 body + Failure[..., ... StatusCode chain) matches the CBFailureEnvelope spelled out in the issue description.

---

3. Intermittency observation

Fresh wolframscript and a long-lived MCP kernel tied to Claude Code sometimes disagree on RelatedWolframAlphaResults/W|A verdicts seconds apart, because gateway 500/IP-check behavior is nondeterministic. The ConfirmBy defect then turns an ordinary upstream flake into an opaque AgentTools fatal whenever Failure is returned — more often than “short smoke” tooling exercises.

---

4. Why a MatchQ-style _Failure arm fixes the surfaced type

Suggested shape (see draft unified diff, not merged here): waRaw from cbRelatedWolframAlphaResults**, then **Replace[..., { _String ..., _Failure :> , _ :> ConfirmBy[...]}]`.

That matches the actual Mathematica types (Failure vs String). $wolframAlphaMissingLLMKitTemplate at Context.wl:~241 is a separate branch keyed on $CloudConnected / subscription messaging — not the Chatbook / W|A KB HTTP error path Failure expresses here — so MatchQ/pattern dispatch belongs at 264, not copying the LLMKit ternary.