Skip to content

Commit 09d6d13

Browse files
committed
feat(ui): render MCP UI widgets (MCP Apps) inline in chat
Add end-to-end support for rendering interactive MCP "App" UI resources inline in the chat instead of raw tool-call JSON. Backend: - registry.CreateToolsets returns the set of MCP-app-capable tool names (MCPAppToolNames) so the agent treats their results specially; tool classification is expressed via a single mcpToolKind classifier. - MakeMCPAppModelResultCallback keeps the rich tool payload in chat history for UI rendering while compacting what is sent to the model (avoids redundant polling/tool churn), wired in agent.go only when MCP-app tools are present; uses the typed MCP CallToolResult from the go-sdk. - /api/mcp-apps/{namespace}/{name}/... endpoints to list tools, call tools, and read ui:// resources from a RemoteMCPServer. UI: - Sandboxed McpAppRenderer (@mcp-ui/client) + ChatMcpAppsContext broker host<->app messaging (sendMessage, visible tool calls, resource/tool-call proxying); ChatLayoutUI mounts ChatMcpAppsProvider so widgets render. - McpAppsInspector view (servers/[ns]/[name]/apps) reachable from the server menu; ToolDisplay/ChatMessage/ToolCallDisplay render app widgets. - sandbox_proxy.html restricts postMessage to the expected parent origin. Tests: Go registry / model-result callback; UI context mapping, mcp-apps server actions, and a ChatLayoutUI provider-mount regression test. Adjacent chat file-upload/minimap UI is included to compile but is out of scope (file-upload backend excluded), per the EP. Includes design/EP-2046-chat-mcp-ui-widgets.md. Signed-off-by: Dmytro Rashko <dmitriy.rashko@amdocs.com>
1 parent a766d22 commit 09d6d13

34 files changed

Lines changed: 3804 additions & 166 deletions
Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# EP-2046: Chat UI support for MCP UI widgets (MCP Apps)
2+
3+
* Issue: [#2046](https://github.com/kagent-dev/kagent/issues/2046)
4+
5+
## Background
6+
7+
The Model Context Protocol is gaining an "Apps"/UI extension
8+
(`@modelcontextprotocol/ext-apps`, rendered via `@mcp-ui/client`) that lets an MCP
9+
server attach an interactive HTML/UI **resource** to a tool. When an agent calls
10+
such a tool, the client can render a live widget instead of (or in addition to) raw
11+
tool-call JSON.
12+
13+
kagent's chat today renders tool calls as collapsible JSON. This EP makes the chat
14+
MCP-App–aware: when a tool call maps to an MCP app resource, the chat renders the
15+
app inline in a sandboxed frame and brokers messages between the app and the chat
16+
(send a message on the user's behalf, surface "visible" tool calls, proxy resource
17+
reads and tool calls back to the originating MCP server).
18+
19+
## Motivation
20+
21+
- Let MCP servers deliver rich, interactive results (forms, boards, charts, live
22+
progress) directly in the kagent chat.
23+
- Provide the in-chat rendering half of the kagent plugin story (the sidebar/plugin
24+
half is EP-2047; the first consumer is the Kanban task-progress widget, EP-2048).
25+
26+
### Goals
27+
28+
- Discover MCP app resources per MCP server and associate them with tool calls.
29+
- Render the app via a sandboxed renderer inside chat messages / tool-call display.
30+
- Broker host↔app messaging: `sendMessage`, visible tool calls, and proxying of
31+
resource reads and tool calls to the backend MCP server.
32+
- Backend endpoints to list an MCP server's tools, read its resources, and call its
33+
tools on behalf of the UI.
34+
35+
### Non-Goals
36+
37+
- The sidebar plugin/registration mechanism (EP-2047).
38+
- Shipping a specific MCP app (the Kanban task-progress app is EP-2048).
39+
- File-upload / artifact handling — note the chat files carry adjacent
40+
file-upload/minimap code (see "Adjacent code" below); that feature is tracked
41+
separately and is **not** part of this EP's scope.
42+
43+
## Implementation Details
44+
45+
### Backend
46+
47+
- **`go/adk/pkg/mcp/registry.go`**`CreateToolsets` now also returns the set of
48+
**MCP-app–capable tool names** (tools whose MCP server advertises a UI resource),
49+
so the agent can treat their results specially.
50+
- **`go/adk/pkg/agent/mcp_apps.go`**`MakeMCPAppModelResultCallback`: for
51+
MCP-app tools, keep the rich tool payload in chat history for UI rendering while
52+
compacting what is sent back to the model (avoids redundant polling/tool churn).
53+
Wired in `agent.go` only when `len(mcpAppToolNames) > 0`.
54+
- **`go/core/internal/httpserver/handlers/mcpapps.go`**`MCPAppsHandler` with
55+
`HandleListTools`, `HandleCallTool`, `HandleReadResource`, exposed under
56+
`/api/mcp-apps/{namespace}/{name}/...`. (Only the MCP-apps hunks of the shared
57+
`server.go`/`handlers.go` are included here; the plugins hunks belong to EP-2047.)
58+
59+
### UI (`ui/src`)
60+
61+
- **`components/mcp-apps/McpAppRenderer.tsx`** — renders an MCP app resource via
62+
`@mcp-ui/client` in a sandbox, wiring its `onUIAction`/resource-read/tool-call
63+
callbacks to the backend; `McpAppsInspector.tsx` is a standalone inspector view
64+
(also surfaced at `app/servers/[namespace]/[name]/apps/page.tsx`, and reachable
65+
from an **"MCP Apps"** entry added to the per-server menu in
66+
`components/mcp/McpServersView.tsx`).
67+
- **`components/chat/ChatMcpAppsContext.tsx`** — context that maps a tool name to its
68+
MCP app (`getMcpAppForTool`) and brokers `sendMessage` / `McpAppVisibleToolCall`
69+
between an app and the chat.
70+
- **`components/chat/ChatLayoutUI.tsx`** — mounts `ChatMcpAppsProvider` around the
71+
chat subtree so the MCP-app context is active for every chat session (without this
72+
mount, tool calls never resolve to apps and no widget renders).
73+
- **`components/chat/ChatInterface.tsx`, `ChatMessage.tsx`, `ToolCallDisplay.tsx`,
74+
`components/ToolDisplay.tsx`** — render the app for MCP-app tool calls and forward
75+
app actions.
76+
- **`app/actions/mcp-apps.ts`** + **`app/api/mcp-apps/.../{resources,tools/.../call}`**
77+
— server actions / BFF routes calling the backend MCP-apps endpoints.
78+
- **`public/sandbox_proxy.html`** — sandbox proxy document for the app iframe.
79+
80+
### New dependencies (`ui/package.json`)
81+
82+
- `@mcp-ui/client` `^7.1.1`
83+
- `@modelcontextprotocol/ext-apps` `^1.7.1`
84+
- `@modelcontextprotocol/sdk` `^1.29.0`
85+
86+
The lockfile (`ui/package-lock.json`) and the generated `ui/public/mockServiceWorker.js`
87+
(MSW worker, bumped `2.14.2``2.14.6`) are regenerated as a side effect of resolving
88+
the new dependency tree.
89+
90+
### Adjacent code
91+
92+
Per the agreed split, the chat files (`ChatInterface.tsx`, `ChatMessage.tsx`,
93+
`messageHandlers.ts`) are taken whole and therefore also carry the chat
94+
**file-upload** (`lib/fileUpload.ts`, `chat/FileAttachment.tsx`) and **minimap**
95+
(`chat/ChatMinimap.tsx`) UI that was developed alongside MCP apps. These are
96+
included so the chat compiles, but are not the subject of this EP; the file-upload
97+
backend (artifact extraction, `save_artifact`) is intentionally **excluded**.
98+
99+
## Test Plan
100+
101+
- **Unit (Go):** `registry_test.go` (MCP-app tool-name detection) and
102+
`mcp_apps_test.go` (model-result callback). `go build ./adk/... ./core/...` and
103+
test compilation pass.
104+
- **Unit (UI):** `getMcpAppForTool` mapping (`ChatMcpAppsContext.test.tsx`); mcp-apps
105+
server actions (`actions/__tests__/mcp-apps.test.ts`); and a regression test
106+
(`chat/__tests__/ChatLayoutUI.test.tsx`) asserting `ChatLayoutUI` mounts
107+
`ChatMcpAppsProvider` around the chat so widgets can render.
108+
- **Manual / e2e:** point the chat at an MCP server exposing a UI resource; confirm
109+
the widget renders inline, `sendMessage` posts to the chat, and resource/tool-call
110+
proxying reaches the server. The Kanban task-progress widget (EP-2048) is the
111+
reference end-to-end case.
112+
113+
## Alternatives
114+
115+
- **Render apps only in a side panel (not inline in chat):** loses the
116+
tool-call→widget association and the conversational flow.
117+
- **Trust the model with full tool payloads:** causes token bloat and tool churn;
118+
hence the model-result compaction callback.
119+
120+
## Open Questions
121+
122+
- Should MCP-app rendering be opt-in per MCP server (a `spec` flag) rather than
123+
inferred from advertised UI resources?
124+
- How should multiple apps in a single conversation share/scope state?

go/adk/pkg/agent/agent.go

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ func CreateGoogleADKAgentWithSubagentSessionIDs(ctx context.Context, agentConfig
5555
if stsPlugin != nil {
5656
dynamicHeaderProvider = stsPlugin.HeaderProvider
5757
}
58-
toolsets := mcp.CreateToolsets(ctx, agentConfig.HttpTools, agentConfig.SseTools, propagateToken, dynamicHeaderProvider)
58+
toolsets, mcpAppToolNames := mcp.CreateToolsets(ctx, agentConfig.HttpTools, agentConfig.SseTools, propagateToken, dynamicHeaderProvider)
5959
subagentSessionIDs := make(map[string]string)
6060

6161
var remoteAgentTools []tool.Tool
@@ -116,6 +116,12 @@ func CreateGoogleADKAgentWithSubagentSessionIDs(ctx context.Context, agentConfig
116116
beforeToolCallbacks = append(beforeToolCallbacks, MakeApprovalCallback(approvalSet))
117117
beforeModelCallbacks = append(beforeModelCallbacks, MakeStripConfirmationPartsCallback())
118118
}
119+
if len(mcpAppToolNames) > 0 {
120+
// For MCP App-capable tools, keep rich tool payloads in chat history for UI rendering,
121+
// but compact what is sent back to the model to avoid redundant polling/tool churn.
122+
log.Info("Wiring MCP App model result callback", "toolCount", len(mcpAppToolNames))
123+
beforeModelCallbacks = append(beforeModelCallbacks, MakeMCPAppModelResultCallback(mcpAppToolNames))
124+
}
119125
beforeToolCallbacks = append(beforeToolCallbacks, makeBeforeToolCallback(log))
120126

121127
llmAgentConfig := llmagent.Config{

go/adk/pkg/agent/mcp_apps.go

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
package agent
2+
3+
import (
4+
"encoding/json"
5+
6+
mcpsdk "github.com/modelcontextprotocol/go-sdk/mcp"
7+
"google.golang.org/adk/agent"
8+
"google.golang.org/adk/agent/llmagent"
9+
adkmodel "google.golang.org/adk/model"
10+
)
11+
12+
// mcpAppRenderedNotice is the terminal message the model sees in place of an
13+
// MCP App tool's render payload. An MCP App tool (one that declares a UI
14+
// resourceUri) produces an interactive view that is displayed to the user and
15+
// refreshes itself in-place via its own app-only tool calls. The model must
16+
// treat a successful render as a completed, self-updating artifact; otherwise it
17+
// tends to re-invoke the rendering tool on every "refresh", flooding the chat
18+
// with duplicate app cards. This notice is protocol-oriented: it applies to any
19+
// tool carrying a UI resourceUri, independent of the tool's name or payload keys.
20+
const mcpAppRenderedNotice = "The interactive UI for this tool has been rendered to the user and now updates live inside the app. Treat this as complete and do not call this tool again unless the user explicitly asks for it."
21+
22+
// MakeMCPAppModelResultCallback replaces what the model sees for MCP App
23+
// (UI-rendering) tool results: instead of the heavy render payload it receives a
24+
// terminal directive (see mcpAppRenderedNotice). The full result is still
25+
// streamed to the UI separately, so this only changes the model's view and
26+
// prevents the model from looping on the rendering tool. Errors are passed
27+
// through so the model can still react to and recover from failures.
28+
func MakeMCPAppModelResultCallback(appToolNames map[string]bool) llmagent.BeforeModelCallback {
29+
return func(_ agent.CallbackContext, req *adkmodel.LLMRequest) (*adkmodel.LLMResponse, error) {
30+
for _, content := range req.Contents {
31+
if content == nil {
32+
continue
33+
}
34+
for _, part := range content.Parts {
35+
if part == nil || part.FunctionResponse == nil || !appToolNames[part.FunctionResponse.Name] {
36+
continue
37+
}
38+
part.FunctionResponse.Response = compactMCPAppModelResponse(part.FunctionResponse.Response)
39+
}
40+
}
41+
return nil, nil
42+
}
43+
}
44+
45+
// compactMCPAppModelResponse rewrites an MCP App tool result for the model.
46+
//
47+
// The model exchanges tool results as a generic map (genai
48+
// FunctionResponse.Response), but the payload is really an MCP
49+
// [mcpsdk.CallToolResult]. We decode it into that typed result so the logic
50+
// works against real fields (IsError, Content, Meta, StructuredContent) rather
51+
// than poking at string keys. If the payload isn't a recognizable MCP result we
52+
// leave it untouched.
53+
func compactMCPAppModelResponse(response map[string]any) map[string]any {
54+
result, err := decodeCallToolResult(response)
55+
if err != nil {
56+
return response
57+
}
58+
59+
if result.IsError {
60+
// On error, keep the original content/meta so the model can
61+
// diagnose and recover; only drop the heavy structured payload.
62+
result.StructuredContent = nil
63+
return encodeCallToolResult(result, response)
64+
}
65+
66+
// On success, collapse the render payload into a terminal directive so the
67+
// model stops re-invoking the rendering tool. Preserve _meta (e.g.
68+
// resourceUri) in case downstream tooling relies on it.
69+
compact := &mcpsdk.CallToolResult{
70+
Meta: result.Meta,
71+
Content: []mcpsdk.Content{&mcpsdk.TextContent{Text: mcpAppRenderedNotice}},
72+
}
73+
return encodeCallToolResult(compact, response)
74+
}
75+
76+
// decodeCallToolResult interprets a generic model-facing response map as a typed
77+
// MCP CallToolResult.
78+
func decodeCallToolResult(response map[string]any) (*mcpsdk.CallToolResult, error) {
79+
raw, err := json.Marshal(response)
80+
if err != nil {
81+
return nil, err
82+
}
83+
var result mcpsdk.CallToolResult
84+
if err := json.Unmarshal(raw, &result); err != nil {
85+
return nil, err
86+
}
87+
return &result, nil
88+
}
89+
90+
// encodeCallToolResult converts a typed CallToolResult back into the generic map
91+
// the model expects, falling back to the original response if conversion fails.
92+
func encodeCallToolResult(result *mcpsdk.CallToolResult, fallback map[string]any) map[string]any {
93+
raw, err := json.Marshal(result)
94+
if err != nil {
95+
return fallback
96+
}
97+
var out map[string]any
98+
if err := json.Unmarshal(raw, &out); err != nil {
99+
return fallback
100+
}
101+
return out
102+
}

go/adk/pkg/agent/mcp_apps_test.go

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
package agent
2+
3+
import (
4+
"testing"
5+
6+
adkmodel "google.golang.org/adk/model"
7+
"google.golang.org/genai"
8+
)
9+
10+
func TestMakeMCPAppModelResultCallbackReplacesRenderPayloadWithNotice(t *testing.T) {
11+
t.Parallel()
12+
13+
req := &adkmodel.LLMRequest{
14+
Contents: []*genai.Content{{
15+
Parts: []*genai.Part{{
16+
FunctionResponse: &genai.FunctionResponse{
17+
Name: "jenkins_monitor_build",
18+
Response: map[string]any{
19+
"content": []map[string]any{{
20+
"type": "text",
21+
"text": "Opened Jenkins Build Monitor for https://example.com/job/demo/1/ (current status: IN_PROGRESS).",
22+
}},
23+
"structuredContent": map[string]any{
24+
"build": map[string]any{
25+
"stages": []any{map[string]any{"name": "Deploy", "status": "IN_PROGRESS"}},
26+
},
27+
"polling_data": "large payload",
28+
},
29+
"_meta": map[string]any{
30+
"ui": map[string]any{
31+
"resourceUri": "ui://jenkins-mcp/build-monitor",
32+
},
33+
},
34+
},
35+
},
36+
}},
37+
}},
38+
}
39+
40+
callback := MakeMCPAppModelResultCallback(map[string]bool{"jenkins_monitor_build": true})
41+
if _, err := callback(nil, req); err != nil {
42+
t.Fatalf("callback returned error: %v", err)
43+
}
44+
45+
got := req.Contents[0].Parts[0].FunctionResponse.Response
46+
47+
// Success render payload should be collapsed into the terminal notice so the
48+
// model stops re-invoking the rendering tool.
49+
content, ok := got["content"].([]any)
50+
if !ok || len(content) != 1 {
51+
t.Fatalf("content not replaced with notice: %#v", got["content"])
52+
}
53+
part, ok := content[0].(map[string]any)
54+
if !ok || part["text"] != mcpAppRenderedNotice {
55+
t.Fatalf("notice text missing: %#v", content[0])
56+
}
57+
58+
// Should strip structuredContent (heavy render payload).
59+
if _, ok := got["structuredContent"]; ok {
60+
t.Fatalf("structuredContent should be stripped, got: %#v", got)
61+
}
62+
63+
// Should preserve _meta
64+
meta, ok := got["_meta"].(map[string]any)
65+
if !ok {
66+
t.Fatalf("_meta not preserved: %#v", got["_meta"])
67+
}
68+
if _, ok := meta["ui"]; !ok {
69+
t.Fatalf("_meta.ui not preserved: %#v", meta)
70+
}
71+
}
72+
73+
func TestMakeMCPAppModelResultCallbackPreservesIsError(t *testing.T) {
74+
t.Parallel()
75+
76+
req := &adkmodel.LLMRequest{
77+
Contents: []*genai.Content{{
78+
Parts: []*genai.Part{{
79+
FunctionResponse: &genai.FunctionResponse{
80+
Name: "jenkins_monitor_build",
81+
Response: map[string]any{
82+
"content": []map[string]any{{
83+
"type": "text",
84+
"text": "Tool execution failed.",
85+
}},
86+
"structuredContent": map[string]any{"error": "connection timeout"},
87+
"isError": true,
88+
},
89+
},
90+
}},
91+
}},
92+
}
93+
94+
callback := MakeMCPAppModelResultCallback(map[string]bool{"jenkins_monitor_build": true})
95+
if _, err := callback(nil, req); err != nil {
96+
t.Fatalf("callback returned error: %v", err)
97+
}
98+
99+
got := req.Contents[0].Parts[0].FunctionResponse.Response
100+
101+
// Should preserve isError
102+
isErr, ok := got["isError"].(bool)
103+
if !ok || !isErr {
104+
t.Fatalf("isError not preserved or false: %#v", got["isError"])
105+
}
106+
107+
// Should still strip structuredContent
108+
if _, ok := got["structuredContent"]; ok {
109+
t.Fatalf("structuredContent should be stripped")
110+
}
111+
}
112+
113+
func TestMakeMCPAppModelResultCallbackLeavesNonAppToolsAlone(t *testing.T) {
114+
t.Parallel()
115+
116+
original := map[string]any{
117+
"output": map[string]any{"answer": 42},
118+
"content": []map[string]any{{
119+
"type": "text",
120+
"text": "Answer is 42",
121+
}},
122+
}
123+
req := &adkmodel.LLMRequest{
124+
Contents: []*genai.Content{{
125+
Parts: []*genai.Part{{
126+
FunctionResponse: &genai.FunctionResponse{
127+
Name: "regular_tool",
128+
Response: original,
129+
},
130+
}},
131+
}},
132+
}
133+
134+
callback := MakeMCPAppModelResultCallback(map[string]bool{"some_app_tool": true})
135+
if _, err := callback(nil, req); err != nil {
136+
t.Fatalf("callback returned error: %v", err)
137+
}
138+
139+
got := req.Contents[0].Parts[0].FunctionResponse.Response
140+
141+
// Non-app tools should pass through unchanged
142+
if _, ok := got["output"]; !ok {
143+
t.Fatalf("non-app tool response modified: %#v", got)
144+
}
145+
}

0 commit comments

Comments
 (0)