feat(agent): experimental in-browser ComfyAI agent

Squashes 60 incremental agent commits into one cohesive change after
prior history accumulated 119 unrelated upstream commits during a
bad rebase. Working tree restored from sno-agent@aec9f9380; new deps
added freshly (no stale lockfile entries); changeTracker API call
sites updated for current sno-frontend-preview (captureCanvasState →
checkState).

What's in this commit:

UI: src/agent/ui/{AgentRoot,AgentFab,XtermPanel,AgentSettings,
useXtermReadline}.vue
- Floating ComfyAI button + draggable xterm panel toggled via 'c'
  keybind; auto-focuses the terminal helper textarea on open.
- xterm-driven readline with Tab completion, Shift+Enter newline,
  Ctrl-A/E/U/K/L/C, history navigation, multiline buffers.
- Liquid-glass theme using Comfy design tokens.
- Settings panel auto-opens when no API key set; compact 3-field
  layout (API base / API key / model) + collapsed advanced section.

Session loop: src/agent/llm/session.ts, composables/useAgentSession.ts
- Vercel AI SDK streamText with run_shell as the only tool.
- IndexedDB-persisted message history (300-message cap), replays
  on reopen with a 'previous session' divider.
- Programmatic guardrails: PROMISSORY_PATTERN auto-continue,
  silent-fail auto-continue, fragile-shell-idiom blocklist
  (Layer 1), definitive-claim verifier registry (Layer 2 — orphans,
  missing-models, queue-state, pre-refusal, punt-to-user).
- Configurable baseURL for OpenRouter / local LLM proxies; default
  gpt-5.4 via OpenAI.

Shell runtime: src/agent/shell/
- POSIX-ish parser (shell-quote based), AsyncIterable pipes,
  redirection, &&/||/; sequencing.
- VFS: in-memory /tmp + UserdataVFS-backed /workflows.
- Coreutils + Comfy.* command dispatch + run-js fallback.

Commands: src/agent/shell/commands/
- comfy/comfyNs: registered command discovery + namespace dispatch.
- workflow: save-as / new-workflow / rename-workflow / clear-workflow
  --force / set-subgraph-{desc,aliases} (modal-bypass equivalents).
- nodeOps: node-search, add-node (smart placement), connect (auto-
  layout on link), disconnect, remove-node, layout, align-nodes,
  distribute-nodes, select, get-widget, toggle-panel.
- graph/state/execution/sweep: introspection + queue + sweep helpers.
- templates / images / install / validate / see: template loading,
  output→input copy, Manager model install, Gemini canvas vision.

Tests: browser_tests/tests/agentTerminal.spec.ts, plus
src/agent/**/*.test.ts unit tests.

Docs: docs/adr/0009-frontend-only-agent-and-local-agent-bridge.md
records the architectural choice + rejected alternatives.

i18n: minimal en additions in src/locales/en/main.json for
agent.{fab,input,panel,settings} namespaces.

Deps added: @ai-sdk/openai, ai, zod, shell-quote, idb-keyval,
@xterm/xterm, @xterm/addon-fit, es-toolkit, @types/shell-quote.

PR #11547 — experimental, draft, expect breaking changes.
Preview: https://pr-11547.comfy-ui.pages.dev/

Author: snomiao

browser_tests/tests/agentTerminal.spec.ts

@@ -0,0 +1,172 @@
+import { expect } from '@playwright/test'
+
+import type { ComfyPage } from '@e2e/fixtures/ComfyPage'
+import { comfyPageFixture as test } from '@e2e/fixtures/ComfyPage'
+
+/**
+ * E2E coverage for the in-browser agent terminal (AgentFab + XtermPanel).
+ *
+ * We exercise the deterministic surface: FAB → open/close, the solid-block
+ * ASCII banner, the "COMFY-AI" title pill, keyboard affordances
+ * (Enter submits, Shift+Enter inserts a literal newline, Tab completes),
+ * and a representative slice of shell commands that back the examples
+ * documented in the LLM system prompt. We intentionally do NOT drive the
+ * LLM itself — typing into xterm runs the shell directly, which is what
+ * the model ends up calling via the `run_shell` tool.
+ */
+
+async function openPanel(comfyPage: ComfyPage): Promise<void> {
+  const fab = comfyPage.page.getByTestId('agent-fab')
+  await expect(fab).toBeVisible()
+  await fab.click()
+  await expect(comfyPage.page.getByTestId('agent-panel')).toBeVisible()
+}
+
+async function readTerminalText(comfyPage: ComfyPage): Promise<string> {
+  // xterm renders into rows under .xterm-rows; concatenate the text content.
+  return await comfyPage.page
+    .getByTestId('agent-terminal')
+    .locator('.xterm-rows')
+    .innerText()
+}
+
+async function typeAndEnter(comfyPage: ComfyPage, text: string): Promise<void> {
+  const helper = comfyPage.page
+    .getByTestId('agent-terminal')
+    .locator('.xterm-helper-textarea')
+  await helper.focus()
+  await comfyPage.page.keyboard.type(text)
+  await comfyPage.page.keyboard.press('Enter')
+}
+
+test.describe('Agent terminal', { tag: '@ui' }, () => {
+  test('FAB opens the panel and shows the COMFY-AI title + prompt', async ({
+    comfyPage
+  }) => {
+    await openPanel(comfyPage)
+
+    await expect(comfyPage.page.getByTestId('agent-panel-title')).toHaveText(
+      'COMFY-AI'
+    )
+
+    // Banner is suppressed for token efficiency — only the shell prompt
+    // should be rendered once the terminal is ready.
+    await expect.poll(() => readTerminalText(comfyPage)).toMatch(/comfy>/)
+  })
+
+  test('Clicking the FAB again closes the panel', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    await comfyPage.page.getByTestId('agent-fab').click()
+    await expect(comfyPage.page.getByTestId('agent-panel')).toBeHidden()
+  })
+
+  test('Enter submits; help command lists built-ins', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    await typeAndEnter(comfyPage, 'help')
+    await expect
+      .poll(() => readTerminalText(comfyPage))
+      .toMatch(/run-js|cmd-list|comfy/)
+  })
+
+  test('Shift+Enter inserts a literal newline (no submit)', async ({
+    comfyPage
+  }) => {
+    await openPanel(comfyPage)
+    const helper = comfyPage.page
+      .getByTestId('agent-terminal')
+      .locator('.xterm-helper-textarea')
+    await helper.focus()
+    await comfyPage.page.keyboard.type('echo one')
+    await comfyPage.page.keyboard.press('Shift+Enter')
+    await comfyPage.page.keyboard.type('echo two')
+    // Still not submitted → now submit the whole multiline buffer.
+    await comfyPage.page.keyboard.press('Enter')
+
+    const out = await readTerminalText(comfyPage)
+    // Both commands ran sequentially via the runtime.
+    expect(out).toContain('one')
+    expect(out).toContain('two')
+  })
+
+  test('Tab completes a partial command', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    const helper = comfyPage.page
+      .getByTestId('agent-terminal')
+      .locator('.xterm-helper-textarea')
+    await helper.focus()
+    // "des" → unique prefix of `describe`
+    await comfyPage.page.keyboard.type('des')
+    await comfyPage.page.keyboard.press('Tab')
+    // Submit; describe w/o arg yields usage text on stderr.
+    await comfyPage.page.keyboard.press('Enter')
+    await expect
+      .poll(() => readTerminalText(comfyPage))
+      .toMatch(/usage: describe/)
+  })
+
+  test('coreutils: pwd / echo / true / false', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    await typeAndEnter(comfyPage, 'pwd')
+    await expect.poll(() => readTerminalText(comfyPage)).toMatch(/^\//m)
+
+    await typeAndEnter(comfyPage, 'echo hello world')
+    await expect
+      .poll(() => readTerminalText(comfyPage))
+      .toContain('hello world')
+  })
+
+  test('comfy namespace lists subcommands', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    await typeAndEnter(comfyPage, 'comfy')
+    await expect
+      .poll(() => readTerminalText(comfyPage))
+      .toMatch(/ComfyUI command namespace/)
+  })
+
+  test('run-js evaluates in the page scope', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    await typeAndEnter(comfyPage, 'run-js return 1 + 2')
+    await expect.poll(() => readTerminalText(comfyPage)).toMatch(/\b3\b/)
+  })
+
+  test('graph summary reports node count for the active graph', async ({
+    comfyPage
+  }) => {
+    await openPanel(comfyPage)
+    await typeAndEnter(comfyPage, 'graph summary')
+    await expect
+      .poll(() => readTerminalText(comfyPage))
+      .toMatch(/node|count|nodes/i)
+  })
+
+  test('queue-status command returns output', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    await typeAndEnter(comfyPage, 'queue-status')
+    await expect
+      .poll(() => readTerminalText(comfyPage))
+      .toMatch(/running|pending|queue/i)
+  })
+
+  test('active-workflow reports path / state', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    await typeAndEnter(comfyPage, 'active-workflow')
+    await expect
+      .poll(() => readTerminalText(comfyPage))
+      .toMatch(/path|modified|persisted|none/i)
+  })
+
+  test('pipe: echo foo | wc -c emits a byte count', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    await typeAndEnter(comfyPage, 'echo foo | wc -c')
+    // "foo\n" = 4 bytes
+    await expect.poll(() => readTerminalText(comfyPage)).toMatch(/\b4\b/)
+  })
+
+  test('unknown command surfaces an error', async ({ comfyPage }) => {
+    await openPanel(comfyPage)
+    await typeAndEnter(comfyPage, 'definitely-not-a-real-command-xyz')
+    await expect
+      .poll(() => readTerminalText(comfyPage))
+      .toMatch(/not found|unknown|no such/i)
+  })
+})

docs/adr/0009-frontend-only-agent-and-local-agent-bridge.md

@@ -0,0 +1,164 @@
+# 9. Frontend-only In-app Agent + Future Local-Agent Bridge
+
+Date: 2026-04-26
+
+## Status
+
+Proposed
+
+## Context
+
+PR #11547 introduces an experimental in-browser agent (`ComfyAI`) that
+lets users drive ComfyUI with natural language. It lives entirely in
+`src/agent/` and runs in the SPA — prompt assembly, tool execution
+(browser-side `run-js` + Comfy API calls), message storage, and IndexedDB
+chat history all happen client-side. The LLM is reached directly from
+the browser via the user's API key (OpenAI / OpenRouter / any
+OpenAI-compatible gateway), with optional Comfy Cloud auth for the
+small set of cloud nodes (Tripo / Tencent / Meshy / Gemini).
+
+This frontend-only architecture is deliberate. It keeps the deployment
+story trivial (no backend changes), keeps the user's API key out of
+ComfyUI's backend, and works whether the backend is local or remote.
+But it raises a coordination problem the moment users want their
+**other agents** — Claude Code, a self-hosted CLI agent, a teammate's
+agent on a different machine — to participate in the same conversation,
+see the same workflow state, or take actions on the user's behalf.
+
+The forces at play:
+
+- **Privacy**: API keys must not leak to ComfyUI's backend or to other
+  observers. The frontend-only model makes this trivially true today.
+- **Source of truth for graph state**: the canonical workflow lives in
+  LiteGraph's in-memory tree inside the SPA. Backend has the queue +
+  history but doesn't track unsaved edits. Any other agent that wants
+  current state must either read from the SPA or read a snapshot the
+  SPA publishes.
+- **Tool affordance**: the agent's `run_shell` tool currently executes
+  in the browser page context (DOM, stores, fetch with same-origin
+  cookies). A local agent has none of that — it would need either a
+  separate REST surface or to drive the SPA remotely.
+- **Identity**: the SPA can hold a Comfy Cloud token; a local agent is
+  a separate principal and should hold its own credentials.
+- **Versioning**: the moment we expose a wire format, breaking changes
+  hurt. Whatever we ship first becomes the contract.
+
+The question this ADR exists to answer: **how should a local agent
+participate in the in-app agent's session, given the frontend-only
+constraint we want to preserve?**
+
+## Decision
+
+**Short term (this PR and the next few): keep the agent strictly
+frontend-only.** Do not add any backend session state, message
+relaying, or local-agent bridge. The current architecture is small,
+auditable, and removes whole categories of risk.
+
+**Long term: when local-agent integration is taken on, prefer Option C
+("opt-in publish bus with execution staying in the SPA") over the
+alternatives.** The detailed shape:
+
+1. Define a small JSON-RPC schema for "agent context" — current
+   workflow id + serialized graph, last N messages, last K tool
+   invocations, agent settings (model + base URL only, never key).
+   Versioned from the start.
+2. SPA exposes a "Share session" toggle in agent settings. When on,
+   it publishes that snapshot to a configurable WS endpoint
+   (default: `ws://localhost:7437/agent`). The user explicitly opts
+   in per session.
+3. Provide a tiny reference subscriber library that local agents use
+   to consume. They get **read-only access by default**; getting
+   write access (post a message back into the user's panel) requires
+   the SPA to authorize via a one-time pairing code shown to the
+   user.
+4. **Tool execution stays in the SPA.** Local agents can _propose_
+   actions ("run this run-js"); the SPA executes and streams the
+   result back. The local agent is a peer that suggests, not an
+   actor that mutates.
+
+**Alternatives considered and rejected (for now):**
+
+- **Option A — ComfyUI backend as session broker.** Push messages to
+  the running ComfyUI server, local agents subscribe via WS or
+  polling. Rejected because ComfyUI is meant to be largely stateless,
+  adding session storage is scope creep, and it puts API keys / chat
+  content in front of the backend (privacy regression).
+- **Option B — browser extension or local sidecar daemon.** A
+  companion daemon reads the SPA's IndexedDB via Chrome DevTools
+  Protocol, or the SPA opens a localhost WS to it. Rejected as the
+  default path because of the cross-platform packaging burden and
+  because it doesn't help when the local agent runs on a different
+  machine than the SPA.
+
+**Comfy Cloud creds reuse (a related future work item):** when the
+user is signed into Comfy Cloud (the `auth_token_comfy_org` flow we
+already use for Tripo/Gemini), the agent could optionally route LLM
+calls through a Comfy-managed inference endpoint instead of OpenAI
+direct. This would gate naturally on the same auth as the cloud
+nodes and simplifies onboarding for users who don't have their own
+OpenAI/OpenRouter key. Out of scope here, but worth noting because
+it interacts with the local-agent identity story above.
+
+## Consequences
+
+### Positive
+
+- **No backend changes today.** PR #11547 lands without touching
+  ComfyUI core. Reviewers don't need to evaluate session-state
+  infrastructure they didn't ask for.
+- **Privacy posture stays strong.** API keys + chat content stay in
+  the user's browser; ComfyUI backend continues to see only what it
+  always saw (queue prompts, file uploads).
+- **Future local-agent path is clear** without committing to a
+  protocol prematurely. When we build it, the SPA stays the
+  source-of-truth + execution sandbox; the local agent is a peer that
+  suggests. Mirrors how editors coexist with Claude Code, GitHub
+  Copilot, etc.
+- **Headroom for multi-subscriber.** Option C naturally supports
+  agent + observer + log-tap subscribers with the same protocol —
+  useful for future debugging tools.
+- **Versioned wire format** means breaking changes are explicit.
+
+### Negative
+
+- **Local agents have no participation today.** Users who want their
+  Claude Code session to see what they're doing in ComfyUI need to
+  copy/paste workflow JSON manually.
+- **When we do build the bridge, it's net-new infrastructure** — a
+  WS server, a pairing flow, a versioning policy, a reference
+  subscriber library. Not trivial.
+- **Tool execution stays in the SPA** even after the bridge ships,
+  which means a local agent on a different machine can't `run-js`
+  against the user's session without the SPA being open. (We accept
+  this as a privacy + simplicity tradeoff.)
+- **The "Share session" toggle is yet another decision the user has
+  to make**, with non-obvious risks. Mitigations: clear UX copy,
+  default off, pairing-code requirement for write access.
+
+## Notes
+
+- The frontend-only constraint also drove several smaller decisions
+  in the PR that are worth recording briefly:
+  - Reasoning guardrails (`PROMISSORY_PATTERN`, `vetScript`,
+    `verifyClaims`) live in the SPA in `src/agent/llm/session.ts`,
+    not in a separate service. They survive prompt drift because
+    they're code, not text.
+  - Chat history is persisted via `useIDBKeyval` to IndexedDB. This
+    is a per-browser-profile store; switching profiles or clearing
+    site data wipes history. Acceptable for the experimental phase;
+    if local-agent bridge ships, the snapshot the SPA publishes
+    becomes another effective "external" history mechanism.
+- The default LLM is `gpt-5.4` via OpenAI's official API. The
+  settings panel exposes a base-URL field so users can target
+  OpenRouter (`https://openrouter.ai/api/v1`) or any OpenAI-compatible
+  gateway. This base-URL flexibility also makes Option C's "Comfy
+  Cloud as inference endpoint" trivially achievable later — it's just
+  another base-URL choice.
+- Concrete near-term TODOs flagged by this PR's stress-testing,
+  _not_ covered by this ADR but related:
+  - Layer 3 of the reasoning guardrails (structured JSON answers
+    with provenance) needs SDK plumbing to surface tool-call IDs
+    alongside text. Currently deferred.
+  - Verifier registry and shell-idiom blocklist are open
+    registries; entries grow as new failure modes surface in real
+    use.

docs/adr/README.md

@@ -8,16 +8,17 @@ An Architecture Decision Record captures an important architectural decision mad
 
 ## ADR Index
 
-| ADR                                                      | Title                                    | Status   | Date       |
-| -------------------------------------------------------- | ---------------------------------------- | -------- | ---------- |
-| [0001](0001-merge-litegraph-into-frontend.md)            | Merge LiteGraph.js into ComfyUI Frontend | Accepted | 2025-08-05 |
-| [0002](0002-monorepo-conversion.md)                      | Restructure as a Monorepo                | Accepted | 2025-08-25 |
-| [0003](0003-crdt-based-layout-system.md)                 | Centralized Layout Management with CRDT  | Proposed | 2025-08-27 |
-| [0004](0004-fork-primevue-ui-library.md)                 | Fork PrimeVue UI Library                 | Rejected | 2025-08-27 |
-| [0005](0005-remove-importmap-for-vue-extensions.md)      | Remove Import Map for Vue Extensions     | Accepted | 2025-12-13 |
-| [0006](0006-primitive-node-copy-paste-lifecycle.md)      | PrimitiveNode Copy/Paste Lifecycle       | Proposed | 2026-02-22 |
-| [0007](0007-node-execution-output-passthrough-schema.md) | NodeExecutionOutput Passthrough Schema   | Accepted | 2026-03-11 |
-| [0008](0008-entity-component-system.md)                  | Entity Component System                  | Proposed | 2026-03-23 |
+| ADR                                                        | Title                                                  | Status   | Date       |
+| ---------------------------------------------------------- | ------------------------------------------------------ | -------- | ---------- |
+| [0001](0001-merge-litegraph-into-frontend.md)              | Merge LiteGraph.js into ComfyUI Frontend               | Accepted | 2025-08-05 |
+| [0002](0002-monorepo-conversion.md)                        | Restructure as a Monorepo                              | Accepted | 2025-08-25 |
+| [0003](0003-crdt-based-layout-system.md)                   | Centralized Layout Management with CRDT                | Proposed | 2025-08-27 |
+| [0004](0004-fork-primevue-ui-library.md)                   | Fork PrimeVue UI Library                               | Rejected | 2025-08-27 |
+| [0005](0005-remove-importmap-for-vue-extensions.md)        | Remove Import Map for Vue Extensions                   | Accepted | 2025-12-13 |
+| [0006](0006-primitive-node-copy-paste-lifecycle.md)        | PrimitiveNode Copy/Paste Lifecycle                     | Proposed | 2026-02-22 |
+| [0007](0007-node-execution-output-passthrough-schema.md)   | NodeExecutionOutput Passthrough Schema                 | Accepted | 2026-03-11 |
+| [0008](0008-entity-component-system.md)                    | Entity Component System                                | Proposed | 2026-03-23 |
+| [0009](0009-frontend-only-agent-and-local-agent-bridge.md) | Frontend-only In-app Agent + Future Local-Agent Bridge | Proposed | 2026-04-26 |
 
 ## Creating a New ADR