|
| 1 | +# Client-Side ComfyUI Agent — Design Notes |
| 2 | + |
| 3 | +## Motivation |
| 4 | + |
| 5 | +Run an LLM agent **inside the user's browser** that can do everything the ComfyUI frontend can do, driven by the existing Pinia stores / composables / `ComfyApi` — no new backend, no new privileges. Because the agent only replays operations the user can already trigger by hand, **the existing backend auth/rate-limits/permissions remain the sole security boundary**. The "frontend bash" adds no attack surface. |
| 6 | + |
| 7 | +Provider-agnostic via Vercel `ai-sdk`. LLMs are already fluent in bash/CLI, so we expose frontend operations as a **yargs-defined CLI** running against a **virtual filesystem** over existing asset APIs. |
| 8 | + |
| 9 | +--- |
| 10 | + |
| 11 | +## Architecture (3 layers) |
| 12 | + |
| 13 | +``` |
| 14 | +┌─────────────────────────────────────────┐ |
| 15 | +│ LLM (ai-sdk, streamText + tools) │ |
| 16 | +├─────────────────────────────────────────┤ |
| 17 | +│ Shell Emulator │ |
| 18 | +│ - argv tokenize, pipe |, && ;, > VAR │ |
| 19 | +│ - stdout/stderr buffers │ |
| 20 | +│ - virtual FS (cwd, paths) │ |
| 21 | +├─────────────────────────────────────────┤ |
| 22 | +│ Tool Adapters │ |
| 23 | +│ ├ commandStore.execute() │ |
| 24 | +│ ├ useGraphNodeManager (add/connect) │ |
| 25 | +│ ├ api.queuePrompt / getQueue / ... │ |
| 26 | +│ ├ workflowStore / widgetValueStore │ |
| 27 | +│ └ userdata / assets / models APIs │ |
| 28 | +└─────────────────────────────────────────┘ |
| 29 | +``` |
| 30 | + |
| 31 | +--- |
| 32 | + |
| 33 | +## 1. Frontend surface map |
| 34 | + |
| 35 | +### Graph / Node management |
| 36 | +- `src/lib/litegraph/src/LGraph.ts`, `LGraphNode.ts`, `LGraphCanvas.ts`, `LLink.ts` |
| 37 | +- Subgraphs: `src/lib/litegraph/src/subgraph/*` |
| 38 | +- Widgets: `src/lib/litegraph/src/widgets/*` |
| 39 | +- Pinia stores: |
| 40 | + - `src/stores/nodeDefStore.ts` — `getNodeDef()`, search |
| 41 | + - `src/stores/canvasStore.ts` |
| 42 | + - `src/stores/widgetStore.ts`, `widgetValueStore.ts` |
| 43 | + - `src/stores/subgraphStore.ts` |
| 44 | +- Composables: |
| 45 | + - `src/composables/graph/useGraphNodeManager.ts` — add/connect/widget |
| 46 | + - `src/composables/graph/useSubgraphOperations.ts` |
| 47 | + - `src/composables/graph/useSelectionOperations.ts` |
| 48 | + |
| 49 | +### Command system |
| 50 | +- `src/stores/commandStore.ts` — `ComfyCommand`, `registerCommand`, `execute(id)` |
| 51 | +- `src/composables/useCoreCommands.ts` — core command set (file/canvas/exec/view) |
| 52 | +- Extensions add more commands via `extensionService.ts:70` (`loadExtensionCommands`) |
| 53 | +- All commands are already idempotent-ish and UI-addressable → perfect CLI targets |
| 54 | + |
| 55 | +### Backend API layer |
| 56 | +- `src/scripts/api.ts` — `ComfyApi extends EventTarget` |
| 57 | +- HTTP: |
| 58 | + - `POST /prompt` → `queuePrompt()` (L854) |
| 59 | + - `GET /queue` → `getQueue()` (L1008) |
| 60 | + - `GET /history` → `getHistory()` (L1024) |
| 61 | + - `GET /object_info` → `getNodeDefs()` (L842) |
| 62 | + - `POST /interrupt` → `interrupt()` (L1099) |
| 63 | + - `DELETE /queue/{id}` → `deleteItem()` (L1082) |
| 64 | + - `GET /system_stats`, `/embeddings`, `/extensions`, `/models/{folder}` |
| 65 | +- WebSocket events: `execution_start`, `executing`, `executed`, `execution_error`, `execution_success`, `progress`, `status`, `b_preview`, `logs` |
| 66 | + |
| 67 | +### Workflow serialization |
| 68 | +- `src/utils/executionUtil.ts:27` — `graphToPrompt(graph)` → `{ workflow, output }` |
| 69 | +- Schemas in `src/platform/workflow/validation/schemas/workflowSchema.ts` |
| 70 | +- Store: `src/platform/workflow/management/stores/workflowStore.ts` |
| 71 | + |
| 72 | +### Extension system |
| 73 | +- `src/types/comfy.ts` — `ComfyExtension` interface |
| 74 | +- `src/services/extensionService.ts` — lifecycle wiring |
| 75 | + |
| 76 | +--- |
| 77 | + |
| 78 | +## 2. Asset / file APIs → fs-like commands |
| 79 | + |
| 80 | +### Userdata (workflows, configs — owned by this user) |
| 81 | +File: `src/scripts/api.ts` |
| 82 | +- `getUserData(file)` — GET `/userdata/{file}` (L1173) |
| 83 | +- `storeUserData(file, data, opts)` — POST `/userdata/{file}` (L1184) |
| 84 | +- `deleteUserData(file)` — DELETE `/userdata/{file}` (L1220) |
| 85 | +- `moveUserData(src, dest, opts)` — POST `/userdata/{src}/move/{dest}` (L1232) |
| 86 | +- `listUserDataFullInfo(dir)` — GET `/userdata?dir=X&recurse=true&full_info=true` (L1246) |
| 87 | + |
| 88 | +Store: `src/stores/userFileStore.ts` — already builds `fileTree` from userdata. |
| 89 | + |
| 90 | +### Cloud assets (new API) |
| 91 | +Service: `src/platform/assets/services/assetService.ts` |
| 92 | +- `GET /assets` — list with tag filters |
| 93 | +- `POST /assets` — upload |
| 94 | +- `PUT /assets/{id}`, `DELETE /assets/{id}` |
| 95 | +- `POST /assets/{id}/tags`, `DELETE /assets/{id}/tags` |
| 96 | +- `POST /assets/download`, `POST /assets/export` |
| 97 | +- `GET /assets/remote-metadata` |
| 98 | + |
| 99 | +### Legacy OSS endpoints |
| 100 | +- `/upload/image`, `/upload/mask` |
| 101 | +- `/view` (image retrieval) |
| 102 | +- `/internal/files/input` |
| 103 | +- `/api/experiment/models`, `/api/experiment/models/{folder}` |
| 104 | +- `/api/view_metadata/{folder}?filename=X` |
| 105 | + |
| 106 | +### Models |
| 107 | +Store: `src/stores/modelStore.ts` — `ModelFolder`, `ComfyModelDef`, `loadModelFolders()`, `getLoadedModelFolder()`. |
| 108 | + |
| 109 | +### Outputs / history |
| 110 | +- `api.getHistory(max_items)` (L1024) |
| 111 | +- `api.getJobDetail(jobId)` (L1045) |
| 112 | +- `src/stores/assetsStore.ts:54` — `mapHistoryToAssets()` |
| 113 | + |
| 114 | +### Existing tree abstraction |
| 115 | +- `src/types/treeExplorerTypes.ts` — `TreeExplorerNode<T>` with rename/delete/addFolder/drop |
| 116 | +- `src/utils/treeUtil.ts:3` — `buildTree()` flat→tree |
| 117 | +- `UserFileStore.fileTree` already presents userdata as a tree |
| 118 | + |
| 119 | +--- |
| 120 | + |
| 121 | +## 3. Virtual filesystem layout |
| 122 | + |
| 123 | +Mount the heterogeneous APIs under a single POSIX-ish tree: |
| 124 | + |
| 125 | +``` |
| 126 | +/ |
| 127 | +├── workflows/ → userdata `workflows/` (list/get/store/move/delete) |
| 128 | +├── userdata/ → full userdata root |
| 129 | +├── models/ |
| 130 | +│ ├── checkpoints/ → modelStore folder |
| 131 | +│ ├── loras/ |
| 132 | +│ └── ... |
| 133 | +├── input/ → /internal/files/input + /upload/image (write) |
| 134 | +├── output/ → history outputs via mapHistoryToAssets |
| 135 | +├── assets/ → cloud /assets (tag-filtered views as subdirs) |
| 136 | +│ ├── by-tag/<tag>/ |
| 137 | +│ └── by-id/<id> |
| 138 | +├── queue/ → /queue (running + pending as pseudo-files) |
| 139 | +├── history/ → /history (each job a dir with outputs) |
| 140 | +├── graph/ → live LGraph (nodes/, links/, groups/) — read-only view |
| 141 | +└── tmp/ → in-memory scratch (Blob|string Map) |
| 142 | +``` |
| 143 | + |
| 144 | +`cwd` lives in shell state. Each mount has a small adapter implementing: |
| 145 | +```ts |
| 146 | +interface FsAdapter { |
| 147 | + list(path): Promise<Entry[]> |
| 148 | + stat(path): Promise<Entry> |
| 149 | + read(path): Promise<Blob | string> |
| 150 | + write(path, data): Promise<void> |
| 151 | + delete(path): Promise<void> |
| 152 | + move(src, dest): Promise<void> |
| 153 | +} |
| 154 | +``` |
| 155 | + |
| 156 | +### fs command → API mapping |
| 157 | + |
| 158 | +| Shell | Impl | |
| 159 | +|---|---| |
| 160 | +| `ls [path]` | `listUserDataFullInfo`, `assetService.list`, `modelStore.getLoadedModelFolder` depending on mount | |
| 161 | +| `cd <path>`, `pwd` | shell state only | |
| 162 | +| `cat <file>` | `getUserData` / fetch `/view` / `assets/{id}` blob | |
| 163 | +| `mv <src> <dst>` | `moveUserData` or assets PUT metadata | |
| 164 | +| `rm <path>` | `deleteUserData` or `DELETE /assets/{id}` or `deleteItem` on queue | |
| 165 | +| `cp <src> <dst>` | read → write (cross-mount allowed) | |
| 166 | +| `find <path> -name X` | recursive `listUserDataFullInfo` + glob | |
| 167 | +| `mkdir` | implicit via `storeUserData` with path | |
| 168 | +| `stat <path>` | mount-specific metadata | |
| 169 | +| `upload <local> <vfs-path>` | `/upload/image` or `POST /assets` | |
| 170 | +| `download <vfs-path>` | blob → trigger browser download | |
| 171 | + |
| 172 | +### Graph/queue commands (not fs) |
| 173 | + |
| 174 | +| Shell | Impl | |
| 175 | +|---|---| |
| 176 | +| `add-node <type> --pos x,y` | `useGraphNodeManager.addNode` | |
| 177 | +| `connect A:0 B:1` | `useGraphNodeManager.connect` | |
| 178 | +| `set-widget <id> <name> <val>` | `widgetValueStore.setWidgetValue` | |
| 179 | +| `rm-node <id>` | `graph.remove` | |
| 180 | +| `nodes [--grep X]` | enumerate `graph._nodes` | |
| 181 | +| `node-defs [pattern]` | `nodeDefStore` search | |
| 182 | +| `queue` | `api.queuePrompt(graphToPrompt(graph))` | |
| 183 | +| `queue-status` | `api.getQueue` | |
| 184 | +| `history [--last N]` | `api.getHistory` | |
| 185 | +| `interrupt` | `api.interrupt` | |
| 186 | +| `cmd <id> [args]` | `commandStore.execute` — exposes the entire `Comfy.*` command set | |
| 187 | +| `export [--format api\|workflow] > /tmp/wf.json` | `graphToPrompt` → vfs write | |
| 188 | +| `open <vfs-path>` | read workflow JSON → `workflowStore.openWorkflow` | |
| 189 | + |
| 190 | +--- |
| 191 | + |
| 192 | +## 4. Shell emulator (minimum viable) |
| 193 | + |
| 194 | +Zero real bash. Just enough sugar to make LLMs feel at home: |
| 195 | + |
| 196 | +- **Tokenizer**: split-respect-quotes, pipes `|`, sequencing `&&` / `;`, redirection `>` / `>>`, var-subst `$VAR` |
| 197 | +- **Variables**: `Map<string, any>` table; `VAR=value cmd` prefix |
| 198 | +- **Redirection**: writes go to vfs `/tmp/*` Blob/string map |
| 199 | +- **Coreutils**: `cat`, `ls`, `echo`, `grep`, `jq` (`jsonpath-plus`), `head`, `tail`, `wc`, `sort`, `uniq` |
| 200 | +- **Exit codes**: non-zero aborts `&&` chains |
| 201 | +- **History + `!!`**: nice-to-have |
| 202 | + |
| 203 | +Each yargs-defined command returns `{ stdout, stderr?, exitCode }`. Shell wires them up. |
| 204 | + |
| 205 | +--- |
| 206 | + |
| 207 | +## 5. ai-sdk integration |
| 208 | + |
| 209 | +Single `run_shell` tool — lets the LLM plan multi-step pipelines in one turn instead of N tool roundtrips. |
| 210 | + |
| 211 | +```ts |
| 212 | +import { streamText, tool } from 'ai' |
| 213 | +import { anthropic } from '@ai-sdk/anthropic' |
| 214 | +import { z } from 'zod' |
| 215 | + |
| 216 | +const shell = createShell() |
| 217 | + |
| 218 | +await streamText({ |
| 219 | + model: anthropic('claude-sonnet-4-6'), |
| 220 | + system: `You operate ComfyUI through a POSIX-like shell. |
| 221 | +Mounts: /workflows /models /input /output /assets /queue /history /graph /tmp |
| 222 | +Coreutils: ls cd pwd cat mv rm cp find stat echo grep jq head tail |
| 223 | +Graph: add-node connect set-widget rm-node nodes node-defs |
| 224 | +Execution: queue queue-status history interrupt cmd export open |
| 225 | +Run 'help' for the full list.`, |
| 226 | + tools: { |
| 227 | + run_shell: tool({ |
| 228 | + description: 'Run one or more shell commands. Supports | && ; > $VAR', |
| 229 | + inputSchema: z.object({ script: z.string() }), |
| 230 | + execute: async ({ script }) => shell.run(script) |
| 231 | + }) |
| 232 | + } |
| 233 | +}) |
| 234 | +``` |
| 235 | + |
| 236 | +--- |
| 237 | + |
| 238 | +## 6. Code layout |
| 239 | + |
| 240 | +``` |
| 241 | +src/agent/ |
| 242 | +├── shell/ |
| 243 | +│ ├── shell.ts # tokenizer, pipes, redirects, varsub |
| 244 | +│ ├── vfs/ |
| 245 | +│ │ ├── index.ts # mount table, path resolution |
| 246 | +│ │ ├── userdata.ts # /workflows, /userdata |
| 247 | +│ │ ├── assets.ts # /assets, /input, /output |
| 248 | +│ │ ├── models.ts # /models/* |
| 249 | +│ │ ├── queue.ts # /queue, /history |
| 250 | +│ │ ├── graph.ts # /graph (live view) |
| 251 | +│ │ └── tmp.ts # in-memory |
| 252 | +│ └── commands/ |
| 253 | +│ ├── coreutils.ts # ls cat mv rm cp find echo grep jq ... |
| 254 | +│ ├── graph.ts # add-node connect set-widget rm-node |
| 255 | +│ ├── queue.ts # queue queue-status interrupt |
| 256 | +│ ├── workflow.ts # open export import |
| 257 | +│ ├── search.ts # search-nodes search-models search-templates |
| 258 | +│ └── cmd.ts # bridge to commandStore |
| 259 | +├── llm/ |
| 260 | +│ ├── provider.ts |
| 261 | +│ └── session.ts # streamText loop |
| 262 | +└── ui/ |
| 263 | + ├── AgentPanel.vue |
| 264 | + └── TerminalView.vue |
| 265 | +``` |
| 266 | + |
| 267 | +--- |
| 268 | + |
| 269 | +## 7. Security posture |
| 270 | + |
| 271 | +- All commands flow through **existing frontend APIs** that the user already has access to. |
| 272 | +- Backend enforces: auth, rate limits, per-endpoint permissions, quotas. |
| 273 | +- No new endpoints. No bypass of validation. |
| 274 | +- The "bash" is a **UX projection** over operations the user could perform manually; LLM simply drives the same buttons. |
| 275 | +- Risky commands (`rm -rf /workflows`, bulk `queue`) should honor `ComfyCommand.confirmation` and support `--dry-run` and `--yes` flags. Default to a confirmation modal for destructive mutations. |
| 276 | + |
| 277 | +--- |
| 278 | + |
| 279 | +## 8. Guardrails |
| 280 | + |
| 281 | +- `streamText` `maxSteps` cap |
| 282 | +- Per-session command count limit |
| 283 | +- Detect runaway loops (same command >N times) |
| 284 | +- Respect `ComfyCommand.confirmation` |
| 285 | +- Optional "read-only" mode for exploration |
| 286 | +- Audit log of every executed script |
| 287 | + |
| 288 | +--- |
| 289 | + |
| 290 | +## 9. PoC scope (2 weeks) |
| 291 | + |
| 292 | +1. Shell runner + tokenizer + vfs core (5 mounts: tmp, workflows, models, input, output) |
| 293 | +2. Coreutils: `ls cat echo grep jq` |
| 294 | +3. Graph commands: `add-node connect set-widget queue queue-status` |
| 295 | +4. `cmd <id>` bridge → unlocks hundreds of existing commands for free |
| 296 | +5. Single `run_shell` tool wired to `ai-sdk` |
| 297 | +6. `AgentPanel.vue` chat + optional `TerminalView.vue` |
| 298 | +7. E2E demo: "make a 512×512 SDXL cat" → search-templates → open → set-widget → queue → `cat /output/latest` |
| 299 | + |
| 300 | +After PoC, expand vfs (assets/queue/history/graph mounts) and add `mv`/`rm`/`cp`/`find`/`upload`/`download`. |
0 commit comments