fix(vscode-lm): enable image support for VS Code LM models

Author: CodeKing

docs/decisions/ADR-005-vscode-lm-image-capability.md

@@ -0,0 +1,46 @@
+# ADR-005: VS Code LM Image Capability Detection
+
+## Status
+
+Accepted (2026-06-27)
+
+## Context
+
+Zoo Code supports the VS Code Language Model API provider. VS Code LM models can expose custom endpoints and provider-specific metadata. Some of these models support image input in native VS Code Chat, but Zoo Code can treat them as text-only.
+
+The current capability mapper in `src/api/providers/vscode-lm.ts` only recognizes:
+
+- `model.capabilities.imageInput`
+- `model.capabilities.vision`
+- static family metadata in `vscodeLlmModels`
+
+When none of those are present, it sets `supportsImages` to `false`. For image-capable custom endpoint models, this false value propagates into task request generation, where image blocks are removed before the VS Code LM request is built.
+
+## Decision
+
+1. Broaden VS Code LM capability detection to recognize additional image/vision metadata shapes exposed by VS Code LM providers.
+2. Keep explicit negative capability metadata authoritative when a model clearly reports image support is disabled.
+3. Use safe fallback behavior for unknown custom VS Code LM models so Zoo Code does not strip images before the VS Code LM API can accept or reject them.
+4. Preserve full VS Code LM model identity in the webview model picker by using `id` when available instead of only `${vendor}/${family}`.
+5. Add regression coverage for:
+    - provider image capability mapping,
+    - image blocks reaching `LanguageModelDataPart` in the final VS Code LM request,
+    - selected model capability propagation in the webview.
+
+## Consequences
+
+### Positive
+
+- Image-capable VS Code LM custom endpoint models are no longer downgraded to text-only inside Zoo Code.
+- Images survive the task request pipeline and can be converted to `LanguageModelDataPart`.
+- Model selection is less prone to collisions when multiple VS Code LM models share vendor/family.
+
+### Negative
+
+- Unknown VS Code LM custom models may show image UI even if the provider later rejects images. This is preferable to silently dropping user images before the API request because VS Code LM is the source of truth for request acceptance.
+
+## Alternatives Considered
+
+1. **Only support `capabilities.imageInput` and `capabilities.vision`**: rejected because it fails for custom endpoints that VS Code Chat can use with images.
+2. **Always disable images for unknown VS Code LM models**: rejected because it causes silent image removal and contradicts VS Code Chat behavior.
+3. **Bypass `maybeRemoveImageBlocks()` for VS Code LM only**: rejected because capability propagation is used by UI, tools, and prompts; fixing only the final request would leave inconsistent behavior.

docs/research/vscode-lm-image-support.md

@@ -0,0 +1,70 @@
+# Research: VS Code LM Image Support
+
+## Status
+
+Completed initial pipeline trace on 2026-06-27.
+
+## Problem
+
+VS Code LM models that accept images in native VS Code Chat can behave as text-only in Zoo Code. The reported example is `customendpoint/gpt-5.5`.
+
+## Pipeline traced
+
+1. **Model discovery**
+
+    - `src/api/providers/vscode-lm.ts` calls `vscode.lm.selectChatModels({})` in `getVsCodeLmModels()` and maps each model through `getVsCodeLmModelInfo()`.
+    - `getVsCodeLmModelInfo()` currently only checks `model.capabilities.imageInput`, `model.capabilities.vision`, static family metadata, then defaults to `false`.
+
+2. **Selected model state**
+
+    - `webview-ui/src/components/settings/providers/VSCodeLM.tsx` receives `vsCodeLmModels` from the extension and stores the selected model object, including `info`, through `vsCodeLmModelSelector`.
+    - The settings UI currently keys VS Code LM models by `${vendor}/${family}`. That can collide for custom endpoints or multiple models with the same vendor/family but different `id` or `version`.
+    - `webview-ui/src/components/ui/hooks/useSelectedModel.ts` reads `vsCodeLmModelSelector.info` and exposes model capabilities to chat UI.
+
+3. **UI gating**
+
+    - `webview-ui/src/components/chat/ChatView.tsx` computes `shouldDisableImages = !model?.supportsImages || selectedImages.length >= MAX_IMAGES_PER_MESSAGE`.
+    - `webview-ui/src/components/chat/ChatTextArea.tsx` disables paste/drop/select image interactions when `shouldDisableImages` is true.
+    - `webview-ui/src/components/chat/ChatRow.tsx` also disables image attachment while editing when `!model?.supportsImages`.
+
+4. **Image ingestion**
+
+    - `src/core/webview/webviewMessageHandler.ts` resolves incoming images through `resolveImageMentions()`.
+    - `resolveImageMentions()` defaults `supportsImages` to `true`; the current webview handler does not pass the current provider capability. This means images are not stripped at this step because of VS Code LM capability detection.
+
+5. **Task request generation**
+
+    - `src/core/task/Task.ts` merges API history and then calls `maybeRemoveImageBlocks(mergedForApi, this.api)` before `buildCleanConversationHistory()` and `this.api.createMessage()`.
+    - `src/api/transform/image-cleaning.ts` converts image blocks into text placeholders when `apiHandler.getModel().info.supportsImages` is false.
+
+6. **VS Code LM request transform**
+    - `src/api/transform/vscode-lm-format.ts` converts Anthropic image blocks to `vscode.LanguageModelDataPart` when that constructor is available.
+    - Existing tests in `src/api/transform/__tests__/vscode-lm-format.spec.ts` already prove image blocks become data parts when they survive to this transform.
+
+## Root cause
+
+The root cause is capability detection and preservation for VS Code LM custom models. `getVsCodeLmModelInfo()` is too narrow: it treats a VS Code LM model as image-capable only when `capabilities.imageInput` or `capabilities.vision` is present, or when static family metadata says images are supported. Custom endpoint models can be image-capable in VS Code Chat without matching these fields/static families, so Zoo Code records `supportsImages: false`.
+
+Once `supportsImages` is false, `Task.attemptApiRequest()` calls `maybeRemoveImageBlocks()`, which replaces image blocks with `[Referenced image in conversation]`. Therefore `convertToVsCodeLmMessages()` never gets an image block and cannot create `LanguageModelDataPart` for the final `client.sendRequest()` call.
+
+A secondary issue is model identity in the settings UI. `VSCodeLM.tsx` uses `${vendor}/${family}` as the picker key, ignoring `id` and `version`. This can preserve or select the wrong `info` when multiple VS Code LM models share vendor/family.
+
+## Responsible files
+
+- `src/api/providers/vscode-lm.ts`
+- `src/api/transform/image-cleaning.ts`
+- `src/core/task/Task.ts`
+- `src/api/transform/vscode-lm-format.ts`
+- `webview-ui/src/components/settings/providers/VSCodeLM.tsx`
+- `webview-ui/src/components/ui/hooks/useSelectedModel.ts`
+- `webview-ui/src/components/chat/ChatView.tsx`
+- `webview-ui/src/components/chat/ChatTextArea.tsx`
+- `webview-ui/src/components/chat/ChatRow.tsx`
+
+## Planned verification
+
+- Add provider tests for broader VS Code LM image-capability shapes and fallback behavior for custom endpoints.
+- Add a provider request test proving Anthropic image blocks reach `sendRequest()` as `LanguageModelDataPart`.
+- Add settings hook/UI tests for preserving `info.supportsImages` and distinct model identity.
+- Run targeted Vitest suites from the correct package directories.
+- Runtime verification with a real VS Code LM image-capable model requires an interactive VS Code extension host and an installed/authenticated model provider. If unavailable in this environment, document exact manual steps and do not claim completed runtime verification.

packages/types/src/provider-settings.ts

@@ -276,6 +276,7 @@ const vsCodeLmSchema = baseProviderSettingsSchema.extend({
 			family: z.string().optional(),
 			version: z.string().optional(),
 			id: z.string().optional(),
+			info: modelInfoSchema.optional(),
 		})
 		.optional(),
 })

packages/types/src/vscode-extension-host.ts

@@ -12,7 +12,7 @@ import type { CloudUserInfo, CloudOrganizationMembership, OrganizationAllowList,
 import type { SerializedCustomToolDefinition } from "./custom-tool.js"
 import type { GitCommit } from "./git.js"
 import type { McpServer } from "./mcp.js"
-import type { ModelRecord, RouterModels } from "./model.js"
+import type { ModelInfo, ModelRecord, RouterModels } from "./model.js"
 import type { OpenAiCodexRateLimitInfo } from "./providers/openai-codex-rate-limits.js"
 import type { SkillMetadata } from "./skills.js"
 import type { TelemetrySetting } from "./telemetry.js"
@@ -136,7 +136,7 @@ export interface ExtensionMessage {
 	openAiModels?: string[]
 	ollamaModels?: ModelRecord
 	lmStudioModels?: ModelRecord
-	vsCodeLmModels?: { vendor?: string; family?: string; version?: string; id?: string }[]
+	vsCodeLmModels?: LanguageModelChatSelector[]
 	mcpServers?: McpServer[]
 	commits?: GitCommit[]
 	listApiConfig?: ProviderSettingsEntry[]
@@ -782,6 +782,7 @@ export interface LanguageModelChatSelector {
 	family?: string
 	version?: string
 	id?: string
+	info?: ModelInfo
 }
 
 export interface ClineSayTool {

src/api/providers/__tests__/vscode-lm.spec.ts

@@ -7,6 +7,14 @@ vi.mock("vscode", () => {
 		constructor(public value: string) {}
 	}
 
+	class MockLanguageModelDataPart {
+		type = "data"
+		constructor(
+			public data: Uint8Array,
+			public mimeType: string,
+		) {}
+	}
+
 	class MockLanguageModelToolCallPart {
 		type = "tool_call"
 		constructor(
@@ -52,6 +60,7 @@ vi.mock("vscode", () => {
 			})),
 		},
 		LanguageModelTextPart: MockLanguageModelTextPart,
+		LanguageModelDataPart: MockLanguageModelDataPart,
 		LanguageModelToolCallPart: MockLanguageModelToolCallPart,
 		lm: {
 			selectChatModels: vi.fn(),
@@ -383,6 +392,44 @@ describe("VsCodeLmHandler", () => {
 			)
 		})
 
+		it("should pass image blocks to VS Code LM as data parts", async () => {
+			const systemPrompt = "You are a helpful assistant"
+			const imageData = Buffer.from("image-data").toString("base64")
+			const messages: Anthropic.Messages.MessageParam[] = [
+				{
+					role: "user" as const,
+					content: [
+						{ type: "text", text: "Describe this image" },
+						{ type: "image", source: { type: "base64", media_type: "image/png", data: imageData } },
+					],
+				},
+			]
+
+			mockLanguageModelChat.sendRequest.mockResolvedValueOnce({
+				stream: (async function* () {
+					yield new vscode.LanguageModelTextPart("Image description")
+					return
+				})(),
+				text: (async function* () {
+					yield "Image description"
+					return
+				})(),
+			})
+
+			const stream = handler.createMessage(systemPrompt, messages)
+			for await (const _chunk of stream) {
+				// Drain stream so sendRequest is invoked.
+			}
+
+			const requestMessages = mockLanguageModelChat.sendRequest.mock.calls[0][0]
+			const userMessage = requestMessages[1]
+			const imagePart = userMessage.content[1]
+
+			expect(imagePart.type).toBe("data")
+			expect(imagePart.mimeType).toBe("image/png")
+			expect(Buffer.from(imagePart.data).toString()).toBe("image-data")
+		})
+
 		it("should handle errors", async () => {
 			const systemPrompt = "You are a helpful assistant"
 			const messages: Anthropic.Messages.MessageParam[] = [
@@ -413,6 +460,56 @@ describe("VsCodeLmHandler", () => {
 			expect(model.info.contextWindow).toBe(4096)
 		})
 
+		it("should mark VS Code LM models with imageInput capability as supporting images", async () => {
+			const mockModel = { ...mockLanguageModelChat, capabilities: { imageInput: true } }
+			;(vscode.lm.selectChatModels as Mock).mockResolvedValue([mockModel])
+			handler["client"] = null
+			await handler.initializeClient()
+
+			const model = handler.getModel()
+			expect(model.info.supportsImages).toBe(true)
+		})
+
+		it("should mark VS Code LM models with supportsImages capability as supporting images", async () => {
+			const mockModel = { ...mockLanguageModelChat, capabilities: { supportsImages: true } }
+			;(vscode.lm.selectChatModels as Mock).mockResolvedValue([mockModel])
+			handler["client"] = null
+			await handler.initializeClient()
+
+			const model = handler.getModel()
+			expect(model.info.supportsImages).toBe(true)
+		})
+
+		it("should mark VS Code LM models with image modality as supporting images", async () => {
+			const mockModel = { ...mockLanguageModelChat, capabilities: { inputModalities: ["text", "image"] } }
+			;(vscode.lm.selectChatModels as Mock).mockResolvedValue([mockModel])
+			handler["client"] = null
+			await handler.initializeClient()
+
+			const model = handler.getModel()
+			expect(model.info.supportsImages).toBe(true)
+		})
+
+		it("should preserve explicit false image capability", async () => {
+			const mockModel = { ...mockLanguageModelChat, capabilities: { imageInput: false } }
+			;(vscode.lm.selectChatModels as Mock).mockResolvedValue([mockModel])
+			handler["client"] = null
+			await handler.initializeClient()
+
+			const model = handler.getModel()
+			expect(model.info.supportsImages).toBe(false)
+		})
+
+		it("should allow images for unknown custom VS Code LM models by default", async () => {
+			const mockModel = { ...mockLanguageModelChat, family: "unknown-custom-family" }
+			;(vscode.lm.selectChatModels as Mock).mockResolvedValue([mockModel])
+			handler["client"] = null
+			await handler.initializeClient()
+
+			const model = handler.getModel()
+			expect(model.info.supportsImages).toBe(true)
+		})
+
 		it("should return fallback model info when no client exists", () => {
 			// Clear the client first
 			handler["client"] = null