Skip to content

fix : image input gating by model capability#5272

Merged
SivanCola merged 5 commits into
esengine:main-v2from
onenewcode:fix/5175-image-input-text-only-model
Jun 26, 2026
Merged

fix : image input gating by model capability#5272
SivanCola merged 5 commits into
esengine:main-v2from
onenewcode:fix/5175-image-input-text-only-model

Conversation

@onenewcode

@onenewcode onenewcode commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Fixes #5175

Summary

  • add a backend-driven image-input capability check to desktop, CLI, and controller paths so native image payloads follow the resolved model entry instead of the old implicit-allow behavior
  • expose imageInputEnabled through desktop tab meta and refresh it after model switches so the composer can warn when the selected model will not receive images directly
  • stop sending native image payloads from the controller for text-only models by gating inputImages(...) on the resolved provider capability, while preserving normal @ref text handling
  • keep pasted and dropped images usable as local refs: text-only models can still work with image attachments through OCR/image/vision MCP tools that read the attached path
  • keep vision-capable models on the direct image-input path, including pasted attachments and workspace image refs
  • update CLI behavior so pasted image refs are not blocked for text-only models; they remain available as tool-readable references while native image payloads stay suppressed
  • add regression coverage for desktop meta propagation, composer image-ref behavior, controller image payload gating, ResolveRefs guidance, and CLI pasted-image behavior
  • restyle warning toasts with theme tokens so the unsupported-direct-image feedback remains readable across desktop themes without affecting composer layout

Behavior

For a text-only model:

  • Reasonix does not send image_url / native image content blocks to the provider.
  • Pasted or dropped images remain attached as local @.reasonix/attachments/... refs.
  • Sending with image attachments is allowed, with a non-blocking desktop warning that the model will not receive images directly.
  • If an OCR/image/vision MCP tool is available, the model can inspect the local attachment path through that tool.

For a vision-capable model:

  • Image attachments and workspace image refs continue to be sent as direct model image input.

Verification

Automated checks run during review:

  • go test ./internal/control ./internal/cli -run ... -count=1
  • go test . -run ... -count=1 from desktop/
  • pnpm --dir desktop/frontend exec tsx src/__tests__/composer-image-capability.test.tsx
  • pnpm --dir desktop/frontend exec tsx src/__tests__/use-controller-meta.test.ts
  • git diff --check

Manual verification guidance:

  1. Configure one text-only model and one vision-capable model for the same workspace; for custom providers, explicitly set vision = true or vision_models = ["..."] on the vision model.
  2. Start the desktop app with cd desktop && wails dev.
  3. Switch to the text-only model, then paste or drag a PNG into the composer.
    • Expect: the image attachment remains in the draft as a local ref.
  4. Send the draft while still on the text-only model.
    • Expect: the turn is allowed, the desktop warning says the model will not receive images directly, and no native image payload is sent to the provider.
  5. With an OCR/image/vision MCP tool available, ask the model to inspect the attached image path.
    • Expect: the tool-readable local ref is available for that workflow.
  6. Switch to the vision-capable model and send the same image attachment again.
    • Expect: the image is sent through the direct model image-input path.
  7. Repeat the text-only and vision-capable flows in the CLI with a pasted image ref.

Local style snapshots

cli
截屏2026-06-25 09 49 16

ui
截屏2026-06-25 11 03 38

Cache impact

Cache-impact: none - image-input gating and toast rendering do not change provider-visible stable prompts, tool schemas, cache keys, or transcript compaction paths.
Cache-guard: focused desktop/CLI image-capability regression coverage plus manual end-to-end validation.
System-prompt-review: N/A

@github-actions github-actions Bot added v2 Go rewrite (1.x) — main-v2 branch, active development desktop Wails desktop app (desktop/**) tui Terminal UI / CLI (internal/cli, internal/control) agent Core agent loop (internal/agent, internal/control) and removed v2 Go rewrite (1.x) — main-v2 branch, active development labels Jun 25, 2026
@onenewcode onenewcode changed the title [codex] gate image input by model capability [codex] fix #5175 image input gating by model capability Jun 25, 2026
@onenewcode onenewcode marked this pull request as ready for review June 25, 2026 03:08
@onenewcode onenewcode changed the title [codex] fix #5175 image input gating by model capability fix #5175 image input gating by model capability Jun 25, 2026
@onenewcode onenewcode changed the title fix #5175 image input gating by model capability fix : image input gating by model capability Jun 25, 2026
@SivanCola

Copy link
Copy Markdown
Collaborator

Thanks, this looks directionally good, but the branch needs to be rebased before review can continue.

I checked the PR against the latest main-v2: it is currently conflicting, with the visible conflict in desktop/frontend/package.json around the frontend test script list. Please rebase onto the latest main-v2, keep the current upstream test entries, and add the new composer-image-capability.test.tsx coverage back into the relevant frontend test scripts.

After resolving that conflict locally, the focused checks I ran against the rebased state passed:

  • go test ./internal/control ./internal/cli ... for the image-input gating tests
  • go test . from desktop/ for the meta capability tests
  • pnpm --dir desktop/frontend exec tsx src/__tests__/composer-image-capability.test.tsx
  • pnpm --dir desktop/frontend exec tsx src/__tests__/use-controller-meta.test.ts

Once the branch is rebased and CI stays green, I will continue the final review.

@SivanCola SivanCola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the merge conflict by merging latest main-v2 into this branch and resolving desktop/frontend/package.json so the current upstream frontend test list is preserved while keeping composer-image-capability.test.tsx in both frontend test scripts.

Review notes:

  • The image input gating is applied at the UI/controller boundary for text-only models, while drafts remain recoverable.
  • The control path still drops image payloads for text-only models, so a frontend miss does not send image input.
  • This is dynamic image capability behavior only; I did not find a stable prompt-prefix or tool-schema cache risk.

Verified:

  • go test ./internal/control ./internal/cli -run ... -count=1
  • go test . -run ... -count=1 from desktop/
  • pnpm --dir desktop/frontend exec tsx src/__tests__/composer-image-capability.test.tsx
  • pnpm --dir desktop/frontend exec tsx src/__tests__/use-controller-meta.test.ts

@SivanCola SivanCola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up pushed in 7da9b0a.

This keeps the original provider-native image guard while avoiding a product regression: text-only models still do not receive direct image bytes, but pasted/dropped images remain usable as local refs so OCR/image/vision MCP tools can inspect them.

What changed:

  • The desktop composer no longer rejects image attach/save/send solely because the selected model lacks native image input. It warns non-blockingly when sending image attachments with a text-only model.
  • The CLI no longer blocks pasted image refs for text-only models.
  • ResolveRefs now describes both paths clearly: direct model image input for vision-capable models, and tool-readable local paths for text-only models.
  • Regression tests now cover the text-only + tool-readable image path while keeping the controller guard that suppresses native image payloads for text-only models.

Verified:

  • go test ./internal/control ./internal/cli -run ... -count=1 passed
  • pnpm --dir desktop/frontend exec tsx src/__tests__/composer-image-capability.test.tsx passed
  • pnpm --dir desktop/frontend exec tsx src/__tests__/use-controller-meta.test.ts passed
  • git diff --check passed

@SivanCola SivanCola disabled auto-merge June 26, 2026 19:54
@SivanCola SivanCola enabled auto-merge June 26, 2026 19:54

@SivanCola SivanCola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approved after merging the latest main-v2 into this branch in 6b788b4.

The new upstream overlap was in the CLI paste tests from #5347. The conflict resolution keeps both contracts:

  • folded paste content is expanded for the turn and memory-compiler source event
  • text-only models can still send pasted image refs for OCR/image/vision tool use while native image payloads stay gated by model capability

Verified after the merge:

  • go test ./internal/cli -run ... -count=1 passed
  • go test ./internal/control -run ... -count=1 passed
  • go test . -run ... -count=1 from desktop/ passed
  • pnpm --dir desktop/frontend exec tsx src/__tests__/composer-image-capability.test.tsx passed
  • pnpm --dir desktop/frontend exec tsx src/__tests__/use-controller-meta.test.ts passed
  • git diff --check passed

@SivanCola SivanCola merged commit 615544e into esengine:main-v2 Jun 26, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control) desktop Wails desktop app (desktop/**) tui Terminal UI / CLI (internal/cli, internal/control) v2 Go rewrite (1.x) — main-v2 branch, active development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] unknown variant image_url, expected text

2 participants