Skip to content

fix(vision): mark auto-generated image descriptions as system-provided#5574

Open
Ninthkitten wants to merge 1 commit into
odysseus-dev:devfrom
Ninthkitten:pr/label-vision-description
Open

fix(vision): mark auto-generated image descriptions as system-provided#5574
Ninthkitten wants to merge 1 commit into
odysseus-dev:devfrom
Ninthkitten:pr/label-vision-description

Conversation

@Ninthkitten

Copy link
Copy Markdown

Summary

The central vision model's image description is injected into the user's message bare, so chat models mistake it for the user's own writing and respond to the prose instead of the image. This PR labels the injected description as auto-generated (naming the vision model) so the chat model knows who is speaking. The label matters even when the same model serves both vision and chat: the description is produced in a separate call that the chat turn has no memory of, so the model still reads its own words as the user's.

Target branch

  • This PR targets dev, not main.

Linked Issue

Fixes #5573

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)

Checklist

  • I searched open issues and open PRs — this is not a duplicate.
  • This PR targets dev
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I actually ran the app (docker compose up) and verified the change works end-to-end. Type-checks and unit tests are not enough.

How to Test

  1. Set a vision model (Settings → AI Defaults) and upload an image in a chat driven by any text-only chat model.
  2. Without this patch, the description is injected as if the user wrote it — models often comment on "your" writing instead of answering about the image.
  3. With this patch, the injected block reads "[Attached image: … — description below was auto-generated by , not written by the user]" and the model answers about the image.
  4. Verify the collapsible "image description" on the user bubble still renders as before.

This fix has been running in a self-hosted deployment since 13 July.

🤖 Generated with Claude Code

The vision model's description is injected as user text, so chat models compliment the user's prose instead of answering about the image. Label the description as auto-generated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the ready for review Description complete — ready for maintainer review label Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vision description is injected as user-authored text — chat models respond to the user's prose instead of the image

1 participant