Add view_image tool so the model can open a local image or http(s) URL on demand, and drop the inline path/URL auto-detection from typed prompts

Author: alexylon

CHANGELOG.md

@@ -4,6 +4,14 @@ All notable changes to Sofos are documented in this file.
 
 ## [Unreleased]
 
+### Added
+
+- **New `view_image` tool lets the model open an image on demand.** Given a local image file path or an `http(s)://` URL, the tool attaches the image to the conversation so the model can describe it. For a folder of images, the model is told to call `list_directory` first and then `view_image` once per file. Supports JPEG, PNG, GIF, and WebP up to 20 MB per local file; URLs are passed through to the model provider, which fetches them on its side. External paths reuse the same Read-permission prompt as `read_file`, so granting access to a directory once covers both tools. Local images larger than 2048 pixels on the long side are downscaled proportionally before they reach the model so a 4K screenshot does not burn through the per-image token budget; smaller images are sent unchanged.
+
+### Changed
+
+- **Image references typed inline in a prompt are no longer auto-attached.** Previously, a path or URL with an image extension typed in a message would be detected, stripped from the text, and attached as an image content block. Two ergonomic problems followed: vague asks such as "look at the image in `assets/`" without a filename did nothing, and unrelated text that happened to end in `.png` could be misread as a path. Attaching an image now goes through the new `view_image` tool, which the model invokes after reading the prompt; clipboard paste (Ctrl-V) continues to attach images directly.
+
 ### Security
 
 - **Shell command and process substitution are now blocked in bash commands.** Constructs such as `echo $(rm bad)`, backtick substitution, and process substitution `<(cmd)` / `>(cmd)` previously slipped past the permission system because only the outer command name was checked. They are now refused before the command runs, with a clear message that names the marker. Single-quoted literals and arithmetic expansion `$((expr))` continue to work.

Cargo.lock

@@ -38,6 +38,7 @@ rand = "0.10"
 base64 = "0.22"
 arboard = "3"
 png = "0.18"
+image = { version = "0.25", default-features = false, features = ["png", "jpeg", "gif", "webp"] }
 unicode-width = "0.2"
 
 # Utilities

Cargo.toml

@@ -62,7 +62,7 @@ Sofos provides an AI assistant inside your terminal with controlled access to yo
 - create, move, copy, and delete files with permission checks;
 - run safe build and test commands;
 - fetch documentation and use provider-native web search;
-- review local, clipboard, or web images;
+- open local image files or remote image URLs through the `view_image` tool, and accept clipboard pastes directly;
 - update a visible task plan during multi-step work;
 - save and resume conversations;
 - connect to external tools through Model Context Protocol servers.
@@ -81,7 +81,7 @@ The assistant can act through tools, but it does not do so silently: tool calls
 - **Strong permission model** — independent Read, Write, and Bash grants for paths outside the workspace.
 - **Bash safety** — allowed, denied, and ask tiers, plus structural checks for parent traversal, redirection, and dangerous git operations.
 - **Safe mode** — read-only native tools for review-only sessions.
-- **Image vision** — local images, web images, and clipboard paste.
+- **Image vision** — `view_image` tool for local files and remote URLs, plus clipboard paste.
 - **MCP integration** — connect additional tool servers through stdio or streamable HTTP.
 - **Session persistence** — saved conversations, resume picker, restored safe mode, restored model where compatible, and persisted cost counters.
 - **Cost visibility** — token totals, cache hit reporting, and provider-specific price estimates.
@@ -214,21 +214,24 @@ sofos -p "Create a high-level summary of this crate" --safe-mode
 
 ### Image vision
 
-Include image paths or URLs directly in your message, or paste images from the clipboard.
+Ask about an image by referring to it in your message. The model calls the `view_image` tool to open the file or URL you mention.
 
 ```text
 What is wrong in ./screenshots/error.png?
-Describe "./docs/architecture diagram.webp".
+Describe ./docs/architecture-diagram.webp.
 Review https://example.com/chart.png
+What do you see in the images in ./assets/?
 ```
 
+For a folder, the model lists the directory first and then opens each image one by one.
+
 Clipboard paste:
 
 ```text
 Ctrl+V    # Inserts a numbered marker such as ①.
 ```
 
-Supported formats: JPEG, PNG, GIF, and WebP. Local images are capped at 20 MB. Paths with spaces should be quoted. Images outside the workspace require Read permission.
+Supported formats: JPEG, PNG, GIF, and WebP. Local images are capped at 20 MB. Images larger than 2048 pixels on the long side are scaled down proportionally before being sent to the model, so a 4K screenshot does not balloon your token budget. Images outside the workspace require Read permission the first time, just like reading a file.
 
 ---
 
@@ -311,10 +314,11 @@ Provider mapping:
 | `delete_directory` | Delete a directory after confirmation. External paths require Write permission. |
 | `execute_bash` | Run approved shell commands through the bash permission system. |
 | `update_plan` | Show the current multi-step task plan with `pending`, `in_progress`, and `completed` statuses. |
+| `view_image` | Attach a local image file or an `http(s)://` URL to the conversation so the model can see it. |
 | `web_fetch` | Fetch a URL and return readable text. |
 | `web_search` | Provider-native web search. |
 
-Image vision is not a tool. Sofos detects supported image paths and URLs in user messages and converts them into image content blocks before sending the request.
+Clipboard pastes are not routed through a tool: pressing Ctrl-V in the prompt attaches the image directly to the message.
 
 ### Safe mode tools
 
@@ -325,6 +329,7 @@ Safe mode is enabled with `--safe-mode` or `/s`. It restricts the native tool se
 - `glob_files`;
 - `search_code` when ripgrep is installed;
 - `update_plan`;
+- `view_image`;
 - `web_fetch`;
 - `web_search`.
 
@@ -574,7 +579,7 @@ See [`RELEASE.md`](RELEASE.md) for the full process.
 | Path denied | Add a `Read`, `Write`, or `Bash` rule, or approve the interactive prompt. |
 | External edit denied | `edit_file` and `morph_edit_file` need both Read and Write for external files. |
 | Code search unavailable | Install `ripgrep` and ensure `rg` is on `PATH`. |
-| Image path with spaces fails | Quote the path: `"path/with spaces/image.png"`. |
+| Image not opening | Mention the image by path or URL in your message; the model will call `view_image`. For a folder, ask it to look in the folder and it will list and open each image. |
 | Terminal does not insert newline with Shift+Enter | Use Alt+Enter or Ctrl+Enter. |
 | Build problems | Run `rustup update`, then `cargo clean` and `cargo build`. |
 

README.md

@@ -275,7 +275,7 @@ src/
 │   ├── codesearch.rs
 │   │   # Ripgrep-backed code search with ignore policy, file-type filters, and output limits.
 │   ├── image.rs
-│   │   # Local and web image detection, validation, encoding, and message-content conversion.
+│   │   # Image loader used by the `view_image` tool: format detection, 20 MB size cap, automatic resize to 2048 pixels on the long side, base64 encoding, and Read-permission integration.
 │   ├── morph_validate.rs
 │   │   # Safety checks that reject suspicious or truncated Morph Apply output before writing files.
 │   ├── plan.rs
@@ -449,7 +449,7 @@ It contains:
 - image size enforcement;
 - numbered marker handling used by the TUI input flow.
 
-It does not own general image path loading. Local and web image detection for user messages lives in `tools/image.rs`.
+It does not own image loading from disk. Local and remote image loading for the `view_image` tool lives in `tools/image.rs`.
 
 ---
 
@@ -995,23 +995,23 @@ Rules:
 
 ### 7.8 `tools/image.rs`
 
-`tools/image.rs` owns image detection and loading for user messages.
+`tools/image.rs` owns the image loader behind the `view_image` tool.
 
 It contains:
 
-- local image path parsing;
-- web image URL detection;
-- supported-format checks;
-- base64 encoding;
-- media-type assignment;
-- size enforcement;
-- integration with workspace and external Read permissions.
+- decode plus optional resize (long side fits within 2048 pixels) before the bytes reach the model;
+- byte-level format detection: PNG, JPEG, GIF, and WebP pass through unchanged when small enough; other decodable formats (e.g. BMP) are re-encoded as PNG;
+- base64 encoding and media-type assignment;
+- the 20 MB per-file size cap on the raw bytes;
+- canonical workspace resolution so inside/outside classification compares the same path shape on both sides;
+- integration with the shared Read-permission grant set, so a single "Allow Read access to /foo?" decision answered for `read_file` also covers `view_image`;
+- a URL passthrough that hands `http(s)://` inputs to the model provider unchanged.
 
 Rules:
 
-- Image paths in user text become image content blocks before provider requests.
-- Unsupported or oversized images should produce clear errors.
-- Failed web-image loading can be retried without discarding the user's text.
+- Local files outside the workspace go through the same interactive Read prompt as `read_file`.
+- Files that fail to decode or exceed the size cap produce errors that name the cause.
+- The loader never fetches remote URLs itself; the model provider does that on its side.
 
 ### 7.9 `tools/morph_validate.rs`
 
@@ -1502,7 +1502,7 @@ Rules:
 | Permission settings and prompts | `tools/permissions/manager.rs` |
 | Permission rule parsing | `tools/permissions/pattern.rs` |
 | Code search | `tools/codesearch.rs` |
-| User-message image loading | `tools/image.rs` |
+| `view_image` tool image loading | `tools/image.rs` |
 | Morph output validation | `tools/morph_validate.rs` |
 | MCP configuration | `mcp/config.rs` |
 | MCP protocol shapes | `mcp/protocol.rs` |

STRUCTURE.md

@@ -21,10 +21,22 @@ pub struct ToolResult {
     pub images: Vec<ImageData>,
 }
 
+/// Image attachment in a tool result. `Url` is fetched by the model
+/// provider; `Base64` is shipped inline.
 #[derive(Debug, Clone)]
-pub struct ImageData {
-    pub mime_type: String,
-    pub base64_data: String,
+pub enum ImageData {
+    Base64 { mime_type: String, data: String },
+    Url { url: String },
+}
+
+impl ImageData {
+    /// Bytes we'd ship inline; 0 for URLs since the provider fetches them.
+    pub fn outbound_size(&self) -> usize {
+        match self {
+            ImageData::Base64 { data, .. } => data.len(),
+            ImageData::Url { .. } => 0,
+        }
+    }
 }
 
 /// Manages multiple MCP server connections and their tools.
@@ -286,12 +298,9 @@ fn format_tool_result(result: CallToolResult) -> ToolResult {
                 text_output.push('\n');
             }
             ToolContent::Image { data, mime_type } => {
-                let size_kb = (data.len() * 3 / 4) / 1024;
+                let size_kb = crate::tools::utils::base64_approx_decoded_kb(data.len());
                 text_output.push_str(&format!("[Image: {} ({} KB)]\n", mime_type, size_kb));
-                images.push(ImageData {
-                    mime_type,
-                    base64_data: data,
-                });
+                images.push(ImageData::Base64 { mime_type, data });
             }
             ToolContent::Resource {
                 uri,

src/mcp/manager.rs

@@ -19,7 +19,6 @@ use crate::error::{Result, SofosError};
 use crate::mcp::McpManager;
 use crate::session::{HistoryManager, SessionState};
 use crate::tools::ToolExecutor;
-use crate::tools::image::ImageLoader;
 use crate::ui::{UI, set_safe_mode_cursor_style};
 use colored::Colorize;
 use std::path::PathBuf;
@@ -61,7 +60,6 @@ pub struct Repl {
     pub(super) client: LlmClient,
     pub(super) tool_executor: ToolExecutor,
     pub(super) history_manager: HistoryManager,
-    pub(super) image_loader: ImageLoader,
     pub(super) ui: UI,
     pub(super) model_config: ModelConfig,
     pub(super) session_state: SessionState,
@@ -129,12 +127,6 @@ impl Repl {
         let has_code_search = tool_executor.has_code_search();
 
         let history_manager = HistoryManager::new(workspace.clone())?;
-        let mut image_loader = ImageLoader::new(workspace.clone())?;
-        image_loader.install_read_path_session(
-            std::io::stdin().is_terminal(),
-            tool_executor.read_path_session_allowed(),
-            tool_executor.read_path_session_denied(),
-        );
 
         // Load custom instructions
         let custom_instructions = history_manager.load_custom_instructions()?;
@@ -214,7 +206,6 @@ impl Repl {
             client,
             tool_executor,
             history_manager,
-            image_loader,
             ui,
             model_config,
             session_state,

src/repl/mod.rs

@@ -444,11 +444,19 @@ impl ResponseHandler {
                     });
 
                     for image in output.images() {
+                        let source = match image {
+                            crate::mcp::manager::ImageData::Base64 { mime_type, data } => {
+                                crate::api::ImageSource::Base64 {
+                                    media_type: mime_type.clone(),
+                                    data: data.clone(),
+                                }
+                            }
+                            crate::mcp::manager::ImageData::Url { url } => {
+                                crate::api::ImageSource::Url { url: url.clone() }
+                            }
+                        };
                         tool_results.push(crate::api::MessageContentBlock::Image {
-                            source: crate::api::ImageSource::Base64 {
-                                media_type: image.mime_type.clone(),
-                                data: image.base64_data.clone(),
-                            },
+                            source,
                             cache_control: None,
                         });
                     }