diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/README.md b/docs/design/agent-workflows/projects/agent-multi-modality/README.md new file mode 100644 index 0000000000..1fb773ce41 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/README.md @@ -0,0 +1,108 @@ +# Agent multi-modality + +This project lets a person share files with an agent and have two things happen at once: +the model reads the file, and the agent can work on the file with its tools. Today neither +happens. A person can attach an image or a document in the agent chat box, but the file is +thrown away just before it reaches the model, so the model never sees it and the agent never +gets it. + +A second goal sits beside the first. Every file a person shares must stay easy to find later, +even after the agent has changed things. If you gave the agent a spreadsheet an hour ago, you +should be able to open exactly that spreadsheet again, unchanged, without hunting through the +agent's scratch files. + +This folder holds the research, the design, and the plan for closing that gap. The work is +staged so that the first stages are small and safe and later stages add the harder modalities +(audio and documents). + +## Glossary + +These terms appear across every file here. Each one names a real thing in our system and says +where it runs. + +- **Agent.** A coding assistant that can read and write files and call tools, driven by a + large language model. In our product it runs inside a sandbox. +- **Model.** The model is the large language model itself (for example Claude or a GPT model). The + model is what perceives an image or a document when the bytes are delivered to it in the right way. +- **Harness.** The program that drives the model through one agent turn (for example Claude + Code or Pi). The harness owns the model conversation. We do not own the harness code. +- **Runner.** Our own Node.js service that starts a sandbox, launches the harness inside it, + and passes each turn's message to the harness. The runner is where the file is dropped + today. It lives at `services/runner/`. +- **Sandbox.** The isolated Linux environment the agent runs in. The agent's files and shell + commands all happen inside the sandbox. +- **Session.** One ongoing conversation with an agent. A session has a stable id that the + front end creates before the first message is sent. +- **Turn.** One request-and-reply within a session. The person sends a message, the agent + works and answers. That is one turn. +- **cwd (working directory).** The folder the agent runs in inside the sandbox. It survives + across turns of the same session. The agent can create, change, and delete anything in it. +- **Mount.** Our file storage unit. A mount is a named area in an object store (S3) with its + own keys and its own access credentials. The `cwd` is backed by a mount. Mounts live in the + API at `api/oss/src/core/mounts/`. +- **geesefs / FUSE mount.** The technology that makes a storage mount appear as a normal + folder inside the sandbox. When a mount is "mounted into the sandbox," the agent sees it as + a directory it can read and write. +- **ACP (Agent Client Protocol).** The message format the runner uses to talk to the harness. + It is an external standard published by Zed Industries. We cannot change it. It defines the + content types a turn can carry (text, image, audio, and so on). +- **Content block.** One piece of a message in ACP: a text block, an image block, an audio + block, or a resource block. A turn is a list of content blocks. +- **Materialize.** The runner materializes a file when it writes a copy of the stored original into + the agent's working directory so the agent's tools can open it. +- **Records.** Records are the API's durable, per-session log of every event in a conversation. The + runner writes to it after every event. +- **Drawer / drive / Files drawer.** The front-end panel that lists the files in a session. It + is a view over one or more mounts. In the code it is the `DriveExplorer`. +- **Attachment resource / attachment_id.** The record the API returns when a file is uploaded. It + holds a server-issued `attachment_id`, the filename, the media type the server verified, and the + size. The API owns the storage location behind the id, so a client never sees it. The + `attachment_id` is the opaque token that travels on the wire, in the saved history, and in the + traces. +- **Reference (attachment reference).** The `attachment_id` plus a little display metadata, sent in + a message in place of the file's bytes. It carries no bytes and no raw storage coordinates; it + points at the attachment resource above, which the runner resolves through the API. +- **Front end / composer.** The front end is the browser app. The composer is its message editor: + the box where a person types, attaches files, and sends a turn. +- **API.** Our FastAPI backend service. It owns storage, permissions, sessions, and records, and it + is the only component that talks to the object store with full rights. +- **SDK.** Our Python library that sits between the API and the runner. It parses incoming messages + into content blocks and writes the wire payload the runner reads. +- **Adapter.** The translation layer inside the harness that converts ACP content blocks into the + model provider's own API calls. We run two: `claude-agent-acp` (maintained by Zed) for Claude, and + `pi-acp` (from the Pi project) for Pi. +- **Object store.** The S3-compatible storage service where mount bytes live. +- **Wire.** The serialized request that travels from the front end through the API to the runner for + one turn. +- **Native modality.** A kind of content (an image, audio, a document) that a model perceives + directly through its own encoder when the bytes are delivered inside the message content. + Everything else is a plain file that only tools can open. +- **Capability.** A yes-or-no fact about what a layer supports, advertised or configured, used to + decide whether a modality can reach the model. +- **Warm session / cold start.** A session is warm when its harness process is still running and can + continue the conversation. A cold start is when that process is gone and the runner must rebuild + the conversation before running the new turn. +- **Cold replay.** The runner's reconstruction of the conversation during a cold start, today by + flattening past messages to text. + +## Read in this order + +1. **[context.md](context.md)**: What a person experiences today, why the file is lost, and + what "done" looks like. Start here if you want the plain story with no design. +2. **[research.md](research.md)**: The findings the design rests on: what the mounts API can + do and who may call it, how different kinds of files are handled by models, how ACP carries + content and what each harness does with it, how other tools (Zed, opencode) handle + attachments, and the state of records and capability flags in our code. Every claim links to + a file and line or to a source. +3. **[design.md](design.md)**: The design itself, written as options and decisions. For each + major choice it gives the alternatives, what breaks under each, the decision, and why. It + keeps every interaction diagram, each with the flow explained in words first. +4. **[plan.md](plan.md)**: The staged implementation plan: what changes in each layer, who + owns each part, how each stage is tested, and how the plan holds up against the review + lenses (responsibilities, engineering practice, tradeoffs, scale, and fit with the rest of + the architecture). +5. **[scope.md](scope.md)**: What is in, what is out, and the follow-up work with a sentence + each on why it waits. +6. **[decisions.md](decisions.md)**: The decision log and the open questions, each with what + is unknown, why it matters, and how to settle it. +7. **[status.md](status.md)**: Where the project stands and what happens next. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/context.md b/docs/design/agent-workflows/projects/agent-multi-modality/context.md new file mode 100644 index 0000000000..479b6b545c --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/context.md @@ -0,0 +1,92 @@ +# Context: what happens today and what we want + +This file tells the story in plain language. It has no design in it. If you want the evidence +behind each claim, read [research.md](research.md). If you want the design, read +[design.md](design.md). + +## What a person experiences today + +A person opens the agent chat, attaches a photo, and types "what is this?" The chat box +accepts the photo. It shows a small preview. The message looks like it was sent with the +image. Then the agent answers as if no image were there. It saw only the words "what is this?" +and nothing else. + +If the person attaches only the photo and no text at all, the turn fails outright with an +error that says there is no message to send. + +Nothing warns the person that this will happen. The attachment looks accepted and then quietly +does nothing. That silent gap is the core problem. + +There is a second, separate product we already ship: the prompt playground, which is not an +agent. That one does handle images and documents correctly. So agents are behind the rest of +the product, and the difference is invisible until you notice the agent ignoring your file. + +## Why the file is lost + +The file travels correctly almost the whole way. The chat box reads the file into memory. The +request carries it. Our SDK carries it. The runner, our service that drives the agent, receives +it intact. Then, at the very last step, the runner builds the message it hands to the harness +using only the text. It writes a single text block and drops everything else. The runner discards +every content block that is not text at that one call. + +That last step was written when agents were text-only. It was never updated when attachments +were wired up through the rest of the path. So the path is complete except for the final step. + +There is a related habit that makes this worse. On every new turn, the front end resends the +whole conversation so far, including every earlier attachment, in full, as raw bytes encoded as +text. A single five megabyte image becomes about seven megabytes of encoded text, and it is +sent again on every following turn. The same full bytes are also saved in the browser's local +storage and written into our tracing data. None of this is needed once a file is sent by +reference instead of by value, and all of it strains the browser, the network, and the traces. + +## What we want to achieve + +We want three outcomes, and all three at the same time. + +**The model reads the file.** When a person shares an image, the model should see the image. +When they share audio, the model should hear it. When they share a document, the model should +read it. This is the part that is broken at the last runner step today. + +**The agent can work on the file.** Beyond the model reading it, the file should be present on +the agent's own working directory so the agent's tools can open it, convert it, edit it, or run +a program over it. Whether the agent changes the file is the person's call in each +conversation, not a fixed rule. + +**The file stays findable.** A person must always be able to return to exactly what they +shared, unchanged, and download it or see it again, even if the agent changes or deletes files in +its working directory. The thing you gave the agent is a durable record, not a temporary +scratch file that the agent might overwrite or delete. + +## What "done" looks like + +A person attaches a spreadsheet and asks the agent to chart the third column. The model reads +the spreadsheet. The agent opens the same spreadsheet with its tools, computes the chart, and +writes a new chart file. The person can still open the original spreadsheet, untouched, from the +files panel, and can also see the new chart the agent produced. If the person attaches a kind of +file the current model cannot handle, the chat box says so before the message is sent, instead +of accepting it and then ignoring it. + +## The three hard constraints the design must respect + +**The protocol to the harness is external.** The runner talks to the harness in ACP, a +standard we do not own. In ACP, for the model to actually perceive an image or audio, the bytes +must be delivered inline in the turn. There is no way to hand the model a link and trust it to +fetch the file. Storing the file in our object store does not remove this rule. It only removes +the need to resend the bytes in the saved conversation history. At the moment the runner hands +the turn to the harness, the bytes must be present. Details and sources are in +[research.md](research.md). + +**We build on mounts, our existing file storage.** We already have a working file storage +system with per-session storage, access controls, upload, download, and listing. The design +reuses it rather than inventing new storage. What a mount is, what it can do, and who may call +it are laid out in [research.md](research.md). + +**The harness adapters we run today deliver images, but not audio or documents.** The runner talks +to the model through two adapter packages (one for Claude, one for Pi). Reading their code shows that +both deliver an image to the model natively, but neither delivers native audio, and both fail to +deliver a document (one drops it, the other turns it into a byte count). So images can be shipped +now, while audio and documents are goals that wait on adapter work. The evidence is in +[research.md](research.md), section 4. + +The full list of verified failure modes, with the exact files and lines, lives in +[research.md](research.md) under "Current state of the code." diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/decisions.md b/docs/design/agent-workflows/projects/agent-multi-modality/decisions.md new file mode 100644 index 0000000000..3efee44e20 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/decisions.md @@ -0,0 +1,110 @@ +# Decisions and open questions + +The reasoning behind each decision is in [design.md](design.md). This file is the compact log so it +can be updated without editing the design narrative. + +## Decision log + +| # | Question | Options | Decision | Reason | +| --- | --- | --- | --- | --- | +| D1 | How does the file reach the model? | inline content blocks; file on disk plus a path; both | Both | Inline gives real perception (required by ACP; a disk file is not reliably read as vision). The disk copy lets the agent's tools work on the file. The two serve different goals. | +| D2 | Does storing the file in the object store remove the need to send bytes to the model? | yes, fully; no, only from the wire and the saved history | No, only from the wire and the saved history | Storage removes bytes from the resent history, but the model boundary still needs the bytes inline at prompt time, rebuilt per turn by the runner from the stored original. | +| D3 | Where does the unchanging original live? | a subfolder in cwd; the agent-files mount; a dedicated session mount kept out of the sandbox; a future project drive | A dedicated session-scoped attachments mount, kept out of the sandbox | The working directory is last-writer-wins, so an original there could be deleted or overwritten by the agent. Findability needs the original out of the agent's reach, and the mount technology exposes whole prefixes, so out-of-reach means its own mount. | +| D4 | One copy or two? | one copy (perceive and edit the same object); two copies | Two copies: an unchanging original and a disposable working copy | One copy cannot be both safe-to-find and freely-editable. Two copies make both goals true and turn an agent edit into a new, visible output rather than a destroyed original. | +| D5 | How does the system know whether a modality will reach the model? | a single capability flag; the intersection of three layers (protocol transport, adapter fidelity, model modalities) | The three-layer intersection, gated at the composer (courtesy, from a pre-send approximation) and the runner (final authority) | A flag can be advertised while the delivery is lossy or dropped, so one flag is not enough. Tool use over the working copy is separate and works regardless of all three layers. | +| D6 | What happens when the model cannot perceive an attached kind? | refuse the attachment; drop it silently; fail the turn; attach it as a workspace-only file with a visible notice | Attach it as a workspace-only file with a visible notice; the runner fails the turn only on a contract violation (asked to deliver a native block the harness cannot accept) | Attach means two things: show it to the model, and put it in the workspace. The workspace half always works, so refusing or failing would block legitimate tool use. Silent dropping is the current trap. A visible notice keeps it honest. | +| D7 | How is the original kept immutable? | keep it out of the sandbox; also write only through a create-only upload route and never sign the mount; also a read-only credential scope | Keep it out of the sandbox, and write only through the create-only upload route with no signed credentials for the mount; read-only scope is a hardening follow-up | Keeping it out of the sandbox stops the agent, the main threat. Originals enter only through the create-only upload route, and the API refuses overwrite and delete of an original as an explicit check, because `write_file` overwrites silently today. No signed credentials are issued for the mount, because any signed credential is read-write today. A read-only scope is the strongest form and needs a signing change, so it waits. | +| D8 | Is audio in scope? | defer; include | Include as a product goal, but blocked on adapter work | Audio stays a goal, and there is no disk-read fallback for it, so it forces a real inline audio block. Neither pinned adapter (`claude-agent-acp`, `pi-acp`) advertises native audio today, so the audio stage cannot ship against the current pins and is blocked on adapter or harness work. | +| D9 | Can the inline-only version grow into the full one? | it is a different architecture; it grows cleanly with no rework; the model-facing seam is preserved but the rest is system-wide | The model-facing seam is preserved, but adopting durable references later is still a system-wide change | Shipping the inline-only version first avoids a rewrite of the one prompt-builder seam. It does not avoid the rest: front-end persistence, API storage ownership, SDK wire types, runner resolution, the record schema, authorization, and cleanup. So the benefit is narrow and honest, not a free migration. | +| D10 | What does the reference on the wire contain? | raw storage coordinates (mount id, path, client media type); an opaque server-issued id | An opaque server-issued attachment id, with the API owning the storage location and the verified metadata | Raw coordinates let a client forge a reference to another session's file and make the client's media type authoritative even though a client can lie. An opaque id keeps storage private to the API, lets the server verify the media type on upload, and makes the session-binding check natural to express. | +| D11 (open) | What does attaching a file promise in the first release? | image perception only (strict limits, no durability); durable agent input (immutable original, workspace copy, findability, records) | OPEN. Recommendation: durable agent input, with workspace-always plus native-when-possible semantics | This is the product owner's call, not an engineering one. The engineering design supports either. The recommendation is durable agent input because it matches the three outcomes in context.md and avoids shipping a storage-less version that would need rework. Marked open until the product owner decides. | + +## Settled by evidence + +Three questions that shaped the design are answered by evidence in the code, and are recorded here so +they are not reopened by accident. + +- **Does a document reach the model natively?** Answered by reading the adapter code, not by a live + test. Neither pinned adapter delivers a document natively today: the Claude adapter drops a blob + resource entirely, and the Pi adapter renders it as a byte count (see [research.md](research.md), + section 4). So documents do not arrive natively today. This is no longer an open question; it is a + Stage 2 blocker (see open question 2 for audio and [plan.md](plan.md), Stage 2). +- **What path convention does the working copy use?** Decided: the id-namespaced visible path + `cwd/attachments//`, which is discoverable by the agent and collision-proof + across same-name files ([design.md](design.md), The working-copy path and edited copies). +- **Does the runner read the original through the API or directly from the object store?** Decided: + through the API download route for the first release, with the session-binding check kept in one + place. Reading the object store directly with a read-only credential scope is the Stage 3 hardening + ([design.md](design.md), decision D7). + +## Open questions + +Each one says what is unknown, why it matters, how to settle it, and what it blocks. + +1. **What does attaching a file promise in the first release? (D11, the product owner's decision.)** + - We do not yet know whether the first release promises image perception only, or durable agent + input with the immutable original, the workspace copy, findability, and records. + - This matters because it sets the scope and the promise of the whole first release, and it is a + product call rather than an engineering one. + - To settle it, the product owner decides on the PR. The design recommends durable agent input with + workspace-always plus native-when-possible semantics (D11). + - This blocks the shape of Stage 1. + +2. **How is native audio delivered at all?** + - We do not yet know whether audio waits on adapter work (an adapter that advertises the audio + capability and delivers an ACP audio block), on moving to a different harness that supports it, or + is deferred outright. + - This matters because neither pinned adapter supports native audio today, so the audio stage + cannot ship against the current pins. + - To settle it, track adapter releases for audio support, or decide to defer audio until one + exists. + - This blocks the audio part of Stage 2. + +3. **What are the cold-replay budget numbers?** + - We do not yet know how many historical attachments, and within what size budget, may be + re-delivered natively on a cold start before falling back to a working copy plus a textual + placeholder. + - This matters because re-sending every past attachment natively on a long conversation would + exceed the provider's per-request size and block limits (see [research.md](research.md), section + 3). + - To settle it, pick a bound in implementation against real provider limits and measure a long + replayed conversation against it. + - This blocks the cold-replay policy in Stage 1. + +4. **When are the old capability names removed across the independently deployed components?** + - We do not yet know the timing of dropping the `fileAttachments` and `file_attachments` aliases + once the front end, API, SDK, and runner all speak the new names. + - This matters because removing an alias before every component is updated breaks the versions in + between, and the four components deploy independently. + - To settle it, remove the aliases only after every component's deployed version emits and accepts + the new names, confirmed per component. + - This blocks the cleanup step of the alias rollout in Stage 2. + +5. **What are the retention rules when a session is archived or deleted?** + - We do not yet know what happens to a session's attachment originals when the session is archived + or deleted, and how long they are kept. + - This matters because attachments are durable originals, so their lifecycle has to be defined + rather than left implicit, both for storage cost and for a person's expectation that a shared file + stays findable. + - To settle it, define retention against the session lifecycle and any tenant data-retention + policy. + - This blocks the findability and cleanup work in Stage 3. + +6. **What is the exact media-type and validation matrix?** + - We do not yet know how the declared media type relates to the type the server inspects from the + bytes when they disagree, and which formats are allowed per kind (image, audio, document). + - This matters because the server verifies the media type rather than trusting the client (D10), so + the rules for a mismatch and the allowed-format list have to be explicit to be enforceable. + - To settle it, write the matrix of declared type versus the type inspected from the bytes and the + allowed formats per kind, and enforce it in the upload route. + - This blocks the server-side validation in Stage 1. + +7. **How are unused uploads cleaned up (the refinement)?** + - We do not yet know when to move from a time-to-live sweep to reference counting against the + conversation records. + - This matters because a file uploaded but never sent leaves a stored object, and reference + counting is more precise than a blind sweep. + - The starting answer is that a time-to-live sweep ships first, in Stage 1. Reference counting is + added only after records reliably carry references, because counting needs the record schema to + hold the reference. + - This blocks the cleanup refinement in Stage 3. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/design.md b/docs/design/agent-workflows/projects/agent-multi-modality/design.md new file mode 100644 index 0000000000..a2d20620dd --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/design.md @@ -0,0 +1,705 @@ +# Design + +This file presents the design as a set of decisions. It explains the major choices in full: for each +it lists the options, says what breaks under each, gives the decision, and explains why. The +complete compact log D1 through D11 lives in [decisions.md](decisions.md); D2 and D8 are small enough +that they are stated inline where they arise. This file keeps every interaction diagram, and each +diagram is explained in words before it is shown. Read [research.md](research.md) first for the facts +these decisions rest on. Each decision carries the same D-number here and in the compact log in +[decisions.md](decisions.md), so the two files never disagree about which decision is which. + +## The idea in one paragraph + +Stop sending file bytes on the wire. When a person attaches a file, the front end uploads it once +through the API's session-mount upload route. The API stores the bytes and returns an **attachment +resource**: a small record it owns, with a server-issued `attachment_id`, the filename, the +media type the server verified, and the size. The message then carries only that `attachment_id` +plus a little display metadata, never the bytes and never raw storage coordinates. Behind that id +the system keeps two copies. One is the **original**, which never changes and which the agent +cannot reach. It is the source of truth for what the model reads, what the files panel lists, and +what a download returns. The other is a **working copy** that the runner writes into the agent's +working directory so the agent's tools can open, convert, or edit it. At the moment the runner +builds a turn, it resolves the `attachment_id` through the API, reads the original, turns it into +the right inline content block, and hands that to the harness in place of the single text block +that is hard-coded today. + +## Where the original and the working copy live + +The design puts the original in a storage mount that is deliberately not made visible to the agent, +and the working copy in the agent's normal working directory. The reference on the wire points at +the original. + +Read the diagram this way. The `attachment_id` travels on the wire, in the saved history, and in +the traces. It names the attachment resource the API owns; only the API knows which object in the +store that resource points at. From the original, three things happen: the runner reads it to build +the model's content block, the runner copies it into the working directory for the agent's tools, +and the files panel lists and downloads it. The working copy is separate and disposable. Note that +the storage key shown as `mounts///...` uses `` only as +the tenant partition of the object key. It does not mean the original lives in a "project mount." +There is no project-scoped mount kind; the attachments mount is scoped to the session. + +The reference on the wire is an opaque, server-issued id, not the file's storage coordinates. This +is deliberate. If the wire carried the raw storage location and the client's own media type, then a +client could forge a reference to another session's file by naming its coordinates, and the client's +media type would be authoritative even though a client can lie about it. An opaque id avoids both +problems. The storage location stays private to the API, the server verifies the media type on +upload, and the id makes one authorization check natural: does this attachment belong to the +session being run. The full options are in decision D10. + +```mermaid +flowchart LR + subgraph store["Object store (S3)"] + orig["Original
attachments mount (session-scoped)
never changes · agent cannot reach it
mounts/<project_id>/<attachments_mount>/…"] + end + subgraph sandbox["Agent sandbox"] + cwd["Working copy
cwd/attachments/<attachment_id>/<filename>
agent may read, edit, or delete"] + end + ref["Attachment reference
{attachment_id, filename, media_type, size}
opaque id on the wire, in history, in traces"] + + ref -.addresses.-> orig + orig ==>|runner reads → base64| acp["ACP image / audio / document block
→ model perceives it"] + orig ==>|runner copies once| cwd + cwd ==>|Read / Bash / edit| tools["agent tools operate on it"] + orig ==>|list + download| ui["conversation + files panel"] + + classDef immut fill:#e8f0fe,stroke:#356,stroke-width:2px; + classDef mut fill:#fdf1e3,stroke:#a63,stroke-width:1px; + class orig immut + class cwd mut +``` + +--- + +## Decision D1: how the file reaches the model + +**The question.** How does the file's content actually reach the model so the model perceives it? + +**Options.** + +- **A. Inline content blocks.** The runner reads the file and puts its bytes into an ACP image, + audio, or document block in the turn. +- **B. File on disk plus a path in the prompt.** The runner writes the file into the working + directory and the prompt text mentions the path, trusting the harness to read it. +- **C. Both, for different purposes.** Deliver native modalities inline for perception, and also + place the file on disk for tool use. + +**What breaks under each.** Option B does not give reliable model perception. ACP has no content +type that hands the model a link and guarantees the model reads it, and Claude Code's own Read tool +does not reliably turn an image file into vision input (see [research.md](research.md), sections 4 +and 5). So with B alone, a person who shares an image and asks "what is this?" can still get an +answer that ignores the image. Option A alone leaves the agent unable to run a tool over the file, +because the file never lands on disk. For audio there is no disk-read fallback at all, so B cannot +serve audio. + +**Decision.** Option C. Deliver native modalities inline so the model perceives them, and also +materialize a working copy on disk so the agent's tools can work on the file. The two are not +redundant; they serve the two separate goals of perception and tool use. + +**Why inline is unavoidable for perception (decision D2 in the log).** In ACP the bytes for an +image, audio clip, or embedded document are required and inline. Storing the file in our object +store removes the bytes from the resent history, but it does not remove the requirement that the +bytes be present at the moment the runner builds the turn. The runner reconstructs the inline block +per turn from the stored original. + +--- + +## The one runner call that changes + +The whole model-facing change is one place in the runner: the call that hands a turn to the harness. + +Today that call sends a single text block. The proposed version resolves the message's references, +reads the originals, materializes working copies, and sends a list of real content blocks. Read the +before-and-after this way: on the left, the runner flattens the message to its text and sends only +that, so the model sees text and a replayed image shows up as the string "[image]"; on the right, +the runner turns each reference into the matching content block and sends the whole list, so the +model perceives the image and the agent can also open the file. + +```mermaid +flowchart TB + subgraph now["TODAY: the file is dropped here"] + n1["wire: content = [text, image(base64)]"] --> n2["messageText() keeps only type == text"] + n2 --> n3["turnText = 'what is this?'"] + n3 --> n4["session.prompt([{type:'text', text: turnText}])"] + n4 --> n5(["model sees text only
· replay shows '[image]'
· image with no text → run rejected"]) + end + subgraph next["PROPOSED"] + p1["wire: content = [text, attachment reference]"] --> p2["resolveContentBlocks(request)"] + p2 --> p3["read original(s) → base64
copy working copy → cwd"] + p3 --> p4["session.prompt([ImageContent(data), TextContent])"] + p4 --> p5(["model perceives image + text
agent can also open the file"]) + end + now -.replace.-> next + + style n5 fill:#fde,stroke:#c33 + style p5 fill:#efe,stroke:#3a3 +``` + +--- + +## Decision D3: where the original lives + +**The question.** Where do we store the unchanging original of a shared file? + +**Options.** + +- **A. A folder inside the working-directory mount, for example `cwd/_uploads/`.** +- **B. The agent-files mount** (the durable, cross-session agent mount). +- **C. A dedicated session-scoped attachments mount that is not made visible to the agent.** +- **D. A future project-level drive.** + +**What breaks under each.** The working directory is last-writer-wins and fully under the agent's +control (see [research.md](research.md), section 2). Concretely, under option A the following breaks +the findability goal: a person shares `report.pdf`, then asks the agent to "clean up the workspace," +and the agent runs `rm -rf _uploads` or a cleanup script, or simply overwrites `report.pdf` with a +derived version. Now the original the person shared is gone, and "always findable" is false. Even +without malice, an agent that reorganizes its files can move or replace the original. Option B avoids +the deletion problem only if agent-files is also kept out of the agent's reach, but agent-files +exists precisely to be visible and writable by the agent across sessions, so using it would either +change its meaning or expose the original to the same last-writer-wins risk. Its cross-session +lifecycle is also wrong for a per-session attachment. Option D does not exist yet and would couple +this feature to unshipped work; a project drive is also the wrong scope, since an attachment belongs +to one conversation. + +**Decision.** Option C. A dedicated attachments mount, scoped to the session, that is created with +the existing `get_or_create_session_mount(session_id, name="attachments")` machinery but is +deliberately never added to the set of mounts made visible in the sandbox. The runner reads it +out-of-band with the signed credentials it already obtains, using a plain object GET rather than a +folder mount (this is possible; see [research.md](research.md), section 2). + +**Why its own mount rather than a subfolder.** The technology that makes a mount visible to the +agent (geesefs) exposes the whole mount prefix as a writable folder. There is no way to expose part +of a mount and hide the rest. So "the agent cannot reach the original" forces the original into a +prefix that is not exposed at all, which means its own mount. The attachments mount reuses every +piece of the mount machinery (create, sign, upload, download, list) and differs from the other +mounts in exactly one way: it is left off the sandbox's visible set. + +**Why lifecycle alone does not decide the location.** On lifecycle alone, an attachment is session-scoped, exactly like +the working directory, so a subfolder inside `cwd` would have been the simplest choice. The +findability requirement, not the lifecycle, is what pushes the original out of the agent's reach and +therefore into its own mount. + +--- + +## Decision D4: one copy or two + +**The question.** Should there be a single copy of the file that the agent both perceives and edits, +or two copies with different rules? + +**Options.** + +- **A. One copy.** The file lives in one place. The agent reads and edits it there. Findability and + editing share the same object. +- **B. Two copies.** An unchanging original plus a disposable working copy. + +**What breaks under A.** If there is one copy and the agent edits it, the person can no longer find +what they originally shared. If there is one copy and it is read-only, the agent cannot edit it, +which breaks the "agent can work on it" goal. A single copy cannot satisfy both goals at once, +because the two goals want opposite things from the same object. + +**Decision.** Option B, two copies. The original never changes and backs findability, download, and +model perception. The working copy is what the agent touches. When the agent edits its working copy, +the result is a new file the agent produces, which shows up under the agent's own origin in the files +panel, while the original stays under "Shared by you." The person then sees both the input they gave +and the output the agent made, which is more useful than having the original silently replaced. + +**Why not make it a policy switch.** A read-only-versus-read-write policy on a single copy cannot +satisfy both requirements at once, because findability wants the file safe and tool use wants it +editable. Two copies dissolve the question. There is no global read-only-versus-read-write policy. +The original is always safe; the agent can always do anything it likes to its copy; and whether the +agent changes the file at all is just what the conversation calls for, decided per conversation +rather than by a platform setting. + +### The working-copy path and edited copies + +The working copy lives at `cwd/attachments//`. The `attachment_id` segment +is what keeps two files from colliding. Two people can each share a file named `report.pdf` in the +same session, and because each one sits under its own id-named folder, neither overwrites the other. + +Re-materialization has one rule: the runner restores a working copy only when the file is missing. +It never overwrites a working copy that is already there, because the agent may have edited it and +that edit is real work. So if a later turn arrives and the working copy is present, the runner +leaves it alone. If the working copy was deleted, the runner writes it again from the original. + +This creates a divergence that the design accepts on purpose. When a later turn references the same +attachment natively, the runner always reads the **original** to build the model's content block, +while the agent's tools continue to see the **edited** working copy. So the model perceives the file +as it was shared, and the tools operate on the file as the agent has changed it. That is the +intended behavior: the original is the immutable source of truth for perception, and the working +copy is the mutable scratch object for tools. + +--- + +## The successful upload and delivery flow, end to end + +Here is the full path for the common case: a person attaches a photo and asks a question, on a warm +turn. Read it as: the session already has an id, so there is no "create the session first" step; the +front end uploads the photo once through the API's session-mount upload route, and the API stores +the bytes and hands back an attachment resource; the front end sends a message that carries the +`attachment_id` instead of the bytes; the runner resolves the id through the API, reads the original +through the API download route, writes a working copy into the agent's directory, builds the real +content blocks, and calls the harness; the model perceives the image and the agent can also open the +file; and the front end renders the photo inline by resolving the reference, not from a giant inline +blob. + +The front end never touches the object store directly and never holds write credentials for the +attachments mount. The upload route is the only way bytes enter, which is what decision D7 relies on. + +```mermaid +sequenceDiagram + autonumber + participant U as User + participant FE as Chat box (front end) + participant API as API (mounts + sessions) + participant S3 as Object store + participant RUN as Runner + participant SBX as Sandbox (cwd) + participant ACP as Harness (ACP) + participant M as Model + + Note over FE: session_id already exists (front end created it) + U->>FE: attach photo.png + type "what is this?" + FE->>API: POST upload (multipart file) to the session attachments mount + API->>API: get_or_create attachments mount, validate size and type + API->>S3: write original bytes (create-only) + API-->>FE: attachment resource {attachment_id, filename, media_type, size} + FE->>API: run turn, message carries the attachment_id (no base64) + API->>RUN: /run (wire: message.content = [text, attachment reference]) + RUN->>API: resolve attachment_id, download original for this turn + API->>API: check the attachment belongs to this session + API->>S3: GET original + S3-->>API: bytes + API-->>RUN: bytes + RUN->>SBX: write working copy to cwd/attachments/attachment_id/photo.png + RUN->>RUN: build ACP blocks [ImageContent(data=base64), TextContent] + RUN->>ACP: session.prompt([...blocks]) + ACP->>M: image + text + M-->>ACP: answer (may open the working copy with a tool) + ACP-->>RUN: stream + RUN-->>FE: stream, rendered + Note over FE: the photo renders inline by resolving the reference through a download, not from base64 +``` + +The one change that makes the model perceive the image is at the "build ACP blocks" and +"session.prompt" steps. The working-copy write happens once per file per session, because the working +directory survives across turns, so later turns skip it. + +--- + +## The lifecycle of one attachment + +An attachment moves through a few clear states. Read the state machine as: the person picks a file; +it is rejected up front only when it is too large or an invalid file, and an unsupported kind is not +rejected but accepted as a workspace-only attachment with a notice; an accepted file is uploaded +once, after which the original is stored and unchanging; when a turn is sent, the `attachment_id` +goes with it; the runner always writes the working copy, and delivers the bytes to the model only +when the capability intersection allows; and from there the attachment is a durable, findable record +whose findability does not depend on what the agent does to its working copy. + +```mermaid +stateDiagram-v2 + [*] --> Selected: user picks / pastes / drops + Selected --> Accepted: passes size and format-validation checks + Selected --> Rejected: too large or invalid file (told before send) + Rejected --> [*] + Accepted --> Uploading: upload once through the API + Uploading --> Stored: original persisted (never changes) + Uploading --> UploadFailed: retry or surface error + UploadFailed --> Uploading: retry + UploadFailed --> [*]: user removes + + Stored --> Referenced: turn sent with the attachment_id + Referenced --> Materialized: runner always writes the working copy + Referenced --> Perceived: native delivery when the capability intersection allows + Referenced --> WorkspaceOnly: intersection does not allow, delivered as a workspace-only attachment with a notice + + Perceived --> Findable + WorkspaceOnly --> Findable + Materialized --> AgentOperates: read / convert / edit / delete + AgentOperates --> Findable + + state Findable { + [*] --> Listed + Listed --> Listed: renders inline · listed in panel · downloadable + } + note right of Findable + The original backs findability. + The agent deleting its working copy + does not change this. + end note + Findable --> [*]: session archived or cleaned up +``` + +--- + +## Decision D5: the layered capability model + +**The question.** How does the system know whether a given modality will actually reach the model? + +**The layered answer.** Whether a modality is perceived natively is the intersection of three +layers. All three must allow it, and the effective capability is the weakest of the three: + +1. **What the ACP transport supports.** The prompt capability flag (`image`, `audio`, + `embeddedContext`) that the adapter advertises. Without the flag the content type cannot even be + carried across the protocol. +2. **What the harness adapter actually delivers natively.** Adapter fidelity. A flag can be + advertised while the delivery is lossy or dropped. This is where the two adapters we run diverge + sharply (see [research.md](research.md), section 4): both pass an image through as a real image, + but the Claude adapter drops a blob resource entirely and the Pi adapter renders it as a byte + count. So a document can clear the transport layer and still never reach the model. +3. **What the selected model perceives.** Model modalities. A non-vision model does not see an image + even when the transport and the adapter would carry it. + +Tool use over the file is a separate, fourth consideration, not part of this intersection. It needs +only the working copy on disk and works regardless of all three layers above. An agent whose model +cannot see an image can still run a program over the image bytes. This is why the design places the +file on disk for every attachment, independent of native perception. + +**Decision on gating (D5).** Compute the intersection and gate in two places. The composer gates for +the person's benefit, so the person learns before sending whether a file will be perceived. The +runner gates as the final authority, because the composer's view can be stale. Both are needed; the +composer gate is a courtesy and the runner gate is the truth. + +Here is the layered capability, per modality, as it stands today. + +| Modality | ACP transport (prompt capability) | Adapter fidelity today | Model modalities | Tool use over the working copy | +| --- | --- | --- | --- | --- | +| Image | `image`, advertised by both adapters | native image on both adapters | needs a vision model | always available | +| Audio | `audio`, advertised by neither adapter | no native audio path on either adapter | needs an audio model | always available | +| Document (PDF and similar) | `embeddedContext`, off by default on Pi | Claude drops the blob, Pi renders a byte count | depends on the model and the format | always available | + +### Two integration gaps to close + +Two facts about the current code mean the layered model needs plumbing that does not exist yet, and +these are stated as work, not as afterthoughts. + +- **Capabilities are surfaced only after a run.** Today the runner returns its capabilities in the + run response, after a run has happened (`protocol.ts`, near line 539). The composer has no way to + discover them before the person sends anything. So the composer needs a pre-send discovery + surface. A static approximation derived from the model and harness catalog is acceptable for the + first release, with the runner remaining the final authority at prompt-build time. +- **The runner error is a plain string.** Today the run error field is a plain string (`error?: + string`, `protocol.ts`, near line 557), not a structured value. The failure case needs a + structured error code the front end can recognize and render, so the person sees a specific, + actionable message rather than a raw string. + +### Renaming capability fields without a simultaneous deploy + +The internal capability flags need new names (`images`, `audio`, `documents`) and the old ones +(`fileAttachments`, `file_attachments`) need to retire. Do **not** do this as a rename that must land +in every component at the same time. The front end, the API, the SDK, and the runner deploy +independently, so a rename that lands +in one component before another breaks the versions in between. Instead, introduce the new fields +alongside the old ones, keep the old names accepted as aliases through the rollout, and remove the +old names later once every component speaks the new ones. The removal timing is an open question +(see [decisions.md](decisions.md)). + +--- + +## Decision D6: what does attaching a file promise, and what happens on an unsupported kind + +**The question.** When a person attaches a file the model cannot perceive, what should happen? +Failing the turn on every unsupported kind would contradict the tool-use goal: an agent can still +process a file its model cannot perceive natively. The resolution is to separate the two meanings of +attach. + +**The resolution: separate the two meanings of attach.** "Attach" means two different things, and +untangling them removes the contradiction: + +- **Show this to the model.** Deliver the file as a native content block the model perceives. +- **Put this file in the agent's workspace.** Materialize it on disk so the agent's tools can open + it. + +**The design's answer.** Attaching a file **always** puts it in the workspace (it is materialized +for tools). Native delivery to the model happens **when the capability intersection from D5 allows +it**. When the intersection does not allow it, the composer says so up front, and the turn still +proceeds with the file as a workspace-only attachment. That is a visible notice to the person, not a +silent drop and not a failed turn. The runner fails the turn only when it is explicitly asked to +deliver a native block the harness cannot accept, which is a contract violation rather than an +ordinary unsupported kind (for example, a stale front end that asks for native audio the adapter +never advertised). So a silent drop never happens, and a hard failure happens only on a real +protocol contract violation. + +**The fuller model, and the first-release simplification.** The richer version of this would attach +a per-attachment intent to each file: *perception required* (fail if the model cannot perceive it), +*perception preferred* (deliver natively when possible, otherwise workspace-only), or *workspace-only* +(never try native delivery). The first release does not build that per-attachment control. It treats +every native kind as *preferred* and offers no per-attachment toggle in the UI. This is the simpler +model that still avoids both the silent drop and the surprise failure. + +**What the first release actually promises is an open product decision.** The exact promise the +first release makes (image perception only, versus durable agent input with the workspace copy and +findability) is the product owner's call, recorded as the open decision D11 in [decisions.md](decisions.md) +and restated below. + +Read the gating diagram as: the harness advertises capabilities at start-up; the runner maps them +and also knows the adapter fidelity; a pre-send discovery surface gives the composer an +approximation; the composer attaches every file, delivering natively where the intersection allows +and attaching workspace-only with a visible notice where it does not; and the runner, as final +authority, always materializes the working copy and fails the turn only when asked to deliver a +native block the harness cannot accept. + +```mermaid +flowchart TD + A["Harness start-up (ACP)"] --> B["prompt capability flags
{ image, audio, embeddedContext }"] + B --> C["runner maps to { images, audio, documents }
and knows the adapter fidelity"] + C --> D["pre-send discovery surface
(static catalog approximation today,
runner is final authority)"] + + subgraph fe["Composer (front end)"] + D --> E{native intersection allows this kind?} + E -->|yes| F["attach and mark for native delivery"] + E -->|no| G["attach as workspace-only
with a visible notice
(not refused, not silently dropped)"] + end + + subgraph run["Runner (final authority)"] + F --> H["always materialize the working copy"] + G --> H + H --> I{asked to deliver a native block
the harness cannot accept?} + I -->|no| J["deliver native where allowed,
workspace-only otherwise"] + I -->|yes, contract violation| K["fail the turn with a
structured error code"] + end + + style G fill:#fef6e0,stroke:#a83 + style K fill:#fde,stroke:#c33 + style J fill:#efe,stroke:#3a3 +``` + +Note the three distinct capabilities: an image maps to `image`, audio maps to `audio`, but a PDF or +document maps to `embeddedContext`, a different flag. Do not treat "supports images" as "supports +documents." + +--- + +## Decision D11 (open): what does attaching a file promise in the first release? + +This is the one product decision left for the product owner, tracked as D11 in [decisions.md](decisions.md). +The engineering design supports either answer; the choice is about what to promise a person first. + +- **Option A: image perception only.** The first release lets the model see images, with strict + size and count limits and no durability promise. The file is not guaranteed to remain findable + after the turn. This is the smallest honest fix for the original bug. +- **Option B: durable agent input.** The first release delivers the full model here: an original + that never changes, rendered inline in the conversation and downloadable at any time (the + Files-drawer listing follows in Stage 3); a working copy for the agent's tools; and a reference + stored in the records. Attaching a file always puts it in the workspace and delivers it natively + when the capability intersection allows. + +**The recommendation** is Option B, durable agent input, with the workspace-always plus +native-when-possible semantics described in D6. It matches the three outcomes in +[context.md](context.md) and avoids shipping a storage-less version that would need rework. The +final call is the product owner's, which is why this is marked open rather than decided. + +--- + +## Decision D7: enforcing that the original does not change + +**The question.** The original must not change. What actually enforces that? + +**Options.** + +- **A. Keep the mount out of the sandbox.** The agent never sees it, so the agent cannot change it. +- **B. Also refuse to hand out any signed credentials for the attachments mount, so the only way + bytes enter is the create-only upload API route.** +- **C. Also add a read-only credential scope so the runner could read the object store directly and + even a leaked credential could not write.** + +**What each gives.** Option A stops the agent, which is the main threat, but it does not stop a bug +elsewhere in our own code that holds credentials. Option B closes the credential path. Recall from +[research.md](research.md), section 2, that any signed credential is read-write today: the signing +call has no read-only variant, so signing a mount at all hands out write access. So immutability +cannot rest on signed credentials. Recall too that `write_file` overwrites silently, so create-only +is not something the storage layer gives for free. Option B therefore has two parts working +together: never issue a signed credential for the attachments mount, and make the upload route +refuse to overwrite or delete an attachment original. That refusal is an explicit API-level check, +not an assumption, precisely because the storage layer would otherwise overwrite without complaint. +Option C is the strongest and is also the path that would later let the runner read the object store +directly, but it needs a read-only variant added to the signing call, which does not exist today. + +**Decision.** Start with A and B together. Originals enter only through the create-only upload route. +The API refuses overwrite and delete for attachment originals, enforced as an explicit check because +`write_file` overwrites silently today. No signed credentials are ever issued for the attachments +mount, because today any signed credential is read-write. A read-only credential scope is the later +hardening that would let the runner read the object store directly; it is noted in +[plan.md](plan.md). Together A and B give real, enforced immutability for the case that matters, +without blocking on the signing change. + +--- + +## Modality to content block mapping + +This shows how each kind of attachment becomes a content block on each side. Read it as: the person's +file has a media type; our front end and SDK turn it into an Agenta content block; and the runner +turns that into the matching ACP block that the harness understands. Small text does not need to be a +resource at all; it can be inlined as text, which every harness supports. Audio stays a product goal +(decision D8 in the log): there is no disk-read fallback for audio, so supporting it forces both the +new `audio` block on our side and the inline ACP audio block. The diagram shows the target shape. +Neither pinned adapter delivers native audio or native documents today (see +[research.md](research.md), section 4), so the audio and document rows describe where these paths +lead once the adapters support them, not what works right now. + +```mermaid +flowchart LR + subgraph in["User attaches"] + img["image/*"] + aud["audio/*"] + pdf["application/pdf"] + txt["text/* · application/json"] + end + subgraph agenta["Agenta content block (add 'audio')"] + aimg["image"] + aaud["audio ⟵ NEW"] + ares["resource"] + end + subgraph acp["ACP content block (external)"] + cimg["ImageContent(data)"] + caud["AudioContent(data)"] + cemb["EmbeddedResource(blob)"] + ctxt["TextContent"] + end + + img --> aimg --> cimg + aud --> aaud --> caud + pdf --> ares --> cemb + txt -->|inline as text| ctxt + + cimg --> gate1["gated on prompt capability image"] + caud --> gate2["gated on prompt capability audio"] + cemb --> gate3["gated on prompt capability embeddedContext"] +``` + +A hard caution carried from [research.md](research.md), section 4: the document path in this diagram +does not work today. The Claude adapter drops a blob resource entirely, and the Pi adapter renders +it as a byte count, so a PDF sent as an embedded resource reaches neither model as a document. The +diagram shows the intended shape, and the document stage is blocked on adapter work (either +adapter-native document handling, or a decision to deliver documents as extracted text). Images map +cleanly and work end to end. Audio has no native path on either adapter yet, so its row is likewise +a target, not a working path. + +--- + +## Where the person finds a shared file + +Findability shows up in three places, all backed by the unchanging original. Read the diagram as: the +original backs a reference saved in the conversation record; that record renders the file inline where +it was sent; the same original is listed in the files panel under a new "Shared by you" origin and can +be downloaded as exact bytes; and separately, when the agent edits its working copy, that produces a +new file under the agent's own origin, so the person sees both what they gave and what the agent made. +Of these surfaces, the inline render in the conversation and the download are part of the first +release; the Files-drawer listing under "Shared by you" is Stage 3 work. + +```mermaid +flowchart TD + orig["Original (never changes)
attachments mount"] --> ref["reference saved in the record"] + ref --> S1 + orig --> S2 + orig --> S3 + + subgraph surfaces["Where the person finds it"] + S1["Inline in the conversation
renders where it was sent
(reference resolves to the original, not base64)"] + S2["Files panel, 'Shared by you' origin
a new origin beside
'session' and 'agent-files'"] + S3["Download
exact bytes, any time
(download endpoint)"] + end + + ao["Agent edits its working copy"] --> art["New file in cwd
shows under the agent's origin"] + art -.different from.-> S2 + note["The person sees both:
what they gave (Shared by you)
and what the agent made (agent origin)"] + S2 -.-> note + art -.-> note +``` + +--- + +## Why the first upload can safely create the mount + +A person might attach a file before the very first message, so there is a fair question about whether +the attachments mount exists yet. It does, because the front end owns the session id before the first +message and the upload route's get-or-create is idempotent. Read the diagram as: the front end +already holds a stable session id when the pane opens; the first upload targets the attachments +mount, and the upload route creates that mount if it is not there yet (an idempotent get-or-create +keyed on the session id); so the mount is ready on the first upload, with no separate "create the +session" step that could run out of order against it. + +```mermaid +sequenceDiagram + autonumber + participant FE + participant API + Note over FE: session_id created when the pane opens (front end owns it) + FE->>API: upload (multipart file) to the session attachments mount + API->>API: get_or_create attachments mount (idempotent on the session id), then write + API-->>FE: attachment resource returned + Note over FE,API: no separate create-session step that could run out of order. The mount is created on the first upload for this session_id +``` + +A file uploaded but never sent leaves an unused stored object. That is a cleanup concern, not an +ordering one, and it is handled by the garbage-collection follow-up in [plan.md](plan.md). + +--- + +## Decision D9: can the inline-only version grow into the full version + +**The question.** If we shipped the inline-only version first, could we grow it into this full +design later, or would we have to rebuild? + +**The inline-only version.** Deliver the file inline to the model only, with no durable storage: the +runner reads the bytes off the wire and builds the content block, exactly at the same seam. This +fixes model perception and nothing else. It does not fix findability, the working copy, the resend +cost, or the storage bloat. + +**Does it grow.** Partly, and it is worth being honest about which part. The one seam that both +versions share is the model-facing prompt-builder: both replace the single hard-coded text block +with a resolved list of content blocks at the same call. Shipping the inline-only version first does not +force a rewrite of that seam later. That is the real benefit, and it is a narrow one. + +What the inline-only version does **not** save is the rest of the work. Adopting durable references later +is still a system-wide change that touches seven places: front-end persistence (saving a reference +instead of bytes), API storage ownership (the attachment resource, the upload route, the download +route), SDK wire types (the reference form of the content block), runner resolution (turning an id +into bytes), the record schema (a field to hold the reference), authorization (the session-binding +check), and cleanup (removing never-referenced uploads). None of that is avoided by shipping thin +first. So the accurate claim is that the inline-only version avoids rework at the prompt-builder seam, not +that it is a smaller version of a migration that the full version merely extends. + +**The one thing not to defer.** The reference-on-the-wire change should not be skipped for long, +because as long as bytes ride the wire and the history, the resend cost, the browser storage +pressure, and the trace bloat all remain. Those are fixed by the reference, which is why the plan +puts the storage and reference work in the first release. + +--- + +## Decision D10: what the reference contains + +**The question.** What actually travels on the wire to name a shared file: its storage location, or +an opaque id? + +**Options.** + +- **A. Raw storage coordinates.** The reference carries the mount id, the object path, and the + client's own media type, filename, and size. The runner reads the object at those coordinates. +- **B. An opaque, server-issued id.** The upload returns an attachment resource, + `{attachment_id, filename, media_type, size}`, where the API owns the storage location and the + authoritative metadata. The wire and the records carry the `attachment_id` plus display metadata. + The runner resolves the id through the API, which enforces that the attachment belongs to the + session being run. + +**What breaks under A.** Two things. First, forged references: a client that knows or guesses +another session's storage coordinates can name them and read files that are not its own, because the +coordinates are the authority. Second, lying metadata: the client-supplied media type becomes +authoritative even though a client can send any media type it likes, so the server would trust a +value it never verified. + +**Decision.** Option B, the opaque server-issued id. The storage location stays private to the API, +so there is nothing for a client to forge. The server verifies the media type on upload, so the +authoritative metadata is the server's, not the client's. And the id makes the key authorization +check natural to express: does this attachment belong to this session. The runner never sees storage +coordinates at all; it hands the id to the API and gets back bytes only when the binding checks out. + +**Reason.** An opaque id is the difference between "trust what the client says about where the file +is and what it is" and "let the server own both." The second is the only one that is safe when the +wire is something a client can write. + +--- + +## Decision log and open questions + +The compact decision log (D1 through D11) and the open questions are in +[decisions.md](decisions.md), so they can be updated without editing this design narrative. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/plan.md b/docs/design/agent-workflows/projects/agent-multi-modality/plan.md new file mode 100644 index 0000000000..e73753eab6 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/plan.md @@ -0,0 +1,308 @@ +# Plan + +This file turns the design into staged work. Each stage says what changes in each layer, who owns +each part, and how it is tested. Stage 1 deliberately packages storage and model perception into one +user-visible release, because shipping storage without perception would recreate the original bug in +a new form; the reason is spelled out in that stage. + +The layers, and who owns each: + +- **Front end** (`web/oss/src/components/AgentChatSlice/`, `web/packages/agenta-playground/`, + `web/oss/src/components/Drives/`): collects the file, gates by capability, uploads it, sends the + reference, and renders it. +- **API** (`api/oss/src/core/mounts/`, `api/oss/src/apis/fastapi/mounts/` and `.../sessions/`): + stores the bytes, serves them, and enforces permissions. +- **Runner** (`services/runner/src/`): reads the original, writes the working copy, builds the content + blocks, and gates on capability. +- **SDK** (`sdks/python/agenta/sdk/agents/`): carries the reference and the new `audio` block, and + maps capabilities. + +Stage 1 as specified below implements the recommended answer to the open product decision D11 +(durable agent input; see [decisions.md](decisions.md)). If the product owner instead chooses image +perception only, Stage 1 shrinks to the inline-only version: the runner builds native image blocks +straight from the bytes on the wire, and the storage, reference, record-schema, and findability work +in this stage is deferred. The rest of the plan is written for the recommended option. + +## Stage 0: block attachments until native delivery works (small, optional) + +**Goal.** Stop the front end from accepting files until the runner can deliver them, by refusing an +attachment that would reach the dead end instead of accepting it. + +- **Front end.** The paperclip is disabled, but paste and drag still add files + (`AgentConversation.tsx`, the paste and drop handlers). Gate those two paths on the same condition + as the paperclip so no file can be attached until the real feature lands. This removes the trap + where a pasted screenshot looks accepted and is then ignored. +- **Tests.** A front-end unit test that a paste or drop of a file is refused while the feature flag + is off. + +This stage can be skipped if Stage 1 lands quickly. + +## Stage 1: the first user-visible release (images perceived, files travel as references) + +This stage packages two bodies of work (the attachment storage and reference handling, and the +runner's model-facing change) into one release, on purpose. + +**Goal.** The model perceives images, and files travel as references rather than bytes. A person +attaches an image, the model sees it, the image is also on the agent's disk, the original stays +findable, and the wire no longer carries base64. + +**Why these ship together and not separately.** Shipping storage alone, visibly, would let the UI +accept files that the model still ignores. That is the original bug in a new form: the person sees an +attachment go through and the agent acts as though it were not there. So the release is packaged so +that model perception and the attachment storage and reference handling land together, and the person +never sees an attachment UI that quietly does nothing. + +**The minimum security and limits work belongs in this release, not later.** Session-binding checks, +forged-reference rejection, server-side media-type verification, per-file and per-turn size and count +limits, and time-to-live cleanup of never-referenced uploads are all part of Stage 1. They are not +polish for a later stage, because the first release already accepts files from a browser and stores +them, and an unauthenticated or unbounded version of that is not shippable. + +**Deployment order inside the release.** The components deploy independently, so the order matters. + +**(a) API first.** Build the attachment resource before anything depends on it. + +- The create-only upload route: get-or-create the attachments mount, validate the file (size, count, + media type verified server-side by inspecting the file bytes, not by trusting the client), write the bytes, and + return `{attachment_id, filename, media_type, size}`. Refuse an overwrite or a delete of an + existing attachment original, because `write_file` overwrites silently today + ([design.md](design.md), decision D7). +- The download route the runner uses to read an original, with the session-binding check: the + attachment must belong to the session being run, and a reference to another session's attachment is + rejected ([design.md](design.md), decision D10). +- Per-file and per-turn size and count limits, enforced server-side. +- Time-to-live cleanup of uploads that are never referenced by a sent turn. +- The record-schema extension so a record can carry an attachment reference. Records are text-only + today (see [research.md](research.md), section 7), so without this a reference cannot survive in the + durable log. This is why the schema change is in the first release rather than deferred. + +**(b) Runner second.** Once the API can store and serve, teach the runner to resolve and deliver. + +- Dual-read during rollout. Dual-read means the runner accepts both the old inline-byte form and the + new attachment-reference form during the rollout, so a runner deploy does not have to be + simultaneous with the front-end switch. +- Resolution through the API: hand the `attachment_id` to the API download route and receive the + bytes only when the binding checks out. The runner never sees storage coordinates. +- Materialize the working copy to the id-namespaced path `cwd/attachments//`, + restoring it only when missing and never overwriting an edited copy ([design.md](design.md), The + working-copy path and edited copies). +- Build ACP image blocks: replace the single text block at the `env.session.prompt(...)` call + (`run-turn.ts`, currently near line 742, the one real call site) with the resolved list of content + blocks. Update `resolvePromptText` and `messageText` callers so an image-with-no-text turn is valid + instead of rejected. +- Structured capability errors: gate the image block on the mapped `images` capability, and on a + contract violation fail the turn with a structured error code the front end can render, reusing the + `assertRequiredCapabilities` and `*_UNSUPPORTED_MESSAGE` pattern in `capabilities.ts`. Never drop + silently. +- A bounded cold-replay policy. Cold replay (rebuilding the conversation on a cold start) must set an + explicit count and byte limit on how many historical attachments are re-delivered natively; beyond + the limit, the working copy and a textual placeholder represent the file. Update the cold-replay + path (`transcript.ts`) so a past image is no longer rendered as the string "[image]", within that + budget (see the cold-replay budget below). + +**(c) Front end last.** Once the API stores and the runner resolves, switch the browser over. + +- Replace the inline base64 flow (`files.ts` `fileToPart`) with upload-through-the-API plus a + reference in the message parts (`agentRequest.ts`). +- Render the person's own attachment by resolving the reference to a download URL, not from the + base64 `data:` URL (`sessions.ts`), which is what removes the browser-storage pressure. +- Enable the attachment UI: the composer accepts a file, attaches it (workspace-only with a visible + notice when the capability intersection does not allow native perception), and shows it. + +**Tracing.** Because the history now carries a reference, the Python span no longer holds base64. +Confirm the trace shows the reference, not the bytes (the Python agent span keeps `messages` on +purpose, and a reference there is small). + +**SDK / wire.** Add the attachment-reference form to the content block: it carries `attachment_id`, +`filename`, `media_type`, and `size`, and no bytes. Update `protocol.ts`, `wire.py`, both contract +tests, and the shared golden fixtures together, as the runner's wire contract requires (see +`services/runner/CLAUDE.md`). + +**Tests.** These are listed in one place at the end of this file, under "Tests across the release," +because they span the API, the runner, and the front end. + +## Stage 2: audio and documents (blocked on adapter work) + +**Goal.** Audio and documents reach the model, gated correctly. + +**This stage is blocked on adapter work and cannot ship on the adapters we pin today.** From +[research.md](research.md), section 4: neither pinned adapter delivers native audio (neither +advertises the audio capability), the Claude adapter drops a document blob entirely, and the Pi +adapter renders a document blob as a byte count. So this stage is a plan for when the adapters +change, not work that can land against the current pins. `claude-agent-acp` is maintained by Zed and +`pi-acp` by the Pi project, so unblocking audio or native documents means an upstream release, a fork +we maintain, or a different harness. + +**What would unblock it.** + +- For audio: an adapter that advertises the audio capability and delivers an ACP audio block to the + model. Until then there is nothing to deliver audio into. +- For documents: either adapter-native document handling (an adapter that turns an embedded resource + into a real document input), or a deliberate decision to deliver documents as extracted text (the + agent, or a pre-step, extracts the text and inlines it), which sidesteps native document delivery + entirely. + +**The work, once unblocked.** + +- **SDK / wire.** Add an `audio` type to the Agenta content block, mapped in both directions in the + Vercel adapter (`messages.py`), and mirrored in `protocol.ts`, `wire.py`, the contract tests, and + the golden fixtures. +- **Runner.** Map audio to an ACP audio block and a document to whichever path the unblocking + decision chooses, gated on the `audio` and `documents` capabilities respectively. +- **Capability names (alias rollout, not a simultaneous rename).** Retire the old `fileAttachments` and + `file_attachments` names in favor of `images`, `audio`, and `documents`, and map ACP + `embeddedContext` to `documents`. Do this as an alias rollout: introduce the new names alongside the + old, keep the old names accepted as aliases through the rollout, and remove them later once every + independently deployed component speaks the new names. A single rename landing in every component at + once would break the versions in between (see [design.md](design.md), decision D5). The removal + timing is an open + question in [decisions.md](decisions.md). +- **Front-end limits.** Replace the placeholder limits (`attachments.ts` `DEFAULT_ATTACHMENT_LIMITS`) + with limits derived from the selected model's real limits (see [research.md](research.md), section + 3), passed down in place of the default, which the file was already written to allow. +- **Tests.** SDK contract test for the `audio` block. Runner tests for audio and document blocks and + their gates. A capability-contract test that the mapped set is consistent across the layers. + Integration tests against whichever adapter version unblocks the stage. + +## Stage 3: the Files-drawer listing and cleanup refinements + +**Goal.** The "Shared by you" origin in the Files drawer, the reference-counting refinement of cleanup, the read-only +credential scope, and the edit-then-find flow verified end to end. The basic time-to-live cleanup +already ships in Stage 1; this stage refines it. + +- **Front end (drawer).** Add "Shared by you" as a third origin over the attachments mount, reusing the + existing provenance tagging in `DriveExplorer` (see [research.md](research.md), section 2). It is a + new origin label, not a new panel. +- **API (cleanup refinement).** Stage 1 ships a time-to-live sweep of never-referenced uploads. Once + records reliably carry references, add reference counting against the conversation records so a + still-referenced upload is never swept and a truly orphaned one is removed promptly (the open + question in [decisions.md](decisions.md) tracks this). +- **Hardening.** Add a read-only credential scope for the attachments mount, the strongest + immutability guarantee and the path that would let the runner read the object store directly instead + of through the API download route ([design.md](design.md), decision D7). +- **Tests.** An end-to-end check that after the agent edits its working copy, the original still opens + unchanged under "Shared by you" and the agent's new file shows under the agent origin. + +## Why the plan is shaped this way + +The split of responsibilities, the reuse choices, and the sizing all follow from a few judgments, +stated here so a reviewer can challenge them. + +**Responsibilities are split cleanly.** The front end collects, gates, uploads, and renders. The API +stores, serves, and enforces permissions. The runner reads the original, materializes the working +copy, and turns references into model content. The SDK carries the reference and the block types. No +layer reaches across into another's job: the runner never talks to the browser's storage, the front +end never builds ACP blocks, the API never knows about ACP. The one place to watch is the capability +mapping, which touches four layers, so the layered contract in [design.md](design.md) is the single +source and the old names retire through an alias rollout, keeping mixed component versions working +during the change. + +**Engineering and architecture practice.** The design reuses the existing mounts substrate rather +than inventing storage, follows the established pattern of a separate mount for a separate lifecycle +(the agent-files precedent), and reuses the existing capability-gate pattern for the failure case. +Nothing here is a new architectural concept; it is a new use of concepts already in the code. + +**Tradeoffs are stated, not hidden.** Every major choice in [design.md](design.md) lists its options +and what breaks under each. The lifecycle tradeoff is the clearest example: an attachment could have +lived in a `cwd` subfolder on lifecycle grounds, and only the findability requirement forces its own +mount. + +**Scale and extensibility, sized for a first version with room to grow.** The reference-on-the-wire +change removes the quadratic resend cost, the browser storage pressure, and the trace bloat, so the +first version already scales better than today. The materialize step runs once per file per session. +The design grows into the harder modalities by adding block types, not by reworking the model-facing +call. Shipping the inline-only version first would avoid rework at that one call, though not the rest +of the system-wide change (decision D9). What is deliberately left for later is stated in +[scope.md](scope.md): video, assistant-produced files, cross-session reuse, and storage optimizations +like deduplication and thumbnails. + +**Fit with the current architecture.** The change lives at the boundaries that already exist: the one +`session.prompt` call in the runner, the existing mount routes, the existing content-block adapter, +the existing capability probe, and the existing provenance tagging in the drawer. It does not +introduce a parallel system beside any of these. + +## Four guarantees the stages must not lose + +These four points are each covered inside a stage above, but they are the kind of detail that gets +dropped during implementation, so they are restated here as explicit checks. + +- **Immutability is enforced server-side, not only by hiding the mount.** Originals enter only + through the create-only upload route, the API refuses overwrite and delete of an original (because + `write_file` overwrites silently today), and no signed credentials are ever issued for the + attachments mount (because any signed credential is read-write today) ([design.md](design.md), + decision D7). A read-only credential scope is the Stage 3 hardening follow-up. +- **Re-materialization is defined behavior.** If the agent deleted its working copy and a later turn + references the file, the runner re-reads the original and writes the copy again; it never overwrites + an edited working copy (Stage 1). Model delivery never depends on the working copy, and always reads + the original. +- **The capability contract is one layered model across every layer.** The layered contract in + [design.md](design.md) is the single source, and the old capability names retire through an alias + rollout (Stage 2), never a rename that must land in every component at the same time, so no + independently deployed component is ever left + speaking a name the others do not. +- **An attach never silently drops, and a turn fails only on a contract violation.** An unsupported + kind becomes a workspace-only attachment with a visible notice, and the runner fails the turn only + when asked to deliver a native block the harness cannot accept ([design.md](design.md), decision D6; + Stage 1). + +## The tracing change + +The tracing improvement in Stage 1 is not a separate task; it follows directly from the +reference-based message format. Once the saved history carries a reference instead of bytes, the +Python agent span (which keeps `messages` on purpose and has no length cap) holds a small reference +rather than a base64 blob. Confirm this in Stage 1 rather than adding a truncation cap, since the +cause is removed at the source. + +## The cold-replay budget + +On a cold start the runner rebuilds the conversation from records, and a historical attachment cannot +be re-sent freely. The policy: on a cold start only the working copies (already on disk) and textual +placeholders represent historical attachments by default. Re-delivering a historical attachment as a +native block again is bounded, and the bound is defined in implementation (for example, only the most +recent few native attachments, or only those within a size budget). The reason is a hard limit: +replaying a long conversation that re-sends every past image natively would exceed the provider's +per-request size and block limits (see [research.md](research.md), section 3). So historical native +re-delivery is capped, and older attachments fall back to their working copy and a textual +placeholder. + +## Tests across the release + +These tests span the API, the runner, and the front end, so they are listed together rather than +under a single stage. + +**Model perception and the seam.** + +- An image-and-text turn produces an image block and a text block. +- An image-only turn is valid instead of rejected. +- A cold replay of a past image no longer emits the string "[image]". +- Run integration tests against the two pinned adapters (`@agentclientprotocol/claude-agent-acp` and + `pi-acp`) confirming that an inline image reaches the model as a real image. + +**Storage, references, and the wire.** + +- Add an SDK contract test for the attachment-reference form against the golden fixture. +- Test that a picked file uploads once and the message carries a reference with no `data:` URL. +- Verify that a saved-and-reloaded message still renders the file from the reference. + +**Security and authorization.** + +- A reference to another session's attachment is rejected (foreign-session rejection). +- A forged reference (an id that does not resolve to the caller's session) is rejected. +- An overwrite attempt against an existing original is refused, and a delete attempt against an + original is refused. +- A browser-supplied media type that disagrees with the type the server inspected from the bytes does + not win; the server's verified type is authoritative. + +**Limits and cleanup.** + +- Per-file and per-turn size and count limits are enforced server-side. +- An upload retry after a transient failure does not create a duplicate or a partial original. +- An abandoned upload (never referenced by a sent turn) is swept by the time-to-live cleanup. + +**Resolution and materialization.** + +- Resolver failure paths: a missing attachment, expired authorization, and a timeout each surface a + structured error, not a silent drop. +- Materialization is atomic and never overwrites an edited working copy. +- Two attachments with the same filename in one session do not collide (the id-namespaced path). +- Warm resume and cold replay both work with references, within the cold-replay budget above. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/research.md b/docs/design/agent-workflows/projects/agent-multi-modality/research.md new file mode 100644 index 0000000000..cc810da2ee --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/research.md @@ -0,0 +1,643 @@ +# Research + +This file collects the findings the design rests on. It is meant to be read top to bottom. Each +section answers one question. Every claim links to a file and line in our code, or to an outside +source. Line numbers drift as files change, so treat them as a starting point and confirm the +surrounding function name. + +Sections: + +1. Section 1 explains where the file is lost today and the side effects it causes. +2. Section 2 describes the mounts system: what it is, every endpoint, who may call each one, and how + the file panel uses them today. +3. Section 3 covers the kinds of files and how models handle them: which kinds a model perceives + natively, which are just files an agent reads with tools, and the size limits per provider. +4. Section 4 explains how ACP carries content and what each harness does with each content type. +5. Section 5 compares two ways to deliver a file to the model. +6. Section 6 surveys how other tools handle attachments: Zed, opencode, and others. +7. Section 7 covers records and replay: what exists and the direction for the runner to rebuild + context. +8. Section 8 covers assistant-produced files: what is declared and what is missing. +9. Section 9 explains the capability flags: where they came from and why they were never used. +10. Section 10 covers the front-end adapter and its current limits. + +--- + +## 1. Current state of the code + +The attachment path is wired from the chat box all the way to the runner. It breaks at the last +step before the harness. + +| Step | Where | Carries the file? | +| --- | --- | --- | +| Chat box picks, pastes, drops, previews the file | `web/oss/src/components/AgentChatSlice/assets/files.ts:20` (`fileToPart` reads the file into a `data:` URL) | Yes, as a base64 `data:` URL in an AI SDK `file` part | +| Request body carries the message parts | `web/packages/agenta-playground/src/state/execution/agentRequest.ts` (history assembled from message parts) | Yes | +| Our SDK parses the part into a content block | `sdks/python/agenta/sdk/agents/adapters/vercel/messages.py:97` (`_part_to_blocks`, the `file` case) | Yes, into a `ContentBlock` of type `image` or `resource` | +| Our SDK writes the wire payload | `sdks/python/agenta/sdk/agents/utils/wire.py` (`ContentBlock.to_wire`) | Yes, base64 intact | +| Runner receives the request | `services/runner/src/server.ts` | Yes, intact | +| **Runner hands the turn to the harness** | `services/runner/src/engines/sandbox_agent/run-turn.ts:742` | **No. It sends `[{ type: "text", text: turnText }]` only.** | + +The single real call that sends a turn to the harness is `env.session.prompt(...)` at +`run-turn.ts:742`. It is hard-coded to one text block. The text it sends comes from +`messageText()` at `protocol.ts:573`, which keeps only blocks whose `type` is `"text"` and joins +them. Everything that is not text is discarded there. + +Three concrete failures follow from this: + +1. **Image and text together.** The model receives only the text. When the conversation is later + replayed from saved history, the image is rendered as the literal string `"[image]"` + (`transcript.ts:176`). +2. **Image with no text.** The turn is rejected. `resolvePromptText()` (`protocol.ts:585`) returns + an empty string, and the run fails with "No user message to send" from `run-plan.ts`. +3. **No warning to the person.** The paperclip button in the chat box is disabled, but pasting or + dragging a file still works and still reaches the dead end. The person is never told the file + will be ignored. + +### Side effects that a reference-based design removes + +- **Resending the whole conversation every turn.** The front end resends the full message history + on each turn (`agentRequest.ts`, the `history` array). A five megabyte image becomes about + seven megabytes of base64 and is re-uploaded on every later turn. The cost grows with the + square of the conversation length. +- **Filling the browser's local storage.** Saved messages hold the base64 inline + (`web/oss/src/components/AgentChatSlice/state/sessions.ts:31`, which notes "a conversation with + large files can approach the localStorage quota"). When the roughly five megabyte browser quota + is exceeded, `writeMessagesWithQuotaGuard` (`sessions.ts:409`) drops other sessions' saved + transcripts to make room. +- **Bloating the traces.** The Python agent span keeps the messages on purpose and there is no + length cap on exported span attributes, so a base64 image lands verbatim in a trace. + +Once the saved history and the traces carry a small reference instead of the bytes, all three go +away, because the bytes are stored once in the object store and never travel in the history again. + +--- + +## 2. The mounts system + +### What a mount is + +A mount is one named storage area in an S3 object store. Each mount has its own set of object +keys under a prefix and can hand out short-lived, prefix-scoped credentials so a client can read +and write its files directly. The code lives in `api/oss/src/core/mounts/service.py` (the service), +`api/oss/src/apis/fastapi/mounts/router.py` (the general routes), and +`api/oss/src/apis/fastapi/sessions/router.py` (the session-scoped routes, class +`SessionMountsRouter` near line 863). + +The object key layout is +`[/]mounts///` (`service.py:358`, `_storage_key`). The +`` segment is the tenant partition of the key. **It is not a "project mount."** There +is no project-scoped mount kind in the system today. Every mount in practice carries a session +scope or an agent scope. In `MountQuery` +(`api/oss/src/core/mounts/dtos.py:25`) both `session_id` and `agent_id` are optional, so the type +itself does not forbid a mount with neither scope. The current code never creates one, so the +accurate claim is that no project-scoped mount kind exists in practice, not that the type makes one +impossible. So in a +storage key like `mounts///photo.png`, the first segment only +says which tenant owns the bytes; the mount itself is still session-scoped. + +There are three kinds of mount today: + +- **A session `cwd` mount.** Backs the agent's working directory for one session. Created on first + use, keyed on the session id (`get_or_create_session_cwd`, `service.py:489`). Mounted into the + sandbox so the agent sees it as its working folder. +- **Extra named session mounts.** A session can have more than one mount, each with its own name + and its own storage prefix (`get_or_create_session_mount(session_id, name=...)`, `service.py:405`). + The comment there says any name other than `cwd` is "an additional session-scoped mount sharing + the same shape with its own prefix." This existing function is what the design uses to create a + separate attachments mount. +- **An agent mount.** Durable across all sessions of one agent. This is the `agent-files` folder + the agent sees. It exists because its lifecycle is different (it outlives the session). It is + mounted into the sandbox beside `cwd` and linked in as `agent-files/` + (`services/runner/src/engines/sandbox_agent/agent-mount.ts:16`, `AGENT_FILES_LINK_NAME`). + +The `cwd` mount is the agent's writable working folder. Its own README, written for the agent, +says: "Concurrent runs share this folder, so the last writer wins for each file" +(`agent-mount.ts:23`). In other words, the agent can overwrite or delete anything in `cwd`. That +fact drives a key design choice (see [design.md](design.md), the decision on where originals live). + +### Every mounts endpoint, who may call it, and what it does + +All of these sit behind the API's normal authentication. A caller is a person (through the web +app, carrying a JWT) or a program (carrying an API key), and either resolves to a user id and a +project id in `request.state`. On top of that, each route checks a role-based permission with +`check_action_access`. The runner is not a special caller: when it needs storage credentials, it +calls the sign endpoint with the same authorization the invocation carried. + +General mounts routes, registered in `MountsRouter.__init__` (`router.py:120`): + +| Route | Method | Permission | Arguments | What it does | +| --- | --- | --- | --- | --- | +| `/mounts/` | POST | `EDIT_MOUNTS` | mount body | Create a mount. | +| `/mounts/query` | POST | `VIEW_MOUNTS` | filters (`session_id`, `agent_id`) | List mounts. | +| `/mounts/{id}` | GET | `VIEW_MOUNTS` | mount id | Fetch one mount. | +| `/mounts/{id}` | PUT | `EDIT_MOUNTS` | mount body | Edit a mount. | +| `/mounts/{id}/sign` | POST | `USE_MOUNTS` | mount id | Mint short-lived S3 credentials scoped to this mount's prefix. | +| `/mounts/agents/sign` | POST | `USE_MOUNTS` | `artifact_id`, `name` | Get-or-create the agent mount and sign it. | +| `/mounts/agents/query` | POST | `VIEW_MOUNTS` | JSON body with `artifact_id`, `name` | Fetch the agent mount without creating it. The arguments arrive in a JSON body, not as query parameters. | +| `/mounts/{id}/archive` / `/unarchive` | POST | `EDIT_MOUNTS` | mount id | Soft archive or restore. | +| `/mounts/{id}/files` | GET | `VIEW_MOUNTS` | `path`, `read`, `order`, `limit`, `depth`, `git_aware`, `include_gitignored`, `with_counts` | List files or read one file's text. `with_counts` adds child counts to the listing. | +| `/mounts/{id}/files` | PUT | `EDIT_MOUNTS` | `path`, raw body | Write one file from a raw body. | +| `/mounts/{id}/files` | DELETE | `EDIT_MOUNTS` | `path` | Delete a file or a folder. | +| `/mounts/{id}/files/folder` | POST | `EDIT_MOUNTS` | `path` | Create an empty folder marker. | +| `/mounts/{id}/files/upload` | POST | `EDIT_MOUNTS` | multipart `file`, `path` | Upload a file through the API. | +| `/mounts/{id}/files/download` | GET | `VIEW_MOUNTS` | `path` | Download exact bytes as a binary response. | +| `/mounts/files/export` | POST | `VIEW_MOUNTS` | list of mounts | Stream a zip of many files ("download all"). | + +Session-scoped routes, registered in `SessionMountsRouter.__init__` (`sessions/router.py:863`). +These require the session permission **and** the mounts permission together (`_check` loops over +both, `sessions/router.py:925`): + +| Route | Method | Permissions | Arguments | What it does | +| --- | --- | --- | --- | --- | +| `/sessions/mounts/` | GET | `VIEW_SESSIONS` + `VIEW_MOUNTS` | `session_id` | List this session's mounts. | +| `/sessions/mounts/query` | POST | `VIEW_SESSIONS` + `VIEW_MOUNTS` | `session_id` | List with a body query. | +| `/sessions/mounts/sign` | POST | `RUN_SESSIONS` + `USE_MOUNTS` | `session_id`, `name` (default `cwd`) | Get-or-create the named session mount and sign it. | +| `/sessions/mounts/{id}/files/upload` | POST | `EDIT_SESSIONS` + `EDIT_MOUNTS` | multipart `file`, `path` | Upload a file into a session mount through the API. | +| `/sessions/mounts/{id}/files/download` | GET | `VIEW_SESSIONS` + `VIEW_MOUNTS` | `path` | Download exact bytes. | + +There is **no move or rename endpoint.** Moving a file is done by writing to the new path and +deleting the old one. Renaming works the same way. + +### Two ways bytes get in and out of a mount + +There are two separate mechanisms, and the difference matters for enforcing that a file cannot be +changed. + +- **Through the API.** The upload, write, download, and delete routes read or write bytes + server-side. Each one enforces its role permission. A person who has `VIEW` but not `EDIT` + cannot write. +- **Directly against S3 with signed credentials.** The sign endpoints hand out short-lived S3 + credentials scoped to the mount's prefix (`sign_mount_credentials`, `service.py:688`). The holder + can then read and write objects under that prefix directly, without going back through the API. + This is how the runner mounts a folder into the sandbox: it signs the `cwd` mount, and geesefs + uses those credentials to present the mount as a writable directory. + +Two consequences for this project: + +1. **The runner can read a mount's bytes without mounting it into the sandbox.** The signed + credentials are ordinary S3 credentials. The runner can do a plain object GET with them. It does + not have to make the mount appear as a folder for the agent. This is what lets an attachments + mount stay out of the agent's view while the runner still reads the original to build the model + turn. +2. **Server-side "cannot be changed" is achievable.** If a mount's bytes only ever enter through + the upload API route, and the system never hands out write-capable signed credentials for that + mount, then nothing but a deliberate API write can change it. That is what makes an attachment + original immutable in fact, not just by convention. Today `sign_mount_credentials` + does not offer a read-only variant, so a read-only scope would be a small addition to the + short-lived storage credentials (STS) signing call. See the hardening step in [plan.md](plan.md), + Stage 3. + +### The write path, precisely + +Three facts about how a write actually happens today shape the design, so they are stated exactly. + +- **A signed credential is read-write.** `sign_mount_credentials` (`service.py:688`) has no + read-only variant, so any credential it issues grants write access to the whole mount prefix. + This matters because it means immutability cannot rest on signed credentials. As long as the + system signs a mount at all, the holder of that signature can write it. Immutability has to come + from not signing the mount and only writing it through the permission-checked upload route. +- **The upload route buffers the whole file in memory.** The API upload helper reads the entire + file into memory before writing it (`api/oss/src/apis/fastapi/mounts/utils.py:45`). This matters + because a large attachment would hold its full size in the API process. A streaming write path is + a later addition once large files are in play. +- **`write_file` overwrites silently.** `write_file` (`service.py:1297`) replaces any object + already at the same path without a check or an error. This matters because "the original is + create-only" cannot be assumed from the storage layer. The API has to enforce create-only + semantics itself, by refusing a write to a path that already holds an attachment original. + +### How the Files drawer uses these today + +The Files drawer is the `DriveExplorer` component (`web/oss/src/components/Drives/DriveExplorer.tsx`). +It shows a session's files as one tree, folding the `cwd` mount and the `agent-files` mount +together. It tags each top-level node by where it came from and shows the origin label only when +the tree mixes origins (`driveHasMixedOrigins` in `ContextRail.tsx`). For file management it calls +the same mount routes above: list to show the tree, upload to add a file, delete to remove one, +and write to save an edit. A "Shared by you" origin for attachments is a new third origin over +this existing tagging, not a new kind of panel. + +The service already hides runner-internal paths and dotfiles from the listings +(`_is_internal_mount_path` at `service.py:154`, `_is_hidden_path` at `service.py:190`), so an +internal attachments area would not clutter the normal view. + +### Responsibilities across a message-attachment flow + +Reading the endpoints above against the three flows that would use them: + +- **Managing files in the drawer** (upload, edit, delete, move). The front end owns this. It calls + the upload, write, delete, and list routes directly. The API enforces permissions and writes the + bytes. This flow exists today for the `cwd` and agent mounts. +- **Attaching a file to a message.** The front end uploads the file once (to an attachments mount) + and then sends only a reference on the wire. The API stores the bytes and later serves them to + the runner. The front end owns the upload and the reference; the API owns storage; the runner + owns turning the reference into model content. +- **Materializing a working copy for the agent.** The runner owns this. It reads the original from + the mount and writes a copy into the agent's `cwd` so the agent's tools can open it. + +--- + +## 3. Kinds of files and how models handle them + +This is the section that answers "which files are special and which are just files." The key idea: +a model has a small set of input types it perceives directly through its own encoder, and +everything else is just bytes that the agent must open with a tool. + +### Native modalities: the model perceives them directly + +For these, the bytes must be delivered to the model in a specific content slot. The model has a +built-in path that turns them into something it understands (an image encoder, an audio encoder, a +document parser). You cannot get this perception by writing the file to disk and asking the agent +to "read" it, because the model's own perception path is only reached through the message content, +not through a tool that returns text. + +- **Images** (PNG, JPEG, GIF, WebP). Perceived by Claude, the GPT-4o and GPT-5 family, and Gemini. +- **Audio** (for example WAV, MP3). Perceived natively by some models (Gemini, and the GPT audio + models). There is no reliable way to make a model "hear" audio by having the agent read the file, + so audio must be delivered as a native audio input. +- **PDF and other documents as a document input.** Claude accepts a PDF as a document content block + and reads both its text and its page images. This is a native path distinct from plain text. +- **Video.** Gemini perceives video natively. Claude and the current GPT chat models do not. Video + is out of scope for this project (see [scope.md](scope.md)). + +"Document" is not a modality of its own. Whether a PDF, a CSV, or a spreadsheet is perceived +natively depends on three things at once: the specific model, the harness adapter that hands the +bytes to that model, and the exact format. So document support is a per-format, per-adapter, +per-model matrix, not a single flag you can turn on. Section 4 shows how far apart the two adapters +we run actually are on this point. + +### Everything else: files an agent reads with tools + +For these, there is no special model path. The right handling is to place the file on the agent's +working directory and let the agent's tools open it, possibly after converting it. + +- **Plain text, source code, CSV, JSON, Markdown.** The agent reads them with its file tools. Small + text can also be inlined directly into the prompt as text, which every harness supports. +- **Office documents** (Excel, Word). The agent converts them first (for example with a library + like openpyxl for a spreadsheet) and then reads the result. The raw binary is not a native model + input for Claude or GPT. +- **Archives** (zip). The agent extracts them and works on the contents. +- **CAD and other binary formats, and folders.** The agent runs whatever tool understands them. + These are never native model inputs. +- **Very large files of any kind.** Even a native modality has size limits (below). Past those, a + file is handled as bytes on disk, not as a native input. + +### An important subtlety: a non-vision model can still use an image + +An agent whose model cannot see images is not useless with an image. Two separate things are +going on: + +1. **Model perception.** Whether the model itself sees the picture. This needs the native image + path and a vision-capable model. +2. **Tool use over the file.** Whether the agent can run a program on the image bytes, for example + to resize it, read its metadata, or feed it to an image library. This needs only the file on + disk. It works even when the model cannot see the image. + +So the file should be placed on disk for the agent regardless of whether the model can perceive it, +and the native delivery to the model should happen only when the model supports that modality. This +is exactly the "two outcomes at once" goal from [context.md](context.md). + +### Provider limits (verify current numbers before relying on them) + +From Anthropic's vision documentation and the Messages API limits: + +- **Whole request.** The Messages API caps a request at about 32 megabytes total. + [Anthropic vision docs](https://platform.claude.com/docs/en/build-with-claude/vision). +- **Images.** The per-image size limit is deployment-dependent, so do not treat any single number + as fixed. The published guidance is to keep each dimension under about 1500 to 2000 pixels and to + send a small number of image-and-document blocks per request to stay within limits. Confirm the + current per-image and pixel limits against the vision docs before relying on them. + [Anthropic vision docs](https://platform.claude.com/docs/en/build-with-claude/vision). +- **PDF documents.** The current documentation gives PDF support up to 600 pages, dropping to 100 + pages for the 200k-context models, within the request size limit. These numbers change, so verify + them. + [Anthropic PDF support docs](https://platform.claude.com/docs/en/build-with-claude/pdf-support). + +Gemini perceives images, audio, and video natively, each with its own per-request limits. +[Gemini image understanding](https://ai.google.dev/gemini-api/docs/image-understanding), +[audio understanding](https://ai.google.dev/gemini-api/docs/audio), +[video understanding](https://ai.google.dev/gemini-api/docs/video-understanding). + +The important design takeaway is not the exact number, which changes, but the shape: there are hard +per-request and per-file limits, they differ by provider, and our front-end limits should be +derived from the selected model's real limits rather than from the arbitrary placeholders we have +today (see section 10). + +--- + +## 4. How ACP carries content, and what each harness does with it + +### The ACP content model + +ACP is the Agent Client Protocol, an external standard published by Zed Industries. The runner +speaks it to the harness through two adapter packages it pins in `services/runner/package.json`: +`@agentclientprotocol/claude-agent-acp` at version `0.58.1` for the Claude harness, and `pi-acp` at +version `0.0.29` for the Pi harness. These two adapters are the ones we actually run. The +specification is at `agentclientprotocol.com`. + +ACP defines these content block types for a prompt turn +([ACP content spec](https://agentclientprotocol.com/protocol/v1/content)): + +- **Text.** Field `text` (required). Every agent must support text. +- **Image.** Fields `data` (base64, required), `mimeType` (required), `uri` (optional). Requires the + `image` prompt capability. +- **Audio.** Fields `data` (base64, required), `mimeType` (required). Requires the `audio` prompt + capability. +- **Embedded resource.** Field `resource` (required), which is either a text variant (`uri` and + `text` required) or a blob variant (`uri` and `blob` base64 required). The spec calls this "the + preferred way to include context in prompts." Requires the `embeddedContext` prompt capability. +- **Resource link.** Fields `uri` (required) and `name` (required), plus optional `mimeType`, + `title`, `description`, `size`. This is a pointer to a resource the agent may fetch, not content + the model is guaranteed to perceive. + +The decisive fact for this project: for an image, an audio clip, or an embedded document, the bytes +are **required** and travel inline as base64 in the turn. There is no content type that hands the +model a bare URL and guarantees the model reads it. A resource link is only a pointer; whether the +model ever sees the file depends on the agent choosing to fetch it with a tool. + +Prompt capabilities are three flags the agent advertises at start-up: `image`, `audio`, and +`embeddedContext`. Each gates the matching content type +([ACP content spec](https://agentclientprotocol.com/protocol/v1/content)). + +### What our runner does with capabilities today + +The runner probes the harness's capabilities through the sandbox-agent library, which surfaces the +ACP capabilities as an `AgentInfo.capabilities` object. The runner maps that into its own +`HarnessCapabilities` shape (`capabilities.ts:68`, `mapCapabilities`), which includes `images` and +`fileAttachments` flags (`protocol.ts:281`). But it only ever branches on the tool-related flags +(`mcpTools`, `toolCalls`); the `images` and `fileAttachments` flags are assigned and then read +nowhere (a grep for `.images` in `services/runner/src` finds only the assignment at +`capabilities.ts:86`). So the runner receives the capability values but never uses them. This +project makes the runner enforce them. More on the history in section 9. + +### What the pinned adapters actually do + +This is where "the model perceives it" meets reality, because each adapter maps ACP content to its +own model API in its own way. Rather than infer this from documentation, the following comes from +reading the two adapter packages the runner pins. + +**The Claude adapter (`@agentclientprotocol/claude-agent-acp` 0.58.1).** The adapter file is +`node_modules/@agentclientprotocol/claude-agent-acp/dist/acp-agent.js`. + +- It advertises `promptCapabilities: { image: true, embeddedContext: true }` (near line 418). It + does **not** advertise audio. +- In its prompt conversion (near lines 3900 to 3925): an ACP image becomes a native Claude image + block; a `resource_link` becomes a text link; a text resource becomes a URI link plus a + `` text block; and a blob resource is **explicitly ignored**, with the code + comment "Ignore blob resources (unsupported)." +- The consequence is blunt. A PDF sent as an ACP blob resource never reaches Claude at all through + this adapter. It is dropped in the conversion. + +**The Pi adapter (`pi-acp` 0.0.29).** The adapter file is `node_modules/pi-acp/dist/index.js`. + +- It advertises `promptCapabilities: { image: true, audio: false, embeddedContext: + process.env.PI_ACP_ENABLE_EMBEDDED_CONTEXT === "true" }` (near line 1696). Audio is off, and + embedded context is off unless an environment variable turns it on. +- In its prompt conversion (near lines 1470 to 1495): images pass through as image inputs; a text + resource is inlined into the message text as `[Embedded Context] uri (mime)\n`; and a blob + resource becomes only the line `[Embedded Context] uri (mime, N bytes)` with no content at all. +- The consequence is that a document sent as a blob to the Pi adapter reaches the model as a byte + count, not as the document. + +### What the adapters support today + +Three consequences follow, stated plainly. + +- Image delivery works end to end on both adapters. An inline ACP image reaches the model as a real + image it perceives. +- Native audio is unsupported by both pinned adapters today. Neither advertises the audio + capability, so there is nothing to deliver audio into. +- Native document delivery does not work today. The Claude adapter drops a blob resource entirely, + and the Pi adapter renders it as a byte count. A PDF sent as a blob reaches neither model as a + document. + +Resource links are handled as pointers by both. The agent may or may not fetch them. + +### An important limit on the "write the file to disk and let the harness read it" idea + +Claude Code's own Read tool does not reliably deliver an image file to the model as vision input. +Feature requests have asked for exactly that behavior. One of them, +[claude-code issue #35866](https://github.com/anthropics/claude-code/issues/35866), is closed as +"not planned"; another, [#30925](https://github.com/anthropics/claude-code/issues/30925), is still +open. So even if we place an image on the agent's disk and the agent runs Read on it, the model is +not guaranteed to see the picture. This is strong evidence that for model perception we cannot rely +on the disk-plus-Read path; the inline content path is the reliable one. The disk copy remains +valuable for tool use over the file, which is the separate outcome. + +--- + +## 5. Two ways to deliver a file to the model, compared + +The design has to choose how the file reaches the model. There are two mechanisms, and they are not +interchangeable. This comparison answers the question of why the design delivers inline ACP content +rather than only writing the file into the sandbox and mentioning its path. + +**Mechanism A: inline ACP content blocks.** The runner reads the file and puts its bytes into an +ACP image, audio, or embedded-resource block in the turn. The harness maps that to the model's +native content path. + +- Gives the model real perception of the file. For images this is confirmed to work through both + pinned adapters (see section 4). +- Requires the bytes at prompt time, and requires the harness to advertise the matching capability. +- Is the only path that could ever work for audio, because there is no tool-read fallback for + audio at all. Note that neither pinned adapter supports native audio today (section 4), so audio + is blocked on adapter work even though inline delivery is its only possible route. + +**Mechanism B: write the file into the sandbox and mention its path in the prompt text.** The runner +writes the file into the agent's working directory and the prompt says something like "the file is +at attachments/photo.png." + +- Lets the agent's tools open, convert, or edit the file. This is real value for non-native + formats. +- Does **not** reliably give the model perception. Whether the model ever sees the picture depends + on the harness having a tool that turns a file read into a vision input, and Claude Code's Read + tool does not do that reliably today (section 4). +- Is the natural path for formats that are not native model inputs anyway (a zip, a CAD file), where + perception is not even the goal. + +The conclusion the design draws: use both, for different purposes. Deliver native modalities inline +(Mechanism A) so the model perceives them, and also place the file on disk (Mechanism B) so the +agent's tools can work on it. This is not redundant. The two mechanisms serve the two separate +outcomes from [context.md](context.md): perception and tool use. + +--- + +## 6. How other tools handle attachments + +Looking at how mature tools handle this gives us patterns for the flow and for who owns each step. + +### Zed + +Zed's editor is the client in ACP. When a person attaches or pastes an image in Zed's agent panel, +Zed sends it as an ACP image content block, which the Claude Code adapter turns into a native Claude +image block. File mentions become resource references. So Zed's own behavior confirms the inline +content path for images. +[Zed on Claude Code via ACP](https://zed.dev/blog/claude-code-via-acp). + +### opencode + +opencode is a terminal-first coding agent. Its attachment behavior +([opencode attachments docs](https://v2.opencode.ai/attachments)): + +- **How files are attached.** By an attach action, by paste, or by drag and drop. In the terminal a + person references a file with `@filename`, which does a fuzzy search in the working directory and + inserts the file's content into the conversation. +- **How they reach the model.** Inline in the prompt, not as a path reference. Text files are decoded + and inserted as text. Images (PNG, JPEG, GIF, WebP) are passed as the provider's native image + input. SVG is treated as text. PDF, audio, and video are not sent to the model at all. +- **Where the bytes live.** opencode does not describe durable storage. Files are materialized at + prompt time from the path or the inline data URL. +- **Who owns what.** The client collects the file and applies a client-side size limit (up to 20 + mebibytes). The server validates the path and enforces a decoded-size limit per attachment. The + model provider applies its own format and size limits. + +The patterns worth taking from opencode: attachments are delivered inline to the model; only a small +set of image formats get native handling; everything else is text or is refused; and there is a clear +split where the client collects and pre-limits, the server validates, and the provider has the final +say on limits. + +### Others + +Claude Code supports `@`-file mentions that pull a file's content into the context, similar to +opencode. The general industry pattern across these tools is: images go as native image content; +plain text and code are inlined as text; unusual binaries are either refused or handled as files an +agent tool must open. None of them rely on handing the model a bare link and trusting it to fetch. + +--- + +## 7. Records and replay + +### What exists + +Our system already keeps a durable, per-session log called **records**. The runner posts every +agent event to the API's record-ingest endpoint, independent of whether any browser is listening +(`services/runner/src/sessions/persist.ts`, whose header comment describes the "producer-driven" +model and the `POST /sessions/records/ingest` endpoint). Each record is tagged with a +`record_source` of `"agent"` for engine events or `"user"` for the inbound user turn. The API has a +records sub-router with an ingest route and a query route (`RecordsRouter` at +`api/oss/src/apis/fastapi/sessions/router.py:435`; the file header lists +`POST /sessions/records/query`). + +Separately, the runner rebuilds a conversation from message history when it starts a session that is +not already warm. This "cold replay" turns each past content block into a text description +(`transcript.ts`), which is why a past image shows up as the literal `"[image]"` there today. + +### Records are text-only today + +The record path carries text and nothing else right now. The runner persists the inbound user +record as text (`services/runner/src/server.ts`, near line 1009), and the record event schema +allows only the shape `{ type: "message", text }` (`services/runner/src/protocol.ts`, near line +325). There is no field in a record for an attachment. On a warm turn the runner sends only the +latest user text; on a cold start it flattens the whole transcript (`run-turn.ts`, near line 146), +and a past image becomes the placeholder string `"[image]"` (`transcript.ts:176`). + +Two things follow for this project. First, carrying an attachment reference in a record requires +extending the record schema, because the schema has no place to put one today. Second, cold replay +currently reduces every past attachment to a placeholder string, so a rebuilt conversation loses +the attachments entirely. This is why the plan makes the record-schema change part of the first +release rather than a later addition: without it, a reference cannot survive in the durable log at +all. + +### The direction + +The product direction is: once the "sessions" work lands, the front end will +send only the last message on each turn instead of resending the whole history. The runner will own +rebuilding the conversation from records when a session cannot be resumed warm (for example after a +crash). This changes who is responsible for history from the front end to the runner. It also means +the reference-based attachment design fits the direction cleanly, because a reference is exactly what +a record should store for a shared file: the record points at the durable original rather than +carrying its bytes. + +### The tracking issue + +This direction is tracked in +[#5443, "(feat) Runner should rebuild session context from records when a session cannot be resumed warm"](https://github.com/Agenta-AI/agenta/issues/5443). +The related warm case is +[#5384, "(feat) Warm sessions should hold client tool calls open instead of replaying the turn"](https://github.com/Agenta-AI/agenta/issues/5384): +#5384 avoids a replay when the session is still alive, while #5443 covers rebuilding from records +when it is not. The rebuild itself is outside this project's scope; what this project contributes is +that records will carry small references to durable originals, which is what a faithful rebuild +needs. + +--- + +## 8. Assistant-produced files + +The runner protocol declares an event type for a file the agent produces: +`{ type: "file"; url: string; mediaType: string }` (`protocol.ts:379`), which the comment says maps +to a Vercel `file` part on the way to the front end. But **nothing emits it.** A search across the +runner source for that event type finds only its declaration, no producer. The protocol defines the +event, but no runner code emits it. + +How this normally works elsewhere: an agent writes the file into its working directory and then +mentions it in its answer, and the client shows a link when it recognizes a file reference. In Zed +and opencode the client renders a file the agent wrote by reading the working directory, not by a +special "here is a file" protocol event. For our product this means assistant-produced files are a +separate piece of work: making the agent write to `cwd` and surfacing the new file in the drawer is +already close to working (the drawer lists `cwd`), while a first-class "assistant returned this file" +inline chip is the not-yet-built part. This project treats assistant-produced files as out of scope +(see [scope.md](scope.md)). + +--- + +## 9. The capability flags + +The neutral capability set, including `images` and `file_attachments`, was introduced with the +original agent-runtime protocol work (`sdks/python/agenta/sdk/agents/dtos.py`, class +`HarnessCapabilities`; introduced in commit `b9e62f99aa`, "feat(sdk): agent runtime ports, adapters, +tool resolution, and messages protocol"). The matching TypeScript shape is `HarnessCapabilities` in +`protocol.ts:281`. + +The intent is documented right in the code. The docstring on the Python `HarnessCapabilities` says +adapters should "deliver tools over MCP only when `mcp_tools` is set, skip image blocks without +`images`" (`dtos.py`). The tool half of that intent was built: `assertRequiredCapabilities` +(`capabilities.ts:169`) refuses a run that carries tools when the harness lacks `mcpTools` and +`toolCalls`. The image half was not built: there is no code that skips or gates image blocks on the +`images` flag, because no code sends image blocks at all yet (section 1). So the flags are not so +much abandoned as built ahead of the feature that would use them. This project is that feature. It +should reuse the same "fail loud when a required capability is missing" pattern the tool gate already +uses, rather than invent a new one. + +Note the difference between two capability vocabularies that are easy to confuse: + +- **ACP prompt capabilities:** `image`, `audio`, `embeddedContext`. External. Advertised by the + harness. These gate ACP content types. +- **Our harness capability flags:** `images`, `fileAttachments`, and the rest in + `HarnessCapabilities`. Ours. Derived from the probe. The design maps the ACP prompt capabilities + into a clear internal set of `images`, `audio`, and `documents` and gates on that. The mapping and + the exact contract across layers are in [design.md](design.md) and [plan.md](plan.md). + +--- + +## 10. The front-end adapter and its current limits + +### What the adapter parses + +Our SDK's Vercel adapter turns an inbound `file` part into a content block +(`sdks/python/agenta/sdk/agents/adapters/vercel/messages.py:97`). The rule is: if the media type +starts with `image/`, the block type is `image`; otherwise it is `resource`. The block carries the +`uri` (from the part's `url`), the `data` (base64, if present), and the `mime_type`. On the way back +out, both `image` and `resource` blocks render as a Vercel `file` part (`_block_to_parts`, +`messages.py:310`). + +So the content-block vocabulary our system produces from the front end is: `text`, `image`, +`resource`, `tool_call`, `tool_result`. There is **no audio block.** Audio, video, and every +document all collapse to `resource` today. Adding a first-class `audio` block is part of this work. + +The front-end helper `fileKind` (`web/oss/src/components/AgentChatSlice/assets/files.ts:11`) already +recognizes image, audio, video, and file, so the front end can tell audio apart even though the wire +cannot carry it as audio yet. + +### The current limits and where they came from + +The front-end attachment limits live in +`web/oss/src/components/AgentChatSlice/assets/attachments.ts:25` (`DEFAULT_ATTACHMENT_LIMITS`): + +- At most 5 files per message. +- At most 5 megabytes per file. +- Accepted types: `image/`, `application/pdf`, `text/`, and `application/json`. + +The file's own comment says these are placeholders meant to be replaced later with limits derived +from the selected model and harness capabilities ("they can later be derived from the selected model +/ harness capabilities ... That wiring is out of scope here; today everything reads the default"). +So the numbers are not grounded in any real model limit. Section 3 has the real provider limits they +should eventually be derived from. Today the front end sends files inline as base64 `data:` URLs +(`files.ts:20`, `fileToPart`), which is exactly the mechanism the reference-based design replaces. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/scope.md b/docs/design/agent-workflows/projects/agent-multi-modality/scope.md new file mode 100644 index 0000000000..e199170829 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/scope.md @@ -0,0 +1,72 @@ +# Scope + +## In scope + +This scope describes the recommended answer to open decision D11 (see [decisions.md](decisions.md)); +if the product owner chooses the smaller first release, the storage and findability items move out of +the first release. + +- A person can attach an image, an audio clip, or a document to an agent message. +- The model perceives images natively (this works end to end on both pinned adapters). Native audio + and native document delivery are goals, but they do not work on the adapters we run today and are + blocked on adapter work (see [research.md](research.md), section 4, and [decisions.md](decisions.md), + D8, open question 2 for audio, and the "Settled by evidence" section for document delivery). +- The agent can always work on the file, because a working copy is written into its working directory + regardless of whether the model can perceive the file. +- Every shared file stays findable: the original never changes, renders inline in the conversation, + and can be downloaded as exact bytes. Listing it under a "Shared by you" origin in the Files drawer + is Stage 3 work. +- Files travel as an opaque, server-issued `attachment_id` on the wire, in the saved history, and in + the traces, not as bytes and not as raw storage coordinates ([decisions.md](decisions.md), D10). +- Attaching a file never silently does nothing. An unsupported kind is attached as a workspace-only + file with a visible notice, and the runner fails a turn only on a contract violation, such as a + stale front end asking for a native block the harness cannot accept + ([decisions.md](decisions.md), D6). The exact promise of the first release is the open product + decision D11. + +## Out of scope (for now) + +- **Video.** No current model we target except Gemini perceives video, and ACP's content model has no + video type. Video needs new protocol and adapter support and waits until there is a clear need and a + supported harness. +- **Assistant-produced files as a first-class chip.** This project will not add an inline element for + files the agent produces, a first-class "here is a file" chip in its answer. The runner protocol + declares a `file` event type but nothing emits it (see [research.md](research.md), section 8). The + agent can already write files into its working directory and the drawer already lists them; the + missing piece is the inline chip, which is separate work. +- **Cross-session attachment reuse.** This project will not let a person pick a file shared in an + earlier session and reattach it. Attachments are session-scoped, so an "attach from a past session" + picker is a later feature. +- **Deduplication by content.** This project will not store one copy for identical uploads when the + same file is shared twice. This is a storage optimization, not a correctness need, so it waits. +- **Thumbnails and previews generated server-side.** The server will not generate preview images for + large files. This is a polish item that does not block the core flow. +- **Transcoding.** This project will not convert files between formats before delivery (for example + turning a video into frames); that work belongs with future video support. + +## Follow-ups + +- **Large pasted text becomes a file.** When a person pastes a very large block of text into the chat + box, it should not be sent through the prompt as text. It should be saved as a file and referenced in + the prompt, the same way an attachment is. Where exactly it is saved is likely answered by the + attachments mount this project builds. This is a natural extension of the reference model and is + listed as a follow-up rather than in the core scope. +- **A clean reference chip in the composer.** When a file is attached, the composer should show the + reference in a clear, readable way (a chip with the filename and kind) rather than a raw preview. + This is a presentation improvement on top of the core flow. +- **A read-only credential scope for the attachments mount.** The strongest form of the immutability + guarantee, noted in [design.md](design.md), decision D7, and [plan.md](plan.md), Stage 3. +- **Front-end limits derived from real model limits.** Replace the placeholder limits with values + computed from the selected model (see [research.md](research.md), section 3). Started in Stage 2 and + can be refined further as provider limits change. + +## Next steps + +- The product owner decides the open product question D11 (what the first release promises) on the PR. +- The document-delivery question is answered: documents do not arrive natively today (the Claude + adapter drops blobs and the Pi adapter renders a byte count), so documents are a Stage 2 blocker on + adapter work, not an open harness check (see [research.md](research.md), section 4). +- The runner-rebuilds-context-from-records direction is tracked in + [issue #5443](https://github.com/Agenta-AI/agenta/issues/5443); it stays outside this project's + scope (see [research.md](research.md), section 7). +- Decide whether Stage 0 ships on its own or folds into Stage 1. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/status.md b/docs/design/agent-workflows/projects/agent-multi-modality/status.md new file mode 100644 index 0000000000..874322d905 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-multi-modality/status.md @@ -0,0 +1,69 @@ +# Status + +This file records the project's current state. + +## State + +The design has one open product decision (D11, what the first release promises) and six open +implementation questions; see [decisions.md](decisions.md) for the list. The problem is verified +against code, the research covers the mounts surface, the modality taxonomy, the ACP content model +read from the two adapters we actually pin, and how other tools handle attachments, and every major +choice is written as options and a decision. Implementation has not started. + +The one-line problem: agent workflows are text-only at the model. A file travels intact from the +chat box to the runner and is dropped at the one call that hands a turn to the harness +(`services/runner/src/engines/sandbox_agent/run-turn.ts`, the `session.prompt` call, currently near +line 742). The image fix is entirely on our side of that call, because both pinned adapters deliver +an image natively. Native audio and native documents are different: neither pinned adapter supports +them today, so those modalities are blocked on adapter work. + +## Reading order + +See [README.md](README.md). In short: [context.md](context.md) for the plain story, +[research.md](research.md) for the findings, [design.md](design.md) for the design and its options, +[plan.md](plan.md) for the staged work, [scope.md](scope.md) for in and out, and +[decisions.md](decisions.md) for the log and open questions. + +## Stage tracker + +| Stage | Scope | State | +| --- | --- | --- | +| 0 | Close the silent-failure gap: refuse paste and drop while the feature is off | not started (optional) | +| 1 | First user-visible release: the attachment resource and storage, the record-schema extension, the runner's resolve-materialize-and-deliver seam for images, structured capability errors, and the minimum security and limits work | not started | +| 2 | Audio and documents: add the `audio` block, map audio and documents, retire the old capability names through an alias rollout, derive front-end limits | blocked on adapter work | +| 3 | Findability polish and cleanup: "Shared by you" origin, reference-counting cleanup refinement, read-only credential scope, verify the edit-then-find flow | not started | + +## Decisions taken + +See the decision log in [decisions.md](decisions.md) (D1 through D11, with D11 still open). In brief: +deliver inline for perception and on disk for tool use; keep the original in a dedicated session mount +out of the sandbox; two copies, an unchanging original and a disposable working copy; compute the +capability as the intersection of transport, adapter fidelity, and model, and gate in the composer and +the runner; never silently drop, attach an unsupported kind as workspace-only, and fail the turn only +on a contract violation; enforce immutability through a create-only upload route with no signed +credentials for the mount; carry an opaque server-issued `attachment_id` on the wire (D10); audio is +a goal but blocked on adapter work. + +## Open questions + +See [decisions.md](decisions.md). Three questions are settled by evidence (document delivery, the +working-copy path, the runner read path); decisions.md records them. The open +questions are: what the first release promises (D11, the product owner's decision); how native audio is +delivered at all (blocked on adapter work); the cold-replay budget numbers; when the old capability +names are removed across independently deployed components; the retention rules when a session is +archived or deleted; the exact media-type and validation matrix; and the cleanup refinement from a +time-to-live sweep to reference counting. + +## Next actions + +- The product owner decides the open product question D11 (what the first release promises) on the PR. +- Decide whether Stage 0 ships on its own or folds into Stage 1. +- The runner-rebuilds-context-from-records direction is tracked in + [#5443](https://github.com/Agenta-AI/agenta/issues/5443) (the warm-case counterpart is #5384). It + stays outside this project's scope. + +## Artifacts + +- [README.md](README.md) · [context.md](context.md) · [research.md](research.md) · + [design.md](design.md) · [plan.md](plan.md) · [scope.md](scope.md) · [decisions.md](decisions.md) +- Branch: `docs/agent-multi-modality` (docs only).