[docs] Plan agent multi-modality

docs/design/agent-workflows/projects/agent-multi-modality/README.md

@@ -0,0 +1,108 @@
+# Agent multi-modality
+
+This project lets a person share files with an agent and have two things happen at once:
+the model reads the file, and the agent can work on the file with its tools. Today neither
+happens. A person can attach an image or a document in the agent chat box, but the file is
+thrown away just before it reaches the model, so the model never sees it and the agent never
+gets it.
+
+A second goal sits beside the first. Every file a person shares must stay easy to find later,
+even after the agent has changed things. If you gave the agent a spreadsheet an hour ago, you
+should be able to open exactly that spreadsheet again, unchanged, without hunting through the
+agent's scratch files.
+
+This folder holds the research, the design, and the plan for closing that gap. The work is
+staged so that the first stages are small and safe and later stages add the harder modalities
+(audio and documents).
+
+## Glossary
+
+These terms appear across every file here. Each one names a real thing in our system and says
+where it runs.
+
+- **Agent.** A coding assistant that can read and write files and call tools, driven by a
+  large language model. In our product it runs inside a sandbox.
+- **Model.** The model is the large language model itself (for example Claude or a GPT model). The
+  model is what perceives an image or a document when the bytes are delivered to it in the right way.
+- **Harness.** The program that drives the model through one agent turn (for example Claude
+  Code or Pi). The harness owns the model conversation. We do not own the harness code.
+- **Runner.** Our own Node.js service that starts a sandbox, launches the harness inside it,
+  and passes each turn's message to the harness. The runner is where the file is dropped
+  today. It lives at `services/runner/`.
+- **Sandbox.** The isolated Linux environment the agent runs in. The agent's files and shell
+  commands all happen inside the sandbox.
+- **Session.** One ongoing conversation with an agent. A session has a stable id that the
+  front end creates before the first message is sent.
+- **Turn.** One request-and-reply within a session. The person sends a message, the agent
+  works and answers. That is one turn.
+- **cwd (working directory).** The folder the agent runs in inside the sandbox. It survives
+  across turns of the same session. The agent can create, change, and delete anything in it.
+- **Mount.** Our file storage unit. A mount is a named area in an object store (S3) with its
+  own keys and its own access credentials. The `cwd` is backed by a mount. Mounts live in the
+  API at `api/oss/src/core/mounts/`.
+- **geesefs / FUSE mount.** The technology that makes a storage mount appear as a normal
+  folder inside the sandbox. When a mount is "mounted into the sandbox," the agent sees it as
+  a directory it can read and write.
+- **ACP (Agent Client Protocol).** The message format the runner uses to talk to the harness.
+  It is an external standard published by Zed Industries. We cannot change it. It defines the
+  content types a turn can carry (text, image, audio, and so on).
+- **Content block.** One piece of a message in ACP: a text block, an image block, an audio
+  block, or a resource block. A turn is a list of content blocks.
+- **Materialize.** The runner materializes a file when it writes a copy of the stored original into
+  the agent's working directory so the agent's tools can open it.
+- **Records.** Records are the API's durable, per-session log of every event in a conversation. The
+  runner writes to it after every event.
+- **Drawer / drive / Files drawer.** The front-end panel that lists the files in a session. It
+  is a view over one or more mounts. In the code it is the `DriveExplorer`.
+- **Attachment resource / attachment_id.** The record the API returns when a file is uploaded. It
+  holds a server-issued `attachment_id`, the filename, the media type the server verified, and the
+  size. The API owns the storage location behind the id, so a client never sees it. The
+  `attachment_id` is the opaque token that travels on the wire, in the saved history, and in the
+  traces.
+- **Reference (attachment reference).** The `attachment_id` plus a little display metadata, sent in
+  a message in place of the file's bytes. It carries no bytes and no raw storage coordinates; it
+  points at the attachment resource above, which the runner resolves through the API.
+- **Front end / composer.** The front end is the browser app. The composer is its message editor:
+  the box where a person types, attaches files, and sends a turn.
+- **API.** Our FastAPI backend service. It owns storage, permissions, sessions, and records, and it
+  is the only component that talks to the object store with full rights.
+- **SDK.** Our Python library that sits between the API and the runner. It parses incoming messages
+  into content blocks and writes the wire payload the runner reads.
+- **Adapter.** The translation layer inside the harness that converts ACP content blocks into the
+  model provider's own API calls. We run two: `claude-agent-acp` (maintained by Zed) for Claude, and
+  `pi-acp` (from the Pi project) for Pi.
+- **Object store.** The S3-compatible storage service where mount bytes live.
+- **Wire.** The serialized request that travels from the front end through the API to the runner for
+  one turn.
+- **Native modality.** A kind of content (an image, audio, a document) that a model perceives
+  directly through its own encoder when the bytes are delivered inside the message content.
+  Everything else is a plain file that only tools can open.
+- **Capability.** A yes-or-no fact about what a layer supports, advertised or configured, used to
+  decide whether a modality can reach the model.
+- **Warm session / cold start.** A session is warm when its harness process is still running and can
+  continue the conversation. A cold start is when that process is gone and the runner must rebuild
+  the conversation before running the new turn.
+- **Cold replay.** The runner's reconstruction of the conversation during a cold start, today by
+  flattening past messages to text.
+
+## Read in this order
+
+1. **[context.md](context.md)**: What a person experiences today, why the file is lost, and
+   what "done" looks like. Start here if you want the plain story with no design.
+2. **[research.md](research.md)**: The findings the design rests on: what the mounts API can
+   do and who may call it, how different kinds of files are handled by models, how ACP carries
+   content and what each harness does with it, how other tools (Zed, opencode) handle
+   attachments, and the state of records and capability flags in our code. Every claim links to
+   a file and line or to a source.
+3. **[design.md](design.md)**: The design itself, written as options and decisions. For each
+   major choice it gives the alternatives, what breaks under each, the decision, and why. It
+   keeps every interaction diagram, each with the flow explained in words first.
+4. **[plan.md](plan.md)**: The staged implementation plan: what changes in each layer, who
+   owns each part, how each stage is tested, and how the plan holds up against the review
+   lenses (responsibilities, engineering practice, tradeoffs, scale, and fit with the rest of
+   the architecture).
+5. **[scope.md](scope.md)**: What is in, what is out, and the follow-up work with a sentence
+   each on why it waits.
+6. **[decisions.md](decisions.md)**: The decision log and the open questions, each with what
+   is unknown, why it matters, and how to settle it.
+7. **[status.md](status.md)**: Where the project stands and what happens next.

docs/design/agent-workflows/projects/agent-multi-modality/context.md

@@ -0,0 +1,92 @@
+# Context: what happens today and what we want
+
+This file tells the story in plain language. It has no design in it. If you want the evidence
+behind each claim, read [research.md](research.md). If you want the design, read
+[design.md](design.md).
+
+## What a person experiences today
+
+A person opens the agent chat, attaches a photo, and types "what is this?" The chat box
+accepts the photo. It shows a small preview. The message looks like it was sent with the
+image. Then the agent answers as if no image were there. It saw only the words "what is this?"
+and nothing else.
+
+If the person attaches only the photo and no text at all, the turn fails outright with an
+error that says there is no message to send.
+
+Nothing warns the person that this will happen. The attachment looks accepted and then quietly
+does nothing. That silent gap is the core problem.
+
+There is a second, separate product we already ship: the prompt playground, which is not an
+agent. That one does handle images and documents correctly. So agents are behind the rest of
+the product, and the difference is invisible until you notice the agent ignoring your file.
+
+## Why the file is lost
+
+The file travels correctly almost the whole way. The chat box reads the file into memory. The
+request carries it. Our SDK carries it. The runner, our service that drives the agent, receives
+it intact. Then, at the very last step, the runner builds the message it hands to the harness
+using only the text. It writes a single text block and drops everything else. The runner discards
+every content block that is not text at that one call.
+
+That last step was written when agents were text-only. It was never updated when attachments
+were wired up through the rest of the path. So the path is complete except for the final step.
+
+There is a related habit that makes this worse. On every new turn, the front end resends the
+whole conversation so far, including every earlier attachment, in full, as raw bytes encoded as
+text. A single five megabyte image becomes about seven megabytes of encoded text, and it is
+sent again on every following turn. The same full bytes are also saved in the browser's local
+storage and written into our tracing data. None of this is needed once a file is sent by
+reference instead of by value, and all of it strains the browser, the network, and the traces.
+
+## What we want to achieve
+
+We want three outcomes, and all three at the same time.
+
+**The model reads the file.** When a person shares an image, the model should see the image.
+When they share audio, the model should hear it. When they share a document, the model should
+read it. This is the part that is broken at the last runner step today.
+
+**The agent can work on the file.** Beyond the model reading it, the file should be present on
+the agent's own working directory so the agent's tools can open it, convert it, edit it, or run
+a program over it. Whether the agent changes the file is the person's call in each
+conversation, not a fixed rule.
+
+**The file stays findable.** A person must always be able to return to exactly what they
+shared, unchanged, and download it or see it again, even if the agent changes or deletes files in
+its working directory. The thing you gave the agent is a durable record, not a temporary
+scratch file that the agent might overwrite or delete.
+
+## What "done" looks like
+
+A person attaches a spreadsheet and asks the agent to chart the third column. The model reads
+the spreadsheet. The agent opens the same spreadsheet with its tools, computes the chart, and
+writes a new chart file. The person can still open the original spreadsheet, untouched, from the
+files panel, and can also see the new chart the agent produced. If the person attaches a kind of
+file the current model cannot handle, the chat box says so before the message is sent, instead
+of accepting it and then ignoring it.
+
+## The three hard constraints the design must respect
+
+**The protocol to the harness is external.** The runner talks to the harness in ACP, a
+standard we do not own. In ACP, for the model to actually perceive an image or audio, the bytes
+must be delivered inline in the turn. There is no way to hand the model a link and trust it to
+fetch the file. Storing the file in our object store does not remove this rule. It only removes
+the need to resend the bytes in the saved conversation history. At the moment the runner hands
+the turn to the harness, the bytes must be present. Details and sources are in
+[research.md](research.md).
+
+**We build on mounts, our existing file storage.** We already have a working file storage
+system with per-session storage, access controls, upload, download, and listing. The design
+reuses it rather than inventing new storage. What a mount is, what it can do, and who may call
+it are laid out in [research.md](research.md).
+
+**The harness adapters we run today deliver images, but not audio or documents.** The runner talks
+to the model through two adapter packages (one for Claude, one for Pi). Reading their code shows that
+both deliver an image to the model natively, but neither delivers native audio, and both fail to
+deliver a document (one drops it, the other turns it into a byte count). So images can be shipped
+now, while audio and documents are goals that wait on adapter work. The evidence is in
+[research.md](research.md), section 4.
+
+The full list of verified failure modes, with the exact files and lines, lives in
+[research.md](research.md) under "Current state of the code."