From 0db3f6ddc546daa2a185c189d085f4c26b0a8edb Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Mon, 20 Jul 2026 17:42:37 +0600 Subject: [PATCH] Add planning documents for progressive tool disclosure project --- .../progressive-tool-disclosure/README.md | 42 +++++++ .../progressive-tool-disclosure/context.md | 75 +++++++++++++ .../progressive-tool-disclosure/design.md | 101 +++++++++++++++++ .../progressive-tool-disclosure/plan.md | 77 +++++++++++++ .../progressive-tool-disclosure/research.md | 104 ++++++++++++++++++ .../progressive-tool-disclosure/status.md | 44 ++++++++ 6 files changed, 443 insertions(+) create mode 100644 docs/design/agent-workflows/projects/progressive-tool-disclosure/README.md create mode 100644 docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md create mode 100644 docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md create mode 100644 docs/design/agent-workflows/projects/progressive-tool-disclosure/plan.md create mode 100644 docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md create mode 100644 docs/design/agent-workflows/projects/progressive-tool-disclosure/status.md diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/README.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/README.md new file mode 100644 index 0000000000..0bdcf042f6 --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/README.md @@ -0,0 +1,42 @@ +# Progressive tool disclosure (playground build kit) + +Status: PLANNING — design workspace only, no implementation. +Date: 2026-07-20 + +The playground advertises ~13 platform-op tool schemas to the model on every turn, before the +model has done anything — ~15K prompt tokens on a bare "hi", plus every extra always-on tool is +a "wander target" that derails runs. Skills already avoid this (on disk, only name+description +in the prompt, body loaded on demand). This project applies the same idea to the platform ops: +advertise a small **discovery meta-toolset** and load a full op schema only when the model asks. +The lazy layer lives at the runner's `advertisedToolSpecs()` seam, so it is harness-agnostic +(Pi + Claude) and touches no committed agent config. + +## Decisions (locked) + +- **Seam = runner advertisement layer.** Intercept the advertised projection at + `services/runner/src/tools/public-spec.ts` (`advertisedToolSpecs`), consumed at exactly two + sites (`pi-assets.ts`, `environment.ts`). Execution + permission stay below it, unchanged. +- **Skills are out of scope.** Already progressive. +- **Op-catalog contents / overlay-cleanup is out of scope.** This work makes *any* op set cheap + to carry; it does not decide which ops belong. +- **Playground overlay only.** No change to any saved/committed agent. +- **Schema diet is complementary, not exclusive.** Shrinking the embedded agent-template schema + is worth doing regardless and lands as its own slice. +- **No commits during planning.** Implementation happens later on its own branch. + +## Deliverables + +- [context.md](context.md) — problem, scope, non-goals, product language, success criteria. +- [research.md](research.md) — how the current advertise/execute path works, with `file:line`, + and the numbered seams the plan pins. +- [design.md](design.md) — the meta-toolset, the execution + permission path, alternatives. +- [plan.md](plan.md) — the sliced implementation plan, each slice with an exit check. +- [status.md](status.md) — living source of truth: locked decisions, open questions, next action. + +## Intended outcome + +A playground author opens an agent and types a message. The model sees two small platform tools +(`agenta_ops` to list what it can do, `agenta_op` to fetch a schema and act) instead of a wall +of op schemas. A no-op turn costs a low-hundreds token constant instead of ~15K, and stays flat +as the catalog grows. Every op still runs with its exact self-targeting binding and approval +gate. Builds are cheaper and, per the internal-tools review, more reliable. diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md new file mode 100644 index 0000000000..a6fc3ef459 --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md @@ -0,0 +1,75 @@ +# Context + +## Problem + +Open a playground agent that shows "Tools: None" and type "hi". The turn costs ~15K prompt +tokens. Nothing the author did explains it: the cost is the playground **build kit**, which +injects ~13 platform-op tool schemas into the agent template and advertises every one of them to +the model on every turn. + +Two costs, not one: + +- **Tokens.** Measured (tiktoken `o200k_base`, 2026-07-17): the ops dominate. `test_run` (~6.5K) + and `commit_revision` (~5.8K) each embed the full ~5.5K-token agent-template delta schema; + `query_spans` ~1.3K; all ops together ~15.4K. Skills are NOT the cost (a ~68-token + announcement only). This is a ~5x tax on every turn, paid before the model is useful. +- **Reliability.** The internal-tools review + (`../builder-agent-reliability/tools-review/part-2-internal-tools.md`) found the same tools are + a *double* cost: "each unused tool is context cost plus a wander target (the capstone showed + extra visible tools derail runs)." Fewer advertised tools is a correctness win, not just a bill. + +It scales the wrong way: adding a catalog op is a one-line data change that ships to every +playground agent unconditionally, so every op we add makes every turn heavier. + +Skills already solved this exact shape — on disk, only name+description in the prompt, body +loaded on demand. The platform ops never got the same treatment. + +## Scope (this delivery) + +- The **runner advertisement layer** for playground platform ops: `advertisedToolSpecs()` in + `services/runner/src/tools/public-spec.ts`, consumed by the Pi and Claude delivery paths. +- A **discovery meta-toolset** (`agenta_ops` + `agenta_op`) that replaces the always-advertised + op schemas: list ops cheaply, fetch one schema on demand, execute against the private spec. +- A **schema diet** for the two heaviest op schemas (`commit_revision`, `test_run`) — + independently valuable, folded in as its own slice. +- A **token + reliability baseline** so before/after is measured, not asserted. +- A **one-line nudge** in the always-loaded `build-an-agent` skill so the model uses the + meta-toolset. + +## Out of scope for the first delivery + +- **Skills.** Already progressive; untouched. +- **Which ops belong in the overlay** (the build-kit-tools-cleanup debate). Orthogonal — this + work makes any op set cheap, which lowers the pressure to prune. +- **External tool discovery (`discover_tools`).** That discovers *Composio* tools to wire into an + agent; it stays as-is and is itself one of the ops we disclose. +- **User / gateway / code / client tools.** The POC targets platform ops (the measured cost). The + mechanism can generalize later. +- **Committed non-playground agents.** They advertise only what their author declared; no problem + today. +- **Dynamic real-name re-advertisement (M2).** Advertising a loaded op under its real name with a + schema-validated signature is a productionization option, evaluated after the POC — not built + here. + +## Product language + +- **Platform op** — an existing Agenta endpoint exposed to the agent as a tool, defined in the + code catalog `op_catalog.py` (e.g. `commit_revision`, `query_spans`). +- **Advertised spec** — the `{name, description, inputSchema, …}` projection the model sees; + distinct from the **private resolved spec** the runner executes from. +- **Discovery meta-toolset** — the small fixed set (`agenta_ops`, `agenta_op`) that stands in for + the op schemas: list, describe-on-demand, invoke. +- **Disclosure** — moving an op's full schema out of the prompt (paid every turn) into a tool + result (paid once, only when fetched). + +## Success criteria + +1. A no-op turn's platform-op prompt cost drops from ~15K to a low-hundreds constant, and stays + flat as ops are added to the catalog. +2. Capability parity: the build-an-agent lab loop (discover → wire → commit → test → schedule) + passes with the disclosed toolset. +3. No safety regression: self-targeting `$ctx` bindings and per-op approval/permission behave + exactly as today, verified per mutating op. +4. Reliability does not regress (target: fewer "wander" failures) on the lab matrix. +5. Cost of laziness is bounded: ≤1 extra round-trip per distinct op used; a schema fetched at + most once per op per conversation. diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md new file mode 100644 index 0000000000..be3ca67de2 --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md @@ -0,0 +1,101 @@ +# Design — the discovery meta-toolset + +## The core move + +Stop advertising N full op schemas every turn. Advertise a small fixed meta-toolset; keep the op +specs resolved but **private** in runner memory; move each op's full schema from the *prompt* +(paid every turn) to a *tool result* (paid once, only for ops actually used). + +Two model-facing tools (names illustrative): + +- **`agenta_ops(query?)`** — returns the op catalog as a compact list: `{op, one_line, + read_only}` per op, no input schemas. Optional `query` filters. A few hundred tokens, flat, + regardless of catalog size. This is the "what can I do on the platform" index. +- **`agenta_op(op, args?)`** — the generic invoker. With no `args` (or `mode:"describe"`) it + returns **one** op's full `inputSchema` as a tool result; with `args` it executes that op. + +The 13 op specs stay in `plan.toolSpecs` and in `toolSpecsByName`. Only the *advertisement* is +replaced. This is the pattern Claude Code uses on its own ~200 tools: names are listed, and a +`ToolSearch` step loads a schema only when needed. + +## Why this is safe by construction + +Execution and permission read the **private** spec (research seams 2–4). The invoker feeds the +target op's private `call` descriptor into the *unchanged* `direct.ts` path, so: + +- **Self-targeting is preserved.** `assembleBody` still fills `call.context` (`$ctx.*`) last, so + `commit_revision` still binds `$ctx.workflow.variant.id` and the model cannot retarget another + variant. The invoker never sees or forwards those fields — they were stripped at resolve time + and are re-applied below the invoker. +- **SSRF guard is preserved.** `directCallUrl` host-locks to the run's Agenta origin. +- **Approval is preserved — if the invoker gates per-op.** This is the one piece that is designed + in, not free (see below). + +## Execution path for `agenta_op(op, args)` + +The runner special-cases the invoker in its dispatch: + +1. **Resolve the target.** Look up `op` in `toolSpecsByName`. Unknown → tool error listing valid + ops (recoverable). +2. **Describe mode** (no `args` / `mode:"describe"`) → return the target spec's + `resolved input schema` as the tool result. No side effects, no approval. +3. **Execute mode** → build the permission **gate from the target op's spec** and run + `decide(gate, plan, stored)` (`permission-plan.ts:138`). `allow` runs; `deny` refuses; + `ask`/undecided pauses the turn and emits the normal `interaction_request(user_approval)` — + identical to calling the op directly today. Then run the target's `call` through + `assembleBody` → `directCallUrl` → `callDirect`, exactly as `executeRelayedTool` does now. +4. **Return** the endpoint response verbatim. + +The invoker is one ordinary advertised tool on both delivery paths (Pi native, Claude MCP), so no +harness-specific advertisement logic is needed. + +## Identifying the disclosure-eligible set + +A platform op is a `callback`-kind spec with a direct `call`; so is a `reference` (workflow) +tool. There is no explicit marker today (research seam 5). Two options: + +- **Heuristic (zero wire change).** Collapse every direct-`call` callback spec into the + meta-toolset; leave builtins, `client`, `code`, `gateway` (callRef), and MCP advertised as-is. + The playground overlay only injects platform ops (+ the two client tools), so this covers the + measured cost. Risk: an author who added a `reference` tool would see it disclosed too. Fine for + a flagged POC. +- **Marker (small wire add).** The platform resolver stamps a `source:"platform"` (or a + `disclosable` group tag) on the resolved spec; the runner collapses exactly that group. Precise, + and lets us disclose gateway/reference later on purpose. Costs a `protocol.ts` + `wire.py` + + golden change. + +Recommendation: heuristic for the POC to prove the numbers, marker before default-on. + +## The catalog summary (`agenta_ops`) + +Build `{op, one_line, read_only}` runner-side from the resolved specs already in memory — the +`description` and `read_only`/`permission` fields ride along; only `inputSchema` is dropped. Zero +new wire fields. (Alternative: thread a summary list from `op_catalog.py`; rejected for the POC as +extra plumbing.) + +## Discoverability + +The always-loaded `build-an-agent` skill (~68 tokens) gains one line: "platform actions are listed +by `agenta_ops`; fetch a schema with `agenta_op(op)` before calling it with args." Mirrors how the +skill already routes the builder; negligible always-on weight. + +## Alternatives considered + +- **M2 — dynamic real-name advertisement.** Advertise names only; a `load_op` call registers the + real op spec into the harness registry mid-session (Pi extension re-register; Claude MCP + `tools/list_changed`) so the model calls the op by its real name with a schema-validated + signature and native per-op permission. Highest fidelity, but needs mid-session re-registration + on both harnesses and, under the cold-replay runtime, reconstruction of the "loaded" set each + turn. Deferred to productionization. +- **Schema diet only.** Keep all ops advertised; replace the embedded ~5.5K agent-template delta + schema in `commit_revision`/`test_run` with an open object + a pointer to the skill's + `references/config-schema.md`. ~11K of ~15K is those two schemas, so this alone is a large, + near-zero-risk win. Complementary — it also shrinks each describe-mode fetch under the invoker. +- **Mode-gating.** Drop the build kit in Chat mode, keep it in Build mode. Removes capability + rather than deferring it; can layer on top, not a substitute. + +## Cost of laziness + +One extra round-trip per *distinct* op used (describe → then execute), paid only for ops the run +actually touches, and cacheable to once per op per conversation. A no-op turn pays nothing beyond +the two meta-tools. Acceptable for a builder flow; measured in Slice 3. diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/plan.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/plan.md new file mode 100644 index 0000000000..a4888c20a0 --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/plan.md @@ -0,0 +1,77 @@ +# Plan — sliced implementation + +## Scope now + +- Behind a flag/env (default OFF) so the disclosed path A/B's against today's always-advertise + path before it becomes default. +- Playground platform ops only; no saved-agent change; no committed-agent behavior change. +- The schema diet (Slice 1) is independently shippable and lands first. +- Each slice leaves the tree working and testable. Order: baseline → diet → mechanism → measure. + +## Slice 0 — Baseline (pin the real cost) + +1. Write a measurement script (tiktoken `o200k_base`) that resolves the default build-kit overlay + and reports per-op advertised token cost + total, using the same `advertisedToolSpecs` + projection the runner ships (or a faithful mirror). +2. Confirm which ops actually advertise live today — resolve with `AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS` + off, and record whether `test_run` is in the advertised set. +3. Add a runner unit test asserting the current behavior: every resolved platform op appears in + `advertisedToolSpecs(plan.toolSpecs)`. This is the invariant later slices intentionally flip. +4. Check a `baseline.md` table into this folder (before-numbers, per op). + +**Exit:** `baseline.md` exists with per-op numbers; the "all platform ops advertised today" test +passes on `main`. + +## Slice 1 — Schema diet (large, low-risk win, independent) + +1. In `op_catalog.py`, replace the embedded agent-template delta schema in `commit_revision` + (and `test_run`) with an open object (`{"type":"object"}`) plus a description pointing at the + skill's `references/config-schema.md`. +2. Update the platform-op / wire contract tests that pin those schemas. +3. Re-run the Slice 0 script; record the drop (expect ~11K). +4. Lab check: an agent can still `commit_revision` with a valid config (the model has the shape + from the skill reference, and the server still validates). + +**Exit:** measured total drops by ~11K; contract tests green; a lab run commits a revision +successfully. + +## Slice 2 — Disclosure mechanism (the POC), flagged + +1. Add the flag/env (default off) that turns disclosure on for a run. +2. Introduce a disclosure transform applied to the advertised set at the two call sites + (`pi-assets.ts:353`, `environment.ts:721`): when on, replace the disclosure-eligible specs + (see design "Identifying the disclosure-eligible set" — heuristic for the POC) with the two + meta-tools `agenta_ops` + `agenta_op`; keep client tools and everything else advertised. + `plan.toolSpecs` / `toolSpecsByName` stay complete. +3. Implement the invoker dispatch: `agenta_ops` returns `{op, one_line, read_only}` built + runner-side from the resolved specs; `agenta_op` describe-mode returns one op's input schema; + `agenta_op` execute-mode builds the gate from the TARGET op's spec, runs `decide()`, then + `assembleBody`/`directCallUrl`/`callDirect` (reuse the `executeRelayedTool` core). +4. Unit tests: (a) with the flag on, only the meta-tools + client tools are advertised; (b) the + full private specs remain in `toolSpecsByName`; (c) per mutating op (`commit_revision`, + `create_schedule`, `remove_*`), `agenta_op` execute-mode produces the SAME approval verdict as + a direct call — no approval regression; (d) `$ctx` binding still fills server-side (the model + cannot retarget); (e) describe-mode has no side effect and no approval. +5. Add the one-line nudge to the `build-an-agent` skill. + +**Exit:** flag on → a lab run completes discover → wire → commit → schedule using only the +meta-tools; every mutating-op approval test passes; `tsc` + `pnpm test` green in `services/runner`. + +## Slice 3 — Measure, decide default, scope M2 + +1. Re-run the Slice 0 script with the flag on; record the no-op turn cost (target: low hundreds). +2. Run the build-an-agent lab / release gate with the flag on vs off; compare pass rate and + "wander" failures. +3. Write a `results.md` comparison (tokens + reliability, before/after). +4. Decide: flip default on? And is M2 (dynamic real-name advertisement) worth it for the ops that + most benefit from schema-validated calls? + +**Exit:** `results.md` checked in with the before/after; a go/no-go recommendation on +default-on and on M2. + +## Not in this plan + +- Marker-based eligibility (Slice 2 uses the heuristic); adding a `source:"platform"` wire marker + is a follow-up if we default-on for gateway/reference too. +- Disclosing gateway/code/client/MCP tools. +- M2 dynamic advertisement implementation. diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md new file mode 100644 index 0000000000..c764f51fed --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md @@ -0,0 +1,104 @@ +# Research — how tools reach the model today + +Verified 2026-07-20 against the code cited. Paths are current (`services/runner/`; some docs +still say `services/agent/`). + +## The path a platform op takes, end to end + +1. **Overlay injects the op set (Python).** The playground build kit is a per-run overlay, not a + committed field. `build_agent_template_overlay()` (`api/oss/src/core/workflows/build_kit.py`) + emits `DEFAULT_BUILD_KIT_OPS` — ~13 platform ops (`discover_tools`, `commit_revision`, + `annotate_trace`, `query_spans`, `test_run`, `discover_triggers`, `create_schedule`, + `create_subscription`, `list_schedules`, `list_deliveries`, `test_subscription`, + `remove_schedule`, `remove_subscription`) plus the `request_connection` / `request_input` + client tools and the `build-an-agent` skill. It is served as the static workflow + `__ag__build_kit` and also delivered as read-only `additional_context` at + `api/oss/src/apis/fastapi/applications/router.py` (~L1916). The frontend merges it per run + (atom `workflowBuildKitOverlayReadyAtomFamily`, on by default via + `web/oss/src/lib/helpers/dynamicEnv.ts`). + +2. **Resolution (Python SDK).** `resolve_tools` + (`sdks/python/agenta/sdk/agents/platform/resolve.py` → `tools/resolver.py`) turns each + declared op into a `CallbackToolSpec` carrying a direct `call{method,path,context,args_into}`. + The op's `context_bindings` (self-targeting fields) are stripped from the model-visible schema + by `PlatformOp.resolved_input_schema()` (`.../platform/op_catalog.py`) and re-emitted as + `call.context`. No HTTP round-trip; the catalog fully describes the op. The resolved set rides + the `/run` wire as `customTools`. + +3. **Runner holds every spec privately.** `buildRunPlan` sets `plan.toolSpecs = request.customTools` + (`services/runner/src/engines/sandbox_agent/run-plan.ts:352`). The runner indexes them by name + with `toolSpecsByName(specs)` (`services/runner/src/tools/public-spec.ts:34`) — the ONE index + the relay execute loop, the internal tool-MCP server, and the ACP approval gate all key on. + +4. **Advertisement is a SEPARATE projection.** `advertisedToolSpecs(specs)` + (`public-spec.ts:57`) maps each spec to `{name, description, inputSchema, kind, render, + timeoutMs}` (`AdvertisedToolSpec`, `public-spec.ts:12`). `inputSchema` is the token weight. + It is called at exactly two sites: + - `services/runner/src/engines/sandbox_agent/pi-assets.ts:353` — Pi path; the specs become + `AGENTA_TOOL_PUBLIC_SPECS`, which the bundled extension (`extensions/agenta.ts`) reads and + registers as native Pi tools. + - `services/runner/src/engines/sandbox_agent/environment.ts:721` — Claude/ACP path; the specs + back the synthetic `agenta-tools` MCP server (`tools/mcp-bridge.ts`, `tool-mcp-http.ts`). + +5. **Execution reads the PRIVATE spec, never the advertisement.** For a platform op (direct + `call`), the relay path runs `executeRelayedTool` (`services/runner/src/tools/relay.ts:318`): + `assembleBody(spec.call, args, runContext)` (`relay.ts:384`) merges model args → static body → + `$ctx` context bindings (last, so a bound field always wins); `directCallUrl(...)` + (`relay.ts:385`, guard in `tools/direct.ts:286`) host-locks to the run's own Agenta origin and + confines to the `/api` mount; `callDirect(...)` (`relay.ts:392`) sends with the caller + credential. All keyed by the private `spec` — advertisement is irrelevant to execution. + +6. **Permission keys on the per-spec gate.** `decide(gate, plan, stored)` + (`services/runner/src/permission-plan.ts:138`) → `effectivePermission` (`:125`) resolves the + spec's own `permission` first, then rule match, then policy default (`allow_reads` → + read-only op runs, write asks). The gate is built from the resolved op's spec, so approval + fidelity lives with the private spec, not the advertised name. + +## Advertisement consumers + +| Harness | Delivery | Consumes `advertisedToolSpecs` at | Notes | +| --- | --- | --- | --- | +| Pi (`pi_core`/`pi_agenta`) | native, via bundled extension | `pi-assets.ts:353` → `AGENTA_TOOL_PUBLIC_SPECS` | No MCP server attached to Pi. | +| Claude (`claude`) | synthetic `agenta-tools` MCP server | `environment.ts:721` | Public metadata only; execution relays back. | + +Both paths consume the same projection function. Intercepting it once (or wrapping it at both +call sites behind a flag) covers both harnesses with no harness-specific logic. + +## Measured token cost (needs Slice 0 re-baseline) + +From the 2026-07-17 investigation (tiktoken `o200k_base`); flagged for re-measure because +`test_run` is handler-gated (`AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS`, default off) and may not +advertise live. + +| Op | Tokens | Note | +| --- | --- | --- | +| `test_run` | ~6,521 | embeds the ~5,462-token agent-template delta schema; handler-gated | +| `commit_revision` | ~5,844 | embeds the same ~5,462-token delta schema | +| `query_spans` | ~1,283 | filtering DSL `$defs` | +| others (10) | remainder | — | +| **all ~13 ops** | **~15,454** | the "hi" cost | + +Two schemas (`commit_revision` + `test_run`) account for ~11K of the ~15K — hence the schema-diet +slice is a large, low-risk win on its own. + +## Seams the plan must pin + +1. **Advertisement projection** (`advertisedToolSpecs`, two call sites). Where disclosure hooks. + Must not alter what execution/permission read. +2. **Private spec index** (`toolSpecsByName`). Must stay COMPLETE — the invoker looks the target + op up here; execution and the approval gate depend on it. +3. **Direct-call execution** (`relay.ts:318` → `direct.ts`). The invoker must reach this + unchanged, feeding the target op's private `call`. +4. **Permission decision** (`decide`/`effectivePermission`). The invoker must build the gate from + the TARGET op's spec, not from itself — otherwise `agenta_op` gates as one tool and writes + lose their approval prompt. +5. **Identifying disclosure-eligible specs.** A platform op is a `callback`-kind spec with a + direct `call`. So is a `reference` (workflow) tool (`direct.ts` header). There is no explicit + "this is a platform op" marker on `ResolvedToolSpec` today, so the runner cannot cleanly tell a + build-kit op from an author's reference tool without one. Two ways out (Open Question 4): + a heuristic (collapse all direct-`call` callback specs), or a small marker added by the platform + resolver (a wire add). +6. **Wire contract mirroring.** If Seam 5 uses a marker, `protocol.ts` + `wire.py` + goldens + change together (`services/runner/CLAUDE.md`, "The wire contract is mirrored"). +7. **Client tools stay advertised.** `request_connection` / `request_input` must remain + model-visible (the browser fulfils them) and are cheap; disclosure skips them. diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/status.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/status.md new file mode 100644 index 0000000000..056f08337e --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/status.md @@ -0,0 +1,44 @@ +# Status + +**Last updated:** 2026-07-20 + +## Current stage + +Planning workspace drafted (context, research, design, plan). No code written, nothing +committed. Ready for team review of the open implementation questions below. Research is +grounded in the current runner code (`file:line` in research.md); the token numbers are from the +2026-07-17 investigation and are explicitly flagged for a Slice 0 re-baseline. + +## Locked decisions + +- Seam = the runner advertisement layer (`advertisedToolSpecs`, `services/runner/src/tools/public-spec.ts`), + consumed at `pi-assets.ts:353` and `environment.ts:721`. Execution + permission stay below it. +- Approach = a discovery meta-toolset (`agenta_ops` + `agenta_op`) that keeps op specs private in + runner memory and moves schemas from the prompt into on-demand tool results. +- Skills, op-catalog contents, external `discover_tools`, and non-platform tool types are out of + scope. +- Playground overlay only — no saved/committed-agent change. +- Schema diet is complementary and lands as its own slice (Slice 1). +- Ship behind a flag, default off, until measured. +- No commits during planning; implementation is a later, separate branch. + +## Open implementation questions + +1. **Invoker shape** — one `agenta_op` with describe-vs-execute mode, or two tools + (`describe_op` + `call_op`)? *Recommendation: one tool* (smaller advertised surface; the mode + is clear from presence of `args` or an explicit `mode`). +2. **Permission fidelity** — confirm `agenta_op` execute-mode builds the gate from the TARGET + op's private spec and runs the existing `decide()`, rather than gating the invoker as a single + tool. *Recommendation: yes, per-op* — otherwise `commit_revision` and other writes lose their + approval prompt. Non-negotiable in the reviewer's eyes IMO, but calling it out explicitly. +3. **Catalog summary source** — build `{op, one_line, read_only}` runner-side from the resolved + specs (zero new wire fields) vs. thread it from `op_catalog.py`. *Recommendation: runner-side.* +4. **Disclosure-eligible identification** — heuristic (collapse all direct-`call` callback specs) + vs. a `source:"platform"` marker on `ResolvedToolSpec` (a `protocol.ts` + `wire.py` + golden + add). *Recommendation: heuristic for the flagged POC; marker before default-on.* +5. **Sequencing** — do Slice 1 (schema diet) and ship it independently first, or hold it and land + the whole thing together? *Recommendation: ship Slice 1 first* — it is ~11K of the win at + near-zero risk and is useful even if the POC needs iteration. +6. **Default-on criteria** — what pass-rate / token target gates flipping the flag (Slice 3)? + Needs a number from the team (e.g. "no regression on the lab matrix and ≥90% token cut on a + no-op turn"). \ No newline at end of file