diff --git a/docs/design/agent-workflows/projects/managed-resource-reconciliation/README.md b/docs/design/agent-workflows/projects/managed-resource-reconciliation/README.md new file mode 100644 index 0000000000..7b8da73651 --- /dev/null +++ b/docs/design/agent-workflows/projects/managed-resource-reconciliation/README.md @@ -0,0 +1,14 @@ +# Managed external resource reconciliation + +Planning workspace for the durable reconciliation layer that follows the process-local Daytona +Secret delivery in PR #5277. This branch contains design documents only; it does not add API code, +migrations, workers, or deployment wiring. + +## Files + +- `context.md` - Problem, goals, non-goals, and success criteria. +- `research.md` - Existing Agenta patterns and candidate reuse cases. +- `design.md` - Proposed reusable managed-resource boundary. +- `api-design.md` - Reviewable API, data, identity, and migration shape. +- `plan.md` - Stack boundaries, design gate, sequencing, and acceptance tests. +- `status.md` - Current state, decisions, open questions, and next action. diff --git a/docs/design/agent-workflows/projects/managed-resource-reconciliation/api-design.md b/docs/design/agent-workflows/projects/managed-resource-reconciliation/api-design.md new file mode 100644 index 0000000000..eb61ce7dbb --- /dev/null +++ b/docs/design/agent-workflows/projects/managed-resource-reconciliation/api-design.md @@ -0,0 +1,138 @@ +# API design + +This document is the proposed lower-level shape for PR B. Names remain reviewable until the CTO +approves the domain boundary. + +## Domain placement + +```text +api/oss/src/apis/fastapi/managed_resources/ +api/oss/src/core/managed_resources/ +api/oss/src/dbs/postgres/managed_resources/ +``` + +Follow the standard Router -> Service -> DAO interface -> Postgres DAO dependency direction. + +## Data model + +### `managed_resource_sets` + +One set represents a controller-owned realization of one product/runtime owner generation. + +| Field | Role | +| --- | --- | +| `id` | Internal resource-set identity | +| `project_id` | Tenant scope | +| `owner_kind`, `owner_id` | Product or runtime owner reference | +| `controller_key` | Routes reconciliation to a typed controller | +| `idempotency_key` | Deduplicates reservation attempts | +| `specification_digest` | Detects owner-generation changes without storing secrets | +| `generation` | Monotonic desired generation | +| `desired_state` | `present` or `absent` | +| `observed_state` | `pending`, `present`, `absent`, `unknown`, or `blocked` | +| `version` | Optimistic compare-and-set version | +| `attempt_count`, `next_attempt_at` | Retry scheduling | +| `error_code` | Bounded non-secret error | +| `claim_id`, `claim_owner`, `claim_generation`, `claim_expires_at` | Fenced worker claim | +| timestamps | Creation, update, and terminal history | + +Unique constraints: + +- `(project_id, controller_key, idempotency_key)`; +- stable cursor index `(created_at, id)`; +- due-work index `(controller_key, desired_state, observed_state, next_attempt_at, created_at, id)`. + +### `managed_resources` + +One row represents one external child resource in a set. + +| Field | Role | +| --- | --- | +| `id`, `resource_set_id` | Internal identity and parent | +| `logical_key` | Stable controller-owned key within the set | +| `provider_key`, `resource_kind` | External system and typed resource kind | +| `external_id`, `external_name` | Provider identity needed for observe/delete | +| `observed_state` | Child observation | +| `version` | Child compare-and-set version | +| `safe_attributes` | Optional versioned, controller-validated, non-secret recovery facts | +| timestamps | Lifecycle history | + +`safe_attributes` must be a discriminated typed payload for registered resource kinds. It is not an +unrestricted metadata bucket. Daytona Secret rows should not store credential values, headers, +provider responses, or control tokens. + +## Internal operations + +The initial API surface should be workload-only: + +```text +POST /internal/managed-resources/reserve +GET /internal/managed-resources/{id} +PATCH /internal/managed-resources/{id}/intent +POST /internal/managed-resources/query-due +POST /internal/managed-resources/{id}/claim +PATCH /internal/managed-resources/{id}/observation +``` + +Exact path naming is less important than keeping the surface internal and avoiding special cases in +global authentication middleware. + +Every mutation includes `expected_version`. Observation writes from a worker also include the claim +ID and claim generation. A stale claim or version returns a conflict without changing state. + +## Workload identity + +PR B must choose one standard internal workload-auth mechanism. The current #5242 approach adds +exact path matching to shared auth middleware and relies on invocation authorization that was +temporarily carried through telemetry context. Do not preserve that coupling. + +The preferred contract is: + +```text +service-stamped run context { + tenant scope + run/session owner + workload authorization or signed assertion +} +``` + +The runner uses its workload identity for control-plane access and the signed context for tenant +scope. Provider credentials remain separate from both. + +## Reconciler execution + +1. Query due sets by `controller_key` and immutable cursor. +2. Atomically claim one set with a new claim generation. +3. Load the current owner generation or typed controller plan. +4. Observe the provider outside a database transaction. +5. Apply the smallest idempotent provider operation. +6. Record child and set observations using version and claim fences. +7. On a retryable failure, record a safe error and bounded backoff. +8. On an invariant violation or ambiguous ownership, mark `blocked` for operator review. + +TaskIQ can wake or schedule controllers. Postgres remains the due-work authority so a lost queue +message cannot permanently lose reconciliation. + +## Daytona mapping + +| Current Daytona field | Proposed location | +| --- | --- | +| session/run owner | resource-set owner reference | +| credential epoch digest | resource-set specification digest | +| Secret provider ID/name | child external ID/name | +| sandbox ID | separate child resource or typed safe attribute | +| allowed host and binding | Daytona controller plan, not universal columns | +| cleanup retry | resource-set retry state | +| janitor claim | generic resource-set claim | + +The Daytona controller may store a versioned safe attribute required to verify exact-host metadata. +That payload remains a typed Daytona shape rather than a column required by other controllers. + +## Migration policy + +Migration `oss000000011_add_agent_secret_leases.py` has not shipped. Replace it before merge. Do +not ship a Daytona schema followed immediately by a production rename migration. + +Preview databases that already ran migration 011 should downgrade to 010 and apply the replacement +011. If a shared preview cannot be reset, use a one-off development repair rather than permanent +production compatibility code for unshipped data. diff --git a/docs/design/agent-workflows/projects/managed-resource-reconciliation/context.md b/docs/design/agent-workflows/projects/managed-resource-reconciliation/context.md new file mode 100644 index 0000000000..57c5031b83 --- /dev/null +++ b/docs/design/agent-workflows/projects/managed-resource-reconciliation/context.md @@ -0,0 +1,30 @@ +# Context + +The original PR #5242 combined two decisions: deliver opaque HTTP credentials through exact-host +Daytona Secrets, and persist crash-safe lifecycle state for external resources. The first decision +does not require an API migration and now lives in PR #5277. The second does require a reviewed +control-plane design and is the subject of this plan-only stacked PR. + +## Goals + +1. Preserve PR #5277 as the functional Daytona isolation layer with no API changes. +2. Keep the lower implementation reusable when durability is added. +3. Design a small managed-external-resource kernel using existing Agenta conventions. +4. Keep product domains such as connections and triggers as their own sources of truth. +5. Keep plaintext credentials and arbitrary provider responses out of the generic control plane. + +## Non-goals + +- Building a generic workflow engine. +- Replacing gateway connections, trigger subscriptions, webhooks, or session ownership. +- Migrating Composio triggers in the first durability PR. +- Enabling the lower PR in production before orphan recovery exists. +- Implementing the durable API, schema, worker, or controller in this planning PR. + +## Success criteria + +- PR #5277 contains no API router, migration, auth-middleware change, or durable janitor. +- PR #5277 proves plaintext opaque credentials do not enter Daytona sandbox configuration. +- The approved implementation adds durability without changing PR #5277's credential plan or + Daytona provider adapter. +- The durable schema uses generic ownership and reconciliation concepts, not Daytona-only columns. diff --git a/docs/design/agent-workflows/projects/managed-resource-reconciliation/design.md b/docs/design/agent-workflows/projects/managed-resource-reconciliation/design.md new file mode 100644 index 0000000000..a15c265f0a --- /dev/null +++ b/docs/design/agent-workflows/projects/managed-resource-reconciliation/design.md @@ -0,0 +1,135 @@ +# Design + +## Boundary + +The shared abstraction is a managed external resource: something Agenta creates in another system, +which may survive the process that created it and must eventually converge to a requested state. + +The kernel owns durable intent and coordination. Controllers own provider behavior. + +```text +Product or runtime owner + -> managed-resource intent + -> durable store + -> controller claim + -> provider operation + -> observed-state compare-and-set +``` + +Product records remain the user-facing source of truth. A trigger subscription remains a trigger +subscription. A connection remains a gateway connection. Managed-resource records are internal +realization records linked to those owners. + +## Common kernel + +The common kernel owns: + +- tenant scope; +- owner reference; +- controller key; +- idempotency key; +- specification digest and generation; +- desired state; +- observed state; +- external resource identity; +- optimistic version; +- attempt count and next attempt time; +- bounded safe error; +- claim identity, generation, and expiry; +- lifecycle timestamps. + +It never stores plaintext credentials, arbitrary provider responses, user-facing connection +configuration, or provider control-plane credentials. + +## Controller boundary + +Each controller understands one resource kind and exposes a small port: + +```text +plan(owner, generation) -> typed provider plan +observe(record) -> provider observation +apply(plan, observation) -> provider result +destroy(record) -> provider result +``` + +The first controller is `daytona.secret_bundle.v1`. It remains in the runner because the runner +already owns resolved model credentials, Daytona credentials, sandbox creation, and warm-session +lifecycle. + +A later Composio trigger controller can live beside the triggers service and run in an API worker. +The kernel does not require every controller to run in the same process. + +## Generic state + +Prefer desired and observed state over a universal transition enum. + +```text +desired: present | absent +observed: pending | present | absent | unknown | blocked +``` + +Provider-specific controllers validate their own transition rules. The shared store only enforces +claim fencing, optimistic versions, valid common states, and safe retry scheduling. + +This avoids pretending that Daytona Secret provisioning, OAuth connection activation, and trigger +subscription enablement share one detailed state machine. + +## PR A process-local seam + +PR A defines the Daytona credential plan and provider adapter independently from durable storage. +It uses a process-local allocation record attached to the existing session environment: + +```text +DaytonaSecretAllocation { + sandboxId + credentialEpochDigest + resources: [{logicalKey, externalId, externalName}] +} +``` + +Normal teardown and failed sandbox creation call the same provider cleanup functions PR B will use. +A hard process crash may still orphan resources. The feature stays disabled for production until +PR B removes that limitation. + +## PR B durable seam + +PR B adds a `ManagedResourcesControl` port. The runner uses an HTTP implementation backed by the +API domain. Tests use an in-memory implementation. The Daytona plan and provider adapter do not +change. + +```text +ManagedResourcesControl { + reserve(intent) + retrieve(id) + setDesiredState(id, state, expectedVersion) + recordObserved(id, result, expectedVersion, claim) + queryDue(controllerKey, cursor, limit) + claim(id, owner, ttl) +} +``` + +The internal API should use one workload-auth dependency or internal router. It should not add +feature-specific path recognition to global authentication middleware. Tenant scope must come from +a typed service-stamped run context or other approved workload identity, not from telemetry fields. + +## Reuse cases + +1. Daytona Secret and sandbox cleanup. +2. Composio trigger instances and project webhook registrations. +3. Provider-side gateway connection initiation and revocation. +4. Future MCP OAuth registrations or managed gateway allocations. +5. Remote relay endpoints, tunnels, or preview sandboxes if they become persistent provider + resources. + +Only Daytona ships in PR B. The trigger controller is the next recommended adoption and the proof +that the kernel is reusable. + +## Guardrails + +- Do not create an arbitrary JSON workflow engine. +- Do not put credentials in `attributes` or `metadata`. +- Do not merge product DTOs with realization DTOs. +- Do not use age alone to decide that a resource is orphaned. +- Do not hold a database transaction while calling a provider. +- Claim work, call the provider outside the transaction, then compare-and-set observed state. +- Every provider create/delete operation must be idempotent or reconciled through observe-before-act. diff --git a/docs/design/agent-workflows/projects/managed-resource-reconciliation/plan.md b/docs/design/agent-workflows/projects/managed-resource-reconciliation/plan.md new file mode 100644 index 0000000000..c03a1c6886 --- /dev/null +++ b/docs/design/agent-workflows/projects/managed-resource-reconciliation/plan.md @@ -0,0 +1,142 @@ +# Plan + +## Stack overview + +The work is split into a functional PR and a stacked design gate: + +1. PR A, #5277, contains the process-local materializer and provider adapter. +2. This PR B contains only the durable reconciliation plan. Implementation starts only after the + design gate below is approved and must reuse PR A without rewriting it. + +## PR A: isolate Daytona credentials without API persistence + +PR: #5277, `[feat] Deliver agent credentials through Daytona Secrets`. + +Branch: `feat/daytona-secret-materialization`; base: `big-agents`. + +### Include + +1. Typed consumer-owned credential contracts in the Python SDK and runner wire. +2. Effective model endpoints and HTTP MCP URLs next to their credentials. +3. Environment/header binding and `opaque_http` usage classification. +4. Exact HTTPS host validation and fail-closed unsupported modes. +5. Daytona Secret planning, deterministic non-sensitive names, create/find/verify/delete adapter. +6. Secret-name attachment to sandbox creation and placeholder materialization for HTTP MCP headers. +7. Process-local allocation records for normal teardown, failed provisioning compensation, and + warm sessions in the current sidecar process. +8. Credential generation fingerprints so a changed credential replaces the warm sandbox. +9. A feature flag or deployment gate that remains off in production until PR B merges. + +### Exclude + +- `api/oss/src/**/agent_secret_leases/`; +- migration 011; +- API router composition; +- global auth middleware changes; +- runner control token and HMAC deployment wiring; +- HTTP lease client; +- durable janitor, cursor, claim, and retry code; +- telemetry-based authorization hotfix; +- unrelated approval-event behavior changes. + +### Acceptance tests + +- Supported model and HTTP MCP credentials become Secret attachments/placeholders. +- Plaintext does not appear in sandbox env, files, command arguments, logs, traces, or persisted + session state. +- Exact allowed host succeeds; a different host receives no plaintext. +- Invalid host, wildcard, IP, local address, empty value, and unsupported usage fail before sandbox + creation. +- Secret creation followed by sandbox failure triggers compensation. +- Normal cold and warm teardown delete sandbox and Secrets. +- Local Pi, local Claude, and self-managed auth behavior remain unchanged. +- The production gate defaults off and the PR states the hard-crash orphan limitation. + +## PR B: durable managed external-resource reconciliation + +Suggested title: `[docs] Plan durable managed-resource reconciliation` + +Branch: `docs/managed-resource-reconciliation-plan`; base: PR A branch. + +This PR includes the seven documents in this workspace and no production code. After approval, the +implementation may continue in a new stacked branch or be added deliberately to this branch after +the review boundary is recorded. + +### Design gate before implementation + +The CTO reviews and approves: + +1. domain name and scope; +2. generic versus controller-specific fields; +3. workload identity and tenant attestation; +4. database ownership and migration; +5. desired/observed state model; +6. claim and fencing semantics; +7. worker topology and operational ownership. + +### Include + +1. New `managed_resources` API/core/Postgres domain. +2. Resource-set and child-resource tables. +3. Internal workload-only routes. +4. Optimistic versions, idempotency, immutable pagination, due-work queries, retries, claims, and + generation fencing. +5. A controller port and in-memory test implementation. +6. A dedicated reconciliation worker path using existing TaskIQ infrastructure for wakeups. +7. Daytona controller integration using PR A's unchanged plan and provider adapter. +8. Crash injection at every provider/database boundary. +9. Production enablement after live Daytona verification and zero leaked resources. + +### Exclude + +- Migrating gateway connections or trigger subscriptions. +- Public managed-resource CRUD. +- A generic workflow language. +- Arbitrary provider JSON persistence. + +### Acceptance tests + +- Repeated reservation returns one resource set. +- Stale versions and stale claim generations cannot mutate state. +- Two workers cannot apply the same generation concurrently. +- A crash after each Secret/sandbox/provider step converges on retry. +- Credential generation changes converge old resources to absent before new resources become + present. +- Queue message loss does not lose due work because Postgres is authoritative. +- Migration upgrade/downgrade works on PostgreSQL. +- Workload auth cannot access unrelated routes or tenant scopes. +- No persisted row, error, or log contains credential plaintext. +- Live QA finishes with the Daytona resource count at baseline. + +## Follow-up controller: Composio triggers + +After PR B proves the kernel, plan a separate adoption PR: + +1. Trigger subscription remains the product source of truth. +2. Desired local subscription state produces a managed-resource intent. +3. The Composio controller creates, observes, enables, disables, or deletes the `ti_*` resource. +4. Local persistence failure can no longer orphan the provider trigger. +5. Provider deletion failure retains a retryable realization record. + +This follow-up is the proof of reuse. It is not part of the initial Daytona rollout. + +## Recut procedure + +1. Completed: freeze new implementation work on #5242 and preserve it as review history. +2. Completed: create PR A from `big-agents` with no API, migration, or durable control-plane files. +3. Completed: run focused SDK, full runner, independent review, and live Daytona QA for PR A. +4. Completed: create a docs-only PR B stacked directly on PR A. +5. Review and approve the reusable domain, workload identity, ownership, claim, and fencing design. +6. Only after approval, implement the domain in reviewable slices and replace the unshipped + migration instead of adding a rename migration. +7. Close or supersede #5242 after both replacement PRs are cross-linked. + +Every branch, commit, and push operation must use GitButler. Do not use a worktree or raw git. + +## Review order + +For PR A: credential contract -> endpoint and host policy -> Daytona provider adapter -> sandbox +attachment -> cleanup and warm-session behavior -> tests. + +For PR B: domain boundary -> data model -> workload identity -> claim/fence rules -> controller +port -> worker loop -> Daytona integration -> failure injection and migration tests. diff --git a/docs/design/agent-workflows/projects/managed-resource-reconciliation/research.md b/docs/design/agent-workflows/projects/managed-resource-reconciliation/research.md new file mode 100644 index 0000000000..73265ca2e0 --- /dev/null +++ b/docs/design/agent-workflows/projects/managed-resource-reconciliation/research.md @@ -0,0 +1,62 @@ +# Research + +## Existing foundations + +Agenta's standard new-domain layering is Router -> Service -> DAO interface -> Postgres DAO. The +composition root wires concrete implementations. Provider-specific behavior in +`api/oss/src/core/gateway/connections/` already uses adapters and a registry. The proposed design +should reuse both patterns. + +Session ownership and evaluation job locks provide useful TTL and fencing examples, but they solve +ephemeral coordination rather than durable external-resource cleanup. TaskIQ and Redis streams can +wake reconcilers, while Postgres remains the authority for due work and observed state. + +References: + +- `api/AGENTS.md` +- `api/oss/src/core/gateway/connections/` +- `api/oss/src/core/sessions/streams/service.py` +- `api/oss/src/dbs/redis/sessions/locks.py` +- `api/oss/src/core/evaluations/runtime/locks.py` +- `api/entrypoints/worker_queues.py` + +## Confirmed second use case + +Trigger subscription creation mints a Composio trigger instance before the local database write. +If persistence fails, the provider resource can remain orphaned. Deletion logs some provider +failures and proceeds with local deletion, which can erase the durable knowledge needed to retry +cleanup. + +References: + +- `api/oss/src/core/triggers/service.py` +- `api/oss/src/core/triggers/providers/composio/adapter.py` + +Gateway connections have the same failure class. Provider initiation happens before persistence, +and provider revocation is best effort before local deletion. The product-level connection should +remain in `gateway_connections`; managed resources can later realize its provider lifecycle. + +## Concepts that remain separate + +- Future MCP authorization relationships belong in `gateway_connections`. +- Outbound webhooks are local subscriptions plus delivery jobs, not external allocations. +- Evaluation reconciliation derives internal graph state. +- Session affinity and job locks should not move into this domain. +- Events are analytics records, not a transactional outbox or saga system. + +## Assessment of #5242 + +The current lease code has reusable mechanics: idempotency, optimistic versions, retry times, +generation-fenced claims, immutable cursors, and safe errors. Its schema is still Daytona-specific: +provider `daytona`; owners run/session; consumers model/HTTP MCP; environment/header bindings; and +columns for sandbox identity, credential epochs, allowed hosts, and provider Secret names. + +Migration 011 is unmerged and has no production data. Replace it before merge rather than shipping +a specialized schema and adding a rename migration immediately afterward. + +## Conclusion + +No generic durable external-resource reconciler exists in Agenta today. Build a small +`managed_resources` domain. Product records remain authoritative, and provider controllers realize +their desired external state through the kernel. Daytona is the first controller. Composio trigger +subscriptions are the strongest later adopter. diff --git a/docs/design/agent-workflows/projects/managed-resource-reconciliation/status.md b/docs/design/agent-workflows/projects/managed-resource-reconciliation/status.md new file mode 100644 index 0000000000..c65eef585f --- /dev/null +++ b/docs/design/agent-workflows/projects/managed-resource-reconciliation/status.md @@ -0,0 +1,37 @@ +# Status + +Date: 2026-07-13 + +## Current state + +- Existing API, worker, connections, triggers, webhooks, sessions, mounts, and evaluation patterns + researched. +- No existing generic durable external-resource reconciler found. +- Composio trigger subscriptions confirmed as a real second consumer. +- PR A opened as #5277 from `feat/daytona-secret-materialization`. +- This plan-only PR B is stacked directly on PR A. +- No production code or existing migration changed in this planning pass. + +## Recommended decisions + +- Keep #5242 as historical review context rather than merge it unchanged. +- Keep PR #5277 free of API and database changes. +- Keep `AGENTA_DAYTONA_OPAQUE_SECRETS=process_local` disabled in production until durable orphan + recovery exists. +- Name the general domain `managed_resources` pending CTO review. +- Keep gateway connections and trigger subscriptions as product sources of truth. +- Replace unshipped migration 011 before merge. +- Use a typed workload identity boundary, not telemetry fields or path-specific auth bypasses. + +## Decisions still needed + +1. Does the team approve managed external resources as the reusable domain? +2. Should the aggregate be named `resource_set`, `realization`, or another term? +3. What standard workload identity should runner and workers use? +4. Does a Daytona sandbox become a child resource or a typed safe attribute of the Secret bundle? +5. Who operates blocked/quarantined resources and what alert opens that workflow? + +## Next action + +Review this stacked plan before adding implementation. Do not begin the generic API, migration, or +worker work until the CTO approves the domain, workload-auth, ownership, and fencing decisions.