From 678e5459e0019622b8a651865d89900d507f6d5c Mon Sep 17 00:00:00 2001 From: Mahmoud Mabrouk Date: Thu, 23 Jul 2026 15:20:47 +0200 Subject: [PATCH 1/2] docs(design): propose free trial credits at signup Proposal for platform-funded first messages for new cloud signups: signup grant record, reservation ledger with atomic charge, server-minted funding provenance, and a small inference gateway composed with the Daytona secret-delivery work (#5223/#5277). Claude-Session: https://claude.ai/code/session_013PybmGbRaSiFzFtzhCrESm --- docs/design/activation-credits/proposal.md | 488 +++++++++++++++++++++ 1 file changed, 488 insertions(+) create mode 100644 docs/design/activation-credits/proposal.md diff --git a/docs/design/activation-credits/proposal.md b/docs/design/activation-credits/proposal.md new file mode 100644 index 0000000000..05da0141de --- /dev/null +++ b/docs/design/activation-credits/proposal.md @@ -0,0 +1,488 @@ +# Free trial credits at signup + +**Status:** proposal, for review +**Scope:** Agenta cloud only. Self-hosted deployments keep today's behavior. + +## Summary and decisions requested + +New signups today must paste an LLM provider API key before the playground sends +their first message. This proposal removes that wall: every new cloud organization +gets a small number of free agent runs, funded by an Agenta-owned provider key on +one cheap model. When the free runs are gone, we ask for the user's key at a moment +when the product has already shown its value. + +The build reuses the existing entitlements meter for consumption and adds four +pieces: a grant record written at signup, a reservation that authorizes each funded +run, a server-side funding policy that decides which surfaces may reserve, and a +small inference gateway so the provider key never enters user-controlled runtime. +The gateway is the one genuinely new component; it is also the first slice of the +credential-isolation boundary we already designed for secrets generally, and the +substrate a future paid managed-credits product would extend. + +Decisions requested from this review: + +1. The trial model (D2) and the starting allowance (D3). +2. The grant-record eligibility design (D4) and its signup-only scope (D5). +3. The credential-protection composition (D6): the gateway as the usage bound, + with the in-flight Daytona secret-delivery work (#5277) as its hardened + transport, built in-house rather than adopted (D7, sized at one to two + engineer-weeks). +4. Sign-off on the guardrail set before any platform key goes live. + +## The problem and the hypothesis + +A new user signs up, lands in the playground, describes the agent they want, and +commits it. At that exact moment the message box grays out, a yellow banner asks +them to connect an API key, and the payoff they just worked for is withheld. + +This proposal takes the funnel problem as its premise: the key wall sits directly +on the signup-to-first-message step, and removing it is the motivating goal of +this work. The experiment below measures how much the wall actually costs us +rather than assuming it. + +Competitors do not have this wall. The AI-first agent platforms we compete with +fund a new user's first AI usage; a signup runs a real agent without seeing a key +form. Only the older workflow-engine incumbents require your own key. Competitor +names and per-product details are withheld from the repo by policy; the named +research notes are available on request. + +**Hypothesis:** letting the first conversation happen before the key ask raises +activation (first successful playground message per signup) without collapsing the +key-connection rate; it moves the wall, it does not remove it. + +## Cost envelope and guardrails + +We measured production traces on EU cloud (2026-07-23, via +`POST /api/tracing/spans/query`, reading the `ag.metrics.tokens.*` span +attributes). Every agent LLM call carries roughly 23,600 tokens of harness context +(system prompt, skills, tool definitions); the user's words and the model's replies +are noise next to it. A 10-turn session totals about 242K input tokens and 1.2K +output tokens. Caveats: the sample was small (8 agent traces from our own team +project; API keys are project-scoped, so organic cross-tenant data needs database +access), and the sampled turns made one LLM call each, so sessions with heavy tool +use will run 2-3x higher. + +Cost per 10-turn session at July 2026 list prices: + +| Model | No caching | With prompt caching | +|---|---|---| +| Gemini 2.5 Flash-Lite | $0.025 | $0.004 | +| DeepSeek V4 Flash | $0.034 | $0.005 | +| OpenAI gpt-5.4-nano | $0.050 | $0.009 | +| Claude Haiku 4.5 | $0.248 | ~$0.05 (verified against a real bill) | + +At 10,000 signups on gpt-5.4-nano this is roughly $500 without caching and under +$100 with caching, against a $3,000 budget. Prompt caching is the largest cost +lever because our workload replays a large stable prefix on every call. + +The budget survives honest users easily; the guardrails exist for dishonest ones. +An agent run is a loop: one user message can trigger many model calls with large +inputs, and the agent's instructions and tools are user-controlled. The guardrails, +all enforced server-side: + +- **The provider key never enters user-controlled runtime.** Agent runs execute in + a sandbox the user's instructions effectively control; a raw key placed there can + be extracted and reused outside every other control. Funded runs therefore talk + to an Agenta inference gateway with a short-lived token instead of a key + (architecture below). This is the load-bearing guardrail; the rest assume it. +- **Per-call and per-run budgets at the request boundary.** The gateway enforces + the model allowlist, a max output size per call, and the reservation's call and + token budgets before forwarding anything to the provider. Enforcement after the + fact cannot cap an oversized first call; enforcement at the request boundary can. + The runner's existing wall-clock limit stays as the outer bound. +- **Reservation before spend:** no reservation, no gateway token, no funded run. +- **Isolated provider project:** the trial key lives in its own provider project + with a hard project-level spend limit (OpenAI supports these; enforcement can lag + slightly), auto-recharge disabled, budget alerts, and easy revocation. +- **Kill switch:** a backend flag that marks managed funding unavailable; the + frontend reads it and falls back to today's connect-a-key behavior. Deleting the + env key is not the mechanism, because the frontend would still show credit and + then fail mid-run. + +Mass fake signups remain bounded by the grant size (cents per org), signup-only +eligibility (D5), and existing email verification; we accept that residual risk for +the MVP. + +## The user experience + +Vocabulary for this section: the **composer** is the playground's message input +box; the **vault** is the per-project store of the user's own provider API keys. + +### First session (free runs remaining) + +Signup and onboarding stay exactly as they are. The one change: when the user +commits their agent, the composer stays enabled and their first message sends. The +run is funded by Agenta on the trial model. The payoff arrives on message one. + +After the first response, a quiet balance badge appears near the composer: "29 free +messages left". It exists so the later wall is expected rather than sudden. The +Billing settings page shows the same balance through its existing usage meters. + +Model choice during the trial: the model picker stays visible, but models other +than the trial model show their normal "connect a key" affordance. Funded runs +never execute on any other model, and the gateway enforces that, not the picker. + +We also fix a related inconsistency: the template setup drawer currently labels +Agenta-managed models "Ready", promising platform keys the runtime does not +provide. After this change the label becomes true for the trial model. + +### The wall (free runs exhausted) + +When the balance reaches zero, the next send is refused before any model call. The +existing yellow banner slot shows: "You've used your free messages. Connect your +API key to keep going, it takes a minute." with the same button that opens the +provider-credentials drawer. The user's typed message must survive this refusal and +send automatically once a key is connected. The playground has a similar +connect-then-send behavior today, but only for the seeded first message; a refused +arbitrary draft currently becomes a transcript error, so retaining and resuming it +is new frontend state, called out in the plan below. + +Users who connected a key never see any of this: their runs resolve their own vault +key and consume nothing. + +### Frontend states + +The gate today is one boolean (vault empty). It becomes a small state machine with +explicit states: loading, user key available, funded run available, allowance +exhausted, funding unavailable (kill switch or outage), and selected model not +funded. Two rules keep it honest: the server's reservation outcome is the authority +on whether a run proceeds and on the remaining balance (the polled usage query is +display-only and can be minutes stale), and a failed or loading balance query must +never be rendered as "you have zero left". + +## Architecture + +### Where credentials come from today, and what we do not build on + +Agent playground runs resolve credentials through `resolve_connection` +(`services/oss/src/agent/app.py`, `sdks/python/agenta/sdk/agents/platform/`): given +the selected model, the resolver fetches the project's vault secrets and picks the +one matching connection. When no connection exists, vault-backed resolution fails +with a typed error (default and self-managed modes instead degrade to +runtime-provided credentials); for a keyless new user on a standard provider, that +failure is today's wall. The resolved credential then travels into the run as +environment variables, which is fine for the user's own key and is exactly what the +gateway exists to avoid for ours. + +An older, model-blind path in the SDK's vault middleware can fall back to provider +keys taken from the service environment. It is a leftover of PR #2957, which built +a first version of platform-funded usage with a `credits_consumed` meter; the agent +service explicitly abandoned that path, its metering call is orphaned, and only +legacy completion workflows still traverse it. This proposal does not build on it. +We reuse its meter machinery for consumption accounting, and we schedule one +hygiene task (out of scope here): audit the cloud service environment for stray +provider keys, since any that exist make legacy-path runs free and unmetered today. + +### In-flight work this composes with: Daytona secret delivery + +Three open PRs already attack the "secrets inside the sandbox" problem and shape +this design. PR #5223 plans, and PR #5277 implements (flag-gated, currently parked +until the runner refactor reshapes the sandbox-provider interface it builds on), +delivery of credentials through Daytona Secrets: the sandbox environment holds +only an opaque placeholder, and Daytona's egress proxy substitutes the real value +into outbound HTTPS requests, but only toward the credential's allowlisted +hostname. PR #5278 proposes (design only, nothing implemented yet) a +reconciliation domain so crashed runners cannot orphan those Daytona Secret +bundles. Together this gives true possession-hiding: no process inside the +sandbox can read the value. + +What it deliberately does not give is usage control. The only lever is the +destination hostname, so a delivered provider key could still be spent without +limit against the provider's own host: any model, any volume, outside our +metering, for the sandbox's lifetime. Its own design doc names the general +solution: move the provider call behind an Agenta gateway. That division of labor +drives D6 below, and this proposal reuses the #5277 machinery wholesale where it +fits: the typed credential contract with fail-closed classification, the exact- +hostname validation, and the per-sandbox secret lease lifecycle. Expired trial +reservations are cleaned by the reservation service's own sweep, described +above; they do not ride #5278's Daytona-scoped reconciliation. + +### The four pieces + +**1. Grant record (eligibility).** When a user signs up, their newly provisioned +personal organization gets an immutable grant row: amount, granted-at, and a +stable campaign identifier (not descriptive prose), with a uniqueness constraint +on (organization, campaign) so a retried signup hook cannot grant twice. No grant, no free runs, which cleanly +excludes every existing organization (a plan-level quota alone would gift the full +allowance to all of them the day we ship). Organizations created later by an +existing user get no grant; that closes the create-orgs-to-farm-credits hole. +The balance is never stored: remaining = grant total minus the consumption meter. +Later gamification ("connect GitHub, earn more runs") and sales promos become +additional grant rows with their own reasons; nothing else changes. + +**2. Reservation (authorization and spend).** A durable record with an explicit +lifecycle: `reserved → started → charged`, or `reserved → refunded`, or +`reserved → expired` for runs that never start, with a reconciliation sweep for +abandoned rows. It is created server-side (not by the frontend), keyed uniquely by +a server-minted invocation identity (below) so retries cannot double-charge, and +bound to the server-authenticated organization, project, and model. One +reservation authorizes exactly one run. The economic invariant: the reservation's +state transition and the consumption-counter change commit in **one database +transaction**, so no crash window can charge twice, charge without counting, or +refund twice; a retried handler lands on the same transition. Consumption uses a +new dedicated lifetime meter (working name `trial_runs_consumed`); the monthly +`credits_consumed` quota, the plan catalog, and the billing projection stay +untouched. The exact sequence: the strict conditional increment and the +reservation's authorizing transition commit in the same transaction, and only +after that commit does the gateway mint a token; the gateway rejects tokens for +any reservation that has not completed it. A refund pairs the terminal transition +with the decrement in the same way. This needs one named refactor: the meter DAO +currently opens and commits its own session, so it must learn to join a +caller-owned transaction (or the conditional upsert moves into the reservation +repository). The reservation service supplies the grant-derived total as the +meter's strict limit, and it fails closed on any infrastructure error, a +deliberate deviation from the entitlements house pattern (feature gates fail +open; spending our money must not). Charge policy: the committed reservation is +the charge; it is refunded if the run dies before its first model call and kept +for anything after. + +**3. Funding policy and provenance (which surfaces may reserve).** Today +interactive playground runs, direct API calls, and evaluations converge on the +same workflow invocation path, and wire metadata is caller-controlled, so the +funding decision cannot rest on anything the client sends. Instead, the API +orchestration layer, at the point where it still knows how the run was invoked, +mints a short-lived signed invocation-purpose claim bound to organization, +project, and a fresh idempotency identity. Only interactive playground and direct +invoke acceptance mint the claim; evaluation, batch, and background paths never +do. The reservation service accepts only this claim, which also supplies the +reservation's unique key (the runner's own turn identifier is created too late to +serve). Request metadata can never turn funding on. + +**4. Inference gateway (the trust boundary).** Funded runs receive no provider +key. The resolver returns a connection whose endpoint is an Agenta gateway URL and +whose credential is an opaque, short-lived, single-reservation token bound to +organization, model, request count, token budget, and expiry. The gateway holds +the real provider key, validates the token, canonicalizes the request (allowed +parameters only, intended provider retention settings forced, expired tokens +refused), enforces the model allowlist, and applies budgets at the request +boundary: each provider call takes an atomic hold against the reservation's +remaining budget before it is forwarded and settles to actual usage after, so +concurrent calls cannot overrun the budget between a check and a decrement. A +token allows several provider calls within its budgets, so replay protection is +per call (a call identity plus a canonical request hash), not per token; a call +that times out ambiguously keeps its hold until reconciliation, so a duplicate +retry cannot be funded twice. Extracting the token from the sandbox yields +minutes of access to one cheap model inside one already-metered budget, instead +of an open provider key. Scope for the MVP: one provider, one model, one API +surface, and we choose the provider's agent-native API as that surface (for +OpenAI, the Responses API), because our runs are tool-using and that is the +protocol the vendor builds agent features on; the pre-launch check below +verifies tool calling end to end rather than trusting either API's +documentation. The runner side needs one contract addition, not new key trust: +the platform-resolved connection must carry an explicit API-dialect field, since +the runner's custom-provider path currently assumes and emits only the +chat-completions dialect and would otherwise call the wrong API through the +gateway. +This is deliberately the smallest version of the credential-isolation gateway +already designed for the secrets roadmap, and the same chokepoint gives us trial +usage metering that does not depend on harness-side instrumentation. Delivery +composes with the Daytona secret work above: the trial connection's endpoint is +the gateway host, and the reservation token rides the #5277 delivery path as an +ordinary opaque credential once that path re-lands on the refactored provider +interface, so even the token is unreadable inside the sandbox (D6). + +### Surface matrix + +| Surface | Trial behavior | +|---|---| +| Agent playground (cloud) | Funded: reservation on the invoke path + gateway token | +| Direct agent API calls (invoke path) | Same as the playground; reservations and rate limits apply | +| Evaluations, batch, background runs | Never funded; user keys only (funding policy) | +| Classic prompt playground | Not funded in MVP; different credential path, near-zero message cost, not the activation surface | +| OSS / self-hosted | No grants, no gateway; resolver behavior unchanged | + +The grant, reservation, policy, and gateway are cloud-only, behind the established +`is_ee()` seams. The SDK and agent-service touchpoints are shared code that no-ops +when the backend issues no grants. + +### Deployment notes + +The consumption counter needs the Python enums, a Postgres enum-value migration, +and usage-API surfacing, deployed migration-first. The grant and reservation +tables are new and small. The gateway is a new small service (or route on an +existing one) holding the only copy of the trial key. The runner needs no trust +changes; its wall-clock limit stays. + +## Decisions + +### D1: What is the unit? + +- **Option A: one unit = one accepted agent run** (one user turn, however many + model calls it makes inside its budgets). Legible in product copy, maps + one-to-one to reservations, and the cost variance between cheap and expensive + turns is bounded by the per-run budgets and small in dollars. +- **Option B: token-denominated credits.** Precise, but meaningless to a new user, + and the 23.6K harness context makes even "hi" cost thousands. + +**Recommendation: A.** Call them "free messages" in product copy for the MVP. We +adopt the word "credits" product-wide only when the unit becomes fungible (earned, +purchased, model-tiered), which is the roadmap but not this build. + +### D2: Which model funds the trial? + +- **Option A: OpenAI gpt-5.4-nano.** Hard project-level spend limits on the + provider side, strong prompt caching, a brand new users recognize so demo + quality is attributed correctly. ~$0.05 per session uncached. Reasoning effort + `none`. +- **Option B: Gemini 2.5 Flash-Lite.** Half the cost, fast, but postpaid billing + with alert-only budgets, so runaway spend depends entirely on our gateway. +- **Option C: DeepSeek V4 Flash.** Cheapest with prepaid billing, but routing + every signup's first prompt through a China-hosted API is an avoidable objection + for GDPR-conscious European prospects. + +**Recommendation: A, conditional on a quality check.** Price and brand argue for +nano, but its vendor positions it for classification-style workloads, not agent +chat, and the first session must feel good, not just cost little. Before launch, +run our agent harness over a handful of representative activation sessions on nano +and Flash-Lite, including real tool use, and pick on quality-per-dollar. Sequence +matters: the candidates are compared directly against the provider APIs first, and +the production gateway is then built for the winner only, so the one-provider MVP +scope holds and we never build a throwaway adapter. "Model-agnostic" applies to +the grant, reservation, and policy pieces; the gateway itself is +provider-specific work. + +### D3: How large is the grant? + +- **Option A: one-time grant, sized around 30 runs.** Covers the measured + activation session (~10 turns) three times over at an expected $0.05-0.15 per + active user. One-time fits our funnel: unlike competitors who sell credits as + the business model, ours exist only to carry users to connecting their own key, + after which usage is unmetered on their key. +- **Option B: monthly recurring grant.** What most competitors do, but they + monetize credits forever; for us it reopens abuse monthly and makes the meter a + permanent product surface. + +**Recommendation: A, with the amount as an experiment parameter, not a constant.** +The grant record carries the amount per cohort, so we can ship cohorts at 20 and +50 and let the funnel data pick. Thirty is a starting point, not a finding. + +### D4: How is eligibility stored? + +- **Option A: plan-level constant.** Simplest, but wrong: lifetime quotas on a + plan grant the full allowance to every existing organization, there is no + cohorting, and no per-org grants later. +- **Option B: mutable per-org balance column.** Supports variable grants but + creates a second writable source of truth beside the meter. +- **Option C: immutable grant record plus the existing consumption meter.** Grants + are append-only rows with amount and reason; the meter stays the only thing that + moves; remaining is computed. New-orgs-only falls out naturally, as do cohorts, + promo grants, and the gamification seam. + +**Recommendation: C.** + +### D5: Who is eligible? + +- **Option A: the personal organization provisioned at signup, only.** Matches the + goal (activation of new users), keeps grant issuance on the one code path that + runs exactly once per signup, and closes the farming hole of creating + organizations for fresh grants. Accepted consequence: an invited teammate joins + an org whose balance may be exhausted, which is fine because such orgs usually + have keys already. +- **Option B: every newly created organization.** Simpler to state, but the + signup and explicit organization-creation provisioning paths are distinct in the + backend, and funding the latter invites farming. + +**Recommendation: A.** Org-scoped accounting (the grain of plans, subscriptions, +and every existing quota), granted only at signup. + +### D6: How is the platform credential protected? + +- **Option A: Daytona Secrets alone (the #5277 path).** The provider key is + delivered as an opaque placeholder that only Daytona's egress proxy resolves, + toward the provider's host. Kills key theft with work that is already built and + QA'd, but leaves usage unbounded: the sandbox can spend the key freely against + the provider host, unmetered, while it lives. Also covers only Daytona runs. +- **Option B: inference gateway alone.** The sandbox receives a short-lived + reservation-bound token; the gateway holds the key and enforces model, budgets, + and metering. Bounds usage on every execution path, but the token sits readable + in the sandbox environment for its lifetime. +- **Option C: composed.** The gateway is the connection's endpoint; the + reservation token is the delivered credential and rides the Daytona Secrets + path where available. Theft of the token requires beating Daytona's egress + proxy, and even then yields minutes of one cheap model inside a metered budget. + +**Recommendation: C**, built in that order: the gateway is the piece the trial +cannot ship without (it is the only usage bound), and the Daytona delivery +hardens it when #5277 re-lands on the refactored provider interface. Option A +alone is not safe for a platform-funded key. + +### D7: Build the gateway or adopt one? + +We surveyed the July 2026 landscape (LiteLLM proxy, BricksLLM, Portkey's +open-source gateway, TensorZero, Envoy AI Gateway, Kong, APISIX, and the small +budget-proxy projects). The decisive finding: none implements our hard +requirement, the reservation-bound atomic hold-and-settle. LiteLLM records spend +asynchronously after calls, so concurrent requests can overshoot a small budget; +the Kubernetes-native gateways do windowed rate buckets, not reservation ledgers; +the closest conceptual match has been unmaintained for 18 months and predates the +Responses API. Whatever we adopt, we write the ledger ourselves. + +- **Option A: adopt LiteLLM.** Buys provider breadth and pass-through we don't + need, costs a second Postgres-backed service with documented memory-growth + issues, and the ledger still gets written by us inside its hook system. +- **Option B: build small.** One FastAPI route in the existing stack: token + lookup, model check, atomic hold via one Redis script or one conditional + Postgres update, byte-faithful streaming pass-through that taps the final + usage event, metering emit, plus a hold reconciler. Roughly 600-900 lines and + one to two engineer-weeks; no new infrastructure, since the stack already runs + Redis and Postgres. + +**Recommendation: B.** The ledger is also the piece that carries into the future +managed-credits product (a trial reservation generalizes to a purchased balance), +and it should live in our code as the budget authority regardless. If that +product later goes multi-provider, a translation layer (Portkey's MIT gateway is +the natural candidate) slots in behind our ledger without rework. + +## Rollout and measurement + +Gate the whole feature on the backend funding flag plus the presence of grants; +no cohort, no change from today. Ship to a percentage of new signups first, watch +spend per signup against the measured envelope, then widen. + +Success metrics: activation rate (first successful playground message per signup), +key-connection rate within 7 days, and platform spend per signup. Kill criteria: +spend per signup exceeding 3x the envelope, or key-connection collapsing while +activation merely shifts messages onto our key. + +Prerequisite fix: agent runs routed through OpenRouter record no token usage in +our tracing (`ag.metrics` and `gen_ai.usage` both empty). The gateway meters trial +usage regardless, but the instrumentation gap blinds our observability for +everything else and should be filed and fixed independently. + +## Out of scope, deliberately + +- **Earned credits (gamification).** One competitor pays out credits for social + follows and for finishing its tutorial, and it visibly works for them. It lands + here as new grant rows with new reasons, once the MVP proves the funnel. +- **Purchasable managed credits.** Reselling metered inference makes credits a + product with billing and margins; the grant, reservation, and gateway substrate + built here is what it would extend. +- **Funding the classic prompt playground.** Different credential path, near-zero + cost per message, not where activation happens. Revisit only if data says + otherwise. +- **Legacy-path hygiene.** Audit the cloud service environment for stray provider + keys and remove the orphaned metering call in the vault middleware, tracked + separately so this feature does not wait on it. + +## Appendix: research this proposal rests on + +- **UX map of the current gate** (frontend): the composer disables while the + project vault is empty; the banner and the connect-then-send machinery live in + the agent playground's conversation components. +- **Prior art** (backend): PR #2957 built platform keys plus credit metering on + the legacy path; the SDK-side gate call was severed in a later refactor; the + entitlements meter machinery survives and is reused here. +- **Entitlements deep dive**: the meter engine supports lifetime counters, atomic + strict enforcement, and refund deltas; the billing UI already renders any + counter the usage API reports. +- **Pricing research** (July 2026): per-token list prices for the candidate + models, provider billing semantics, and competitor free-tier structures. The + AI-first competitors grant on the order of hundreds to a few thousand + credit-units per month (roughly $1-5 of model cost), several with earn-more + actions or user-set spend caps; the workflow-engine incumbents fund nothing. + Named competitor details stay out of the repo by policy; ask for the local + research notes. +- **Trace measurement** (EU cloud, 2026-07-23): the 23.6K harness context and + session token totals, reproducible against `POST /api/tracing/spans/query` + filtering agent-run spans and reading `ag.metrics.tokens.*`; also the source of + the OpenRouter usage-recording gap. From 994e6782ee3961d7c0476835266c16cda82cf393 Mon Sep 17 00:00:00 2001 From: Mahmoud Mabrouk Date: Thu, 23 Jul 2026 16:28:41 +0200 Subject: [PATCH 2/2] docs(design): expand activation-credits into a full design workspace Split the single proposal into context, research, design, and rfc, and add an implicit-decisions register listing every requirement that entered the design beyond the original ask, with its origin and the cost of striking it. Claude-Session: https://claude.ai/code/session_013PybmGbRaSiFzFtzhCrESm --- docs/design/activation-credits/context.md | 129 +++++ docs/design/activation-credits/design.md | 311 +++++++++++ .../activation-credits/implicit-decisions.md | 114 ++++ docs/design/activation-credits/proposal.md | 488 ------------------ docs/design/activation-credits/research.md | 359 +++++++++++++ docs/design/activation-credits/rfc.md | 116 +++++ 6 files changed, 1029 insertions(+), 488 deletions(-) create mode 100644 docs/design/activation-credits/context.md create mode 100644 docs/design/activation-credits/design.md create mode 100644 docs/design/activation-credits/implicit-decisions.md delete mode 100644 docs/design/activation-credits/proposal.md create mode 100644 docs/design/activation-credits/research.md create mode 100644 docs/design/activation-credits/rfc.md diff --git a/docs/design/activation-credits/context.md b/docs/design/activation-credits/context.md new file mode 100644 index 0000000000..9e4fead77a --- /dev/null +++ b/docs/design/activation-credits/context.md @@ -0,0 +1,129 @@ +# Context: why we are funding new users' first messages + +## The situation + +A person signs up for Agenta cloud. After the signup survey they land on `/apps`, +which redirects first-time users into the onboarding playground. There they +describe the agent they want, or pick a template, and hit Create. The agent is +committed, a first prompt is seeded into the conversation, and at that exact +moment the product locks: the message input grays out with the placeholder +"Connect a model to start chatting…", and a yellow banner says "Connect +{provider} to run this agent with your own key." Until the user pastes an LLM +provider API key, nothing runs. The moment they do, the seeded prompt auto-sends; +the code comment for that behavior reads "Connecting the key IS the go-ahead." + +So the product walks a new user all the way up to the payoff and then demands +they leave, create an account with OpenAI or Anthropic, generate a key, and come +back. Most people who hit a wall like this during their first five minutes do not +come back. This is the activation problem this project attacks: the single +largest point of friction sits exactly on the step we most need people to +complete, which is seeing their agent respond once. + +The wall is not a backend limitation. It is one client-side boolean +(`gateActive`) that disables the composer whenever the project's key vault is +empty. The backend below it, however, currently has no safe way to fund a run +with our own key, and building that safely is where the real work is. The +research and design documents in this folder cover both halves. + +## Why now + +Two reasons, one internal and one competitive. + +Internal: activation is the metric of the quarter. Signups are arriving and not +converting to a first successful playground message, and the key wall sits +directly on that step. + +Competitive: Agenta now sells into the agent-platform market. Every AI-first +competitor in that market lets a new signup run a real agent immediately, on +credits the platform funds, with no key form anywhere in the first session. Only +the older workflow-engine incumbents still demand your own key, and their +first-session experience is visibly the weaker for it. Funded starter usage is +not a differentiator we could add; it is a gap we currently have. (Named +competitor analysis stays out of the repo by policy; research.md carries the +anonymized findings, and the named notes are available on request.) + +## What we are building, in one paragraph + +Every new cloud organization gets a small, one-time balance of free playground +messages, roughly thirty. While the balance lasts, playground runs execute on an +Agenta-owned key against one cheap model, and the user sees a quiet countdown. +When the balance hits zero, the existing connect-your-key moment appears, with +the user's typed message preserved and auto-sent once they connect. Users who +bring their own key never touch any of this. The trial exists to move the key +wall past the payoff moment, not to remove it. + +## Goals + +- A new user's first message sends and gets an answer with zero setup. +- The wall moves to after the aha moment and converts better there, measured by + activation rate and key-connection rate. +- Total program cost stays bounded: $3,000 covers roughly 10,000 signups with + hard worst-case limits per account. +- Everything reuses the existing entitlements and metering machinery where it + fits, and what is new (the gateway, the grant, the reservation) is sized for + an MVP while being the substrate later credit products extend. + +## Non-goals for this iteration + +- Earned credits (connect GitHub, finish the tutorial). Validated in the market, + wanted later, designed for as a seam, not built now. +- Purchasable managed credits (reselling metered inference). Same: the ledger + built here is what it would extend. +- Funding the classic prompt playground, evaluations, or any background + execution. The trial funds interactive playground runs only. +- Any change for self-hosted deployments. This is cloud-only. + +## Constraints that shaped the design + +- **The runtime is user-controlled.** Playground agents execute in a Daytona + sandbox where the user's instructions and tools decide what runs. Anything + placed in that sandbox, including credentials, must be assumed readable and + usable by the user. This single fact forces most of the architecture. +- **Agent runs are token-heavy.** Every LLM call replays roughly 23,600 tokens + of harness context (system prompt, skills, tool definitions). Naive + chat-message cost math is off by two orders of magnitude; all budgeting works + from measured agent-run numbers. +- **MVP discipline.** Mahmoud's standing direction: reuse the entitlements + architecture, do not build parallel systems, keep credits as abstract units + rather than money, ship the simple case first without foreclosing the later + ones. +- **In-flight work overlaps.** PRs #5223/#5277/#5278 (Daytona secret delivery) + attack the credentials-in-sandbox problem and are part of this design's + composition. PR #2957 built a first version of platform-funded usage whose + metering half we reuse. + +## Glossary + +Terms the rest of this folder uses without re-explaining. + +- **Playground / composer**: the agent chat surface; the composer is its message + input box. +- **Vault**: the per-project store of the user's own provider API keys + ("secrets"). "Vault empty" is what triggers today's wall. +- **BYOK**: bring your own key; the user pays their provider directly. Agenta's + normal mode. +- **Self-managed connection**: a mode where the harness signs in with the user's + own Claude/ChatGPT subscription instead of an API key; no vault key needed. +- **Harness**: the agent runtime (Pi and peers) that owns the system prompt, + skills, and tools, and makes the actual LLM calls. +- **Runner / sandbox**: the service that executes agent runs inside Daytona + cloud sandboxes; the sandbox environment is controlled by the user's agent. +- **Entitlements**: the EE quota system answering "may this org do X, and how + much". Its counters are **meters**: rows incremented atomically in Postgres, + with limits defined per plan (a **quota**). A quota with no period never + resets; that is a lifetime counter. +- **Grant**: (this design) an append-only record giving an org a trial + allowance. +- **Reservation**: (this design) the durable record that authorizes and charges + exactly one funded run. +- **Gateway**: (this design) the small Agenta-side proxy that holds the real + provider key and forwards funded model calls under budget enforcement. +- **Daytona Secrets**: Daytona's mechanism where a sandbox env var holds an + opaque placeholder and Daytona's egress proxy substitutes the real value only + into HTTPS requests toward an allowlisted host. +- **Responses API**: OpenAI's agent-native API surface (successor to plain chat + completions for tool-using workloads); relevant because the gateway must speak + it. +- **Prompt caching**: provider-side reuse of a repeated prompt prefix at a + fraction of the input price; our replayed 23.6K harness context makes this the + dominant cost lever. diff --git a/docs/design/activation-credits/design.md b/docs/design/activation-credits/design.md new file mode 100644 index 0000000000..ad25dd65c0 --- /dev/null +++ b/docs/design/activation-credits/design.md @@ -0,0 +1,311 @@ +# Design: platform-funded trial runs + +This document specifies the architecture. context.md explains the situation and +defines terms; research.md carries the evidence each choice rests on; rfc.md +lists the decisions requested. Numbers cited here (costs, token counts, file +references) are sourced in research.md. + +## The shape of the problem + +Three facts, together, dictate the architecture: + +1. **The run executes in a runtime the user controls.** Playground agents run in + a Daytona sandbox driven by the user's instructions and tools. A provider key + placed there must be assumed stolen: today's delivery injects resolved keys + into the sandbox environment, and "print your environment" is a valid agent + instruction. A stolen platform key bypasses every counter, cap, and kill + switch we could build elsewhere. So the platform key can never enter the + sandbox, which forces a proxy of some kind between the sandbox and the + provider. +2. **Spending must be authorized per run, atomically.** The trial balance is + small (about thirty runs). Two browser tabs racing for the last run, a + network retry of the same send, or a crash between "charged" and "counted" + must not double-spend or spend uncounted. The entitlements meter gives + atomic counting; the design must give it a spend lifecycle around it. +3. **Only some surfaces may spend.** Interactive playground runs and direct + invoke calls should draw on the trial; evaluations, batch, and background + invocations must never. But all of these converge onto the same workflow + invocation path in the backend, and everything the client sends is + forgeable. The funding decision needs provenance minted server-side, before + the paths converge. + +Everything else is reuse: the meter engine counts, the billing UI displays, the +existing banner and auto-send machinery carry the UX, and the in-flight Daytona +secret-delivery work hardens the transport. + +## Component 1: the grant record (who is eligible, for how much) + +At signup, the newly provisioned personal organization gets one append-only +grant row: organization, amount, granted-at, and a stable campaign identifier, +with a uniqueness constraint on (organization, campaign) so a retried signup +hook cannot grant twice. + +Why a grant record and not the two obvious alternatives: + +- **Not a plan-catalog quota.** Quota limits are plan-wide code constants. A + lifetime quota on the free plan would hand the full allowance to every + existing organization on that plan the day it ships; there is no per-org + override mechanism to carve out "new signups only". The grant row is that + mechanism, minimal. +- **Not a mutable balance column.** A writable "remaining" beside the meter + creates two sources of truth that drift. The balance is computed: grant total + minus meter value. Only the meter moves. + +What the grant row buys beyond eligibility: per-cohort amounts (ship cohorts at +20 and 50 and let funnel data pick the number), a rollout gate (no grant, no +feature, so shipping to a percentage of signups is just granting to a +percentage), and the future seams: earned credits and sales promos become more +grant rows with different campaign identifiers, touching nothing else. + +Eligibility is deliberately narrow: the personal organization provisioned by +the **signup** path only. The backend distinguishes signup provisioning from +explicit organization creation (they are different code paths), and funding +every new org would invite farming orgs for fresh grants. Accepted consequence: +an invited teammate can join an org whose balance is spent; such orgs +overwhelmingly have keys already. + +## Component 2: the reservation (authorizing and charging one run) + +A durable record with an explicit lifecycle: + +``` +reserved ── started ── charged + │ + ├── refunded (run died before its first model call) + └── expired (never started; reconciliation sweep) +``` + +Created server-side when a funded run is accepted, keyed uniquely by a +server-minted invocation identity (component 3), bound to the authenticated +organization, project, and model. One reservation authorizes exactly one run. + +**The economic invariant.** The meter's strict conditional increment and the +reservation's authorizing transition commit in one database transaction, and +the gateway mints a token only for a committed reservation. A refund pairs the +terminal transition with the decrement the same way. Consequences of the +invariant: a retried request lands on the same reservation row (idempotent by +the unique key) instead of charging twice; a crash between increment and +transition is impossible because they are one commit; a crashed refund handler +retries into the same transition instead of refunding twice. Without this, all +three failure modes are live: increment-then-crash charges without +authorizing, charge-then-fail-to-count spends unmetered, and a double refund +mints free runs. + +One named refactor makes it possible: `MetersDAO.adjust` currently opens and +commits its own session, so it must learn to join a caller-owned transaction +(or the conditional upsert moves into the reservation repository). The strict +SQL predicate itself is kept exactly as is. + +The consumption counter is a new lifetime meter (working name +`trial_runs_consumed`, `period=None`). The existing monthly `credits_consumed` +is not reused: a monthly counter would silently refill the trial every month. +The reservation service passes the grant-derived total as the meter limit, so +the plan catalog is untouched. + +**Failure policy is fail-closed.** The entitlements house pattern fails open on +infrastructure errors, which is right for feature gates (a Redis blip should +not lock users out of features) and wrong here (a Redis blip should not spend +our money unmetered). The reservation service is a deliberate, documented +deviation. + +**Charge policy.** The committed reservation is the charge. It is refunded if +the run dies before its first model call (validation failure, runner startup +crash, provider down), and kept for anything after: partial runs consumed real +tokens. + +## Component 3: funding provenance (which surfaces may reserve) + +The API orchestration layer, at the point where it still knows how a run was +invoked, mints a short-lived signed invocation-purpose claim bound to +organization, project, and a fresh idempotency identity. Only interactive +playground and direct invoke acceptance mint it; the evaluation, batch, and +background paths never do. The reservation service accepts only this claim, and +the claim's identity is the reservation's unique key. + +Why this indirection instead of a simpler flag: the invocation paths converge +before credential resolution, wire metadata like a `run_kind` field is +caller-controlled, and the runner's own turn identifier is created after +connection resolution, too late to serve as the idempotency key. The claim is +the one artifact that is both unforgeable (signed, server-minted) and early +enough (minted at acceptance). + +## Component 4: the inference gateway (the trust boundary) + +Funded runs receive no provider key. The resolver returns a connection whose +endpoint is the gateway's URL and whose credential is an opaque, short-lived +token minted from the committed reservation and bound to it: organization, +model, request count, token budget, expiry. + +The gateway holds the only copy of the trial provider key and, per request: + +- validates the token and rejects expired ones; +- canonicalizes the request: allowed parameters only, forced provider + data-retention settings, exactly the allowlisted model; +- takes an **atomic hold** against the reservation's remaining budget before + forwarding, and settles the hold to actual usage after the response. Check + followed by decrement is not enough: two concurrent calls both pass the + check; the hold is the decrement, taken conditionally. A call that times out + ambiguously keeps its hold until reconciliation (OpenAI stores responses, so + late settlement can read the true usage), which prevents a duplicate retry + from being funded twice; +- enforces replay protection per provider call (a call identity plus a + canonical request hash), not per token, because one token legitimately makes + several calls within its budgets; +- streams the response through byte-faithfully (no frame re-encoding, no + buffering) while tapping the final usage event, and emits the usage record to + our metering. This also closes an observability hole: harness-side + instrumentation provably drops usage for at least one provider today, and the + gateway meters at the only chokepoint that cannot be bypassed. + +**API surface.** One provider, one model, one API surface for the MVP, and the +surface is the provider's agent-native API (for OpenAI, the Responses API), +because the funded runs are tool-using and that is the protocol the vendor +builds agent features on. The pre-launch quality test exercises real tool calls +end to end rather than trusting either API's documentation. The runner needs +one contract addition, not new key trust: the platform-resolved connection +carries an explicit API-dialect field, because the runner's custom-provider +path currently assumes and emits only the chat-completions dialect and would +otherwise speak the wrong protocol through the gateway. + +**Build, not adopt.** No surveyed OSS gateway implements reservation-bound +hold-and-settle (research.md §8); adopting one means operating someone else's +platform while still writing the only hard part inside their plugin system. The +build is one FastAPI route plus the ledger, roughly 600-900 lines, one to two +engineer-weeks, on Redis and Postgres we already run. The ledger is also the +piece a future paid managed-credits product extends, and it stays the budget +authority if a multi-provider translation layer ever slots in behind it. + +**What a leaked token is worth.** Extracted from the sandbox, it buys minutes +of one cheap model inside an already-metered budget. That is the whole point of +the boundary: theft of the credential is no longer catastrophic, so the rest of +the guardrails only have to keep honest users honest. + +## Composition with Daytona secret delivery + +The in-flight Daytona work (#5223 design, #5277 implementation, #5278 +reconciliation proposal) delivers credentials as placeholders that only +Daytona's egress proxy resolves, toward one allowlisted host. It hides the +value completely (possession) and bounds nothing about how it is used toward +that host (usage). Research.md §7 has the mechanism and its limits. + +The two compose without redesign: the trial connection's endpoint is the +gateway host, and the reservation token rides the #5277 delivery path as an +ordinary `opaque_http` credential once that path re-lands on the refactored +provider interface. Then even the token is unreadable inside the sandbox, and +the gateway remains the usage bound. Shipping on Daytona Secrets alone, with a +raw provider key and the provider's host allowlisted, is explicitly rejected: +it would hide a key that can still be spent without limit. Expired trial +reservations are cleaned by the reservation service's own sweep; they do not +ride #5278's Daytona-scoped reconciliation. + +## Frontend + +The single boolean gate becomes a small state machine: loading, user key +available, funded run available, allowance exhausted, funding unavailable +(kill switch or outage), selected model not funded. Rules that keep it honest: + +- The reservation outcome is the authority on whether a run proceeds and what + remains. The polled usage query (2-minute stale time) is display data for the + countdown, never authorization. +- A failed or loading balance query renders as unknown, never as zero. Nothing + is worse than showing a new user "0 free messages" because a query flaked. +- On exhaustion, the send is refused before any model call; the banner slot + shows the connect-your-key message; the typed draft is retained and auto-sent + once a key lands. The existing auto-send covers only the seeded first prompt + today (a refused arbitrary draft becomes a transcript error), so draft + retention is new state with its own tests. +- The countdown appears after the first response, not before the first send. + Nothing about the free experience should add friction before the aha moment. +- The model picker stays visible; non-trial models show the normal + connect-a-key affordance. The picker is presentation; the gateway enforces. +- The template drawer's "Agenta-managed / Ready" label becomes true for the + trial model instead of being the false promise it is today. + +## Surface matrix + +| Surface | Behavior | +|---|---| +| Agent playground (cloud) | Funded: provenance claim → reservation → gateway token | +| Direct agent API calls (invoke path) | Same as playground; reservations and throttles apply | +| Evaluations, batch, background | Never funded; no claim is ever minted for them | +| Classic prompt playground | Unchanged; different credential path, near-zero cost, not the activation surface | +| OSS / self-hosted | No grants, no gateway; resolver behavior unchanged | + +EE/OSS seams: grants, reservations, provenance, and the gateway are cloud-only +behind the established `is_ee()` pattern. The SDK and agent-service touchpoints +(the dialect field, the resolver's managed-connection branch) are shared code +that no-ops when the backend issues no grants. + +## Guardrails, stacked + +From inner to outer, each assuming the previous failed: + +1. Gateway budgets: per-call caps, per-run call and token budgets, model + allowlist, at the request boundary (an oversized first call is refused, not + observed after the fact; the runner's existing limits are time-only). +2. Reservation: no committed reservation, no token, no run. +3. Daily throttle per organization on trial runs. Note the existing throttle + middleware is a per-minute token bucket; a calendar-day cap is either a + small daily meter or an accepted approximation. +4. Isolated provider project: hard project-level spend limit (enforcement lag + acknowledged), auto-recharge off, alerts, one-click revocation. +5. Kill switch: a backend flag marking managed funding unavailable, read by the + frontend state machine. Deleting the env key is not the mechanism: the + frontend would still show balance and then fail mid-run. +6. Grant size itself: worst case per fake account is cents. Mass fake signups + stay an email-verification problem, deliberately. + +## Deployment and migration order + +1. Postgres enum migration for the new meter key; grant and reservation tables + (small, new). Migration before code. +2. `MetersDAO` caller-owned-transaction refactor. +3. Reservation service + provenance claim minting + gateway service (compose + stack, holds the only copy of the trial key). +4. SDK/agent-service: dialect field, managed-connection branch in the resolver. +5. Frontend state machine, countdown, exhausted state, draft retention. +6. Usage API surfacing (the billing page renders the new counter without + changes once reported). +7. Trial provider account setup: isolated project, hard cap, no auto-recharge, + prepaid load sized to the experiment ($500 to start). + +Prerequisites tracked separately: the OpenRouter usage-instrumentation gap +(file an issue; the gateway meters trial runs regardless, but observability +stays blind for everything else), and the audit of stray `*_API_KEY` values on +the cloud completion service (legacy-path runs would be free and unmetered +today if any are set). + +## Rollout, measurement, kill criteria + +Gate on backend funding flag plus grants; grant to a percentage of new signups +first, watch spend per signup against the measured envelope, widen. + +Measure: activation rate (first successful playground message per signup), +key-connection rate within 7 days, platform spend per signup, and trial-to-key +conversion at the exhaustion moment. Kill when spend per signup exceeds 3x the +envelope, or when key-connection collapses while activation merely shifts +messages onto our key (the trial must move the wall, not become the product). + +## Risks and open questions + +- **Trial model quality.** The cheap models are positioned for classification + workloads; if the first session feels dumb, activation gains evaporate. The + pre-launch quality test (real tool-using sessions, both candidate models, + through the gateway path) is the gate on D2, and it is the one place this + design says "measure before deciding". +- **Prompt caching through the gateway.** The cost envelope's best case assumes + caching works through our pass-through (stable prefix, provider-side cache + keys). If it does not, costs run 5x higher, still inside budget, but worth + verifying in the same pre-launch test. +- **#5277 timing.** The composition's hardening half depends on the Daytona + delivery re-landing after the runner refactor. The gateway does not wait for + it; the token-in-env interim is acceptable because the token is scoped and + expiring. +- **Session lifetime vs token expiry.** Warm sandboxes outlive single runs; + token expiry and per-reservation budgets must line up with session semantics + (a token per run, not per sandbox). +- **The exhaustion moment is the product.** Everything funnels to one screen: + "you used your free messages, connect a key". If that screen converts, the + program pays for itself; if it reads as a paywall trap, we bought nothing. + Copy and behavior (draft preserved, one-click to the drawer, auto-send on + connect) deserve design attention disproportionate to their code size. diff --git a/docs/design/activation-credits/implicit-decisions.md b/docs/design/activation-credits/implicit-decisions.md new file mode 100644 index 0000000000..5e7c2b25d0 --- /dev/null +++ b/docs/design/activation-credits/implicit-decisions.md @@ -0,0 +1,114 @@ +# Implicit decisions and requirements register + +What Mahmoud asked for: new users send their first playground messages on our +key; cap after some number of requests; a cheap model; $3,000 covering ~10,000 +signups; reuse the entitlements architecture; credits as abstract units, simple +first, extensible later; competitor names out of the repo; take the Daytona +secrets PR (#5277) into account; prefer a lightweight gateway if one is needed. + +Everything else in this folder entered the design some other way. This file +lists each such decision, its origin, and what striking it would cost, so the +reviewer can cut scope deliberately. Origin codes: + +- **[inference]** I introduced it from research or product judgment. +- **[review]** the external review loop (five adversarial passes) forced it. +- **[codebase]** the existing code leaves no real choice. + +## Load-bearing (striking these changes safety or correctness) + +1. **The platform key never enters the sandbox; a gateway fronts funded calls.** + [review, then verified] The sandbox is user-controlled and today's delivery + puts resolved keys in its environment; a platform key there is stealable and + its theft bypasses every cap. Striking this means accepting that any user + can extract our key. The Daytona-secrets path alone hides the key but does + not bound its use (research.md §7), so it cannot substitute. +2. **A new lifetime counter instead of reusing `credits_consumed`.** [codebase] + The existing counter is monthly; reusing it refills every trial monthly. + No design freedom here beyond the counter's name. +3. **Eligibility via a grant record, not a plan-catalog quota.** [review] + A plan-level lifetime quota grants the full allowance to every existing org + on the plan at ship time; there is no per-org override mechanism. Striking + the grant record means either gifting all existing orgs or building some + other new-orgs-only mechanism, which is the same thing under another name. +4. **Atomic spend: counter and authorization commit together.** [review] + Without it, a crash window can charge twice, spend uncounted, or refund + twice. The blast radius of striking it is small in dollars (a trial run + costs fractions of a cent) but it corrupts the balance users see; I would + keep it because the cost is one transaction boundary, not a system. +5. **Funding decided server-side, never from client-sent metadata.** [review] + Anything the client sends is forgeable, and evaluations converge onto the + same invoke path. Striking this means a crafted request can fund + non-playground workloads. + +## Defensible but negotiable (engineering conservatism; strippable with eyes open) + +6. **The signed invocation-purpose claim.** [review] The hardened form of #5. + A simpler form exists: an ordinary server-side check at the invoke route + that creates the reservation there, with a plain idempotency key. The claim + adds unforgeability across service hops; the MVP has few hops. Strippable + to the simple form for MVP. +7. **Per-call replay protection (call identity + canonical request hash).** + [review] Guards against a duplicated provider call inside one token's + budget. The per-run budget already bounds the damage to the run's own + budget. Strippable; keep the per-run budget. +8. **Holds retained on ambiguous timeout + late settlement + reconciler.** + [review] Prevents a duplicate retry being funded twice mid-run. Worst case + if stripped: a run occasionally gets one extra funded call. Strippable to + "expire holds after N minutes" for MVP. +9. **Reservation lifecycle with refunds.** [review] The refund path (run died + before first model call) protects users from losing a run to our own + crashes. Strippable to "no refunds, runs are cheap" at the cost of + occasional support tickets and a worse-feeling countdown. +10. **Explicit API-dialect field on the platform connection.** [review + + codebase] Needed because the runner's custom-provider path speaks only + chat-completions today and the gateway should speak the agent-native API. + Strippable only by pinning the gateway to chat-completions and validating + tool calling works there for the chosen model; that validation replaces + the field. +11. **Fail-closed reservation checks.** [inference] House pattern is + fail-open; I flipped it for money. Striking it means an infrastructure + blip silently funds unmetered runs; keeping it means a blip blocks trial + runs (users with keys unaffected). I would keep it; it is one branch. +12. **Kill switch as a backend flag rather than deleting the env key.** + [review] Deleting the key mid-flight strands users who still see balance. + Strippable if we accept ugly mid-run failures during an emergency stop. + +## Product choices I made without being asked (review these as product, not engineering) + +13. **Thirty runs, one-time, never recurring.** [inference] Thirty ≈ 3x the + measured 10-turn session; one-time because our credits end at BYOK, + unlike competitors who sell credits forever. The grant record makes the + amount a per-cohort parameter, so this is a starting value, not a design + commitment. +14. **Unit = one message/run, shown as "free messages", not tokens or money.** + [inference] Matches "credits as abstract units" and the competitor + legibility findings. Token-denominated units are meaningless to new users. +15. **Org-scoped, signup-only grants.** [inference + review] Follows the grain + of subscriptions; signup-only closes org-farming. Consequence: invited + teammates can meet an exhausted balance. +16. **gpt-5.4-nano as primary candidate, decided by a tool-using quality + test.** [inference; review added the tool-test condition] Price and brand + argue for it; vendor positioning argues for testing before committing. +17. **Countdown appears after the first response; picker stays open with + connect-affordances on non-trial models; exhausted state preserves the + draft and auto-sends on key connect.** [inference] UX judgment calls, + reviewable from the design.md frontend section. +18. **Build the gateway (600-900 lines) rather than adopt LiteLLM/other.** + [inference, answering Mahmoud's direct question] Grounded in the survey: + nothing implements reservation hold-and-settle, so adoption still means + writing the hard part. research.md §8 has the evidence. +19. **Daily per-org throttle on trial runs; per-call token caps.** [inference] + Abuse hygiene sized from the pricing research. +20. **Rollout by cohort with kill criteria (3x spend envelope, key-connection + collapse).** [inference] The specific thresholds are guesses to be tuned. + +## What the review loop cost + +For honesty: passes 1-2 corrected real errors (wrong enforcement path; key +leak into the sandbox) and those corrections are the load-bearing section +above. Passes 3-5 mostly added items 6-12: correct engineering, but each is a +requirement Mahmoud never asked for, and together they roughly doubled the +specified surface. rfc.md's "smallest safe version" shows the design with +items 6-9 and 12 stripped; the delta between that and the full design is the +price of the review loop's conservatism, and it is the reviewer's call whether +to pay it now, later, or never. diff --git a/docs/design/activation-credits/proposal.md b/docs/design/activation-credits/proposal.md deleted file mode 100644 index 05da0141de..0000000000 --- a/docs/design/activation-credits/proposal.md +++ /dev/null @@ -1,488 +0,0 @@ -# Free trial credits at signup - -**Status:** proposal, for review -**Scope:** Agenta cloud only. Self-hosted deployments keep today's behavior. - -## Summary and decisions requested - -New signups today must paste an LLM provider API key before the playground sends -their first message. This proposal removes that wall: every new cloud organization -gets a small number of free agent runs, funded by an Agenta-owned provider key on -one cheap model. When the free runs are gone, we ask for the user's key at a moment -when the product has already shown its value. - -The build reuses the existing entitlements meter for consumption and adds four -pieces: a grant record written at signup, a reservation that authorizes each funded -run, a server-side funding policy that decides which surfaces may reserve, and a -small inference gateway so the provider key never enters user-controlled runtime. -The gateway is the one genuinely new component; it is also the first slice of the -credential-isolation boundary we already designed for secrets generally, and the -substrate a future paid managed-credits product would extend. - -Decisions requested from this review: - -1. The trial model (D2) and the starting allowance (D3). -2. The grant-record eligibility design (D4) and its signup-only scope (D5). -3. The credential-protection composition (D6): the gateway as the usage bound, - with the in-flight Daytona secret-delivery work (#5277) as its hardened - transport, built in-house rather than adopted (D7, sized at one to two - engineer-weeks). -4. Sign-off on the guardrail set before any platform key goes live. - -## The problem and the hypothesis - -A new user signs up, lands in the playground, describes the agent they want, and -commits it. At that exact moment the message box grays out, a yellow banner asks -them to connect an API key, and the payoff they just worked for is withheld. - -This proposal takes the funnel problem as its premise: the key wall sits directly -on the signup-to-first-message step, and removing it is the motivating goal of -this work. The experiment below measures how much the wall actually costs us -rather than assuming it. - -Competitors do not have this wall. The AI-first agent platforms we compete with -fund a new user's first AI usage; a signup runs a real agent without seeing a key -form. Only the older workflow-engine incumbents require your own key. Competitor -names and per-product details are withheld from the repo by policy; the named -research notes are available on request. - -**Hypothesis:** letting the first conversation happen before the key ask raises -activation (first successful playground message per signup) without collapsing the -key-connection rate; it moves the wall, it does not remove it. - -## Cost envelope and guardrails - -We measured production traces on EU cloud (2026-07-23, via -`POST /api/tracing/spans/query`, reading the `ag.metrics.tokens.*` span -attributes). Every agent LLM call carries roughly 23,600 tokens of harness context -(system prompt, skills, tool definitions); the user's words and the model's replies -are noise next to it. A 10-turn session totals about 242K input tokens and 1.2K -output tokens. Caveats: the sample was small (8 agent traces from our own team -project; API keys are project-scoped, so organic cross-tenant data needs database -access), and the sampled turns made one LLM call each, so sessions with heavy tool -use will run 2-3x higher. - -Cost per 10-turn session at July 2026 list prices: - -| Model | No caching | With prompt caching | -|---|---|---| -| Gemini 2.5 Flash-Lite | $0.025 | $0.004 | -| DeepSeek V4 Flash | $0.034 | $0.005 | -| OpenAI gpt-5.4-nano | $0.050 | $0.009 | -| Claude Haiku 4.5 | $0.248 | ~$0.05 (verified against a real bill) | - -At 10,000 signups on gpt-5.4-nano this is roughly $500 without caching and under -$100 with caching, against a $3,000 budget. Prompt caching is the largest cost -lever because our workload replays a large stable prefix on every call. - -The budget survives honest users easily; the guardrails exist for dishonest ones. -An agent run is a loop: one user message can trigger many model calls with large -inputs, and the agent's instructions and tools are user-controlled. The guardrails, -all enforced server-side: - -- **The provider key never enters user-controlled runtime.** Agent runs execute in - a sandbox the user's instructions effectively control; a raw key placed there can - be extracted and reused outside every other control. Funded runs therefore talk - to an Agenta inference gateway with a short-lived token instead of a key - (architecture below). This is the load-bearing guardrail; the rest assume it. -- **Per-call and per-run budgets at the request boundary.** The gateway enforces - the model allowlist, a max output size per call, and the reservation's call and - token budgets before forwarding anything to the provider. Enforcement after the - fact cannot cap an oversized first call; enforcement at the request boundary can. - The runner's existing wall-clock limit stays as the outer bound. -- **Reservation before spend:** no reservation, no gateway token, no funded run. -- **Isolated provider project:** the trial key lives in its own provider project - with a hard project-level spend limit (OpenAI supports these; enforcement can lag - slightly), auto-recharge disabled, budget alerts, and easy revocation. -- **Kill switch:** a backend flag that marks managed funding unavailable; the - frontend reads it and falls back to today's connect-a-key behavior. Deleting the - env key is not the mechanism, because the frontend would still show credit and - then fail mid-run. - -Mass fake signups remain bounded by the grant size (cents per org), signup-only -eligibility (D5), and existing email verification; we accept that residual risk for -the MVP. - -## The user experience - -Vocabulary for this section: the **composer** is the playground's message input -box; the **vault** is the per-project store of the user's own provider API keys. - -### First session (free runs remaining) - -Signup and onboarding stay exactly as they are. The one change: when the user -commits their agent, the composer stays enabled and their first message sends. The -run is funded by Agenta on the trial model. The payoff arrives on message one. - -After the first response, a quiet balance badge appears near the composer: "29 free -messages left". It exists so the later wall is expected rather than sudden. The -Billing settings page shows the same balance through its existing usage meters. - -Model choice during the trial: the model picker stays visible, but models other -than the trial model show their normal "connect a key" affordance. Funded runs -never execute on any other model, and the gateway enforces that, not the picker. - -We also fix a related inconsistency: the template setup drawer currently labels -Agenta-managed models "Ready", promising platform keys the runtime does not -provide. After this change the label becomes true for the trial model. - -### The wall (free runs exhausted) - -When the balance reaches zero, the next send is refused before any model call. The -existing yellow banner slot shows: "You've used your free messages. Connect your -API key to keep going, it takes a minute." with the same button that opens the -provider-credentials drawer. The user's typed message must survive this refusal and -send automatically once a key is connected. The playground has a similar -connect-then-send behavior today, but only for the seeded first message; a refused -arbitrary draft currently becomes a transcript error, so retaining and resuming it -is new frontend state, called out in the plan below. - -Users who connected a key never see any of this: their runs resolve their own vault -key and consume nothing. - -### Frontend states - -The gate today is one boolean (vault empty). It becomes a small state machine with -explicit states: loading, user key available, funded run available, allowance -exhausted, funding unavailable (kill switch or outage), and selected model not -funded. Two rules keep it honest: the server's reservation outcome is the authority -on whether a run proceeds and on the remaining balance (the polled usage query is -display-only and can be minutes stale), and a failed or loading balance query must -never be rendered as "you have zero left". - -## Architecture - -### Where credentials come from today, and what we do not build on - -Agent playground runs resolve credentials through `resolve_connection` -(`services/oss/src/agent/app.py`, `sdks/python/agenta/sdk/agents/platform/`): given -the selected model, the resolver fetches the project's vault secrets and picks the -one matching connection. When no connection exists, vault-backed resolution fails -with a typed error (default and self-managed modes instead degrade to -runtime-provided credentials); for a keyless new user on a standard provider, that -failure is today's wall. The resolved credential then travels into the run as -environment variables, which is fine for the user's own key and is exactly what the -gateway exists to avoid for ours. - -An older, model-blind path in the SDK's vault middleware can fall back to provider -keys taken from the service environment. It is a leftover of PR #2957, which built -a first version of platform-funded usage with a `credits_consumed` meter; the agent -service explicitly abandoned that path, its metering call is orphaned, and only -legacy completion workflows still traverse it. This proposal does not build on it. -We reuse its meter machinery for consumption accounting, and we schedule one -hygiene task (out of scope here): audit the cloud service environment for stray -provider keys, since any that exist make legacy-path runs free and unmetered today. - -### In-flight work this composes with: Daytona secret delivery - -Three open PRs already attack the "secrets inside the sandbox" problem and shape -this design. PR #5223 plans, and PR #5277 implements (flag-gated, currently parked -until the runner refactor reshapes the sandbox-provider interface it builds on), -delivery of credentials through Daytona Secrets: the sandbox environment holds -only an opaque placeholder, and Daytona's egress proxy substitutes the real value -into outbound HTTPS requests, but only toward the credential's allowlisted -hostname. PR #5278 proposes (design only, nothing implemented yet) a -reconciliation domain so crashed runners cannot orphan those Daytona Secret -bundles. Together this gives true possession-hiding: no process inside the -sandbox can read the value. - -What it deliberately does not give is usage control. The only lever is the -destination hostname, so a delivered provider key could still be spent without -limit against the provider's own host: any model, any volume, outside our -metering, for the sandbox's lifetime. Its own design doc names the general -solution: move the provider call behind an Agenta gateway. That division of labor -drives D6 below, and this proposal reuses the #5277 machinery wholesale where it -fits: the typed credential contract with fail-closed classification, the exact- -hostname validation, and the per-sandbox secret lease lifecycle. Expired trial -reservations are cleaned by the reservation service's own sweep, described -above; they do not ride #5278's Daytona-scoped reconciliation. - -### The four pieces - -**1. Grant record (eligibility).** When a user signs up, their newly provisioned -personal organization gets an immutable grant row: amount, granted-at, and a -stable campaign identifier (not descriptive prose), with a uniqueness constraint -on (organization, campaign) so a retried signup hook cannot grant twice. No grant, no free runs, which cleanly -excludes every existing organization (a plan-level quota alone would gift the full -allowance to all of them the day we ship). Organizations created later by an -existing user get no grant; that closes the create-orgs-to-farm-credits hole. -The balance is never stored: remaining = grant total minus the consumption meter. -Later gamification ("connect GitHub, earn more runs") and sales promos become -additional grant rows with their own reasons; nothing else changes. - -**2. Reservation (authorization and spend).** A durable record with an explicit -lifecycle: `reserved → started → charged`, or `reserved → refunded`, or -`reserved → expired` for runs that never start, with a reconciliation sweep for -abandoned rows. It is created server-side (not by the frontend), keyed uniquely by -a server-minted invocation identity (below) so retries cannot double-charge, and -bound to the server-authenticated organization, project, and model. One -reservation authorizes exactly one run. The economic invariant: the reservation's -state transition and the consumption-counter change commit in **one database -transaction**, so no crash window can charge twice, charge without counting, or -refund twice; a retried handler lands on the same transition. Consumption uses a -new dedicated lifetime meter (working name `trial_runs_consumed`); the monthly -`credits_consumed` quota, the plan catalog, and the billing projection stay -untouched. The exact sequence: the strict conditional increment and the -reservation's authorizing transition commit in the same transaction, and only -after that commit does the gateway mint a token; the gateway rejects tokens for -any reservation that has not completed it. A refund pairs the terminal transition -with the decrement in the same way. This needs one named refactor: the meter DAO -currently opens and commits its own session, so it must learn to join a -caller-owned transaction (or the conditional upsert moves into the reservation -repository). The reservation service supplies the grant-derived total as the -meter's strict limit, and it fails closed on any infrastructure error, a -deliberate deviation from the entitlements house pattern (feature gates fail -open; spending our money must not). Charge policy: the committed reservation is -the charge; it is refunded if the run dies before its first model call and kept -for anything after. - -**3. Funding policy and provenance (which surfaces may reserve).** Today -interactive playground runs, direct API calls, and evaluations converge on the -same workflow invocation path, and wire metadata is caller-controlled, so the -funding decision cannot rest on anything the client sends. Instead, the API -orchestration layer, at the point where it still knows how the run was invoked, -mints a short-lived signed invocation-purpose claim bound to organization, -project, and a fresh idempotency identity. Only interactive playground and direct -invoke acceptance mint the claim; evaluation, batch, and background paths never -do. The reservation service accepts only this claim, which also supplies the -reservation's unique key (the runner's own turn identifier is created too late to -serve). Request metadata can never turn funding on. - -**4. Inference gateway (the trust boundary).** Funded runs receive no provider -key. The resolver returns a connection whose endpoint is an Agenta gateway URL and -whose credential is an opaque, short-lived, single-reservation token bound to -organization, model, request count, token budget, and expiry. The gateway holds -the real provider key, validates the token, canonicalizes the request (allowed -parameters only, intended provider retention settings forced, expired tokens -refused), enforces the model allowlist, and applies budgets at the request -boundary: each provider call takes an atomic hold against the reservation's -remaining budget before it is forwarded and settles to actual usage after, so -concurrent calls cannot overrun the budget between a check and a decrement. A -token allows several provider calls within its budgets, so replay protection is -per call (a call identity plus a canonical request hash), not per token; a call -that times out ambiguously keeps its hold until reconciliation, so a duplicate -retry cannot be funded twice. Extracting the token from the sandbox yields -minutes of access to one cheap model inside one already-metered budget, instead -of an open provider key. Scope for the MVP: one provider, one model, one API -surface, and we choose the provider's agent-native API as that surface (for -OpenAI, the Responses API), because our runs are tool-using and that is the -protocol the vendor builds agent features on; the pre-launch check below -verifies tool calling end to end rather than trusting either API's -documentation. The runner side needs one contract addition, not new key trust: -the platform-resolved connection must carry an explicit API-dialect field, since -the runner's custom-provider path currently assumes and emits only the -chat-completions dialect and would otherwise call the wrong API through the -gateway. -This is deliberately the smallest version of the credential-isolation gateway -already designed for the secrets roadmap, and the same chokepoint gives us trial -usage metering that does not depend on harness-side instrumentation. Delivery -composes with the Daytona secret work above: the trial connection's endpoint is -the gateway host, and the reservation token rides the #5277 delivery path as an -ordinary opaque credential once that path re-lands on the refactored provider -interface, so even the token is unreadable inside the sandbox (D6). - -### Surface matrix - -| Surface | Trial behavior | -|---|---| -| Agent playground (cloud) | Funded: reservation on the invoke path + gateway token | -| Direct agent API calls (invoke path) | Same as the playground; reservations and rate limits apply | -| Evaluations, batch, background runs | Never funded; user keys only (funding policy) | -| Classic prompt playground | Not funded in MVP; different credential path, near-zero message cost, not the activation surface | -| OSS / self-hosted | No grants, no gateway; resolver behavior unchanged | - -The grant, reservation, policy, and gateway are cloud-only, behind the established -`is_ee()` seams. The SDK and agent-service touchpoints are shared code that no-ops -when the backend issues no grants. - -### Deployment notes - -The consumption counter needs the Python enums, a Postgres enum-value migration, -and usage-API surfacing, deployed migration-first. The grant and reservation -tables are new and small. The gateway is a new small service (or route on an -existing one) holding the only copy of the trial key. The runner needs no trust -changes; its wall-clock limit stays. - -## Decisions - -### D1: What is the unit? - -- **Option A: one unit = one accepted agent run** (one user turn, however many - model calls it makes inside its budgets). Legible in product copy, maps - one-to-one to reservations, and the cost variance between cheap and expensive - turns is bounded by the per-run budgets and small in dollars. -- **Option B: token-denominated credits.** Precise, but meaningless to a new user, - and the 23.6K harness context makes even "hi" cost thousands. - -**Recommendation: A.** Call them "free messages" in product copy for the MVP. We -adopt the word "credits" product-wide only when the unit becomes fungible (earned, -purchased, model-tiered), which is the roadmap but not this build. - -### D2: Which model funds the trial? - -- **Option A: OpenAI gpt-5.4-nano.** Hard project-level spend limits on the - provider side, strong prompt caching, a brand new users recognize so demo - quality is attributed correctly. ~$0.05 per session uncached. Reasoning effort - `none`. -- **Option B: Gemini 2.5 Flash-Lite.** Half the cost, fast, but postpaid billing - with alert-only budgets, so runaway spend depends entirely on our gateway. -- **Option C: DeepSeek V4 Flash.** Cheapest with prepaid billing, but routing - every signup's first prompt through a China-hosted API is an avoidable objection - for GDPR-conscious European prospects. - -**Recommendation: A, conditional on a quality check.** Price and brand argue for -nano, but its vendor positions it for classification-style workloads, not agent -chat, and the first session must feel good, not just cost little. Before launch, -run our agent harness over a handful of representative activation sessions on nano -and Flash-Lite, including real tool use, and pick on quality-per-dollar. Sequence -matters: the candidates are compared directly against the provider APIs first, and -the production gateway is then built for the winner only, so the one-provider MVP -scope holds and we never build a throwaway adapter. "Model-agnostic" applies to -the grant, reservation, and policy pieces; the gateway itself is -provider-specific work. - -### D3: How large is the grant? - -- **Option A: one-time grant, sized around 30 runs.** Covers the measured - activation session (~10 turns) three times over at an expected $0.05-0.15 per - active user. One-time fits our funnel: unlike competitors who sell credits as - the business model, ours exist only to carry users to connecting their own key, - after which usage is unmetered on their key. -- **Option B: monthly recurring grant.** What most competitors do, but they - monetize credits forever; for us it reopens abuse monthly and makes the meter a - permanent product surface. - -**Recommendation: A, with the amount as an experiment parameter, not a constant.** -The grant record carries the amount per cohort, so we can ship cohorts at 20 and -50 and let the funnel data pick. Thirty is a starting point, not a finding. - -### D4: How is eligibility stored? - -- **Option A: plan-level constant.** Simplest, but wrong: lifetime quotas on a - plan grant the full allowance to every existing organization, there is no - cohorting, and no per-org grants later. -- **Option B: mutable per-org balance column.** Supports variable grants but - creates a second writable source of truth beside the meter. -- **Option C: immutable grant record plus the existing consumption meter.** Grants - are append-only rows with amount and reason; the meter stays the only thing that - moves; remaining is computed. New-orgs-only falls out naturally, as do cohorts, - promo grants, and the gamification seam. - -**Recommendation: C.** - -### D5: Who is eligible? - -- **Option A: the personal organization provisioned at signup, only.** Matches the - goal (activation of new users), keeps grant issuance on the one code path that - runs exactly once per signup, and closes the farming hole of creating - organizations for fresh grants. Accepted consequence: an invited teammate joins - an org whose balance may be exhausted, which is fine because such orgs usually - have keys already. -- **Option B: every newly created organization.** Simpler to state, but the - signup and explicit organization-creation provisioning paths are distinct in the - backend, and funding the latter invites farming. - -**Recommendation: A.** Org-scoped accounting (the grain of plans, subscriptions, -and every existing quota), granted only at signup. - -### D6: How is the platform credential protected? - -- **Option A: Daytona Secrets alone (the #5277 path).** The provider key is - delivered as an opaque placeholder that only Daytona's egress proxy resolves, - toward the provider's host. Kills key theft with work that is already built and - QA'd, but leaves usage unbounded: the sandbox can spend the key freely against - the provider host, unmetered, while it lives. Also covers only Daytona runs. -- **Option B: inference gateway alone.** The sandbox receives a short-lived - reservation-bound token; the gateway holds the key and enforces model, budgets, - and metering. Bounds usage on every execution path, but the token sits readable - in the sandbox environment for its lifetime. -- **Option C: composed.** The gateway is the connection's endpoint; the - reservation token is the delivered credential and rides the Daytona Secrets - path where available. Theft of the token requires beating Daytona's egress - proxy, and even then yields minutes of one cheap model inside a metered budget. - -**Recommendation: C**, built in that order: the gateway is the piece the trial -cannot ship without (it is the only usage bound), and the Daytona delivery -hardens it when #5277 re-lands on the refactored provider interface. Option A -alone is not safe for a platform-funded key. - -### D7: Build the gateway or adopt one? - -We surveyed the July 2026 landscape (LiteLLM proxy, BricksLLM, Portkey's -open-source gateway, TensorZero, Envoy AI Gateway, Kong, APISIX, and the small -budget-proxy projects). The decisive finding: none implements our hard -requirement, the reservation-bound atomic hold-and-settle. LiteLLM records spend -asynchronously after calls, so concurrent requests can overshoot a small budget; -the Kubernetes-native gateways do windowed rate buckets, not reservation ledgers; -the closest conceptual match has been unmaintained for 18 months and predates the -Responses API. Whatever we adopt, we write the ledger ourselves. - -- **Option A: adopt LiteLLM.** Buys provider breadth and pass-through we don't - need, costs a second Postgres-backed service with documented memory-growth - issues, and the ledger still gets written by us inside its hook system. -- **Option B: build small.** One FastAPI route in the existing stack: token - lookup, model check, atomic hold via one Redis script or one conditional - Postgres update, byte-faithful streaming pass-through that taps the final - usage event, metering emit, plus a hold reconciler. Roughly 600-900 lines and - one to two engineer-weeks; no new infrastructure, since the stack already runs - Redis and Postgres. - -**Recommendation: B.** The ledger is also the piece that carries into the future -managed-credits product (a trial reservation generalizes to a purchased balance), -and it should live in our code as the budget authority regardless. If that -product later goes multi-provider, a translation layer (Portkey's MIT gateway is -the natural candidate) slots in behind our ledger without rework. - -## Rollout and measurement - -Gate the whole feature on the backend funding flag plus the presence of grants; -no cohort, no change from today. Ship to a percentage of new signups first, watch -spend per signup against the measured envelope, then widen. - -Success metrics: activation rate (first successful playground message per signup), -key-connection rate within 7 days, and platform spend per signup. Kill criteria: -spend per signup exceeding 3x the envelope, or key-connection collapsing while -activation merely shifts messages onto our key. - -Prerequisite fix: agent runs routed through OpenRouter record no token usage in -our tracing (`ag.metrics` and `gen_ai.usage` both empty). The gateway meters trial -usage regardless, but the instrumentation gap blinds our observability for -everything else and should be filed and fixed independently. - -## Out of scope, deliberately - -- **Earned credits (gamification).** One competitor pays out credits for social - follows and for finishing its tutorial, and it visibly works for them. It lands - here as new grant rows with new reasons, once the MVP proves the funnel. -- **Purchasable managed credits.** Reselling metered inference makes credits a - product with billing and margins; the grant, reservation, and gateway substrate - built here is what it would extend. -- **Funding the classic prompt playground.** Different credential path, near-zero - cost per message, not where activation happens. Revisit only if data says - otherwise. -- **Legacy-path hygiene.** Audit the cloud service environment for stray provider - keys and remove the orphaned metering call in the vault middleware, tracked - separately so this feature does not wait on it. - -## Appendix: research this proposal rests on - -- **UX map of the current gate** (frontend): the composer disables while the - project vault is empty; the banner and the connect-then-send machinery live in - the agent playground's conversation components. -- **Prior art** (backend): PR #2957 built platform keys plus credit metering on - the legacy path; the SDK-side gate call was severed in a later refactor; the - entitlements meter machinery survives and is reused here. -- **Entitlements deep dive**: the meter engine supports lifetime counters, atomic - strict enforcement, and refund deltas; the billing UI already renders any - counter the usage API reports. -- **Pricing research** (July 2026): per-token list prices for the candidate - models, provider billing semantics, and competitor free-tier structures. The - AI-first competitors grant on the order of hundreds to a few thousand - credit-units per month (roughly $1-5 of model cost), several with earn-more - actions or user-set spend caps; the workflow-engine incumbents fund nothing. - Named competitor details stay out of the repo by policy; ask for the local - research notes. -- **Trace measurement** (EU cloud, 2026-07-23): the 23.6K harness context and - session token totals, reproducible against `POST /api/tracing/spans/query` - filtering agent-run spans and reading `ag.metrics.tokens.*`; also the source of - the OpenRouter usage-recording gap. diff --git a/docs/design/activation-credits/research.md b/docs/design/activation-credits/research.md new file mode 100644 index 0000000000..9f56a69b98 --- /dev/null +++ b/docs/design/activation-credits/research.md @@ -0,0 +1,359 @@ +# Research: what we verified before designing + +Eight research threads ran against the codebase, production traces, provider +documentation, and the market (2026-07-23). Each section states what was found, +where the evidence lives, and what it forced in the design. File references are +repo-relative with line numbers from the day of research. + +## 1. How the key wall works today (frontend) + +The yellow banner is `ConnectModelBanner` +(`web/oss/src/components/AgentChatSlice/components/ConnectModelBanner.tsx:44-57`), +rendered above the composer in +`web/oss/src/components/AgentChatSlice/AgentConversation.tsx:2190`. Its copy: +"Connect {provider} to run this agent with your own key", with a "Set up +credentials" button that opens the model-and-harness drawer. + +The gate condition lives in +`web/oss/src/components/AgentChatSlice/hooks/useAgentModelKeyStatus.ts:98-99`: + +``` +gateActive = !loading && vaultEmpty && !selfManaged && !keySetupDone && !!providerEntry +``` + +The load-bearing facts: + +- `vaultEmpty` means the project vault has **zero secrets of any kind**. The + gate is project-wide, not per-provider or per-model. +- `keySetupDone` is a persisted localStorage flag + (`agenta:provider-key-setup-done`); once a user has ever completed key setup, + the gate never fires again on that browser. +- Entitlements and billing play **no part** in the decision. The wall is purely + "vault empty". +- While `gateActive`, the composer is disabled client-side + (`AgentConversation.tsx:750, 2237-2248`); the run is never attempted. + +The onboarding flow stages the wall precisely at the payoff. Signup lands on +`/apps` (`web/oss/src/state/url/postLoginRedirect.ts`, EE inserts a post-signup +survey); with no agents yet, `OnboardingEntry` redirects into the ephemeral +onboarding playground, where the banner is deliberately suppressed +(`AgentConversation.tsx:2186-2190`). The user describes an agent, commits it, +a first prompt is seeded, and only then does the gate fire. When a key is +connected, the seeded prompt auto-sends (`AgentConversation.tsx:1692-1739`, +comment: "Connecting the key IS the go-ahead"). + +Two adjacent findings: + +- The template setup drawer already **promises** platform keys: `ModelRow.tsx` + labels Agenta-managed models "Ready" with the comment "Agenta-managed is Ready + by default (platform keys, no input)". The runtime does not honor that + promise; the product half-claims the feature this project builds. +- The classic prompt playground (`web/oss/src/components/Playground/`) has no + client-side key gate at all; a keyless run there fails at the backend. + +**Design consequence.** The frontend change is small: the gate becomes "no vault +key AND no trial balance", plus a countdown, an exhausted state, and draft +retention. The auto-send machinery is reusable but only covers the seeded first +prompt; retaining an arbitrary refused draft is new state. The real work is +below the frontend. + +## 2. Prior art: PR #2957 and the severed wire (backend) + +The platform-funded idea was built once. PR #2957 ("Limit usage of agenta +provided keys via credits") shipped two halves: + +**Key supply, still live.** The SDK's vault middleware +(`sdks/python/agenta/sdk/middlewares/running/vault.py`, `get_secrets()` around +lines 282-425) reads provider keys from the service process environment +(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, …), injects them as "local secrets", and +merges them under the user's vault keys: the user's key wins, the platform key +fills gaps. The deploy files carry the slots +(`hosting/docker-compose/oss/.env.oss.gh:80-91`, +`hosting/docker-compose/ee/env.ee.gh.example:256-268`). + +**Credit gate, orphaned.** `_allow_local_secrets` (`vault.py:126`) calls +`GET /api/access/permissions/check?resource_type=local_secrets`, which on EE +meters a `credits_consumed` counter and denies when exhausted +(`api/oss/src/apis/fastapi/access/router.py:65-90`). The denial copy still +exists: "Out of credits. Please set your LLM provider API keys or contact +support." But `git grep _allow_local_secrets` returns exactly one hit, the +definition. The call site was lost in the SDK reorganization (last live in +commit `55bb45468a`). The server half works; nothing calls it. + +**The part that invalidates simple reuse.** The agent playground does not use +this path at all. `services/oss/src/agent/secrets.py` says it outright: the +`/invoke` path "no longer calls this: it resolves ONE least-privilege connection +for the configured model via `resolve_connection` … instead of the model-blind +whole-vault dump." The agent resolver +(`sdks/python/agenta/sdk/agents/platform/connections.py:537+`) fetches only +vault secrets and fails with a typed error when none match (default and +self-managed modes instead degrade to runtime-provided credentials, +`sdks/python/agenta/sdk/agents/handler.py:160`). There is no platform fallback +on the agent path, which is why the frontend wall exists. + +Two loose ends worth acting on regardless of this project: + +- If any `*_API_KEY` values are set on the cloud completion service today, + **legacy-path** runs can use them unmetered. Audit and remove. +- A separate, newer `sandbox_credits` system meters Daytona compute for agent + sandboxes. It is unrelated to LLM spend; do not confuse the two. + +**Design consequence.** "Reconnect the severed wire" was the original plan and +it is wrong for the surface that matters. The enforcement point must live on the +agent path's connection-resolution seam, where the selected model is known. The +meter machinery from #2957 is the piece we keep. + +## 3. The entitlements engine (what we reuse) + +The EE entitlements domain (`api/ee/src/core/access/entitlements/`, +`api/ee/src/core/meters/`, `api/ee/src/dbs/postgres/meters/`) is a generic quota +engine with exactly the primitives a trial needs: + +- **Vocabulary**: boolean `Flag`s, metered `Counter`s, level `Gauge`s, and + rate-limit `Throttle`s, with per-plan `Quota(free, limit, strict, scope, + period)` in a code-constant catalog (`types.py:321-675`). Hobby already + carries `credits_consumed: Quota(free=100, limit=100, strict=True, + period=MONTHLY)` (`types.py:348`). +- **Atomicity**: `MetersDAO.adjust` (`dao.py:376-518`) is a single + `INSERT … ON CONFLICT DO UPDATE SET value = greatest(value+delta, 0) WHERE + RETURNING`. The limit check happens in SQL, so concurrent + requests cannot jointly overshoot; negative deltas (refunds) clamp at zero. +- **Lifetime counters exist natively**: the meter row's identity is a + deterministic UUID of (scope, period, key) (`meters/types.py:96-141`). A + monthly quota "resets" because the month rolls the identity to a fresh row; a + quota with `period=None` is one row forever. No reset job anywhere. +- **Checks are explicit calls**, not middleware: handlers call + `check_entitlements(key=…, delta=…)` (`service.py:272`). On infrastructure + errors it **fails open** (`service.py:302-314`), a sensible policy for + feature gates and the wrong one for spending money. +- **Signup provisioning**: `create_organization_and_project` + (`api/oss/src/services/commoners.py:252`) hands EE + `provision_signup_subscription`, which starts either a reverse trial or the + free plan (`cloud_v0_hobby`). Notably, **explicit org creation runs a + different provisioning path** (`api/ee/src/core/organizations/service.py:931`) + than signup; the two are distinguishable. +- **Frontend**: `/billing/usage` reports every counter with + `{value, limit, free, period, scope}`; `UsageProgressBar` on the Billing page + renders any counter present, and the TypeScript types already declare + `credits_consumed`. A quota-exceeded 403 today falls through to a generic + error toast (`axiosConfig.ts:147-173` intercepts only auth-upgrade codes); + nothing renders a "you hit the limit" state. + +Two impedance points that matter: + +- `credits_consumed` is monthly. Reusing it for a lifetime trial would silently + refill the trial every month. A new counter with `period=None` is required. +- Quota limits are plan constants. There is **no per-org override mechanism**, + and a plan-level lifetime quota would grant the full allowance to every + existing org on the plan the day it ships, not just new signups. + +**Design consequence.** Consumption accounting reuses the meter engine as-is +(new lifetime counter). Eligibility cannot come from the plan catalog; it needs +a per-org grant record written at signup. The exhausted-state UX needs a real +frontend state, not the 403 fallthrough. + +## 4. What an activation session actually costs (measured) + +We queried production traces on EU cloud +(`POST /api/tracing/spans/query`, token metrics in span attributes +`ag.metrics.tokens.{incremental,cumulative}.*`). The API key is project-scoped, +so the sample is our own team project: 148 spans / 80 traces over 180 days, of +which 8 are agent-playground traces. Small and non-organic, but it contains the +number that matters. + +The one fully instrumented agent call (claude-haiku-4-5, user message "hey"): +`input_tokens=10`, `output_tokens=119`, `total_tokens=23,621`, billed cost +$0.02997. The arithmetic closes only as a **~23.5K-token cache write of harness +context** (23,492 × $1.25/M write + 10 × $1/M + 119 × $5/M = $0.0300). Every +agent call carries the harness (system prompt, skills, tool definitions); +99.5% of activation cost is our own context, not the conversation. + +For contrast, the legacy prompt playground's calls are trivial: median 40 input +/ 24 output tokens, about $0.000025 per call. + +A 10-turn activation session, modeled from these measurements (history grows +~130 tokens/turn): roughly **242K input tokens, 1.2K output tokens**. Cost per +session at July 2026 prices, without and with prompt caching (85% input +discount): + +| Model | No caching | Cached | +|---|---|---| +| Gemini 2.5 Flash-Lite | $0.025 | $0.004 | +| DeepSeek V4 Flash | $0.034 | $0.005 | +| OpenAI gpt-5.4-nano | $0.050 | $0.009 | +| Claude Haiku 4.5 | $0.248 | ~$0.05 (cross-checked against the real Anthropic cache math: ~$0.057) | + +Caveats: the sampled turns made one LLM call each; real tool-using sessions run +2-3x. Two data-quality findings came out of the measurement: + +- **Agent runs routed through OpenRouter record no token usage at all** + (`ag.metrics` and `gen_ai.usage` both empty). Cheap-model spend is invisible + to our own observability today. This needs an issue filed regardless. +- Cache tokens are inconsistently folded into `total_tokens` across providers, + so `total` is not a billed-token count. + +**Design consequence.** 10,000 signups cost roughly $500 uncached / under $100 +cached on the cheap models: inside the $3,000 budget with room for the 2-3x +tool-use multiplier. Prompt caching is the dominant lever, so the trial model +must cache well through our stack. Any per-user cap denominated in tokens must +be in the millions, which is why the design counts runs instead. And the +gateway must do its own usage metering, because harness-side instrumentation +provably drops it for at least one provider. + +## 5. Model pricing, provider mechanics, abuse math + +Verified prices (per 1M tokens in/out, official pages, 2026-07-23): + +| Model | Input | Output | Notes that matter | +|---|---|---|---| +| DeepSeek V4 Flash | $0.14 ($0.0028 cache-hit) | $0.28 | Prepaid wallet, hard stop at zero. `deepseek-chat` alias dies 2026-07-24; thinking mode is on by default and must be disabled. PRC processing is a sales objection for EU-conscious prospects. | +| Groq Llama 3.1 8B | $0.05 | $0.08 | Cheapest, 840 tok/s, but 8B-class quality would make the first session feel bad; false economy. | +| Gemini 2.5 Flash-Lite | $0.10 | $0.40 | Newer Flash-Lite generations cost 3-6x more. Free tier trains on data (never route users through it). Paid is postpaid: budgets alert, they do not cap. | +| Mistral Small 3.2 | $0.10 | $0.30 | EU vendor; official page omits per-model rates (lower confidence). | +| OpenAI gpt-5.4-nano | $0.20 | $1.25 | Reasoning model; supports effort `none`. Prepaid at low tiers plus project-level **hard spend limits** (enforcement can lag). Auto-recharge is on by default and delayed cutoffs can overdraw, so prepaid is defense-in-depth, not a hard cap. Vendor positions nano for classification-style workloads: agent-chat quality needs testing, not assuming. | +| Claude Haiku 4.5 | $1.00 | $5.00 | 10-20x the alternatives; not the value pick for a trial. | + +Abuse math. These models accept up to 1M-token contexts; uncapped, one crafted +DeepSeek message costs $0.14 and a scripted account multiplies that. Capping +output per call bounds almost nothing for an agent, because one user turn can +trigger many large-input calls. The bound has to sit at the request boundary: +per-call input/output caps plus a per-account total budget. With sane caps the +worst case per account lands at cents on the cheap models; the residual risk is +mass fake signups, which is bounded by grant size (cents each) and signup +verification, not by pricing. + +**Design consequence.** gpt-5.4-nano and Gemini 2.5 Flash-Lite are the two +candidates (D2 in the RFC), decided by a real tool-using quality test. Provider +spend controls are a backstop only; our own gateway budget enforcement is the +actual bound. The trial provider account must be isolated, hard-capped, and +have auto-recharge disabled. + +## 6. Competitor free-tier patterns (anonymized) + +Six direct and adjacent competitors were analyzed by their pricing pages, docs, +and community forums. Names are withheld from the repo by policy; the named +notes exist locally. + +The pattern among AI-first agent platforms is uniform: **grant a platform-funded +allowance at signup, let the first session run a real agent with zero key setup, +stop or throttle at zero, upsell**. Monthly grants land between a few hundred +and a few thousand credit-units, roughly **$1-5 of model cost per month**. The +two workflow-engine incumbents are the holdouts (bring-your-own-key for all +workflow AI; one offers a token-sized one-time promo credential), and their +time-to-first-agent-magic is documented in reviews as the worse for it. + +Denominations cluster three ways: a single abstract credit pool with model-tier +multipliers (simple to show, generates "why did this cost 200 credits" tickets); +two meters splitting platform actions from LLM spend at zero markup (most +honest); and no AI meter at all (the incumbents). Bring-your-own-key semantics +diverge sharply: one platform charges half-credits even on your own key +(documented resentment in its forum), others bypass metering entirely on BYOK, +one has no BYOK at all and sells model choice as a plan feature. + +Documented mistakes worth avoiding: hard stops that do not auto-resume after +top-up (a support rage point), hiding the denominator ("3x more usage" with no +numbers; reviews weaponize it), killing a free tier outright (every review +since leads with it), and one-time-only grants as the entire free tier (a cliff +before habit forms; fine as a bonus). One platform runs an earn-credits +onboarding (credits for social follows, a large tutorial-completion reward), +converting free-tier COGS into distribution; nobody else does; it validates the +gamification roadmap. + +Nobody publishes abuse handling. The observable mitigations are structural: +small grants, cheap default models, per-account concurrency caps, daily message +caps, no rollover, email-verified signups. + +**Design consequence.** Our funnel differs from the credit-sellers in one +decisive way: their credits are the business model; ours end at "connect your +own key", after which usage is unmetered on the user's key. That makes a +one-time grant correct for us where recurring grants are correct for them. The +denominator stays visible ("N free messages left"), exhaustion degrades +gracefully with the draft preserved, and the unit stays legible (a message, not +a token). + +## 7. Daytona secret delivery (#5223 / #5277 / #5278) + +Today the runner injects resolved provider keys straight into the sandbox +environment (`services/runner/src/engines/sandbox_agent/environment-setup.ts:176` +does `Object.assign(env, plan.secrets)`; `daytona.ts` spreads them into the +Daytona create call). Anything in the sandbox is readable by the user's agent. + +PR #5223 (design) and PR #5277 (implementation, flag +`AGENTA_DAYTONA_OPAQUE_SECRETS=process_local`, off by default) change the +delivery: the SDK contract becomes a typed +`modelConnection.credentials[{binding, value, usage}]` where `usage` is +`opaque_http` or `local_use`, classified fail-closed. For `opaque_http` +credentials the runner creates one Daytona organization Secret per binding with +a random name and `hosts: [exactHttpsHost]` (IP literals, wildcards, non-443 +ports, and localhost are rejected), and passes only the secret name to the +sandbox. Inside the sandbox the env var holds a `dtn_secret_*` placeholder. +**Daytona's egress proxy substitutes the real value into outbound HTTPS +requests, and only toward the allowlisted host.** No process inside the sandbox +can read the value; that is real possession-hiding, verified in the PR's live +QA. PR #5278 (design only, nothing implemented) proposes a reconciliation +domain so a crashed runner cannot orphan those Daytona Secrets. + +What the mechanism cannot do: bound **usage**. The only lever is the +destination hostname. A trial agent could loop +`curl https://api.openai.com/... -H "Authorization: Bearer $KEY"` and the proxy +would happily substitute the platform key: any model on that host, any volume, +outside all Agenta metering, for the sandbox's lifetime (15-minute auto-stop, +30-minute auto-delete, renewable by starting runs). The #5223 design says it +itself: "Move the provider call behind an Agenta gateway. This is the general +solution." + +Status: #5277 is deliberately parked (maintainer comment 2026-07-17): the +in-flight runner refactor reshapes the sandbox-provider interface it builds on, +so it will re-land on the new seam. It is currently conflicting with its base. + +**Design consequence.** Daytona Secrets solve possession, not usage. Funding a +trial on #5277 alone would hide the key while letting it be spent without +limit; the gateway is still mandatory. But the two compose exactly: the trial +connection's endpoint is the gateway host, the delivered credential is a +short-lived reservation token instead of any provider key, and #5277's typed +contract, host validation, and secret lifecycle carry it. Then even the token +is unreadable inside the sandbox, and a token that leaks anyway buys minutes of +one cheap model inside a metered budget. + +## 8. Gateway landscape: adopt or build + +Surveyed against five requirements: (R1) validate our reservation-bound opaque +token; (R2) model allowlist; (R3) atomic per-call budget hold settled to actual +usage, hold kept on ambiguous timeout; (R4) transparent streaming pass-through +of the provider's agent API with key injection; (R5) usage records into our +metering. + +The decisive finding: **no surveyed system implements R3.** LiteLLM's spend +tracking is post-hoc and batched (concurrent calls can overshoot a small +budget); its custom-auth hook and native Responses API support are real, but it +is a heavy operational dependency (its own Postgres, documented memory-growth +issues, community sizing of 4 vCPU / 8 GB) whose weight buys provider breadth +we do not need. BricksLLM, conceptually closest (TTL keys, budgets enforced via +Redis), is dead: last commit January 2025, before the Responses API existed, +and its budgets are dollar-denominated. Portkey's OSS gateway is light, MIT, +actively maintained, with excellent Responses API pass-through, but it is +deliberately stateless: keys, budgets, and usage logs are the hosted product, +so we would write R1/R2/R3/R5 inside their plugin framework. TensorZero was +archived June 2026. Envoy AI Gateway v1.0 has genuine token-budget primitives +but is a Kubernetes platform (we deploy docker compose on EC2) and its quotas +are windowed buckets, not reservation holds. Kong's equivalent is +enterprise-only; APISIX drags in the nginx+etcd platform. The handful of tiny +OSS "budget proxy" projects validate the hold-estimate/forward/reconcile +pattern but support no Responses API and are not production-grade. + +DIY sizing, grounded in the repo: FastAPI, httpx 0.28, and redis are already in +the API's dependency set, with existing Redis locking/caching/throttling +utilities (`api/oss/src/utils/`). The build is one route plus a small ledger: +token lookup (~50 lines), model check (~10), atomic hold via one Redis Lua +script or one conditional Postgres `UPDATE … RETURNING` (~150 plus a +reconciler; ambiguous timeouts keep the hold, and OpenAI's stored responses +allow late settlement via `GET /v1/responses/{id}`), byte-faithful SSE +pass-through that taps the final usage event without re-encoding frames +(~150-250), metering emit (~30). Roughly 600-900 lines plus tests; one to two +engineer-weeks; zero new infrastructure. + +**Design consequence.** Build it (D7 in the RFC). Whatever we adopted, we would +still write the hold-and-settle ledger, and the ledger is the asset that +carries into a future paid managed-credits product. If that product later goes +multi-provider, a translation layer (Portkey's OSS gateway is the natural +candidate) slots in behind our ledger. diff --git a/docs/design/activation-credits/rfc.md b/docs/design/activation-credits/rfc.md new file mode 100644 index 0000000000..d514bd2ea8 --- /dev/null +++ b/docs/design/activation-credits/rfc.md @@ -0,0 +1,116 @@ +# RFC: free trial credits at signup + +**Status:** for review. Reviewer: Mahmoud. +**Reading order:** context.md (situation, glossary) → this file (decisions) → +design.md (full architecture) → research.md (evidence) → +implicit-decisions.md (what entered the design beyond the original ask, and +what each item costs to strike). + +## The proposal in three sentences + +New cloud signups get about thirty free playground messages on an Agenta-funded +cheap model, so their first conversation happens before any key form. At zero, +the existing connect-your-key moment appears with their draft preserved and +auto-sent once a key lands. It costs us roughly $500-1,500 per 10,000 signups +uncached and a fraction of that with caching, against the $3,000 budget. + +## The smallest safe version + +Because the full design accreted requirements beyond the original ask +(implicit-decisions.md is the register), here is the floor: the version that +cannot responsibly be shrunk further, stripping every negotiable hardening +item. + +- A grant row at signup (new orgs only; anything less gifts every existing + org). +- A new lifetime counter; charge = one conditional increment in the same + transaction as a minimal reservation row (idempotency key, no refund states, + holds expire on a timer). +- A server-side funding check on the invoke route only (no signed claim). +- The gateway, minimal: token validation, one model, per-call and per-run + caps, streaming pass-through, usage record. This piece is not strippable; + the sandbox is user-controlled and a raw platform key inside it is + stealable, which no counter elsewhere can fix. +- Frontend: relax the gate while balance remains, countdown pill, exhausted + banner, draft retention. +- Isolated provider account with a hard spend limit; backend kill-switch flag. + +Estimate for this floor: the gateway's one-to-two engineer-weeks plus roughly a +week across backend and frontend. The full design in design.md adds the +hardened reservation lifecycle, signed provenance, replay protection, and +refund semantics; the delta is deferrable and listed item by item in +implicit-decisions.md §6-12. + +## Decisions requested + +**D1. Unit.** One unit = one message (an accepted agent run), shown as "free +messages". Token units are meaningless to a new user; the 23.6K-token harness +context makes even "hi" cost thousands. The word "credits" is reserved for the +future fungible unit (earned, purchased). **Recommend: yes.** + +**D2. Model.** gpt-5.4-nano (prepaid controls, caching, recognizable brand, +~$0.05/session uncached) vs Gemini 2.5 Flash-Lite (half the cost, postpaid +alert-only billing) vs DeepSeek V4 Flash (cheapest, PRC-hosted, a sales +objection). **Recommend: nano, conditional on a pre-launch quality test of +real tool-using sessions on both finalists; the vendor positions nano for +classification workloads, so first-session quality is measured, not assumed.** + +**D3. Amount.** One-time ~30 runs (3x the measured session; our credits end at +BYOK, so recurring grants serve competitors' business model, not ours), with +the amount as a per-cohort experiment parameter. **Recommend: yes, cohorts at +20 and 50.** + +**D4. Eligibility storage.** Append-only grant record + existing meter; +balance is computed, never stored. Rejected: plan-catalog quota (gifts all +existing orgs; no per-org override exists) and a mutable balance column +(second source of truth). **Recommend: grant record.** + +**D5. Scope.** Org-scoped, signup-provisioned orgs only (the signup and +explicit-org-creation code paths are distinct, so this is enforceable; it +closes grant farming). Consequence: invited teammates can meet an exhausted +balance. **Recommend: yes.** + +**D6. Credential protection.** Daytona Secrets alone hide the key but leave +its use unbounded toward the provider host; gateway alone bounds use but the +token sits readable in the sandbox; composed, the token rides the #5277 +delivery once it re-lands, and theft requires beating Daytona's egress proxy +for a prize of minutes of one cheap model. **Recommend: gateway first (it is +the only usage bound), composition when #5277 re-lands.** + +**D7. Build vs adopt the gateway.** No surveyed OSS gateway implements +reservation-bound hold-and-settle; LiteLLM is a heavy platform whose weight +buys unneeded provider breadth; the closest match is dead; the light one is +stateless where we need state. Building is 600-900 lines on infrastructure we +run. **Recommend: build; if managed credits later go multi-provider, put a +translation layer behind our ledger.** + +**D8. Scope of hardening for the MVP.** The smallest safe version above, or +the full design.md treatment. This is the "which of the accreted requirements +do we pay for now" decision and it is deliberately yours; +implicit-decisions.md prices each item. **Recommend: smallest safe version +plus fail-closed checks (item 11), deferring items 6-9 and 12 to the managed- +credits iteration.** + +## Cost and guardrails, condensed + +Measured: ~23.6K tokens of harness context per LLM call dominates; a 10-turn +session is ~242K input tokens; $0.025-0.05 per session on the candidate models +uncached, under a cent cached. 10K signups ≈ $250-500 expected, worst-case +bounded by: per-call and per-run gateway budgets, the ~30-run balance, a daily +per-org throttle, an isolated provider account with a hard spend limit and +auto-recharge off, and a backend kill switch. Mass fake signups cost cents +each and remain an email-verification problem. + +## Measurement + +Activation rate (first successful playground message per signup), +key-connection within 7 days, spend per signup, conversion at the exhaustion +screen. Kill at 3x the spend envelope or a key-connection collapse. The trial +moves the wall; it must not become the product. + +## Out of scope + +Earned credits (new grant rows later), purchasable managed credits (extends +this ledger), the classic prompt playground, self-hosted. Prerequisite fixes +tracked separately: the OpenRouter token-usage instrumentation gap, and the +audit for stray provider keys on the cloud service environment.