Skip to content

Commit ba3148c

Browse files
glitch003claude
glitch003
and
claude
committed
docs(plans): private-apps framework + lambda-parity gateway plan
Brings the two design plans from PR #446 into this branch so they ship with the lambda-parity implementation they describe, and #446 can be closed. - plans/chipotle-lambda-parity.md — prioritized gateway work (this PR implements P0.0/P0.1/P0.2/P1) for per-key blast-radius parity with a public Lambda URL. - plans/private-apps-backend.md — the backend-as-Lit-Actions framework these gateway controls unlock (relay becomes optional). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 743e98e commit ba3148c

2 files changed

Lines changed: 767 additions & 0 deletions

File tree

‎plans/chipotle-lambda-parity.md‎

Lines changed: 351 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,351 @@
1+
# Chipotle: Lambda-Parity for Frontend-Callable Actions
2+
3+
Status: design proposal / work plan
4+
Author: Chris (with Claude)
5+
Date: 2026-06-04
6+
Related: [private-apps-backend.md](./private-apps-backend.md) — this is the
7+
gateway-side dependency that lets that plan drop its relay requirement.
8+
9+
## Goal
10+
11+
Make a Chipotle **usage API key safe to embed directly in a frontend**, with
12+
abuse/spend blast-radius control on par with a **public AWS Lambda Function URL**.
13+
Achieving this lets "private apps" host only a frontend and call Lit Actions
14+
directly — no relay needed.
15+
16+
## The framing (why this is achievable)
17+
18+
A public Lambda URL is itself a credential-free, internet-callable endpoint.
19+
Anyone can hit it; AWS's protection is **not secrecy**, it's **bounded,
20+
configurable blast radius**: the URL invokes only that one function, throttled,
21+
concurrency-capped, with budgets + alarms. A determined griefer can still run up
22+
*capped* cost — parity means the worst case is a number you chose and got alerted
23+
about, not "drains the account."
24+
25+
Two things are notably **out of scope of the risk** and must stay that way:
26+
27+
- **Confidentiality is unaffected.** Action plaintext/secrets are TEE- and
28+
CID-gated. Griefing a public key burns money/availability; it can never read
29+
encrypted data. Nothing here touches that guarantee.
30+
- **CID scoping already bounds *what* runs.** Usage keys are group-scoped
31+
(`execute_in_groups`), groups pin exact CIDs, and execute permission is checked
32+
on-chain (`accounts/mod.rs::can_execute_action`). A leaked key on a group that
33+
pins only your public actions can run **only those actions** — nothing else,
34+
no account management. This is the "Function URL → one function" property, and
35+
it already exists.
36+
37+
So the gaps are all about bounding **rate, concurrency, and spend per key**, plus
38+
defense-in-depth.
39+
40+
## What already exists (verify + document, don't rebuild)
41+
42+
| Capability | Where | Status |
43+
|---|---|---|
44+
| Per-key CID scoping (leak radius = pinned actions) | `accounts/mod.rs::can_execute_action`; group `cid_hashes` | ✅ works |
45+
| Global overload shedding → 429 | `core/v1/guards/cpu_overload.rs` (CPL-202, commit d2743fdb) | ✅ but global, not per-key |
46+
| Account-wide prepaid credits → 402 when empty | `core/v1/guards/billing.rs::BilledLitActionApiKey`; `stripe.rs` | ✅ but account-wide, crude |
47+
| Per-second execution charge | `actions/client/execution.rs::flush_unbilled_seconds`; `stripe.rs` `COST_LIT_ACTION_PER_SECOND_CENTS` | ✅ |
48+
49+
## Architecture: where state lives & the zero-latency path
50+
51+
The hot path today is already built on one principle, and everything here follows
52+
it: **authoritative data lives in a slow store (chain or Stripe); the request path
53+
only ever touches an in-process [moka](https://docs.rs/moka) cache with TTL +
54+
stale-while-revalidate (SWR) + request coalescing; staleness is tolerated within
55+
the TTL; money/usage settles asynchronously off the response path.** We are not
56+
inventing an architecture — we are adding cached fields that obey the existing one.
57+
58+
What the code already establishes (verified):
59+
60+
- **On-chain permission reads are already cached on the hot path.**
61+
`accounts/blockchain_cache.rs` caches `canExecuteAction` / `canUseWalletInAction`
62+
/ wallet derivation (60-min TTL, per-account **generation-counter** invalidation
63+
bumped on any write). The key's on-chain record is in memory when a request runs.
64+
- **Stripe reads are cached too.** `stripe.rs`: `wallet_cache` (key→wallet, 1h),
65+
`customer_cache` (10m), `balance_cache` (10m TTL + **SWR** + background refresh +
66+
coalescing). The credit check in `BilledLitActionApiKey::from_request` is a
67+
memory read after warmup.
68+
- **Charging is off the response path.** `stripe::charge()` reads the cached
69+
balance, does an **optimistic in-memory decrement**, records the event, and
70+
`spawn`s the real Stripe balance transaction fire-and-forget (there's a
71+
`billing.charge.settlement_failed` metric for when it doesn't land).
72+
`flush_unbilled_seconds` awaits only the cache ops, not the network. So
73+
per-request charging costs microseconds, not a round trip.
74+
75+
### The zero-latency gate
76+
77+
The on-chain key record is **already read and cached** on the hot path (it's what
78+
`canExecuteAction` returns). So we pack a single **`hasSpendingRules` bit** into
79+
that record. Reading it costs **zero** extra network and zero extra cache lookup —
80+
it rides along in data already in memory. Fetched in the **same multicall** as
81+
`canExecuteAction` (trivial Solidity change — return multiple values), even a cold
82+
cache miss adds no extra round trip.
83+
84+
```
85+
guard (before execution):
86+
(canExec, hasSpendingRules) = blockchain_cache.permissions(keyHash) # 1 cached multicall
87+
if !hasSpendingRules: return Success # ← the 99% with no caps: ZERO added latency
88+
else: enforce rules (all in-memory; see below)
89+
```
90+
91+
### Where each kind of state lives
92+
93+
The three things we store have different shapes (config vs. counter, write-rarely
94+
vs. write-per-request), so they don't share one home:
95+
96+
| Data | Shape | Home | Hot-path read |
97+
|---|---|---|---|
98+
| `hasSpendingRules` flag | 1 bit, toggled rarely | **On-chain** (opt 2), in the key record | Free — already in `BlockchainCache` |
99+
| Cap + window, rate/burst, concurrency, IP/origin allowlist | Config, edited occasionally | **lit-payments DB** (opt 3), edited via its backend+frontend | New moka cache (TTL+SWR), **only read when flag set** |
100+
| Per-key cumulative spend (rolling window) | High-write counter, durable | **lit-payments DB**, written on the existing **async** charge path | Optimistic in-memory decrement (mirrors `balance_cache`) |
101+
| Rate-limit token buckets, concurrency counts, per-IP counters | High-write, ephemeral, rolling | **In-process memory** (per-node) | Native — already in memory |
102+
103+
**Why this split:**
104+
105+
- **Flag on-chain, not the cap value.** The flag is a permission (belongs with the
106+
others) and is the one thing that must be free to read for *everyone*. Toggled
107+
only when a key gains/loses its first rule (1 tx, rare). Keeping the cap *value*
108+
off-chain means tuning caps in the lit-payments UI needs no gas/tx, and the DB
109+
read is gated behind the flag so non-cap accounts never touch it.
110+
- **Rule details + per-key usage in the DB, not Stripe metadata.** Stripe metadata
111+
is per-*customer* (account), not per-key; writing it is an API call; it's the
112+
wrong tool for rolling windows, rate config, or analytics. Keep Stripe for the
113+
money relationship that already exists (opt 1 stays as-is).
114+
- **Counters in memory, per-node.** Rate buckets / concurrency counts can't go
115+
on-chain (per-request writes) or in Stripe. Per-node is acceptable because the
116+
**durable spend cap is the real backstop**; per-node rate limiting yields an
117+
effective cluster rate of ≈ limit × N nodes, which is fine for griefing control
118+
(AWS API Gateway throttling is approximate too). Start per-node; add a shared
119+
store (Redis-like) only if cluster-exact limits are ever required.
120+
121+
### Request flow for an opted-in key
122+
123+
```
124+
1. blockchain_cache: (canExecuteAction, hasSpendingRules) 1 cached multicall
125+
2. flag set → rules_cache.get(keyHash) memory; DB read only on cold miss (SWR)
126+
3. spend_cache.get(keyHash) vs rolling cap memory; reject 402 if over
127+
4. rate bucket + concurrency counter memory; reject 429/503 if over
128+
5. origin / IP allowlist check memory; reject 403 if not allowed
129+
6. execute
130+
7. POST-response, in the existing spawned settlement task:
131+
- Stripe balance txn (already there)
132+
- increment per-key rolling spend in DB + optimistic in-mem decrement
133+
```
134+
135+
Steps 1–5 are memory reads (low microseconds). Step 7 is entirely off the response
136+
path. An opted-in key pays one cold DB read per ~TTL window; a no-cap key pays
137+
nothing. The staleness tolerance is the same bargain CPL-246 already accepted for
138+
balances: a tiny possible overspend inside the TTL window in exchange for a fast
139+
hot path.
140+
141+
### Resolved design decisions
142+
143+
1. **Cap semantics → rolling window** (match AWS Budgets / Lambda), not a lifetime
144+
balance. The per-key spend counter resets on the window boundary.
145+
2. **Rules-cache freshness → SWR** (short TTL + background refresh, no cross-service
146+
invalidation plumbing), mirroring `balance_cache`. A cap edit takes effect
147+
within the TTL window.
148+
3. **Counter durability → per-key spend reloads from the DB on cache miss**
149+
(durable across deploys/restarts); rate/concurrency buckets may reset on restart
150+
(acceptable — the spend cap is the durable backstop).
151+
4. **`hasSpendingRules` is fetched in the same contract multicall as
152+
`canExecuteAction`** so a cold cache miss adds no extra round trip (a simple
153+
Solidity change to return multiple values).
154+
5. **"Turn rules on" ordering:** write the DB rule first, *then* flip the on-chain
155+
bit. The bit going true is what activates enforcement, so this ordering avoids a
156+
window where the flag is set but rules aren't loaded.
157+
158+
## Work items, in priority order
159+
160+
Priority = how much it bounds the worst case per dollar of effort. P0 items are
161+
the ones that turn "drains the account" into "burns a number you chose."
162+
163+
---
164+
165+
### P0.0 — `hasSpendingRules` flag + multicall (foundation for everything below)
166+
167+
**What.** A single `hasSpendingRules` bit on the on-chain key record, returned in
168+
the **same multicall** as `canExecuteAction`, surfaced through `BlockchainCache`,
169+
and a request guard that short-circuits to "no enforcement" when it's false.
170+
171+
**Why first.** This is the zero-latency gate. Every per-key control below
172+
(spend cap, rate, concurrency, origin) reads it to decide whether to do *any* extra
173+
work. Land it first so the rest can assume "if we got here, rules exist." It also
174+
guarantees the headline property: **keys with no rules pay zero added latency.**
175+
176+
**Build.**
177+
- Solidity: extend the `canExecuteAction` read to also return `hasSpendingRules`
178+
(decision 4 — just more return values).
179+
- `accounts/blockchain_cache.rs`: cache the bit alongside the existing
180+
`execute_action` / `execute_and_wallet` results (same generation-counter
181+
invalidation, so flipping it bumps the generation and is picked up immediately).
182+
- A request guard (sibling to `BilledLitActionApiKey` / `cpu_overload`) that reads
183+
the cached bit and returns `Success` immediately when false.
184+
185+
**Acceptance.** A key with no rules executes with no extra reads beyond today's
186+
cached permission check; flipping the bit on-chain takes effect on the next request
187+
(generation bump), no redeploy.
188+
189+
---
190+
191+
### P0.1 — Per-key spend cap with auto-disable ⭐ highest leverage
192+
193+
**What.** A usage key carries its own budget (e.g. credits or a cents cap over a
194+
window). Each execution decrements it; when exhausted the key is rejected (and
195+
flagged disabled), independent of the account's overall balance.
196+
197+
**Why it's #1.** This is the single control that converts the failure mode from
198+
"a leaked frontend key can spend the whole account" into "a leaked key can spend
199+
*its* budget, then stops." It's the analog of an AWS per-function/per-budget cap.
200+
201+
See [Architecture: where state lives](#architecture-where-state-lives--the-zero-latency-path)
202+
for the storage split this builds on. The cap is a **rolling window** (decision 1):
203+
a per-key spend-to-date counter that resets on the window boundary, checked against
204+
a cap configured in the lit-payments DB, gated by the on-chain `hasSpendingRules`
205+
flag (P0.0) so non-cap keys pay zero latency.
206+
207+
**Build.**
208+
- **On-chain:** the `hasSpendingRules` bit lands via P0.0 (fetched in the
209+
`canExecuteAction` multicall). The existing no-op `balance` field on the key
210+
(`add_usage_api_key` in `core/v1/models/request.rs`; hardcoded `10_000_000` in
211+
`account_management.rs`, never read) can be repurposed as the flag or retired.
212+
- **DB (lit-payments):** store the cap + window per key; track the per-key rolling
213+
spend-to-date. New endpoints on the lit-payments backend to read/set them.
214+
- **Hot path (`guards/billing.rs`):** when the flag is set, read the cached rules +
215+
cached per-key spend (both in-memory; cold miss = one DB read, SWR) and reject
216+
with **402** if the rolling spend would exceed the cap. Mark the key disabled for
217+
the remainder of the window so subsequent calls short-circuit.
218+
- **Async (off response path):** in the existing spawned settlement task that
219+
already writes Stripe (`stripe::charge`), also increment the per-key rolling
220+
spend in the DB and apply the optimistic in-memory decrement (mirror
221+
`balance_cache`). Per-key spend reloads from the DB on cache miss (decision 3).
222+
223+
**Acceptance.** A key with a $X/window cap, hammered in a loop, stops at ~$X spent
224+
within the window and is rejected until the window rolls; the account's other keys
225+
and overall balance are untouched; a key with no rules sees no added latency.
226+
227+
---
228+
229+
### P0.2 — Per-key + per-IP rate limiting (rate + burst)
230+
231+
**What.** Token-bucket throttle keyed on the API key, and a coarser one keyed on
232+
client IP, on the `/core/v1/lit_action` path. Configurable rate + burst per key.
233+
234+
**Why.** Today the only throttle is the **global** CPU-overload 429 — it protects
235+
the node, not your wallet, and one abusive key can still spend fast within global
236+
capacity. This is API Gateway's per-key/per-stage throttling.
237+
238+
**Build.**
239+
- A request guard (sits with the other `core/v1/guards/`) that runs before
240+
execution, after key resolution. Reuse the existing 429 response plumbing from
241+
CPL-202 so OpenAPI/docs stay consistent.
242+
- Limits per key default to sane values, overridable per key (store next to
243+
`balance`).
244+
- Per-IP limit as a second bucket (defense against many anonymous callers on one
245+
public key).
246+
247+
**Acceptance.** A key exceeding its configured rate gets 429 with a `Retry-After`;
248+
other keys unaffected; the global CPU guard still independently sheds load.
249+
250+
**Decided.** Bucket store is **in-process, per-node** (decision in Architecture):
251+
effective cluster rate ≈ limit × N nodes, acceptable for griefing control since
252+
the spend cap (P0.1) is the durable backstop. Add a shared store (Redis-like) only
253+
if cluster-exact limits are ever required. Buckets gated behind `hasSpendingRules`
254+
(P0.0) so non-rule keys are untouched.
255+
256+
---
257+
258+
### P1 — Per-key concurrency cap (reserved-concurrency analog)
259+
260+
**What.** Cap simultaneous in-flight executions per key.
261+
262+
**Why.** Rate limits bound requests/sec; a concurrency cap bounds *simultaneous*
263+
spend and node load — the thing AWS reserved concurrency exists for. Cheap ins-
264+
urance once P0.2 exists; together they tightly bound burn velocity.
265+
266+
**Build.** A counter (incremented at execution start, decremented at end/timeout)
267+
in the execution dispatch path (`actions/client/execution.rs` / the dispatch in
268+
`lit-actions/server`), checked against the key's configured max. Over-cap →
269+
429/503.
270+
271+
**Acceptance.** A key with concurrency=N never has >N actions running at once;
272+
the N+1th waits or is rejected per policy.
273+
274+
---
275+
276+
### P2.1 — Per-key origin allowlist (replace wildcard CORS)
277+
278+
**What.** Each key (esp. public ones) carries an allowed-origins list; the gateway
279+
sets/validates CORS against it instead of today's wildcard.
280+
281+
**Why.** Today CORS is `AllowedOrigins::all()` (`main.rs`). An origin allowlist is
282+
standard hygiene for a browser-callable endpoint and stops casual cross-site
283+
reuse. **Defense-in-depth only** — `Origin`/`Referer` are browser-enforced and
284+
trivially spoofed by non-browser clients, so it rides *on top of* P0.1/P0.2,
285+
never instead of them.
286+
287+
**Build.** Add `allowed_origins` to the key model; per-request CORS reflection +
288+
rejection in the Rocket CORS layer / a guard. Empty list = same-as-today (or
289+
deny, for `public` keys — see P2.2).
290+
291+
**Acceptance.** A public key configured for `app.example.com` rejects browser
292+
calls from other origins; server-to-server calls still work (documented as not a
293+
security boundary).
294+
295+
---
296+
297+
### P2.2 — "Public / frontend-safe" key type + Dashboard guardrails
298+
299+
**What.** A flag marking a key as intended for the browser. The Dashboard (and
300+
API) then *require* the blast-radius controls to be set before issuing it: a spend
301+
cap (P0.1), rate limits (P0.2), concurrency cap (P1), and an origin allowlist
302+
(P2.1).
303+
304+
**Why.** Makes the safe path the default path. Without this, someone ships an
305+
unbounded key to the browser and we're back to square one. This is the DX glue
306+
that makes the whole effort land.
307+
308+
**Build.** A `public: bool` (or key class) on the key model; validation that
309+
public keys have non-default caps; Dashboard UX that surfaces "this key is
310+
frontend-safe; here's your max blast radius: $X/day, N rps, M concurrent."
311+
312+
**Acceptance.** Creating a public key without caps is refused with a clear message;
313+
the Dashboard shows the bounded worst case for a public key at a glance.
314+
315+
---
316+
317+
### P3 — Optional hardening (post-parity)
318+
319+
- **Alerting / budget alarms.** Spend-velocity alerts and a notification when a
320+
key auto-disables (the CloudWatch-alarm analog). Parity = bounded worst case
321+
**+ you find out**.
322+
- **App-check / PoW / captcha** for *unauthenticated* public endpoints, to raise
323+
the cost of scripted abuse before it hits the spend cap.
324+
- **Per-IP WAF-style rules** (geo, known-bad ranges, anomaly) beyond the P0.2
325+
bucket.
326+
- **Gateway authorizer hook** for teams that want auth enforced at the edge in
327+
addition to in-action.
328+
329+
## Suggested sequencing
330+
331+
```
332+
P0.0 hasSpendingRules flag + multicall ← foundation; the zero-latency gate
333+
P0.1 per-key spend cap (rolling) + auto-disable ← biggest blast-radius cut
334+
P0.2 per-key + per-IP rate/burst ← parallel-able with P0.1, both gated by P0.0
335+
└─ P1 per-key concurrency cap ← small add once P0.2 lands
336+
P2.1 per-key origin allowlist ← independent; defense-in-depth
337+
P2.2 public key type + Dashboard caps ← depends on P0.0/P0.1/P0.2/P1/P2.1 existing
338+
P3 alarms / app-check / WAF / authz ← post-parity hardening
339+
```
340+
341+
After **P0.0 + P0.1 + P0.2 + P1 + P2.2**, a usage key is safe to ship in a frontend with
342+
bounded, configurable blast radius — Lambda parity — and
343+
[private-apps-backend.md](./private-apps-backend.md) can make its relay optional.
344+
345+
## The honest caveat (same one AWS has)
346+
347+
None of this makes a public endpoint un-griefable. A determined attacker can run
348+
up cost *within the caps you set*. Parity is **bounded, configurable worst case +
349+
alerting**, not zero risk. The caps (P0/P1) make the worst case a number you chose;
350+
the alarms (P3) make sure you hear about it. That is exactly what AWS reserved
351+
concurrency + Budgets + CloudWatch deliver — and what this plan brings to Chipotle.

0 commit comments

Comments
 (0)