|
| 1 | +# Zero-leak state stores — implementation roadmap |
| 2 | + |
| 3 | +Opened 2026-08-01. Consumes `000_state_store_inventory.md` (36 stores, verdicts) |
| 4 | +and the prior unit's stream-path work (260731_macos_rss_retention, landed). |
| 5 | +Push to origin/dev is user-authorized for this unit. |
| 6 | + |
| 7 | +## Design goal |
| 8 | + |
| 9 | +Among production TS-based LLM proxies on GitHub, opencodex should have the |
| 10 | +strongest theoretical no-leak posture — while carrying translation duty |
| 11 | +(bidirectional tool conversion, MCP namespace flatten/restore, image |
| 12 | +normalization, reasoning-signature carry) that pure relays do not have. |
| 13 | +The competitive evidence (060) defines "strongest" per category; the phases |
| 14 | +below close every UNBOUNDED and the material CONDITIONALLY-UNBOUNDED verdicts. |
| 15 | + |
| 16 | +## External evidence (Luna lanes, source-opened 2026-08-01) |
| 17 | + |
| 18 | +- Portkey Gateway: no post-stream replay state, but NO upstream abort on client |
| 19 | + disconnect and unbounded local transform buffers (streamHandler.ts, |
| 20 | + streamHandlerUtils.ts). No process-wide byte budget. |
| 21 | +- Claude Code Router: per-request AbortController + upstream cancel (good); |
| 22 | + context-archive store enforces TTL+count+aggregate-body-bytes (the only |
| 23 | + competitor with a 3-dimension eviction contract). |
| 24 | +- mcp-proxy (punkpeye): replay event store FIFO 1,000 events, no TTL/bytes; |
| 25 | + session map cleanup only on transport close. |
| 26 | +- LiteLLM: added per-entry max cache item size (v1.63.14) after OOMs; |
| 27 | + streaming retention issue #6404 open. one-api: externalizes state |
| 28 | + (Redis/DB), memory cache default off. |
| 29 | +- Eviction-contract precedent: lru-cache maxSize+maxEntrySize+TTL+LRU; |
| 30 | + cacache for content-addressed disk spill (eviction separate). |
| 31 | + |
| 32 | +Conclusion: no surveyed TS proxy has a process-wide app-owned byte budget. |
| 33 | +Landing one (wp5) plus per-store byte caps positions opencodex to be stronger |
| 34 | +than every surveyed competitor — a claim that stays PROVISIONAL until 060 |
| 35 | +supplies the full source ledger (audit round 1). |
| 36 | + |
| 37 | +## Phase map (dependency-ordered; one decade doc per work-phase) |
| 38 | + |
| 39 | +| Doc | WP | Content | Depends | |
| 40 | +|---|---|---|---| |
| 41 | +| `010_continuation_hard_cap.md` | wp2 | continuation last-entry cap hole → true hard cap; oversized entries demote to disk stub ONLY after atomic per-entry spill (fsync + no-replace publication); missing/corrupt spill = explicit continuation failure, never silent naked-delta (R1-2); contract redefinition | — | |
| 42 | +| `020_blob_and_replay_caps.md` | wp3 | Cursor blob byte cap split by PROVENANCE (local-regenerated evictable, remote-origin pinned within TTL — R1-3), Antigravity inner-call bounds, vision-LRU clamp-before-insert, image zero-weight sentinel accounting (R1-1) | — | |
| 43 | +| `030_eviction_mechanisms.md` | wp4 | store-by-store cleanup matrix with three mechanisms: TTL sweep / config-generation reconciliation / admission caps (R1-4). Covers: subagent quota-failure records, key/combo cooldowns, provider+codex quota history, routing health, warning memos, model-cache provider history, pool/combo history, XAI verdicts, guardian backoff, GCP source tokens, ownership/PID memos, OAuth/reauth reconciliation. Windows ACL memo: eviction-after-rename keyed by actual temp path, NO destination-level reuse (R1-5); explicit security review required | — | |
| 44 | +| `035_registry_admission_caps.md` | wp4b | remaining operational registries/flights: debug-subscriber count cap, active turns/sockets/workers hard admission CAP (reject beyond cap with a coherent busy error — S3-3) + leak metric, credential-refresh flights bounded distinct-grant admission (cap on concurrent grant fingerprints) + staleness replacement (S3-3), usage-read flight staleness guard PLUS bounded parse (stream/limit the full-log read so the in-flight value itself is byte-capped — S2-1), Cursor model-discovery response byte cap + gather-flight concurrency cap, OAuth pending-code value byte cap + auth flow/probe admission caps, MiMo JWT value byte cap, Cursor MCP manager payload caps, crash-ring + fixed-slot diagnostic + AFFINITY value-byte truncation (S2-2) | — | |
| 45 | +| `040_app_bytes_observability.md` | wp5 | appOwnedBytes payload on /api/system/memory (continuation, blobs, rings, caches) + process-wide budget with oldest-first demotion contract | 010,020,035 (their accounting hooks; 030 provides sweeping only) | |
| 46 | +| `050_translator_stream_bounds.md` | wp6 | per-stream translator caps for ALL inventory §Translator-layer accumulators: tool-arg aggregate bytes (adapters keep interleaving), Responses→Chat/Claude output collectors, Kiro deferred/text state, reasoning carry, item-ID maps, tool-search sources, request-direction copy high-water, cursor producer queue backpressure + frame-size admission, image-retention/oauth/grok tail admission bounds | — (parallel-safe) | |
| 47 | +| `055_background_shell_lifecycle.md` | wp6b | Cursor background-shell store (UNBOUNDED when enabled, R1-1): session ownership, max-live admission, idle/absolute lifetime, controlled termination | — (parallel-safe) | |
| 48 | +| `060_proxy_benchmark.md` | wp7 | competitive comparison table with source URLs; per-category verdicts; gap list must be empty or converted to work-phases; superiority claims live ONLY here | 010–055 landed | |
| 49 | +| `070_close_and_push.md` | wp8 | final gates, docs sync, push, HEAD parity | all | |
| 50 | + |
| 51 | +### Budget scope split (second-audit R1-6) |
| 52 | + |
| 53 | +The 040 process-wide budget governs EVICTABLE RETAINED stores only |
| 54 | +(continuation, blobs, rings, caches). Translator per-stream buffers and |
| 55 | +serialized-tail backlogs are OBSERVED (high-water fields in appOwnedBytes) |
| 56 | +but never budget-evicted — in-flight conversion state cannot be evicted |
| 57 | +coherently; it is bounded by 050's per-stream caps instead. With this split |
| 58 | +050 is genuinely parallel to 040, and 040's dependency is on the accounting |
| 59 | +hooks of 010/020/035 (030 provides sweeping only, no accounting interface). |
| 60 | + |
| 61 | +### Store-by-store mechanism lock for 030 (S2-5) |
| 62 | + |
| 63 | +| Store | Mechanism | Why | |
| 64 | +|---|---|---| |
| 65 | +| subagent quota-failure records | (a) TTL sweep | getters already treat expired rows as absent | |
| 66 | +| key/combo cooldowns | (a) TTL sweep | same | |
| 67 | +| Anthropic routing health | (a) TTL sweep | same | |
| 68 | +| XAI verdicts | (a) TTL sweep | 30 s TTL with exact-key lazy expiry today (S3-2); reconciliation would leave expired churn until config changes | |
| 69 | +| warning-key memos (all) | (b) reconciliation ONLY | no expiry semantics; TTL would resurrect suppressed warnings | |
| 70 | +| codex quota rows | (b) reconciliation ONLY | 6 h is disk-hydration admission, not a live TTL; live getters serve rows indefinitely (quota.ts:261/298) | |
| 71 | +| provider quota history, model-cache provider history, pool/combo history, guardian backoff, GCP source tokens, ownership/PID memos, OAuth/reauth keys | (b) reconciliation | dead keys unreachable from current config/accounts | |
| 72 | +| Windows ACL success memos | (c) delete-after-rename | R1-2 contract; destination re-keying forbidden | |
| 73 | + |
| 74 | +Scope guards: provider adapter LOGIC unchanged (bounds are added at the |
| 75 | +adapter-local accumulation sites without altering event semantics — the |
| 76 | +"translation duty" stores are bounded, never deleted); #820 architecture |
| 77 | +(turn lease/session lanes/scheduler) stays out; stream-path work from the |
| 78 | +prior unit is not reopened. |
| 79 | + |
| 80 | +## Locked design decisions |
| 81 | + |
| 82 | +1. **Continuation (010):** keep the store authoritative for replay; single |
| 83 | + oversized entries spill to a DEDICATED per-entry file (not the debounced |
| 84 | + monolithic snapshot, which is async/best-effort/2 MiB-skipping — R1-2). |
| 85 | + Demotion ordering is locked: RAM row becomes a stub ONLY after an atomic |
| 86 | + successful spill (write-fsync + atomic NO-REPLACE publication — link or |
| 87 | + exclusive copy; plain rename replaces existing destinations and is |
| 88 | + unsuitable, 010 round-6). Spill FAILURE (disk permissions, |
| 89 | + exhaustion, I/O) evicts the row from RAM and records a small |
| 90 | + `spill-failed` tombstone for the response id: later continuation |
| 91 | + against it returns a terminal structured 400 telling the caller to resend |
| 92 | + the full context without `previous_response_id` — caller-driven recovery, |
| 93 | + not an automatic client fallback. Replay through a missing/ |
| 94 | + corrupt stub likewise returns an EXPLICIT structured continuation error, |
| 95 | + never silent forwarding of the naked delta. The last-entry exemption is |
| 96 | + deleted; `storedResponseBytes > byteCap()` always spills or |
| 97 | + tombstone-evicts within a bounded in-flight window — the RAM cap is |
| 98 | + unconditionally hard. |
| 99 | + UNSAFE boundary: if spill-through cannot preserve replay, stop and report |
| 100 | + rather than truncating conversations. |
| 101 | +2. **Blobs (020):** per-blob admission cap + aggregate byte cap with LRU; |
| 102 | + eviction split by provenance (R1-3): locally-regenerated blobs (root/turn, |
| 103 | + rebuilt each request per protobuf-request.ts) are LRU-evictable; REMOTE |
| 104 | + `setBlobArgs` blobs are pinned within TTL — evicted only at TTL expiry |
| 105 | + unless protocol-level re-send is proven by regression. The aggregate cap |
| 106 | + is enforced AT INSERT for every provenance: if admitting a remote |
| 107 | + blob would exceed the cap after evicting all evictable entries, the |
| 108 | + insert is REJECTED and the hash takes the explicit-miss path (identical |
| 109 | + protocol surface to an evicted-hash miss) — pinning orders eviction |
| 110 | + preference, it never overrides the cap. Explicit miss on |
| 111 | + evicted hash. Antigravity inner maps get count+byte bounds with |
| 112 | + clear-on-invalid preserved. Image zero-weight sentinels get key-byte |
| 113 | + accounting and a count cap (R1-1). |
| 114 | +3. **Sweeper (030):** ONE shared `sweepExpired()` utility with per-store |
| 115 | + registration, run on a coarse timer (60 s, unref) and on store-write — |
| 116 | + but with THREE mechanism classes per the 030 matrix (R1-4): clock-TTL |
| 117 | + predicates, config-generation reconciliation for generation-owned keys, |
| 118 | + and admission caps for active resources. No behavior change for live keys. |
| 119 | + Windows ACL success memo: evict the temp-path row after rename completes; |
| 120 | + never let one temp's success vouch for another (R1-5, security review). |
| 121 | +4. **Budget (040):** observability FIRST (appOwnedBytes payload), then a |
| 122 | + documented demotion contract: when the global budget is exceeded, evict |
| 123 | + oldest unpinned entries store-by-store in a fixed priority order |
| 124 | + (logs/rings → caches → blobs → continuation spill). Never reject new work |
| 125 | + admission as the first lever. |
| 126 | +5. **Translator bounds (050):** aggregate-byte caps sized for 20+ parallel |
| 127 | + tool calls as NORMAL (per prior handoff): per-call args cap 2 MiB, per-turn |
| 128 | + aggregate 32 MiB, overflow FAILS the turn coherently (never truncated JSON). |
| 129 | + Cursor frame admission rejects announced sizes above cap before buffering. |
| 130 | + |
| 131 | +## Open questions carried into phase docs |
| 132 | + |
| 133 | +- 010: spill file naming/GC — per-entry content-addressed vs id-keyed files; |
| 134 | + orphan cleanup on startup. Decide in the phase doc after reading the disk |
| 135 | + path (the debounced snapshot is NOT reused for spill — locked, R1-2). |
| 136 | +- 040: budget default size and config surface (fixed vs configurable). |
| 137 | +- 050: which tails (image/oauth/grok) get admission caps vs busy-rejection. |
| 138 | + |
| 139 | +## Regression-class requirements per phase (audit round 1) |
| 140 | + |
| 141 | +- 010: spill durability-before-demotion; missing/ |
| 142 | + corrupt spill → explicit error; spill-failure tombstone eviction + counter; |
| 143 | + restart replay; provider metadata + tool- |
| 144 | + result replay through stubs; replacement/TTL cleanup of spill files; orphan |
| 145 | + cleanup; concurrent flush/demotion; disk-permission failure. |
| 146 | +- 020: local vs remote blob provenance; exact boundary admission; aggregate |
| 147 | + accounting; replacement/re-store LRU; explicit misses; Antigravity live-call |
| 148 | + preservation; image sentinel metadata accounting; pinned-saturation |
| 149 | + (remote blobs collectively at cap → new inserts rejected with explicit miss). |
| 150 | +- 030: fake-clock global expiry; live-key preservation; generation |
| 151 | + reconciliation; timer singleton/unref/stop; write-trigger sweep; Windows |
| 152 | + repeated-temp ACL proof (second temp for same destination is re-hardened). |
| 153 | +- 035: subscriber admission cap (rejects beyond cap, existing listeners |
| 154 | + unaffected); active-registry admission counter + leak metric monotonicity; |
| 155 | + active-registry HARD admission cap (beyond-cap request receives a coherent |
| 156 | + busy error, existing work unaffected — S3-3); flight staleness (a stuck |
| 157 | + flight is replaced, not joined, after the guard) plus bounded |
| 158 | + distinct-grant admission (concurrent grant fingerprints capped — S3-3); |
| 159 | + usage-read bounded parse (oversized log yields a capped result, |
| 160 | + not an unbounded in-flight value); discovery response byte cap + |
| 161 | + concurrent-gather cap; OAuth pending-code value cap + flow/probe admission; |
| 162 | + MiMo JWT byte cap; MCP payload caps; value-byte truncation for crash-ring, |
| 163 | + fixed-slot diagnostics, and affinity entries (truncated string still |
| 164 | + useful, marker appended). |
| 165 | +- 040: observe-only metrics; exact replacement/eviction accounting; pinned/ |
| 166 | + no-evictable fallback; single-entry-over-budget; privacy-safe payloads. |
| 167 | +- 050: 20+ interleaved calls normal; per-call and aggregate boundaries; |
| 168 | + coherent overflow (no truncated JSON, turn fails); upstream cancellation; |
| 169 | + Cursor announced-frame rejection before allocation; queue ordering/ |
| 170 | + backpressure; stalled-tail busy/admission. |
| 171 | +- 055: max-live admission; idle + absolute lifetime termination; session |
| 172 | + ownership (no cross-session kill); controlled termination drains output; |
| 173 | + disabled-by-default posture unchanged. |
0 commit comments