Skip to content

Commit 679dcd3

Browse files
nnemirovskynnemirovsky
committed
docs(plans): add OAuth dynamic phantom swap implementation plan
Bidirectional OAuth token management through sluice's MITM proxy. Response interception captures real tokens from OAuth endpoints, stores in vault, replaces with deterministic phantoms before they reach the agent. Supports concurrent refresh dedup via singleflight.
1 parent faf6dfb commit 679dcd3

1 file changed

Lines changed: 350 additions & 0 deletions

File tree

‎docs/plans/20260407-oauth-phantom-swap.md‎

Lines changed: 350 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,350 @@
1+
# OAuth Dynamic Phantom Swap
2+
3+
## Overview
4+
5+
Extend sluice's phantom token system to handle OAuth credentials bidirectionally. Currently, phantom swap is request-only (phantom -> real in outbound requests). This adds response-side interception: when an OAuth token endpoint returns new access/refresh tokens, sluice captures the real tokens, stores them in the vault, and replaces them with phantom tokens in the response body before it reaches the agent.
6+
7+
This enables the agent to use OAuth-based providers (OpenAI Codex subscriptions, Google OAuth, etc.) while never seeing real credentials. The entire OAuth lifecycle (initial auth, token refresh, token rotation) is handled transparently through phantom tokens.
8+
9+
## Context
10+
11+
- `internal/proxy/inject.go` -- MITM request handler with three-pass phantom injection. `injectCredentials()` (line 468) handles request-side swap. goproxy `OnResponse()` available but only used for WebSocket upgrades currently.
12+
- `internal/vault/store.go` -- Age-encrypted credential storage. `Get()` returns `SecureBytes`, `Add()` stores encrypted.
13+
- `internal/vault/phantom.go` -- Phantom token generation. `GeneratePhantomToken()` creates format-matching placeholders.
14+
- `internal/vault/binding.go` -- `BindingResolver` maps (destination, port, protocol) to credentials. `CredentialsForDestination()` returns all bound credentials.
15+
- `internal/proxy/server.go` -- `StoreResolver()` and `StoreEngine()` do atomic hot-reload of policy and bindings.
16+
- `internal/store/store.go` -- SQLite store with `rules`, `bindings`, `config` tables. Schema in `migrations/000001_init.up.sql`.
17+
- `internal/mcp/inspect.go` -- `ContentInspector` with `RedactResponse()` pattern for response modification.
18+
- `internal/proxy/quic.go` -- Response body reading pattern (line 464-483): `io.ReadAll` + size limit + redact rules.
19+
20+
## Development Approach
21+
22+
- **Testing approach**: Regular (code first, then tests)
23+
- **CRITICAL: every task MUST include new/updated tests**
24+
- **CRITICAL: all tests must pass before starting next task**
25+
- Run `go test ./... -timeout 30s` after each change
26+
- Maintain backward compatibility. Existing non-OAuth credentials must work unchanged.
27+
28+
## Testing Strategy
29+
30+
- **Unit tests**: Go tests for OAuth credential type, response interception, token parsing, vault update, phantom generation
31+
- **Integration tests**: MITM proxy with mock OAuth server returning token responses
32+
- **E2e tests**: Deferred to manual testing
33+
34+
## Progress Tracking
35+
36+
- Mark completed items with `[x]` immediately when done
37+
- Add newly discovered tasks with + prefix
38+
- Document issues/blockers with ! prefix
39+
40+
## Solution Overview
41+
42+
### Credential types
43+
44+
Add a `type` field to credentials. Two types:
45+
- `static` (default, current behavior): simple phantom -> real swap in requests
46+
- `oauth`: bidirectional swap with response interception
47+
48+
OAuth credentials store a JSON blob in the vault (real tokens only, no phantom values):
49+
50+
```json
51+
{
52+
"access_token": "real-access-token",
53+
"refresh_token": "real-refresh-token",
54+
"token_url": "https://auth0.openai.com/oauth/token",
55+
"expires_at": "2026-04-07T12:00:00Z"
56+
}
57+
```
58+
59+
Phantom tokens are deterministic, derived at runtime from the credential name using the existing `SLUICE_PHANTOM:` prefix scheme:
60+
- Access phantom: `SLUICE_PHANTOM:credname.access`
61+
- Refresh phantom: `SLUICE_PHANTOM:credname.refresh`
62+
63+
The agent sees these deterministic phantoms. The MITM proxy resolves them to real tokens by loading the OAuth JSON from vault and extracting the corresponding field.
64+
65+
### Response interception flow
66+
67+
1. Agent sends request to token URL with phantom refresh token
68+
2. Sluice request handler: swaps `SLUICE_PHANTOM:cred.refresh` -> real refresh token
69+
3. Upstream returns JSON response: `{ "access_token": "new-real", "refresh_token": "new-real-refresh", "expires_in": 3600 }`
70+
4. Sluice response handler detects this is a response from a configured `token_url`
71+
5. Sluice extracts `access_token` and `refresh_token` from response JSON
72+
6. Sluice replaces real tokens with deterministic phantoms in response body **first** (before any I/O)
73+
7. Sluice returns the modified response to the agent immediately
74+
8. Sluice **asynchronously** updates vault with new real tokens and writes phantom files to shared volume
75+
76+
Token replacement in the response body is independent of vault persistence. If the vault write fails, the agent still receives phantom tokens (not real ones). The vault write is retried or logged, and the next refresh cycle will correct the state.
77+
78+
### Concurrent refresh protection
79+
80+
Many OAuth providers invalidate refresh tokens on use (rotation). If two requests trigger simultaneous refresh, the second refresh would use an already-invalidated token. Solution: use `golang.org/x/sync/singleflight` keyed on credential name. Only one refresh response is processed at a time per credential. Concurrent requests reuse the first result.
81+
82+
### Token URL matching
83+
84+
The `token_url` from the OAuth credential is parsed at credential-add time. During MITM response handling, sluice checks if the response's request URL matches any configured `token_url`. Only matching responses are intercepted.
85+
86+
### CLI interface
87+
88+
```bash
89+
# Add OAuth credential (tokens read from stdin/prompt, not CLI flags, to avoid shell history exposure)
90+
sluice cred add openai_oauth \
91+
--type oauth \
92+
--token-url https://auth0.openai.com/oauth/token \
93+
--destination api.openai.com \
94+
--ports 443
95+
# Prompts for: access token, refresh token (optional)
96+
97+
# List shows type
98+
sluice cred list
99+
# NAME TYPE DESTINATION
100+
# openai_oauth oauth api.openai.com
101+
# github_pat static api.github.com
102+
```
103+
104+
### Data model changes
105+
106+
**Vault**: OAuth credentials stored as JSON blob (same `.age` file, different content format). `Get()` returns the full JSON. New `ParseOAuth()` function parses and returns structured `OAuthCredential`. Vault stores only real token data, not phantom values (phantoms are deterministic).
107+
108+
**Store**: New `credential_meta` table (not on `bindings` table, since one credential can have multiple bindings). One row per credential: `name` (PK), `cred_type`, `token_url`. Avoids duplication across bindings.
109+
110+
**Phantoms**: OAuth credentials get two phantom files written to the phantoms volume: `CRED_NAME_ACCESS` and `CRED_NAME_REFRESH`. For the MITM proxy, deterministic `SLUICE_PHANTOM:name.access` / `SLUICE_PHANTOM:name.refresh` tokens are used.
111+
112+
## Technical Details
113+
114+
### Response handler registration
115+
116+
```go
117+
// In NewInjector(), add response handler alongside existing WebSocket handler
118+
proxy.OnResponse().DoFunc(inj.interceptOAuthResponse)
119+
```
120+
121+
### Response interception function
122+
123+
```go
124+
func (inj *Injector) interceptOAuthResponse(resp *http.Response, ctx *goproxy.ProxyCtx) *http.Response {
125+
// 1. Check if request URL matches any configured token_url
126+
// 2. If not, return resp unchanged
127+
// 3. Read response body (same size limit as requests)
128+
// 4. Parse JSON, extract access_token/refresh_token
129+
// 5. Update vault with new real tokens
130+
// 6. Generate new phantom tokens
131+
// 7. Replace real tokens with phantoms in response body
132+
// 8. Update response body and content-length
133+
// 9. Trigger phantom file reload
134+
return resp
135+
}
136+
```
137+
138+
### Token URL index
139+
140+
```go
141+
// oauthIndex maps token URLs to credential names for fast lookup
142+
type oauthIndex struct {
143+
entries []oauthEntry
144+
}
145+
146+
type oauthEntry struct {
147+
tokenURL *url.URL // parsed token URL
148+
credential string // credential name in vault
149+
}
150+
```
151+
152+
Stored in `Injector` as `*atomic.Pointer[oauthIndex]` for hot-reload. Rebuilt when credentials change.
153+
154+
### Phantom token mapping for OAuth
155+
156+
Phantom tokens are deterministic, derived from the credential name at runtime:
157+
- Access phantom: `SLUICE_PHANTOM:credname.access`
158+
- Refresh phantom: `SLUICE_PHANTOM:credname.refresh`
159+
160+
Vault JSON stores only real tokens (no phantom values):
161+
162+
```json
163+
{
164+
"access_token": "real-access",
165+
"refresh_token": "real-refresh",
166+
"token_url": "https://auth0.openai.com/oauth/token",
167+
"expires_at": "2026-04-07T12:00:00Z"
168+
}
169+
```
170+
171+
For phantom files on the shared volume (what the agent loads as env vars), `GeneratePhantomToken()` produces format-matching values (e.g., `sk-phantom-xxx` for OpenAI). These are separate from the MITM deterministic tokens. The MITM proxy handles `SLUICE_PHANTOM:` tokens. The container phantoms are for the agent's SDK initialization.
172+
173+
### Security considerations
174+
175+
- Token replacement in response body happens **before** vault write (response-first, persist-async)
176+
- If vault write fails, agent still receives phantom tokens, not real ones
177+
- Real tokens handled via `SecureBytes` with `Release()` after vault write
178+
- Response body modification uses same size limits as request body (16 MiB)
179+
- Token URL matching is exact (scheme + host + path), not prefix
180+
- Responses with status 200-299 are intercepted. Both `application/json` and `application/x-www-form-urlencoded` content types are supported (per RFC 6749).
181+
- When replacing response body, clear `Transfer-Encoding: chunked` and set explicit `Content-Length`
182+
- `singleflight` prevents concurrent refresh race conditions
183+
- If refresh_token is missing from a token response, the existing refresh_token is preserved in vault
184+
185+
## What Goes Where
186+
187+
- **Implementation Steps**: vault changes, store schema, CLI, response handler, phantom generation, hot-reload
188+
- **Post-Completion**: manual testing with real OAuth provider, OpenClaw integration testing
189+
190+
## Implementation Steps
191+
192+
### Task 1: Add OAuth credential type to vault
193+
194+
**Files:**
195+
- Create: `internal/vault/oauth.go`
196+
- Modify: `internal/vault/store.go`
197+
198+
- [ ] Create `internal/vault/oauth.go` with `OAuthCredential` struct (AccessToken, RefreshToken, TokenURL, ExpiresAt). No phantom values in the struct (phantoms are deterministic, derived from credential name at runtime).
199+
- [ ] Add `ParseOAuth(data []byte) (*OAuthCredential, error)` to parse JSON blob from vault
200+
- [ ] Add `(*OAuthCredential) Marshal() ([]byte, error)` to serialize back to JSON
201+
- [ ] Add `(*OAuthCredential) UpdateTokens(access, refresh string, expiresIn int)` that updates real tokens and computes ExpiresAt. If refresh is empty, preserve existing refresh_token.
202+
- [ ] Add `IsOAuth(data []byte) bool` function that checks if credential content is valid OAuth JSON (has access_token + token_url fields)
203+
- [ ] Write tests for OAuthCredential parse/marshal round-trip
204+
- [ ] Write tests for UpdateTokens (both tokens, access only, refresh preserved)
205+
- [ ] Write tests for IsOAuth detection (positive, negative, malformed JSON)
206+
- [ ] Run tests: `go test ./internal/vault/ -timeout 30s`
207+
208+
### Task 2: Add credential_meta table to store schema
209+
210+
**Files:**
211+
- Create: `internal/store/migrations/000002_credential_meta.up.sql`
212+
- Create: `internal/store/migrations/000002_credential_meta.down.sql`
213+
- Modify: `internal/store/store.go`
214+
215+
- [ ] Create migration: new `credential_meta` table with `name TEXT PRIMARY KEY`, `cred_type TEXT NOT NULL DEFAULT 'static'`, `token_url TEXT`, `created_at DATETIME DEFAULT CURRENT_TIMESTAMP`
216+
- [ ] Add `CredentialMeta` struct to store (Name, CredType, TokenURL, CreatedAt)
217+
- [ ] Add `AddCredentialMeta(name, credType, tokenURL string) error`
218+
- [ ] Add `GetCredentialMeta(name string) (*CredentialMeta, error)`
219+
- [ ] Add `ListCredentialMeta() ([]CredentialMeta, error)`
220+
- [ ] Add `RemoveCredentialMeta(name string) error` (cascade with credential removal)
221+
- [ ] Write tests for migration (up and down)
222+
- [ ] Write tests for CRUD operations on credential_meta
223+
- [ ] Run tests: `go test ./internal/store/ -timeout 30s`
224+
225+
### Task 3: Extend CLI for OAuth credentials
226+
227+
**Files:**
228+
- Modify: `cmd/sluice/cred.go`
229+
230+
- [ ] Add `--type` flag to `sluice cred add` (default: "static", options: "static", "oauth")
231+
- [ ] Add `--token-url` flag (required when type=oauth)
232+
- [ ] When type=oauth: prompt for access token and refresh token via stdin/terminal (not CLI flags, to avoid shell history exposure). Support stdin pipe for scripted use.
233+
- [ ] When type=oauth: create OAuthCredential JSON, store in vault, create credential_meta row, create binding with destination
234+
- [ ] Generate two phantom files for OAuth: `CRED_NAME_ACCESS` and `CRED_NAME_REFRESH`
235+
- [ ] Update `sluice cred list` to show credential type column (join with credential_meta)
236+
- [ ] Update `sluice cred remove` to also delete credential_meta row
237+
- [ ] Write tests for CLI flag parsing and validation
238+
- [ ] Write tests for OAuth credential creation flow
239+
- [ ] Run tests: `go test ./cmd/sluice/ -timeout 30s`
240+
241+
### Task 4: Build OAuth token URL index
242+
243+
**Files:**
244+
- Create: `internal/proxy/oauth_index.go`
245+
- Modify: `internal/proxy/inject.go`
246+
247+
- [ ] Create `internal/proxy/oauth_index.go` with `OAuthIndex` struct: maps token URLs to credential names
248+
- [ ] Add `NewOAuthIndex(metas []store.CredentialMeta) *OAuthIndex` that filters oauth-type entries and parses token URLs
249+
- [ ] Add `Match(requestURL *url.URL) (credName string, ok bool)` for exact URL matching (scheme + host + path)
250+
- [ ] Add `oauthIndex *atomic.Pointer[OAuthIndex]` field to `Injector` struct
251+
- [ ] Build and store index during `NewInjector()` initialization
252+
- [ ] Add `UpdateOAuthIndex(metas []store.CredentialMeta)` for hot-reload (called from StoreResolver path)
253+
- [ ] Write tests for index building and matching (exact match, no match, multiple entries)
254+
- [ ] Write tests for hot-reload (index swap)
255+
- [ ] Run tests: `go test ./internal/proxy/ -timeout 30s`
256+
257+
### Task 5: Implement response-side OAuth token interception
258+
259+
**Files:**
260+
- Create: `internal/proxy/oauth_response.go`
261+
- Modify: `internal/proxy/inject.go`
262+
263+
- [ ] Add `golang.org/x/sync/singleflight` dependency: `go get golang.org/x/sync`
264+
- [ ] Add `refreshGroup singleflight.Group` field to `Injector` struct for concurrent refresh dedup
265+
- [ ] Register response handler in `NewInjector()`: `proxy.OnResponse().DoFunc(inj.interceptOAuthResponse)`
266+
- [ ] Implement `interceptOAuthResponse(resp *http.Response, ctx *goproxy.ProxyCtx) *http.Response`:
267+
- Check response status is 200-299
268+
- Match request URL against OAuth index. If no match, return unchanged.
269+
- Use `singleflight.Do(credName, ...)` to prevent concurrent refresh race
270+
- Read response body (same 16 MiB limit)
271+
- Parse response body: support both `application/json` and `application/x-www-form-urlencoded` (per RFC 6749)
272+
- Extract `access_token`, `refresh_token` (optional), `expires_in`
273+
- Replace real tokens with deterministic phantoms (`SLUICE_PHANTOM:cred.access`, `SLUICE_PHANTOM:cred.refresh`) in response body **first**
274+
- Clear `Transfer-Encoding` header, set explicit `Content-Length`
275+
- Update `resp.Body` with modified body
276+
- **Asynchronously** (goroutine): load OAuthCredential from vault, call `UpdateTokens()`, store back, write phantom files, signal agent reload
277+
- Use `SecureBytes` for real token handling in the async goroutine, `Release()` after vault write
278+
- [ ] Handle edge cases: missing refresh_token in response (preserve existing), non-JSON/non-form content type (pass through), read errors (pass through unchanged)
279+
- [ ] If vault write fails: log error. Agent already has phantom tokens. Next refresh cycle will correct.
280+
- [ ] Write tests with mock goproxy context and httptest response (JSON format)
281+
- [ ] Write tests for form-encoded token responses
282+
- [ ] Write tests for partial responses (only access_token, no refresh_token)
283+
- [ ] Write tests for concurrent refresh dedup (singleflight)
284+
- [ ] Write tests for error cases (non-2xx, non-JSON, oversized body, vault write failure)
285+
- [ ] Run tests: `go test ./internal/proxy/ -timeout 30s`
286+
287+
### Task 6: Request-side OAuth phantom swap
288+
289+
**Files:**
290+
- Modify: `internal/proxy/inject.go`
291+
292+
- [ ] Extend `injectCredentials()` to detect OAuth credentials: check if credential name exists in OAuth index (or if vault content is OAuth JSON via `IsOAuth()`)
293+
- [ ] For OAuth credentials: parse vault JSON via `ParseOAuth()`, extract real access_token and refresh_token, build two phantom pairs: `[{SLUICE_PHANTOM:cred.access, realAccess}, {SLUICE_PHANTOM:cred.refresh, realRefresh}]`
294+
- [ ] Add OAuth phantom pairs to the scoped replacement list alongside static phantom pairs
295+
- [ ] Ensure OAuth phantoms (`SLUICE_PHANTOM:*.access`, `SLUICE_PHANTOM:*.refresh`) are included in the unbound strip pass (pass 3) via extended `stripUnboundPhantoms()`
296+
- [ ] Use `SecureBytes` for real tokens, `Release()` after replacement
297+
- [ ] Write tests: request with OAuth phantom access token gets swapped to real
298+
- [ ] Write tests: request with OAuth phantom refresh token gets swapped to real
299+
- [ ] Write tests: mixed static + OAuth credentials on same request
300+
- [ ] Write tests: unbound OAuth phantom tokens are stripped
301+
- [ ] Run tests: `go test ./internal/proxy/ -timeout 30s`
302+
303+
### Task 7: Hot-reload and phantom file management
304+
305+
**Files:**
306+
- Modify: `internal/proxy/server.go`
307+
- Modify: `internal/vault/phantom.go`
308+
309+
- [ ] Extend `GeneratePhantomEnv()` to handle OAuth credentials: detect OAuth JSON in vault, write two files per OAuth credential (`CRED_ACCESS`, `CRED_REFRESH`) with format-matching phantom values
310+
- [ ] Add `WriteOAuthPhantoms(dir string, cred *OAuthCredential, name string) error` to write phantom files for a single OAuth credential (called from async goroutine in response handler)
311+
- [ ] Ensure async phantom file write in response handler (Task 5) does not block HTTP response delivery
312+
- [ ] Rebuild OAuth index on `StoreResolver()` calls (when credential_meta changes)
313+
- [ ] Write tests for OAuth phantom file generation (two files created, correct naming)
314+
- [ ] Write tests for hot-reload path (credential_meta change triggers index rebuild)
315+
- [ ] Run tests: `go test ./... -timeout 30s`
316+
317+
### Task 8: Verify acceptance criteria
318+
319+
- [ ] Verify static credentials still work unchanged (backward compatibility)
320+
- [ ] Verify OAuth credential can be added via CLI with --type oauth
321+
- [ ] Verify phantom files are generated for OAuth (access + refresh)
322+
- [ ] Verify request-side swap works for OAuth access token
323+
- [ ] Verify request-side swap works for OAuth refresh token
324+
- [ ] Verify response interception captures new tokens from token endpoint
325+
- [ ] Verify vault is updated with rotated tokens
326+
- [ ] Verify new phantoms are written and agent reload signaled
327+
- [ ] Verify unbound OAuth phantoms are stripped from requests
328+
- [ ] Run full test suite: `go test ./... -v -timeout 30s`
329+
- [ ] Run e2e tests: `go test -tags=e2e ./e2e/ -v -count=1 -timeout=300s`
330+
331+
### Task 9: [Final] Update documentation
332+
333+
- [ ] Update CLAUDE.md with OAuth credential type documentation
334+
- [ ] Update README.md: add section on dynamic OAuth/JWT token management (transparent access/refresh token rotation through phantom swap, subscription-based auth support)
335+
- [ ] Move this plan to `docs/plans/completed/`
336+
337+
## Post-Completion
338+
339+
**Manual verification:**
340+
- Test with real OpenAI Codex OAuth flow (onboard locally, feed tokens to sluice)
341+
- Test token refresh cycle end-to-end (let access token expire, verify transparent refresh)
342+
- Test with OpenClaw in Docker container behind sluice MITM
343+
- Verify phantom tokens never appear in audit log
344+
345+
**Future work:**
346+
- REST API support for OAuth credential CRUD (OpenAPI spec + handlers)
347+
- Sluice-driven proactive token refresh (refresh before expiry, not on 401)
348+
- OAuth discovery (auto-detect token_url from .well-known/openid-configuration)
349+
- Multiple OAuth providers per agent (OpenAI + Google + Anthropic simultaneously)
350+
- Token usage metrics in audit log (track refresh frequency, expiry patterns)

0 commit comments

Comments
 (0)