Date: 2026-03-22
Status: Living Document — current state documented; proposed changes marked
Workflow: ../../workflows/control-plane/control-plane.workflow.md — implementation waves, gap table, build commands
The Ambient Control Plane (CP) and the Runner are two cooperating runtime components that sit between the api-server and the actual Claude Code execution. Together they implement the execution half of the session lifecycle: provisioning Kubernetes resources, starting Claude, delivering messages in both directions, and persisting the conversation record.
User / CLI
│ REST / gRPC
▼
ambient-api-server ← data model, auth, RBAC, DB
│ gRPC WatchSessions
▼
ambient-control-plane (CP) ← K8s provisioner + session watcher
│ K8s API + env vars
▼
Runner Pod ← FastAPI + ClaudeBridge + gRPC client
│ Claude Agent SDK
▼
Claude Code CLI (subprocess)The api-server is the source of truth for all persistent state. The CP and Runner have no databases of their own. They read from the api-server via the Go SDK and write back via PushSessionMessage gRPC and UpdateStatus REST.
The CP is a standalone Go service (ambient-control-plane) that:
- Watches the api-server for session events via gRPC
WatchSessions - Provisions Kubernetes resources for each session (namespace, secret, service account, pod, service)
- Assembles the start context (Project.prompt + Agent.prompt + Inbox messages + Session.prompt) and injects it as
INITIAL_PROMPTenv var into the runner pod - Updates session phase via
sdk.Sessions().UpdateStatus()as pods transition through states
The CP does not proxy traffic. It does not fan out events. It does not hold any persistent state. It is a pure Kubernetes reconciler driven by the api-server event stream.
Maintains one gRPC WatchSessions stream per resource type (sessions, projects, project_settings). Reconnects with exponential backoff (1s → 30s) on stream failure. Dispatches each event to a buffered channel consumed by the Informer.
Performs an initial list+watch sync at startup. Converts proto events to SDK types. Buffers events (256 capacity) and dispatches them to all registered reconcilers.
Handles session ADDED and session MODIFIED (phase=Pending) events by provisioning:
- Namespace (named
{project_id}) - K8s Secret with
BOT_TOKEN(the runner's api-server credential) - ServiceAccount (no automount)
- Pod (runner image + env vars)
- Service (ClusterIP on port 8001 pointing at the pod)
- RoleBinding granting
system:image-builderClusterRole tosession-{id}-sa(enables push to the OpenShift internal image registry)
On phase=Stopping → calls deprovisionSession (deletes pods).
On DELETED → calls cleanupSession (deletes pod, secret, service account, service, namespace).
Mints and caches per-project SDK clients. Each project uses the same bearer token but different project context. Also provides namespaceForSession, phase constants, and label helpers.
Thin wrapper over k8s.io/client-go dynamic client. Provides typed Create/Get/Delete methods for Pod, Service, Secret, ServiceAccount, Namespace, RoleBinding. Eliminates raw unstructured map construction from reconciler code.
The CP creates a Pod (not a Job) for each session. Key pod attributes:
| Attribute | Value | Reason |
|---|---|---|
restartPolicy |
Never |
Sessions are single-run; no automatic restart |
imagePullPolicy |
IfNotPresent for localhost/ images, Always otherwise |
kind uses local containerd — Always breaks localhost/ image pulls |
serviceAccountName |
session-{id}-sa |
Session-scoped; no cross-session access |
automountServiceAccountToken |
true |
Runner uses the SA token to authenticate to the CP token endpoint |
| CPU request/limit | 500m / 2000m | Generous for Claude Code |
| Memory request/limit | 512Mi / 4Gi | Claude Code is memory-intensive |
The CP binds the system:image-builder ClusterRole to session-{id}-sa via a namespace-scoped RoleBinding at provision time. This grants the runner pod push access to the OpenShift internal image registry (image-registry.openshift-image-registry.svc:5000), enabling agents to build and push images using daemonless tools such as crane. Pull access is provided automatically to all SAs in the namespace by OpenShift via the system:image-pullers RoleBinding created at namespace initialization.
assembleInitialPrompt builds INITIAL_PROMPT from four sources in order:
1. Project.prompt — workspace-level context (shared by all agents in this project)
2. Agent.prompt — who this agent is (if session has AgentID)
3. Inbox messages — unread InboxMessage.Body items addressed to this agent
4. Session.prompt — what this specific run should doEach section is joined with \n\n. Empty sections are omitted. If all four are empty, INITIAL_PROMPT is not set and the runner waits for a user message via gRPC.
| Var | Value | Purpose |
|---|---|---|
SESSION_ID |
session.ID | Primary session identifier |
PROJECT_NAME |
session.ProjectID | Project context |
WORKSPACE_PATH |
/workspace |
Claude Code working directory |
AGUI_PORT |
8001 |
Runner HTTP listener port |
BACKEND_API_URL |
CP config | api-server base URL |
AMBIENT_GRPC_URL |
CP config | api-server gRPC address |
AMBIENT_GRPC_USE_TLS |
CP config | TLS flag for gRPC |
AMBIENT_CP_TOKEN_URL |
CP config | CP token endpoint URL (e.g. http://ambient-control-plane.{ns}.svc:8080/token) |
INITIAL_PROMPT |
assembled prompt | Auto-execute on startup |
IS_RESUME |
"true" |
Set when session.StartTime != nil (session has been started before); tells the runner to skip INITIAL_PROMPT auto-execute |
RESUME_AFTER_SEQ |
max seq from session_messages |
Set alongside IS_RESUME when messages exist; runner's gRPC listener starts watching from this seq to prevent replay of historical messages |
USE_VERTEX / ANTHROPIC_VERTEX_PROJECT_ID / CLOUD_ML_REGION |
CP config | Vertex AI config (when enabled) |
GOOGLE_APPLICATION_CREDENTIALS |
/app/vertex/ambient-code-key.json |
Vertex service account path |
LLM_MODEL / LLM_TEMPERATURE / LLM_MAX_TOKENS |
session fields | Per-session model config |
CREDENTIAL_IDS |
JSON map {provider: credential_id} |
Resolved credentials for this session; runner calls /credentials/{id}/token per provider |
When the CP provisions a runner pod for a session that has been started before (session.StartTime != nil), it sets restart-specific env vars to prevent the runner from replaying the initial prompt and historical messages:
if session.StartTime != nil:
1. Set IS_RESUME=true
2. Query api-server: GET /api/ambient/v1/session_messages?search=session_id='...'&orderBy=seq desc&size=1
3. If messages exist, set RESUME_AFTER_SEQ=<max_seq>The CP uses the Go SDK's SessionMessages().List() with size=1, orderBy=seq desc to resolve the maximum sequence number. This translates to a GET /api/ambient/v1/session_messages call against the api-server.
On the runner side:
IS_RESUME=truecausesINITIAL_PROMPTto be skipped (no auto-execute on startup)RESUME_AFTER_SEQ=Ncauses the gRPC listener to startWatchSessionMessagesfromlast_seq=N, skipping all messages withseq <= N
This ensures a restarted session picks up from where it left off without replaying the initial prompt or re-processing historical user messages.
The Runner is a Python FastAPI application (ambient-runner) that runs inside each session pod. It:
- Owns the Claude Code execution lifecycle (start, run, interrupt, shutdown)
- Bridges between the AG-UI protocol (HTTP SSE) and the gRPC message store
- Listens to the api-server gRPC stream for inbound user messages
- Pushes conversation records back to the api-server via
PushSessionMessage - Exposes a local SSE endpoint for live AG-UI event observation
One runner pod runs per session. The pod is ephemeral — it exists only while the session is active.
app.py ← FastAPI application factory + lifespan
│
├── endpoints/
│ ├── run.py ← POST / (AG-UI run endpoint)
│ ├── events.py ← GET /events/{thread_id} (SSE tap — NEW)
│ ├── interrupt.py ← POST /interrupt
│ ├── health.py ← GET /health
│ └── ... (capabilities, repos, workflow, mcp_status, content)
│
├── bridges/claude/
│ ├── bridge.py ← ClaudeBridge (PlatformBridge impl)
│ ├── grpc_transport.py ← GRPCSessionListener + GRPCMessageWriter
│ ├── session.py ← SessionManager + SessionWorker
│ ├── auth.py ← Vertex AI / Anthropic auth setup
│ ├── mcp.py ← MCP server config
│ └── prompts.py ← System prompt builder
│
├── _grpc_client.py ← AmbientGRPCClient (codegen)
├── _session_messages_api.py ← SessionMessagesAPI (codegen, hand-rolled proto codec)
│
└── middleware/
└── grpc_push.py ← grpc_push_middleware (HTTP path fire-and-forget)When AMBIENT_GRPC_URL is set (standard deployment):
1. app.py lifespan() starts
2. RunnerContext created from env vars (SESSION_ID, WORKSPACE_PATH)
3. bridge.set_context(context)
4. bridge._setup_platform() called eagerly:
- SessionManager initialized
- Vertex AI / Anthropic auth configured
- MCP servers loaded
- System prompt built
- GRPCSessionListener instantiated and started
→ If RESUME_AFTER_SEQ set: listener initializes last_seq from env var,
skipping all historical messages (seq <= RESUME_AFTER_SEQ)
→ WatchSessionMessages RPC opens
→ listener.ready asyncio.Event set
5. await bridge._grpc_listener.ready.wait()
(blocks until WatchSessionMessages stream is confirmed open)
6. If INITIAL_PROMPT set and not IS_RESUME:
_auto_execute_initial_prompt(prompt, session_id, grpc_url)
→ _push_initial_prompt_via_grpc()
→ PushSessionMessage(event_type="user", payload=prompt)
→ listener receives its own push → triggers bridge.run()
7. yield (app is ready, uvicorn serving)
8. On shutdown: bridge.shutdown() → GRPCSessionListener.stop()Subscribes to WatchSessionMessages for this session via a blocking iterator running in a ThreadPoolExecutor. For each inbound message:
event_type == "user"→ parse payload asRunnerInput→ callbridge.run()→ fan out events- All other types → logged and skipped (runner only drives runs on user messages)
Sets self.ready (asyncio.Event) once the stream is open. Reconnects with exponential backoff on stream failure. Tracks last_seq to resume after reconnect.
Fan-out during a turn:
bridge.run() yields events
├── bridge._active_streams[thread_id].put_nowait(event) ← SSE tap queue
└── writer.consume(event) ← GRPCMessageWriterAccumulates MESSAGES_SNAPSHOT events during a turn. On RUN_FINISHED or RUN_ERROR, calls PushSessionMessage(event_type="assistant") with the assembled payload.
Current payload format (proposed for change — see below):
{
"run_id": "...",
"status": "completed",
"messages": [
{"role": "user", "content": "..."},
{"role": "reasoning", "content": "..."},
{"role": "assistant", "content": "..."}
]
}This payload includes the user echo and reasoning content, making it verbose and difficult to display in the CLI.
Wraps the HTTP run endpoint event stream. Calls PushSessionMessage once per AG-UI event as events flow out of bridge.run(). Fire-and-forget. Active only on the HTTP POST / path, not the gRPC listener path.
Note: With the gRPC listener as the primary path, grpc_push_middleware fires only when a run is triggered via HTTP (external POST). This is a secondary path for backward compatibility; the gRPC listener path is preferred.
| Stream | Source | Content | Persistence | Purpose |
|---|---|---|---|---|
WatchSessionMessages (gRPC DB stream) |
api-server DB | event_type=user and event_type=assistant rows |
Persisted; replay from seq=0 | Durable conversation record; CLI, history |
GET /events/{thread_id} (SSE tap) |
Runner in-memory queue | All AG-UI events: tokens, tool calls, reasoning chunks, status events | Ephemeral; runner-local; lost on reconnect | Live UI; streaming display; observability |
Added to endpoints/events.py. Registered as a core (always-on) endpoint.
Behavior:
- Registers
asyncio.Queue(maxsize=256)intobridge._active_streams[thread_id] - Streams every AG-UI event as SSE until
RUN_FINISHED/RUN_ERRORor client disconnect - Sends
: keepalivepings every 30s to hold the connection - On exit (any reason), removes the queue from
_active_streams
This endpoint is read-only. It never calls bridge.run() or modifies any state. It is a pure observer.
thread_id in the runner corresponds to the session ID (same value as SESSION_ID env var).
event_type=user payload: plain string — the user's message text.
event_type=assistant payload: JSON blob containing:
run_id— the run that produced this turnstatus—"completed"or"error"messages— array of all MESSAGES_SNAPSHOT messages including:role=user(echo of the input)role=reasoning(extended thinking content)role=assistant(Claude's reply)
This is verbose, inconsistent with the user payload format, and leaks reasoning content into the durable record.
event_type=user payload: plain string — unchanged.
event_type=assistant payload: plain string — the assistant's reply text only.
Specifically: extract only the role=assistant message's content field from the final MESSAGES_SNAPSHOT and store that as the payload. Symmetric with event_type=user.
What moves where:
role=reasoningcontent → flows throughGET /events/{thread_id}SSE only (ephemeral, live)role=assistantcontent → stored as plain string inevent_type=assistantDB rowrole=userecho → already inevent_type=userDB row; no need to repeat
Rationale:
- CLI can display
event_type=userandevent_type=assistantidentically — both are plain strings - Reasoning is observability data, not conversation record data
- Payload size drops dramatically (reasoning can be 10x longer than the reply)
- Replay via
WatchSessionMessagesreturns a clean conversation thread
Current:
payload = json.dumps({
"run_id": self._run_id,
"status": status,
"messages": self._accumulated_messages,
})Proposed:
assistant_text = next(
(m.get("content", "") for m in self._accumulated_messages
if m.get("role") == "assistant"),
"",
)
payload = assistant_textThe runner's GET /events/{thread_id} is only accessible within the cluster (pod-to-pod via ClusterIP Service). External clients need a proxy through the api-server.
The CP creates a session-{id} Service (ClusterIP, port 8001) pointing at the runner pod. The api-server can reach it at:
http://session-{kube_cr_name}.{kube_namespace}.svc.cluster.local:8001/events/{kube_cr_name}The proposed GET /api/ambient/v1/sessions/{id}/events endpoint on the api-server:
- Looks up the session from DB — gets
kube_cr_nameandkube_namespace - Constructs the runner URL
- Opens an HTTP GET with
Accept: text/event-stream - Streams the runner's SSE body verbatim to the client response
- Passes keepalive pings through unchanged
- Closes the client stream when the runner closes or client disconnects
This endpoint is implemented in plugins/sessions/plugin.go as GET /sessions/{id}/events → sessionHandler.StreamRunnerEvents (status: ✅ implemented).
plugins/proxy/plugin.go (ambient-api-server) forwards every request whose path does NOT start with /api/ambient/ verbatim to BACKEND_URL (default http://localhost:8080). Method, path, query string, headers (including Authorization), and body are forwarded unchanged. The response — headers, status code, body — is copied back unchanged.
Implementation: pkgserver.RegisterPreAuthMiddleware wraps the entire HTTP server before routing. Native paths (/api/ambient/..., /metrics, /favicon.ico) fall through to the next handler; all others are proxied.
Status: ✅ implemented — plugins/proxy/plugin.go; blank-imported in cmd/ambient-api-server/main.go.
Streams live AG-UI events for a session via GET /sessions/{id}/events.
acpctl session events <session-id>Behavior:
- Opens SSE connection to api-server
/sessions/{id}/events - Renders each event type distinctly:
TEXT_MESSAGE_CONTENT→ print token to stdout (no newline — streaming)RUN_STARTED/RUN_FINISHED/RUN_ERROR→ status lineTOOL_CALL_START/TOOL_CALL_END→ tool name + status: keepalive→ ignored
- Exits on
RUN_FINISHED,RUN_ERROR, or Ctrl+C
Status: 🔲 planned
Runner pods authenticate to the api-server gRPC interface using a BOT_TOKEN injected at pod start and refreshed by the CP every 4 minutes via a K8s Secret update. In OIDC environments (e.g. S0), BOT_TOKEN is an OIDC client-credentials JWT with a 15-minute TTL.
This creates a three-way async race:
- CP ticker writes a fresh token to the Secret every 4 minutes
- Kubelet propagates the Secret update to the pod's file mount (30–60s delay in busy clusters)
- Runner reads the file mount on gRPC reconnect
When the CP writes a token that is already close to expiry — because its in-memory OIDCTokenProvider cache had a short buffer — the runner reconnects with an already-expired token and enters an UNAUTHENTICATED loop.
The fundamental issue is that the Secret-write model is an async push with no synchronization guarantee between when the token is written and when the runner reads it.
The CP exposes a lightweight HTTP endpoint that runners call synchronously on demand to obtain a guaranteed-fresh token. This eliminates the async race entirely.
GET /token- Served by a new
net/httplistener on the CP (port 8080, separate from any existing listener) - Runner authenticates using its K8s service account token (mounted at
/var/run/secrets/kubernetes.io/serviceaccount/token) — validated by the CP via the K8sTokenReviewAPI - CP calls
tokenProvider.Token(ctx)at request time and returns the result — always fresh, always valid TTL - Response:
{"token": "<value>", "expires_at": "<RFC3339>"}
The runner's K8s SA token is a signed JWT issued by the K8s API server. The CP validates it using the K8s authentication/v1 TokenReview resource:
POST /apis/authentication.k8s.io/v1/tokenreviews
{
"spec": { "token": "<runner SA token>" }
}A successful TokenReview returns status.authenticated=true and status.user.username (e.g. system:serviceaccount:ambient-code--myproject:session-abc123-sa). The CP verifies the username prefix matches a known runner SA pattern before returning a token.
This approach uses credentials already present in every pod — no new secrets required.
The CP token endpoint is the sole source of the api-server bearer token for all runner pods. There is no Secret write loop and no BOT_TOKEN env var or file mount.
| Phase | Mechanism |
|---|---|
| Initial startup | GET /token from CP endpoint — called in lifespan before gRPC channel opens |
| gRPC reconnect | GET /token from CP endpoint — synchronous, guaranteed fresh |
The CP is critical infrastructure. It creates the runner pod, so it is running before the runner makes its first token request. If the CP is unreachable, the runner cannot function regardless (the CP is also responsible for all K8s provisioning). No fallback is needed or provided.
The CP adds a minimal net/http server alongside its existing K8s controller loop:
mux := http.NewServeMux()
mux.HandleFunc("/token", tokenHandler)
mux.HandleFunc("/healthz", healthHandler)
http.ListenAndServe(":8080", mux)The server runs in a goroutine alongside runKubeMode. It shares the existing tokenProvider and k8sClient from the main CP config.
_grpc_client.py reconnect() is updated to call the CP token endpoint instead of re-reading the Secret file:
def reconnect(self) -> None:
fresh_token = _fetch_token_from_cp() # GET AMBIENT_CP_TOKEN_URL/token with SA token
self.close()
self._token = fresh_tokenAMBIENT_CP_TOKEN_URL is injected by the CP as an env var when creating the runner pod. In local dev environments where the CP is not present, BOT_TOKEN env var may be set directly and the runner skips the CP endpoint call.
| Package | Purpose |
|---|---|
internal/tokenserver/server.go |
HTTP server setup and graceful shutdown |
internal/tokenserver/handler.go |
GET /token handler — TokenReview validation + tokenProvider call |
Status: 🔲 planned — RHOAIENG-56711
The runner fetches provider credentials at session start before invoking Claude. Credentials are resolved by the CP and injected into the runner pod as CREDENTIAL_IDS — a JSON-encoded map of provider → credential_id:
CREDENTIAL_IDS={"gitlab": "01JX...", "github": "01JY...", "jira": "01JZ..."}The CP builds this map from the Credential Kind RBAC resolver: for each provider, walk agent → project → global scope and take the most specific matching credential. Credentials not visible to this session are excluded.
The runner calls GET /api/ambient/v1/credentials/{id}/token for each provider present in CREDENTIAL_IDS. The token endpoint is gated by credential:token-reader — the CP grants this role to the runner pod's service account at session start for each injected credential ID.
Token response shape:
{ "provider": "gitlab", "token": "glpat-...", "url": "https://gitlab.myco.com" }
{ "provider": "github", "token": "github_pat_...", "url": "https://github.com" }
{ "provider": "jira", "token": "ATATT3x...", "url": "https://myco.atlassian.net", "email": "bot@myco.com" }
{ "provider": "google", "token": "{\"type\":\"service_account\", ...}" }token is always present. url and email are included when set on the Credential. The runner maps each response to environment variables and on-disk files consumed by Claude Code and its tools.
| Provider | Env vars set | Files written |
|---|---|---|
google |
USER_GOOGLE_EMAIL |
credentials.json (token value is full SA JSON) |
jira |
JIRA_URL, JIRA_API_TOKEN, JIRA_EMAIL |
— |
gitlab |
GITLAB_TOKEN |
/tmp/.ambient_gitlab_token |
github |
GITHUB_TOKEN |
/tmp/.ambient_github_token |
| Var | Value | Purpose |
|---|---|---|
CREDENTIAL_IDS |
JSON map {provider: id} |
Resolved credential IDs for this session; runner uses to call /credentials/{id}/token |
Status: ✅ implemented — Credential Kind live (PR #1110); CP integration pending (Wave 5)
The CP's cleanupSession calls kube.DeleteNamespace(). This currently fails in kind with:
namespaces "bond" is forbidden: User "system:serviceaccount:ambient-code:ambient-control-plane" cannot delete resource "namespaces" in API group "" in the namespace "bond"The ambient-control-plane ServiceAccount does not have delete on namespaces at cluster scope. The namespace is left behind after session cleanup.
Proposed fix: Add a ClusterRole with delete on namespaces and bind it to ambient-control-plane SA in the deployment manifests.
| Decision | Rationale |
|---|---|
| CP provisions Pods, not Jobs | Sessions are single-run; operator-style Job retry semantics don't apply |
| CP assembles INITIAL_PROMPT, not api-server | CP has K8s access and can read the full start context; api-server does not know which pod to address |
| gRPC listener started eagerly, not lazily | Prevents chicken-and-egg: listener must be subscribed before INITIAL_PROMPT push |
| Runner self-pushes INITIAL_PROMPT via gRPC | Avoids HTTP call to old backend; ensures message is durable before Claude runs |
WatchSessionMessages as the inbound trigger |
User messages arrive once (persisted in DB); listener replays from last_seq on reconnect |
MESSAGES_SNAPSHOT as the assistant accumulator |
Claude Agent SDK emits periodic full snapshots; last snapshot before RUN_FINISHED is the complete turn |
SSE tap via _active_streams dict |
Zero-copy fan-out from listener loop to any subscribed HTTP client; no additional gRPC round-trip |
| assistant payload → plain string | Symmetric with user payload; reasoning is observability data not conversation record |
| GET /events is runner-local | Runner has the event queue; api-server proxies it; no second fan-out layer needed |
| Namespace per project, not per session | Sessions within a project share a namespace; secrets and RBAC are project-scoped |
| CP token endpoint over Secret-write renewal | Secret writes are async push with no synchronization guarantee vs. token TTL; synchronous pull from CP eliminates the race entirely |
| Runner SA token for CP auth | K8s SA tokens are already mounted in every pod, long-lived, and K8s-managed — no new secrets or out-of-band key distribution required |
| CP is sole token source — no BOT_TOKEN Secret | CP creates the runner pod, so it is always reachable before the runner's first token request; retaining a Secret adds complexity and a second failure mode with the same blast radius |
system:image-builder bound to session SA at provision time |
Agents need push access to the internal image registry to build and distribute images; OpenShift grants pull automatically via system:image-pullers at namespace init but push requires an explicit RoleBinding; co-locating it with the other session SA grants keeps all RBAC provisioning in one place |