Ambient Model: Implementation Guide

Date: 2026-03-20 Status: Living Document — updated continuously as the workflow is executed and improved Spec: ambient-model.spec.md — all Kinds, fields, relationships, API surface, CLI commands

This Document Is Iterative

This document is updated as the workflow runs. Each time the workflow is invoked, start from the top, follow the steps, and update this document with what you learned — what worked, what broke, what the step actually requires in practice.

The goal is convergence, not perfection on the first run. Expect failures. Expect missing steps. Expect that the workflow itself needs fixing. When something breaks, fix this document and the relevant development context before moving on. Lessons learned are not archived in separate run log files — they are corrected in the spec, this guide, and the component context files. Git history is the audit trail.

We start from the top each time. We update as we go. We run it until it Just Works™.

Re-reading this guide at the start of each run is valuable precisely because it may have been improved by the previous run.

Overview

This document describes a reusable autonomous workflow for implementing changes to the Ambient platform. The workflow is spec-driven: the data model doc is the source of truth, and agents reconcile code status against it, plan implementation work, and execute in parallel across components.

Each invocation starts from Step 1 and works through the steps in order. Steps are updated to reflect reality as it is discovered. The workflow does not require a clean-slate implementation — it is designed to run repeatedly until code and spec converge.

The Pipeline

Changes flow downstream in a fixed dependency order:

Spec (ambient-model.spec.md)
  └─► API (openapi.yaml)
        └─► SDK Generator
              └─► Go SDK (types, builders, clients)
                    ├─► BE  (REST handlers, DAOs, migrations)
                    ├─► CLI (commands, output formatters)
                    ├─► CP  (gRPC middleware, interceptors)
                    ├─► Operator (CRD reconcilers, Job spawning)
                    ├─► Runners (Python SDK calls, gRPC push)
                    └─► FE  (TypeScript API layer, UI components)

Each stage depends on the stage above it being settled. Agents must not implement downstream work against an unstable upstream.

Workflow Steps

Each invocation: start from Step 1. Update this document before moving to the next step if anything is wrong or missing.

Step 1 — Acknowledge Iteration

Before doing anything else, internalize that this run may not succeed. The workflow is the product. If a step fails, edit this document to capture the failure and what the step actually requires, then retry.

Checklist:

Read this document top to bottom
Read the spec (ambient-model.spec.md) header to check the Last Updated date
Confirm you are working on the correct branch and project
Verify the target cluster: kubectl config current-context and confirm the namespace (kind cluster: podman ps | grep kind; UAT/staging: check kubectl get ns)

Step 2 — Read the Spec

Read specs/api/ambient-model.spec.md in full.

Extract and hold in working memory:

All entities and their fields
All relationships
All API routes
CLI table (✅ implemented / 🔲 planned)
Design decisions
Session start context assembly order

This is the desired state. Everything else is measured against it.

Step 3 — Assess What Has Changed

Compare the spec against the current state of the code. For each component, ask:

Component	What to check
API	Does `openapi/openapi.yaml` have all spec entities, routes, and fields? Read the actual fragments. For every Kind, check the schema `required[]` array against the spec ERD — field-level, not just route-level.
SDK	Do generated types/builders/clients exist for all spec entities?
BE	Read `plugins/<kind>/model.go` for every Kind. Compare field-by-field against the Spec. Drift here is the most common source of gaps.
CP	Does middleware handle new RBAC scopes and auth requirements?
CLI	Does `acpctl` implement every route marked ✅ in the spec CLI table?
Operator	Do CRD reconcilers handle Agent-scoped session start?
Runners	Does the runner drain inbox at session start and push correct event types?
FE	Do API service layer, queries, and components exist for all new entities?

The gap table must compare Spec against every component simultaneously, field by field. A field removal touches API, SDK, BE (model + migration), and CLI — all four must be in the gap table from the start. Do not discover mid-wave that the CLI still has a flag the API no longer accepts.

Check all three directions for every Kind:

Spec ERD → model.go — spec says field exists; is it in the model?
model.go → Spec ERD — model has a field; is it documented in the spec?
OpenAPI required[] → Spec ERD — OpenAPI marks it required; does the spec agree?

Produce a gap table:

ENTITY          COMPONENT   STATUS      GAP
Agent           API         missing     no routes in openapi.yaml
Agent           SDK         missing     no generated type
Agent           BE          missing     no DAO, no handler, no migration
Inbox           BE          missing     no DAO, no handler
Inbox           CLI         missing     no acpctl commands
Session.prompt  BE          present     —

The gap table is the implementation backlog. The Lessons Learned section at the bottom of this guide captures the accumulated state from previous runs — start from there.

Step 4 — Break It Into Work by Agent

Decompose the gap table into per-agent work items, sequenced by pipeline order:

Wave 1 — Spec consensus (no code; human approval)

Confirm gap table is complete and agreed upon
Freeze spec for this run

Wave 2 — API (gates everything downstream)

Update openapi/openapi.yaml for all new entities and routes
Register routes in routes.go
Add handler stubs (501 Not Implemented) to complete the surface
Security gate: new routes use environments.JWTMiddleware; no user token logged; RBAC scopes documented in openapi
Implementation detail: see .claude/context/api-server-development.md
Acceptance: make test passes, make binary succeeds, make lint clean

Wave 3 — SDK (gates BE, CLI, FE)

Run SDK generator: cd components/ambient-sdk && make generate-sdk
The generator reads components/ambient-api-server/openapi/openapi.yaml and produces Go, Python, and TypeScript clients
Commit generated types, builders, client methods
Verify TS and Python client paths for any nested resource — the generator uses the first path segment as base path; nested resources require hand-written extension files
For top-level resources (like Application at /applications), the generator handles paths correctly — no extension files needed for standard CRUD
For sub-resource actions (/sync, /refresh, /status), hand-written extension methods are required in go-sdk/client/<kind>_extensions.go

Build commands:

cd components/ambient-sdk
make build-generator     # compile generator binary
make generate-sdk        # generate all three SDKs
make verify-sdk          # generate + verify Go/Python/TS compile

Acceptance: make verify-sdk passes (Go go build ./..., Python import check, TS npm run build)

Wave 4 — BE + CP (parallel after Wave 3)

BE: migrations, DAOs, service logic, gRPC presenters
CP: runner fan-out compatibility verified (see CP ↔ Runner Compatibility Contract)
Security gate: all handler paths check user token via service layer; no token values in logs; input validated before DB write
Implementation detail: BE → .claude/context/backend-development.md; CP → .claude/context/control-plane-development.md
Acceptance: make test passes, go vet ./... && golangci-lint run clean

Wave 5 — CLI + Operator + Runners (parallel after Wave 3 + BE)

CLI: implement all 🔲 commands that are now unblocked — see .claude/context/cli-development.md
Operator: CRD reconciler updates for Agent start — see .claude/context/operator-development.md
Runners: inbox drain at session start, correct event types — see .claude/context/control-plane-development.md
Security gate (Operator): all Job pods set SecurityContext with AllowPrivilegeEscalation: false, capabilities dropped; OwnerReferences set on all child resources
Acceptance: CLI make test passes; Operator go vet ./... && golangci-lint run clean; Runner python -m pytest tests/ passes; all tested in kind cluster

Wave 6 — FE (after Wave 4 BE)

API service layer and React Query hooks for new entities
UI components: Agent list, Inbox, Project Home
Implementation detail: see .claude/context/frontend-development.md
Security gate: no tokens or credentials in frontend state or logs; all API errors surface structured messages, not raw server responses
Acceptance: npm run build — 0 errors, 0 warnings

Wave 7 — Integration

End-to-end smoke: start Agent session → watch session stream → send message → confirm response
make test and make lint across all components
Final step: push new image to kind cluster (see Mandatory Image Push Playbook)

Each wave is a gate.

Step 5 — Verify the Work

After each wave is complete:

Re-run the gap table (Step 3) for that component only
If gaps remain, return to Step 4 for that wave
If clean, mark that wave item as complete and proceed to the next wave

When all waves are complete and the gap table is empty, the workflow run is done. Update the Lessons Learned section of this guide with any new rules or corrections discovered during the run.

Invocation

Current

One session working through the full pipeline:

Human reads the spec and produces the gap table (Step 3)
Human works wave by wave through the pipeline, executing code changes and verifying each wave before proceeding
Each wave's component guide is read before implementation begins

Future

A standing Overlord agent monitors for spec changes and automatically invokes the workflow — one session per wave, gating downstream waves on upstream completion.

CP ↔ Runner Compatibility Contract

The Control Plane (CP) is a fan-out multiplexer — it sits between the api-server and runner pods. Multiple clients can watch the same session; the runner pushes once. CP must preserve these invariants on every change:

Concern	Runner expects	CP must preserve
Session start	Job pod scheduled by operator	CP does not reschedule
Event emission	Runner pushes AG-UI events via gRPC	CP forwards in order, never drops
`RUN_FINISHED`	Emitted once at end	CP forwards exactly once — not duplicated
`MESSAGES_SNAPSHOT`	Emitted periodically	CP forwards in order
Token	Runner receives token from K8s secret	CP does not touch runner token
Non-JWT tokens	test-user-token has no username claim	CP skips ownership check when JWT username absent

Runner compat test (run before any CP PR):

acpctl create session --project my-project --name test-cp "echo hello"
acpctl session messages -f --project my-project test-cp

Expected: RUN_STARTED → TEXT_MESSAGE_CONTENT (tokens) → RUN_FINISHED. No connection errors, no dropped events, no duplicate RUN_FINISHED.

Runner Pod Addressing

The api-server does not have a built-in proxy to runner pods. Runner pods are addressed by Kubernetes cluster-internal DNS, constructed at request time from the session's stored fields:

http://session-{KubeCrName}.{KubeNamespace}.svc.cluster.local:8001

The Session model stores KubeCrName and KubeNamespace — both are available from the DB. The runner listens on port 8001 (set via AGUI_PORT env var by the operator; default in runner code is 8000 but the operator overrides it).

This pattern is used by components/backend/websocket/agui_proxy.go (the V1 backend) and by plugins/sessions/handler.go in the ambient-api-server. The sessions plugin implements proxyToRunner(w, r, url) which copies method, headers, body, and response verbatim. All workspace, files, git, repos/status, and AGUI sub-resource endpoints use this pattern. When the runner is unavailable, handlers return a stub (empty body) or 503 Service Unavailable.

Implementing `GET /sessions/{id}/events` (Runner SSE Proxy)

This endpoint proxies the runner pod's GET /events/{thread_id} SSE stream through to the client. The runner's /events/{thread_id} endpoint:

Registers an asyncio queue into bridge._active_streams[thread_id]
Streams every AG-UI event as SSE until RUN_FINISHED / RUN_ERROR or client disconnect
Sends 30s keepalive pings (: ping) to hold the connection
Cleans up the queue on exit regardless of how it ends

thread_id in the runner corresponds to the session ID (the value stored in Session.KubeCrName).

Implementation steps for BE agent (Wave 4):

Look up session from DB by id path param — get KubeCrName and KubeNamespace
Construct runner URL: http://session-{KubeCrName}.{KubeNamespace}.svc.cluster.local:8001/events/{KubeCrName}
Open an HTTP GET to that URL with Accept: text/event-stream
Set response headers: Content-Type: text/event-stream, Cache-Control: no-cache, Connection: keep-alive, X-Accel-Buffering: no
Stream the runner's SSE body directly to the client response writer, flushing after each \n\n boundary
On runner disconnect or RUN_FINISHED / RUN_ERROR event, close the client stream

Key differences from /sessions/{id}/messages:

	`/messages`	`/events`
Source	api-server DB + gRPC fan-out	Runner pod SSE (live only)
Persistence	Persisted; supports replay from `seq=0`	Ephemeral; runner-local in-memory queue
Reconnect	Resume via `Last-Event-ID` / `after_seq`	No replay; live only
Keepalive	30s ticker `: ping`	30s ticker from runner; proxy must pass through

SSE proxy pattern (follow plugins/sessions/message_handler.go:streamMessages for SSE writer setup):

func (h *eventsHandler) StreamRunnerEvents(w http.ResponseWriter, r *http.Request) {
    id := mux.Vars(r)["id"]
    session, err := h.sessionSvc.Get(r.Context(), id)
    if err != nil {
        // 404
        return
    }
    runnerURL := fmt.Sprintf("http://session-%s.%s.svc.cluster.local:8001/events/%s",
        *session.KubeCrName, *session.KubeNamespace, *session.KubeCrName)

    req, _ := http.NewRequestWithContext(r.Context(), http.MethodGet, runnerURL, nil)
    req.Header.Set("Accept", "text/event-stream")

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        // 502
        return
    }
    defer resp.Body.Close()

    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")
    w.Header().Set("X-Accel-Buffering", "no")
    w.WriteHeader(http.StatusOK)
    if f, ok := w.(http.Flusher); ok {
        f.Flush()
    }

    io.Copy(w, resp.Body) // SSE frames pass through verbatim
    if f, ok := w.(http.Flusher); ok {
        f.Flush()
    }
}

sessionsRouter.HandleFunc("/{id}/events", eventsHandler.StreamRunnerEvents).Methods(http.MethodGet)

Constraints

Pipeline order is strict: no downstream agent starts a wave until the upstream wave is merged and SDK is regenerated
One active session per wave: a session started for a wave runs to completion before the next wave begins
Spec is frozen during execution: no spec changes while a wave is in flight; queue changes for next cycle
PRs are atomic per wave per component: one PR per agent per wave; avoids merge conflicts across components
Agents stay in their lane: cross-component edits require a spec change and a new wave assignment

Code Generation

All new Kinds must use the generator and templates in components/ambient-api-server/templates/. Do not hand-write plugin boilerplate.

go run ./scripts/generator.go \
  --kind Agent \
  --fields "project_id:string:required,name:string:required,prompt:string,current_session_id:string" \
  --project ambient \
  --repo github.com/ambient-code/platform/components \
  --library github.com/openshift-online/rh-trex-ai

Templates of interest:

Template	Produces
`generate-openapi-kind.txt`	OpenAPI paths + schemas for a Kind
`generate-plugin.txt`	`plugin.go` init registration
`generate-dao.txt`	DAO interface + implementation
`generate-services.txt`	Service layer
`generate-handlers.txt`	HTTP handlers
`generate-presenters.txt`	OpenAPI ↔ model converters
`generate-migration.txt`	Gormigrate migration
`generate-mock.txt`	Mock DAO for tests

The SDK generator (components/ambient-sdk/) consumes the updated openapi.yaml — run it after any openapi change.

SDK Generator Pitfalls

These issues have caused failures in past runs. Check for them after every generator invocation.

Nested resource base path (TS + Python): The generator uses the first path segment of a resource's routes as the base path for generated client methods. For resources nested under /projects/{id}/agents/..., the generator emits /projects as the base path for all methods — wrong. Fix: write hand-crafted extension files that override the generated class.

Go SDK: go-sdk/client/agent_extensions.go — non-CRUD methods (start, sessions, inbox)
TS SDK: ts-sdk/src/project_agent_api.ts, ts-sdk/src/inbox_message_api.ts — complete rewrites
After fixing the generator, delete the extension files and re-verify

Auxiliary DTO schemas in sub-spec files: Each openapi.*.yaml sub-spec file must have exactly one primary resource schema. That schema must use allOf. Auxiliary DTOs (request/response bodies, view models) that don't end in List, PatchRequest, or StatusPatchRequest must live in openapi.yaml main components — not in sub-spec files. The generator picks schemas alphabetically; if the first candidate lacks allOf, the entire parse fails.

Generator directory naming: Generator creates directory named {kindLowerPlural}. For InboxMessage this is inboxMessages, not inbox. Copy and rename manually when the desired package name differs.

Generated handler variable names: mux.Vars key must match the route variable name. Nested routes use {msg_id}, not {id} — generated handlers always use {id}. Fix after generation.

Generated middleware import: RegisterRoutes callback must use environments.JWTMiddleware, not auth.JWTMiddleware. Generated code always gets this wrong. Fix after generation.

Generated integration tests for nested routes: Integration tests generated by the code generator reference flat openapi client methods that don't exist when routes are nested. Stub with t.Skip and mark for future update.

Known Code Invariants

These rules were established by fixing bugs found in production. Violating them causes panics, security holes, or incorrect behavior. Apply them when reading or writing any handler/presenter/SDK code.

gRPC presenter completeness: grpc_presenter.go sessionToProto() must map every field that exists in both the DB model and proto message. Missing fields cause downstream consumers (CP, operator) to receive zero values silently.

Inbox handler scoping: inbox/handler.go List must always inject agent_id = 'X' from URL pa_id into the TSL search filter. Never return cross-agent data. listArgs.Search is string, not *string — use empty-string checks, not nil checks.

Inbox handler agent_id enforcement: inbox/handler.go Create must set inboxMessage.AgentId = mux.Vars(r)["pa_id"] from the URL, ignoring any agent_id in the request body. Prevents body spoofing.

Inbox presenter nil safety: Nil-guard UpdatedAt independently from CreatedAt. They can be nil independently; treating them as a pair causes panics.

InboxMessagePatchRequest scope: Only Read *bool is permitted. No other fields. Prevents privilege escalation via PATCH.

StartResponse field name: ignition_prompt is the canonical field name across openapi, Go SDK, TS SDK. Not ignition_context.

Start HTTP status: Start returns HTTP 201 on new session creation, HTTP 200 when session already active. SDK doMultiStatus() accepts both.

Nested resource URL encoding: All nested resource URLs must use encodeURIComponent (TS) / url.PathEscape (Go) on every path segment.

proto field addition: Edit .proto → make proto → verify *.pb.go regenerated → wire through presenter. Do not edit *.pb.go directly.

Mandatory Image Push Playbook

This sequence is required after every wave or bug-fix batch. Do not mark a wave complete until the rollout succeeds.

# 0. Find the running cluster name
CLUSTER=$(podman ps --format '{{.Names}}' | grep 'kind' | grep 'control-plane' | sed 's/-control-plane//')
echo "Cluster: $CLUSTER"

# 1. Build without cache (cache misses source changes when go.mod/go.sum unchanged)
podman build --no-cache -t localhost/vteam_api_server:latest components/ambient-api-server

# 2. Load into kind via ctr import (kind load docker-image fails with podman localhost/ prefix)
podman save localhost/vteam_api_server:latest | \
  podman exec -i ${CLUSTER}-control-plane ctr --namespace=k8s.io images import -

# 3. Restart and verify
kubectl rollout restart deployment/ambient-api-server -n ambient-code
kubectl rollout status deployment/ambient-api-server -n ambient-code --timeout=60s

Why kind load docker-image fails with podman: It calls docker inspect internally and cannot resolve localhost/ prefix images. The podman save | ctr import approach bypasses kind's image loader and writes directly to containerd's k8s.io namespace inside the control-plane container.

Why --no-cache is required: The Dockerfile copies source in layers. If go.mod/go.sum are unchanged, the go build step hits cache and emits the old binary.

gRPC Local Port-Forward

The Go SDK derives the gRPC address from the REST base URL hostname + port 9000. When pointing at http://127.0.0.1:8000, it derives 127.0.0.1:9000. If local port 9000 is occupied (e.g. minio), gRPC streaming fails.

Fix for local development:

kubectl port-forward svc/ambient-api-server 19000:9000 -n ambient-code &
export AMBIENT_GRPC_URL=127.0.0.1:19000

The TUI's PortForwardEntry for gRPC maps to local port 19000 — use this consistently.

Long-term: add grpc_url to pkg/config/config.go so it can be set once via acpctl config set grpc_url 127.0.0.1:19000.

Component Development Guides

Each wave maps to one or more component development guides. Read the guide for that component before implementing — it contains file locations, code patterns, pitfalls, build commands, and acceptance criteria.

Wave	Component	Development Guide
Wave 2	API Server	`.claude/context/api-server-development.md`
Wave 3	SDK	`.claude/context/sdk-development.md`
Wave 4	Control Plane	`.claude/context/control-plane-development.md`
Wave 5	CLI	`.claude/context/cli-development.md`
Wave 5	Operator	`.claude/context/operator-development.md`
Wave 5	Runner	`.claude/context/control-plane-development.md` (Runner ↔ CP contract section)
Wave 6	Frontend	`.claude/context/frontend-development.md`

The old Gin/K8s backend (components/backend/) is covered by .claude/context/backend-development.md — only relevant if you are modifying the V1 backend.

Artifacts

Artifact	Location	Owner
Spec	`specs/api/ambient-model.spec.md`	Human / consensus
This workflow	`workflows/sessions/ambient-model.workflow.md`	Updated each run
OpenAPI spec	`components/ambient-api-server/openapi/openapi.yaml`	API wave
Generated SDK	`components/ambient-sdk/go-sdk/`	SDK wave
Wave PRs	GitHub, tagged by wave and component	Per wave

Lessons Learned (Run Log — 2026-03-22)

The Spec Can Lag the Code

The spec CLI table had every Agent and Inbox command marked 🔲 planned. The code had 456-line agent/cmd.go and 301-line inbox/cmd.go — fully implemented. Always verify with wc -l and go build before assuming a gap is real. The spec table is maintained manually; the code moves faster.

Fix applied: Added the Implementation Coverage Matrix to the end of the spec as the authoritative cross-component index. Update it whenever code ships, not after the next review cycle.

SDK Extension Methods Must Be Symmetric

The apply command needed GetInProject, ListInboxInProject, and SendInboxInProject on ProjectAgentAPI — methods the generator would never emit for a nested resource. These had to be hand-written in agent_extensions.go. The pattern: any method that uses a nested URL (/projects/{p}/agents/{a}/...) must live in an extensions file, not in generated code.

Rule: When adding a new nested operation to the CLI or a new command that calls an API endpoint, check agent_extensions.go (or the relevant *_extensions.go) first. If the method isn't there, add it before writing the CLI command.

CLI `events.go` Should Use SDK, Not Raw HTTP

The first implementation of acpctl session events used net/http directly, bypassing the SDK client (no auth header construction from config, no X-Ambient-Project header). This was a shortcut taken because SessionAPI.StreamEvents didn't exist yet.

Fix applied: Added StreamEvents(ctx, sessionID) (io.ReadCloser, error) to session_messages.go. Refactored events.go to use connection.NewClientFromConfig() + client.Sessions().StreamEvents(). Now the auth and project headers are handled consistently with all other SDK calls.

Rule: Never bypass the SDK client in CLI commands. If a method is missing, add it to the SDK first, then write the CLI command against it.

`StreamEvents` Cannot Use `do()` — It Must Return the Body

The SDK's do() and doMultiStatus() methods unmarshal the response body into a typed result and close the connection. For SSE streams, you need the body open and streaming. StreamEvents uses a.client.httpClient.Do(req) directly and returns resp.Body as io.ReadCloser. The caller closes it.

This means StreamEvents needs access to a.client.baseURL, a.client.token, and a.client.httpClient — all unexported fields. Since session_messages.go is in the same client package, this works without accessors.

Rule: SSE / streaming endpoints require a separate implementation pattern. Do not try to fit them into do(). Return io.ReadCloser from the SDK; let the CLI layer scan it with bufio.Scanner.

`gopkg.in/yaml.v3` Was Missing From CLI `go.mod`

The apply command imported yaml.v3 but the CLI go.mod didn't declare it. The build failure message (missing go.sum entry) was clear but required running go get gopkg.in/yaml.v3 to resolve. The dependency was already transitively available (via the SDK), but Go modules require explicit declaration for direct imports.

Rule: When adding a new file to the CLI that imports a new package, run go build ./... immediately. Fix go.mod before committing.

Generic Proxy Is a Pre-Auth Middleware, Not a Route

plugins/proxy/plugin.go forwards all non-/api/ambient/ requests to BACKEND_URL (default http://localhost:8080). It must use pkgserver.RegisterPreAuthMiddleware — the plugin's RegisterRoutes callback only receives the /api/ambient/v1 subrouter and cannot intercept paths outside that prefix. Pre-auth middleware wraps the entire HTTP server before gorilla mux routing, so it sees every path.

Rule: Any endpoint that lives outside /api/ambient/v1/ (e.g. /health, /api/projects/..., /api/auth/...) must be handled via RegisterPreAuthMiddleware. It cannot be registered as a route in a plugin's RegisterRoutes.

Spec Coverage Matrix Is the Right Indexing Artifact

The gap between what the spec said (🔲 everywhere for agents/inbox) and what the code had (full implementations) was only discoverable by reading actual source files. An implementation coverage matrix embedded in the spec — with direct references to SDK method names and CLI commands — turns the spec into a live index that can be scanned in seconds.

Rule: The coverage matrix in ambient-model.spec.md is the primary index. Update it immediately when a component ships a feature. Do not rely on the CLI table alone — it maps REST→CLI but doesn't tell you what the SDK exposes.

Lessons Learned (Run Log — 2026-05-09)

Step 3 Gap Analysis Must Go Field-Level, Not Just Route-Level

The first pass of Step 3 during this run checked only for route existence in openapi.yaml and handler stubs. It found no gaps. The real gaps were at the field level:

Agent model had 22 fields in model.go but the spec ERD documented only 10.
ScheduledSession had 4 runtime fields (timeout, inactivity_timeout, stop_on_run_finished, runner_type) not in the spec.
Session.llm_max_tokens, timeout, sdk_restart_count were int in the spec but int32 in the model.
triggered_by_user_id existed only in openapi.sessions.yaml — not in model.go and not in the spec. It was an OpenAPI schema artifact from an earlier design that was never implemented.

Fix applied to Step 3: The gap table instructions now explicitly say: "Read plugins/<kind>/model.go for every Kind. Compare field-by-field against the Spec. Drift here is the most common source of gaps."

Rule: The field-level diff of model.go is the highest-signal check in Step 3. Route existence is a necessary but not sufficient check. An API can have all routes defined and still have significant undocumented fields, wrong types, or phantom fields in the OpenAPI that don't exist in the model.

Spec and OpenAPI Can Diverge From Each Other Too

triggered_by_user_id was present in openapi.sessions.yaml but absent from both the Go model and the spec. The spec → code comparison caught the spec gap, but the OpenAPI → model comparison caught a different gap: the OpenAPI had a field the model never had.

Rule: The gap analysis must check all three directions:

Spec → model (spec says field exists; does the model have it?)
Model → spec (model has field; is it documented in the spec?)
OpenAPI → model (OpenAPI declares field; is it in the model?)

`display_name` Was Removed From Project — Proto Lags Behind

The display_name field was removed from plugins/projects/model.go, openapi.projects.yaml, grpc_handler.go, and grpc_presenter.go. A drop-column migration (202505090001) was added. However, proto/ambient/v1/projects.proto still declares display_name because the buf tool is not installed in the development environment and proto regeneration was deferred.

Consequence: The proto wire format still carries display_name as an optional field, but Go code no longer populates it. Per proto3 semantics this is safe — the field transmits as zero value (empty string) and is silently ignored by consumers. It is not a protocol break.

Rule: Before removing any field from the Go model, check proto/ambient/v1/*.proto. If the field appears in proto, the removal requires buf generate to regenerate *.pb.go. Do not edit *.pb.go manually. If buf is unavailable, document the proto lag explicitly in the commit message and as a follow-up task.

Follow-up task: Remove display_name from proto/ambient/v1/projects.proto and regenerate when buf is available.

Agent Fields Propagated Through ignite_handler, Not start_handler

The Agent model has 8 fields (repo_url, workflow_id, llm_model, llm_temperature, llm_max_tokens, bot_account_name, resource_overrides, environment_variables) that are copied to Session when an agent is ignited. This happens in plugins/agents/ignite_handler.go, not in start_handler.go. start_handler.go does minimal field copying.

Rule: When auditing field propagation from Agent → Session, read ignite_handler.go specifically. The two handlers have different responsibilities: start_handler.go handles the idempotency check and session status; ignite_handler.go handles the full field copy and session initialization.

Credential Scoping — Migrated to Global (2026-05-12)

Credentials were previously project-scoped at /api/ambient/v1/projects/{id}/credentials with a required project_id field on the model. The spec targets global resources at /api/ambient/v1/credentials.

Migration applied in one wave touching 7 files:

openapi.credentials.yaml — paths changed to /credentials; project_id removed from schema
model.go — removed ProjectID string field
migration.go — added dropProjectIDMigration() with ALTER TABLE IF EXISTS ... DROP COLUMN IF EXISTS project_id; kept prior addProjectIDMigration() so existing DBs apply and then immediately undo
handler.go — removed all projectID := mux.Vars(r)["id"] guards and project-filter injection in List
presenter.go — removed ProjectId from PresentCredential; updated ConvertCredential signature
plugin.go — changed subrouter from /projects prefix to /credentials; registered dropProjectIDMigration()
factory_test.go — removed ProjectID: "test-project" from factory struct

SDK impact: Generator produced /credentials base path correctly. credential_extensions.go used a.basePath() which no longer exists for top-level resources (generator inlines paths). Fixed by replacing with literal /credentials/{id}/token.

Rule: When the generator is used for top-level (non-nested) resources, it does NOT generate a basePath() method — it inlines paths. Any hand-written extension file that calls a.basePath() must be updated to use the literal path after changing a resource from nested to global.

Factory Test Files Must Be Checked When Removing Model Fields

Removing a field from a model struct causes go vet to fail on any test factory that references the field as a struct literal key. plugins/projectSettings/factory_test.go had DisplayName: stringPtr("Test Project") that was not found by the initial grep targeting only the projects/ plugin directory.

Rule: When removing a field from plugins/{kind}/model.go, grep the entire plugins/ tree (not just the plugin directory) for the field name. Test factories in other plugins that create the model as a dependency will reference it too.

Step 3 Must Produce a Field-Level Diff, Not Just a Route-Level Diff

During the credentials globalization run, Step 3 found the route /projects/{id}/credentials in openapi.yaml and marked it ✅. It did not compare the OpenAPI schema's required array (["project_id", "name", "provider"]) against the spec's Credential ERD (no project_id). The implementation diverged from the spec for the entire life of the feature, undetected.

Rule: For every Kind, Step 3 must check all three directions:

Spec ERD → model.go (spec says field exists; does the model have it?)
model.go → Spec ERD (model has a field; is it in the spec ERD?)
OpenAPI required[] → Spec ERD (OpenAPI marks a field required; does the spec agree?)

A route existing in openapi.yaml is necessary but not sufficient. Field-level drift is the most common and hardest-to-catch form of divergence.

Integration Tests Must Assert the Spec-Correct Route, Not an Implementation-Convenient One

plugins/credentials/factory_test.go created credentials with ProjectID: "test-project" (wrong field) and Provider: "test-provider" (invalid enum value). These tests encoded the wrong implementation shape. Because no test ever called POST /credentials (the spec-correct global path), the divergence from spec went undetected for the entire history of the feature.

Rule: Integration tests for a resource must use the route the spec defines, not a route that happens to work in the current implementation. For a global resource (no project scope), the factory and every integration test must call the global path. If the test is written against a nested path for a resource the spec defines as global, the test is wrong — fix the test, not just the implementation.

Corollary: Factory test Provider values must use valid enum values. A factory that passes "test-provider" will compile and run even if the API rejects it — use "github" or another value from the spec's provider enum table.

Lessons Learned (Run Log — 2026-06-11)

Generator `--project` Flag Controls API Path Prefix

The scripts/generator.go --project flag is used to construct the API path prefix: /api/{project}/v1/{kind}. Using --project ambient-api-server produces paths like /api/ambient-api-server/v1/applications instead of the correct /api/ambient/v1/applications. The flag value should match the desired URL path segment, not the component directory name.

Rule: Always use --project ambient (not --project ambient-api-server) when running the generator. The workflow's Step 5 command already specifies this correctly.

Generator Output Requires `make generate` Before Build

The generator's --skip-generate flag (used when the generator itself runs) means the OpenAPI Go client types (openapi.Application, openapi.ApplicationList, openapi.ApplicationPatchRequest) are not created. After running the generator, make generate must run before go build ./... will succeed.

Rule: After scripts/generator.go runs, always run make generate in the api-server directory to regenerate pkg/api/openapi/ from the updated specs.

Generator Omits DELETE from OpenAPI Spec

The generator creates GET, LIST, POST, and PATCH endpoints in the OpenAPI spec but does NOT generate a DELETE endpoint. If the Kind needs delete support, the DELETE endpoint must be manually added to openapi.{kind}.yaml.

Rule: After running the generator, check whether the Kind requires DELETE. If so, add the delete section to the /{id}: path in the OpenAPI spec following the session/credential pattern.

Generator Misses `time` Import for `*time.Time` Fields

When the model has *time.Time fields (e.g., LastSyncedAt), the generator adds the field correctly but sometimes omits the "time" import in migration.go and adds an unused "time" import in presenter.go.

Rule: After running the generator, check imports in migration.go (needs "time" if *time.Time is used) and presenter.go (remove unused "time" import).

Test `testmain_test.go` Must Import All Plugin Dependencies

Generated testmain_test.go only imports the test helper. If the plugin's migration seeds roles or references tables from other plugins, those plugins must be imported via side-effect imports. Without them, testcontainer initialization crashes with pq: relation "roles" does not exist.

Rule: Copy the full side-effect import block from plugins/credentials/testmain_test.go into any new plugin's testmain_test.go. This ensures all migrations run in the correct order.

POST Sub-Resource Handlers Without Body Must Use `HandleGet`

handlers.Handle() expects a request body and will return 400 if the POST has no body. Sub-resource actions like /sync and /refresh that don't need a request body should use handlers.HandleGet() instead.

Rule: For POST handlers that perform an action without request body input, use handlers.HandleGet(w, r, cfg), not handlers.Handle(w, r, cfg, http.StatusOK). The session Start and Stop handlers follow this pattern.

Deployment: Migration Init Container Image Must Be Updated Separately

kubectl set image only updates the main container. The migration init container uses a separate image reference and must be updated via a JSON patch or kubectl get/apply roundtrip.

Rule: When deploying a new api-server image, update BOTH the main container (api-server) and the migration init container (migration). The migration init container runs ambient-api-server migrate and must contain the new migration code.

RBAC: `pathToAction` Must Map Sub-Resource Verbs

The RBAC middleware's pathToAction() function maps URL path suffixes to permission actions. Without entries for sync and refresh, POST requests to those sub-resources map to create instead of update.

Rule: When adding sub-resource endpoints with non-CRUD semantics, add their path suffixes to pathToAction() in pkg/rbac/middleware.go.

RBAC: `isListEndpoint` Must Include New Resource Plural Names

The RBAC middleware's isListEndpoint() function has a hardcoded list of plural resource names used to distinguish list vs singleton GET requests. New resources must be added to this list.

Rule: When adding a new resource, add its plural name to the isListEndpoint() switch case in pkg/rbac/scope.go.

UAT Deployment: `CREDENTIAL_ENCRYPTION_ALLOW_PLAINTEXT` Required

The current branch includes credential encryption checks that the production v0.2.13 image did not have. When deploying to a cluster without a configured keyring, set CREDENTIAL_ENCRYPTION_ALLOW_PLAINTEXT=true as an environment variable on the deployment.

Integration Test Method Names Follow API Path Prefix

The generated integration tests use method names derived from the API path (e.g., ApiAmbientV1ApplicationsGet). If the generator was run with the wrong --project flag, the test method names will reference non-existent methods. After fixing the OpenAPI paths, regenerate with make generate and update the test method names accordingly.

CI Runs `golangci-lint` — Must Pass Before PR Merge

The CI pipeline runs golangci-lint (v2.x) on both ambient-api-server and ambient-cli with --timeout=5m. Common issues caught:

ineffassign: Assigning to err without checking it. In integration tests where you send a malformed body and only care about the HTTP status code, use restyResp, _ := resty.R()... (blank identifier) instead of restyResp, err := resty.R()....
staticcheck ST1000: Every Go package must have a package comment. New command packages (e.g., package application) need // Package application implements... above the package declaration.
unused: Package-level variables that are declared but never referenced anywhere in the package. Remove them or use blank identifier _.

Rule: Run gofmt -w . and golangci-lint run --timeout=5m in each changed Go component before pushing. The CI checks gofmt separately from golangci-lint — both must pass. The CI config uses only-new-issues: false, so pre-existing lint issues in unchanged files will also block the PR.

Uh oh!

FilesExpand file tree

ambient-model.workflow.md

Latest commit

History