openclaw substrate support (#1939)

support running openclaw on substrate.

---------

Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Co-authored-by: Eitan Yarmush <eitan.yarmush@solo.io>

Author: peterj

.github/workflows/ci.yaml

@@ -81,7 +81,7 @@ jobs:
       - name: Install agent-sandbox
         run: |
           kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/manifest.yaml"
-          kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=90s
+          timeout 90s bash -c 'until [ "$(kubectl get crd sandboxes.agents.x-k8s.io -o jsonpath="{.status.conditions[?(@.type==\"Established\")].status}" 2>/dev/null)" = "True" ]; do sleep 1; done'
           kubectl rollout status deployment/agent-sandbox-controller -n agent-sandbox-system --timeout=120s
           kubectl wait --for=condition=Ready pod -l app=agent-sandbox-controller -n agent-sandbox-system --timeout=120s
 

docs/substrate-agentharness-lifecycle.md

@@ -0,0 +1,60 @@
+# Substrate AgentHarness Lifecycle
+
+This branch should use a single ownership model for `runtime: substrate` harnesses.
+
+## Ownership
+
+- Platform/Helm owns `WorkerPool` capacity.
+- kagent owns the generated per-harness `ActorTemplate`.
+- kagent owns the per-harness actor lifecycle through `ate-api`.
+- Substrate owns the `WorkerPool` deployment and the `ActorTemplate` golden snapshot process.
+
+kagent should not create or delete `WorkerPool` resources from the `AgentHarness` reconciler. A chart may optionally install a default `WorkerPool`, and the controller may use that default when `spec.substrate.workerPoolRef` is unset.
+
+## Spec Shape
+
+`AgentHarness.spec.substrate` should contain only harness-level inputs:
+
+- `workerPoolRef`, optional; falls back to the configured controller default.
+- `snapshotsConfig`, optional; defaults to `gs://ate-snapshots/<namespace>/<name>`.
+- `workloadImage`, optional.
+- exactly one of `gatewayToken` or `gatewayTokenSecretRef`.
+
+There is no `actorTemplateRef`. kagent always generates the `ActorTemplate`, so adopting an external template is not part of the workflow.
+
+## Status
+
+Use top-level Kubernetes conditions for progress:
+
+- `Accepted`
+- `ActorTemplateReady`
+- `ActorReady`
+- `Ready`
+
+`Ready` is the aggregate condition. Specific blockers should be reflected in `reason` and `message`.
+
+Do not store ownership booleans or cleanup markers in annotations or status. Ownership is deterministic:
+
+- `WorkerPool` is external.
+- generated `ActorTemplate` is owned by the `AgentHarness` through an owner reference.
+
+## Reconcile
+
+The substrate reconcile path should:
+
+1. Resolve `workerPoolRef` from spec or controller default.
+2. Verify the `WorkerPool` exists.
+3. Create or update the generated `ActorTemplate` with an owner reference to the `AgentHarness`.
+4. Wait for `ActorTemplate.status.phase == Ready`.
+5. Create or resume the actor through `ate-api`.
+6. Mark `ActorReady` and aggregate `Ready`.
+
+## Delete
+
+The finalizer should:
+
+1. Delete the harness actor recorded in `status.backendRef.id`.
+2. Read the generated `ActorTemplate` and delete `status.goldenActorID`, if present.
+3. Remove the finalizer.
+
+Kubernetes garbage collection deletes the generated `ActorTemplate` through the owner reference. kagent does not delete `WorkerPool`.

examples/substrate-openclaw/README.md

@@ -0,0 +1,144 @@
+# OpenClaw on Agent Substrate
+
+## 1. Install Substrate on your Kind cluster
+
+You can clone the kagent fork of substrate [here](https://github.com/kagent-dev/substrate).
+
+These instructions use a Kind cluster called `kind` (`KIND_CLUSTER_NAME=kind`).
+
+```bash
+cd substrate
+
+./hack/create-kind-cluster.sh
+./hack/install-ate-kind.sh --deploy-ate-system
+```
+
+`--deploy-ate-system` installs the **control plane only** (ate-api, ate-controller, atelet, atenet, …). Your registry catalog will show `ateapi-*`, `atelet-*`, etc., but **not** ateom until you build it.
+
+Build and push **ateom-gvisor** (required for the WorkerPool `ateomImage`):
+
+```bash
+# build the ateom-gvisor image from the substrate repo root
+export KO_DOCKER_REPO=localhost:5001
+export KO_DEFAULTPLATFORMS=linux/$(go env GOARCH)
+./hack/run-tool.sh ko build -B ./cmd/ateom-gvisor
+```
+
+## kagent AgentHarness with substrate runtime
+
+kagent generates a per-harness `ActorTemplate` and uses an existing `WorkerPool`.
+
+Install kagent (Substrate must already be running in the cluster):
+
+```bash
+export KIND_CLUSTER_NAME=kind
+make helm-install KAGENT_HELM_EXTRA_ARGS="\
+  --set controller.substrate.enabled=true \
+  --set controller.substrate.ateApiEndpoint=dns:///api.ate-system.svc:443 \
+  --set controller.substrate.ateApiInsecure=true \
+  --set substrateWorkerPool.create=true \
+  --set substrateWorkerPool.ateomImage=localhost:5001/ateom-gvisor:latest"
+```
+
+The generated `ActorTemplate` uses `controller.substrate.pauseImage`, `controller.substrate.runscAMD64URL`, `controller.substrate.runscAMD64SHA256`, `controller.substrate.runscARM64URL`, and `controller.substrate.runscARM64SHA256` from the Helm values Override them with `--set` or a values file when you need to pin a different gVisor build.
+
+Create a harness. If `snapshotsConfig` is omitted, kagent defaults it to `gs://ate-snapshots/<namespace>/<agentharnessname>`.
+
+- **Worker pool** — reference an existing pool (`workerPoolRef`) or configure a controller default WorkerPool
+- **Gateway token** — required per harness with either `gatewayToken` or `gatewayTokenSecretRef`
+
+```yaml
+apiVersion: kagent.dev/v1alpha2
+kind: AgentHarness
+metadata:
+  name: peterj-claw
+  namespace: kagent
+spec:
+  runtime: substrate
+  backend: openclaw
+  description: OpenClaw on Agent Substrate
+  modelConfigRef: default-model-config
+  substrate:
+    # Optional: defaults to gs://ate-snapshots/kagent/peterj-claw
+    # snapshotsConfig:
+    #   location: gs://ate-snapshots/kagent/peterj-claw
+
+    # Required unless the controller has a default WorkerPool configured.
+    workerPoolRef:
+      name: kagent-default
+
+    # Required: configure the OpenClaw gateway token for this harness.
+    # Use either gatewayToken or gatewayTokenSecretRef. The Secret must contain key "token".
+    gatewayToken: test-token
+
+    # gatewayTokenSecretRef:
+    #   name: openclaw-gateway-token
+
+    # Optional: override the sandbox image used in the ActorTemplate (must be digest-pinned).
+    # workloadImage: ghcr.io/kagent-dev/nemoclaw/sandbox-base@sha256:d52bee415dc4c0dba7164f9eabe727574c056d4f211781f20af249707883a3b4
+```
+
+kagent creates an `ActorTemplate` that looks roughly like this:
+
+```yaml
+apiVersion: ate.dev/v1alpha1
+kind: ActorTemplate
+metadata:
+  name: peterj-claw
+  namespace: kagent
+  labels:
+    app.kubernetes.io/managed-by: kagent
+    kagent.dev/agent-harness: peterj-claw
+spec:
+  pauseImage: gcr.io/gke-release/pause@sha256:bcbd57ba5653580ec647b16d8163cdd1112df3609129b01f912a8032e48265da
+  runsc:
+    amd64:
+      url: gs://gvisor/releases/nightly/2026-05-19/x86_64/runsc
+      sha256Hash: a397be1abc2420d26bce6c70e6e2ff96c73aaaab929756c56f5e2089ea842b63
+    arm64:
+      url: gs://gvisor/releases/nightly/2026-05-19/aarch64/runsc
+      sha256Hash: 1ba2366ae2efceba166046f51a4104f9261c9cb72c6db8f5b3fe2dc57dea86b9
+  workerPoolRef:
+    name: peterj-claw-wp
+    namespace: kagent
+  snapshotsConfig:
+    location: gs://ate-snapshots/kagent/peterj-claw
+  containers:
+  - name: openclaw
+    image: ghcr.io/kagent-dev/nemoclaw/sandbox-base@sha256:d52bee415dc4c0dba7164f9eabe727574c056d4f211781f20af249707883a3b4
+    ports:
+    - containerPort: 80
+    command:
+    - /bin/sh
+    - -c
+    - |
+      # Generated by kagent:
+      # 1. writes ~/.openclaw/openclaw.json from modelConfigRef/channels/gateway token
+      # 2. configures gateway.controlUi.basePath for the kagent proxy path
+      # 3. starts `openclaw gateway run --port 80 --allow-unconfigured`
+      # 4. waits for the gateway and tails the log
+    env:
+    - name: HOME
+      value: /root
+```
+
+The generated `command` contains a base64-encoded `openclaw.json`, so the live object will be more verbose than the abbreviated example above. `pauseImage`, runsc URLs and hashes, and the default workload image come from controller/Helm configuration unless overridden on the `AgentHarness`; the gateway token comes from `spec.substrate.gatewayToken` or `gatewayTokenSecretRef`. kagent also sets `gateway.controlUi.basePath` to `/api/agentharnesses/<namespace>/<name>/gateway` so OpenClaw serves the Control UI under the same path kagent proxies.
+
+When `modelConfigRef` or `spec.channels` are set, credentials are **not** copied into the ActorTemplate or `openclaw.json` as plaintext. kagent writes `valueFrom.secretKeyRef` (or inline `value` for harness inline tokens) on the ActorTemplate container env; Substrate `ate-api` resolves those refs at actor resume. In `openclaw.json`, kagent uses OpenClaw [env SecretRefs](https://docs.openclaw.ai/gateway/secrets) (`{source:"env",provider:"default",id:"<VAR>"}`) for `models.providers.*.apiKey`, `channels.telegram.accounts.*.botToken`, and `channels.slack.accounts.*.botToken` / `appToken`. Rotate a Secret and recreate the ActorTemplate golden snapshot when keys change.
+
+With `controller.substrate.enabled=true`, the kagent Helm chart installs a namespace-scoped Role and RoleBinding so `ate-api-server` (in `ate-system` by default) can `get` Secrets and ConfigMaps referenced by generated ActorTemplates. Harnesses in other namespaces need that namespace listed in `rbac.namespaces` (or a matching RoleBinding applied manually).
+
+Port-forward the UI:
+
+```bash
+kubectl port-forward -n kagent svc/kagent-ui 8001:8080
+```
+
+Navigate to the deployed agent harness. If the OpenClaw Control UI asks for a gateway connection, use:
+
+- Gateway URL: `http://localhost:8001/api/agentharnesses/kagent/peterj-claw/gateway/`
+- Gateway token: `test-token`
+
+The gateway URL must include the trailing slash. The token is the value configured in `spec.substrate.gatewayToken`, or the Secret value referenced by `spec.substrate.gatewayTokenSecretRef`; enter it in the token/credentials field rather than relying on a `token` query parameter.
+
+kagent proxies UI traffic to the actor OpenClaw gateway through Substrate's **atenet-router** (Envoy) using the actor `Host` header (`<actor-id>.actors.resources.substrate.ate.dev`). The default router URL is `http://atenet-router.ate-system.svc:80`; override with `controller.substrate.atenetRouterURL` when needed.

go/api/config/crd/bases/kagent.dev_agentharnesses.yaml

@@ -19,6 +19,9 @@ spec:
   scope: Namespaced
   versions:
   - additionalPrinterColumns:
+    - jsonPath: .spec.runtime
+      name: Runtime
+      type: string
     - jsonPath: .spec.backend
       name: Backend
       type: string
@@ -511,6 +514,75 @@ spec:
                       type: string
                     type: array
                 type: object
+              runtime:
+                default: openshell
+                description: Runtime selects the harness provisioning stack. Defaults
+                  to openshell when unset.
+                enum:
+                - openshell
+                - substrate
+                type: string
+              substrate:
+                description: Substrate is required when runtime is substrate.
+                properties:
+                  gatewayToken:
+                    description: |-
+                      GatewayToken is the OpenClaw gateway Bearer token for this harness.
+                      Prefer gatewayTokenSecretRef for production secrets.
+                    minLength: 1
+                    type: string
+                  gatewayTokenSecretRef:
+                    description: |-
+                      GatewayTokenSecretRef references a Secret key holding the OpenClaw gateway Bearer token.
+                      The Secret must contain a "token" key.
+                    properties:
+                      apiGroup:
+                        type: string
+                      kind:
+                        type: string
+                      name:
+                        type: string
+                    required:
+                    - name
+                    type: object
+                  snapshotsConfig:
+                    description: |-
+                      SnapshotsConfig configures actor memory snapshots. Defaults to
+                      gs://ate-snapshots/<namespace>/<agentharnessname> when unset.
+                    properties:
+                      location:
+                        description: |-
+                          Location is the GCS URI prefix for golden and incremental snapshots.
+                          Example: gs://ate-snapshots/kagent/my-namespace/my-harness/
+                        pattern: ^gs://
+                        type: string
+                    required:
+                    - location
+                    type: object
+                  workerPoolRef:
+                    description: |-
+                      WorkerPoolRef references an existing ate.dev WorkerPool in the harness namespace.
+                      When unset, the controller uses its configured default WorkerPool.
+                    properties:
+                      apiGroup:
+                        type: string
+                      kind:
+                        type: string
+                      name:
+                        type: string
+                    required:
+                    - name
+                    type: object
+                  workloadImage:
+                    description: WorkloadImage overrides the default nemoclaw/openclaw
+                      sandbox image in the ActorTemplate.
+                    type: string
+                type: object
+                x-kubernetes-validations:
+                - message: Exactly one of gatewayToken or gatewayTokenSecretRef must
+                    be specified
+                  rule: (has(self.gatewayToken) && !has(self.gatewayTokenSecretRef))
+                    || (!has(self.gatewayToken) && has(self.gatewayTokenSecretRef))
             required:
             - backend
             type: object
@@ -520,6 +592,10 @@ spec:
                 || (has(c.slack) && ((self.backend == ''hermes'' && has(c.slack.hermes)
                 && !has(c.slack.openclaw)) || ((self.backend == ''openclaw'' || self.backend
                 == ''nemoclaw'') && has(c.slack.openclaw) && !has(c.slack.hermes)))))'
+            - message: spec.substrate may only be set when runtime is substrate
+              rule: '!has(self.substrate) || self.runtime == ''substrate'''
+            - message: spec.substrate is required when runtime is substrate
+              rule: self.runtime != 'substrate' || has(self.substrate)
           status:
             description: AgentHarnessStatus is the observed state of an AgentHarness.
             properties:

go/api/httpapi/substrate.go

@@ -0,0 +1,60 @@
+package httpapi
+
+// SubstrateStatusResponse aggregates Agent Substrate control-plane and Kubernetes state.
+type SubstrateStatusResponse struct {
+	// Enabled is true when the controller is configured with an ate-api endpoint.
+	Enabled bool `json:"enabled"`
+	// AteAPIError is set when ate-api list calls fail (actors/workers may be partial or empty).
+	AteAPIError string `json:"ateApiError,omitempty"`
+
+	WorkerPools    []SubstrateWorkerPoolEntry    `json:"workerPools"`
+	ActorTemplates []SubstrateActorTemplateEntry `json:"actorTemplates"`
+	Actors         []SubstrateActorEntry         `json:"actors"`
+	Workers        []SubstrateWorkerEntry        `json:"workers"`
+}
+
+// SubstrateWorkerPoolEntry is a ate.dev WorkerPool CR.
+type SubstrateWorkerPoolEntry struct {
+	Namespace  string `json:"namespace"`
+	Name       string `json:"name"`
+	Replicas   int32  `json:"replicas"`
+	AteomImage string `json:"ateomImage"`
+}
+
+// SubstrateActorTemplateEntry is a ate.dev ActorTemplate CR.
+type SubstrateActorTemplateEntry struct {
+	Namespace       string `json:"namespace"`
+	Name            string `json:"name"`
+	Phase           string `json:"phase,omitempty"`
+	GoldenActorID   string `json:"goldenActorId,omitempty"`
+	GoldenSnapshot  string `json:"goldenSnapshot,omitempty"`
+	WorkerPoolRef   string `json:"workerPoolRef,omitempty"`
+	HarnessName     string `json:"harnessName,omitempty"`
+	ManagedByKagent bool   `json:"managedByKagent"`
+}
+
+// SubstrateActorEntry is runtime state from ate-api (redis).
+type SubstrateActorEntry struct {
+	ActorID                string `json:"actorId"`
+	Status                 string `json:"status"`
+	ActorTemplateNamespace string `json:"actorTemplateNamespace,omitempty"`
+	ActorTemplateName      string `json:"actorTemplateName,omitempty"`
+	AteomPodNamespace      string `json:"ateomPodNamespace,omitempty"`
+	AteomPodName           string `json:"ateomPodName,omitempty"`
+	AteomPodIP             string `json:"ateomPodIp,omitempty"`
+	LastSnapshot           string `json:"lastSnapshot,omitempty"`
+	InProgressSnapshot     string `json:"inProgressSnapshot,omitempty"`
+	Version                int64  `json:"version,omitempty"`
+}
+
+// SubstrateWorkerEntry is a worker assignment from ate-api (redis).
+type SubstrateWorkerEntry struct {
+	WorkerNamespace string `json:"workerNamespace"`
+	WorkerPool      string `json:"workerPool"`
+	WorkerPod       string `json:"workerPod"`
+	ActorNamespace  string `json:"actorNamespace,omitempty"`
+	ActorTemplate   string `json:"actorTemplate,omitempty"`
+	ActorID         string `json:"actorId,omitempty"`
+	IP              string `json:"ip,omitempty"`
+	Version         int64  `json:"version,omitempty"`
+}

go/api/httpapi/types.go

@@ -144,6 +144,17 @@ type OpenshellAgentHarnessListEntry struct {
 	Endpoint           string                           `json:"endpoint,omitempty"`
 }
 
+// SubstrateAgentHarnessListEntry is set when runtime is substrate.
+type SubstrateAgentHarnessListEntry struct {
+	Backend        v1alpha2.AgentHarnessBackendType `json:"backend"`
+	Runtime        v1alpha2.AgentHarnessRuntime     `json:"runtime"`
+	ActorID        string                           `json:"actorId,omitempty"`
+	GatewayUIPath  string                           `json:"gatewayUIPath,omitempty"`
+	ModelConfigRef string                           `json:"modelConfigRef,omitempty"`
+	BackendRefID   string                           `json:"backendRefId,omitempty"`
+	Endpoint       string                           `json:"endpoint,omitempty"`
+}
+
 type AgentResponse struct {
 	ID    string         `json:"id"`
 	Agent *AgentResource `json:"agent"`
@@ -157,6 +168,7 @@ type AgentResponse struct {
 	Accepted              bool                            `json:"accepted"`
 	WorkloadMode          v1alpha2.WorkloadMode           `json:"workloadMode,omitempty"`
 	OpenshellAgentHarness *OpenshellAgentHarnessListEntry `json:"openshellAgentHarness,omitempty"`
+	SubstrateAgentHarness *SubstrateAgentHarnessListEntry `json:"substrateAgentHarness,omitempty"`
 }
 
 // Session types