Skip to content

Commit 4701353

Browse files
ianhodgeoz-agent
andauthored
Adding k8 backend to self hosted worker (#36)
Co-authored-by: Oz <oz-agent@warp.dev>
1 parent 1d32ae4 commit 4701353

23 files changed

Lines changed: 2742 additions & 23 deletions

.github/workflows/ci.yml

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,9 @@ jobs:
1111
runs-on: ubuntu-latest
1212
steps:
1313
- uses: actions/checkout@v4
14-
- uses: actions/setup-go@v4
14+
- uses: actions/setup-go@v6
1515
with:
16-
go-version: '1.25.0'
16+
go-version-file: 'go.mod'
1717
- name: Check code formatting
1818
run: |
1919
if [ -n "$(gofmt -s -l .)" ]; then
@@ -27,17 +27,38 @@ jobs:
2727
run: go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
2828
- name: Run golangci-lint
2929
run: golangci-lint run
30+
test:
31+
runs-on: ubuntu-latest
32+
steps:
33+
- uses: actions/checkout@v4
34+
- uses: actions/setup-go@v6
35+
with:
36+
go-version-file: 'go.mod'
37+
- name: Run tests
38+
run: go test ./...
3039

3140
build:
3241
runs-on: ubuntu-latest
3342
steps:
3443
- uses: actions/checkout@v4
35-
- uses: actions/setup-go@v4
44+
- uses: actions/setup-go@v6
3645
with:
37-
go-version: '1.25.0'
46+
go-version-file: 'go.mod'
3847
- name: Build
3948
run: go build -v ./...
4049

50+
helm:
51+
runs-on: ubuntu-latest
52+
steps:
53+
- uses: actions/checkout@v4
54+
- uses: azure/setup-helm@v4
55+
with:
56+
version: v4.1.3
57+
- name: Lint Helm chart
58+
run: helm lint ./charts/oz-agent-worker --set worker.workerId=ci-worker --set image.tag=ci
59+
- name: Render Helm chart
60+
run: helm template oz-agent-worker ./charts/oz-agent-worker --namespace agents --set worker.workerId=ci-worker --set image.tag=ci >/tmp/oz-agent-worker-chart.yaml
61+
4162
docker:
4263
runs-on: ubuntu-latest
4364
if: github.event_name == 'push' && github.ref == 'refs/heads/main'

.gitignore

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ go.work
1616
go.work.sum
1717

1818
# Binary
19-
oz-agent-worker
19+
/oz-agent-worker
20+
/oz-agent-worker.exe
2021

2122
# Dependency directories
2223
vendor/

Dockerfile

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,17 @@ COPY . .
1111
RUN CGO_ENABLED=0 GOOS=linux go build -o oz-agent-worker .
1212

1313
# Runtime stage
14-
FROM alpine:latest
14+
FROM alpine:3.22
1515

16-
# Install ca-certificates for HTTPS connections
17-
RUN apk --no-cache add ca-certificates
16+
# Install ca-certificates for HTTPS connections and create a non-root runtime user
17+
RUN apk --no-cache add ca-certificates \
18+
&& addgroup -S oz \
19+
&& adduser -S -D -u 10001 -G oz oz
1820

1921
WORKDIR /app
2022

2123
# Copy the binary from builder
2224
COPY --from=builder /app/oz-agent-worker .
25+
USER oz
2326

2427
ENTRYPOINT ["./oz-agent-worker"]

README.md

Lines changed: 118 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,12 @@ Self-hosted worker for Oz cloud agents.
1010

1111
## Requirements
1212

13-
- Docker daemon (accessible via socket or TCP)
1413
- Service account API key with team scope
1514
- Network egress to warp-server
15+
- One supported execution backend:
16+
- Docker daemon access for the Docker backend
17+
- Local `oz` CLI access plus a writable workspace root for the Direct backend
18+
- Kubernetes API access plus cluster credentials for the Kubernetes backend
1619

1720
## Usage
1821

@@ -28,6 +31,118 @@ docker run -v /var/run/docker.sock:/var/run/docker.sock \
2831

2932
> **Note:** Mounting the Docker socket gives the container access to the host's Docker daemon. This is required for the worker to create and manage task containers.
3033
34+
### Direct
35+
36+
The direct backend executes tasks directly on the host instead of inside Docker or Kubernetes. It requires the `oz` CLI to be available on `PATH` (or configured explicitly with `backend.direct.oz_path`) and stores per-task workspaces under `backend.direct.workspace_root` (default: `/var/lib/oz/workspaces`).
37+
38+
Example config:
39+
40+
```yaml
41+
worker_id: "my-worker"
42+
backend:
43+
direct:
44+
workspace_root: "/var/lib/oz/workspaces"
45+
oz_path: "/usr/local/bin/oz"
46+
```
47+
48+
### Kubernetes
49+
50+
The Kubernetes backend creates one Job per task. Cluster selection is controlled by the Kubernetes client config:
51+
52+
- `backend.kubernetes.kubeconfig` points to an explicit kubeconfig file
53+
- if `kubeconfig` is omitted, the worker uses in-cluster config when running inside Kubernetes
54+
- otherwise it falls back to the default kubeconfig loading rules and uses the current context
55+
56+
Example config:
57+
58+
```yaml
59+
worker_id: "my-worker"
60+
backend:
61+
kubernetes:
62+
kubeconfig: "/path/to/kubeconfig"
63+
namespace: "agents"
64+
unschedulable_timeout: "2m"
65+
pod_template:
66+
nodeSelector:
67+
kubernetes.io/os: linux
68+
containers:
69+
- name: task
70+
resources:
71+
requests:
72+
cpu: "2"
73+
memory: 4Gi
74+
```
75+
76+
Notes:
77+
78+
- `namespace` selects the namespace inside the chosen cluster; it does not choose the cluster itself, and defaults to `default` when omitted
79+
- `unschedulable_timeout` controls how long a Pod may remain unschedulable before the task is failed early; it defaults to `30s`, and `0s` disables that fail-fast behavior
80+
- `image_pull_policy` defaults to `IfNotPresent`
81+
- the Kubernetes backend requires creating Pods with a root init container to materialize sidecars into `emptyDir` volumes
82+
- the worker runs a short-lived startup preflight Job and waits for either preflight Pod creation or an early controller failure, so incompatible Pod Security or admission policy failures surface before the worker starts accepting tasks
83+
- `preflight_image` defaults to `busybox:1.36`; set it if your cluster only allows pulling startup-preflight images from an internal or allowlisted registry
84+
- `pod_template` accepts standard Kubernetes PodSpec YAML and is the declarative way to configure task pod scheduling, service accounts, image pull secrets, resources, and environment
85+
- when using `pod_template`, define a container named `task` if you want to customize the main task container directly; otherwise the worker appends its own `task` container to the PodSpec
86+
- use `valueFrom.secretKeyRef` inside `pod_template` to inject Kubernetes Secret values into task container environment variables:
87+
88+
```yaml
89+
pod_template:
90+
containers:
91+
- name: task
92+
env:
93+
- name: MY_SECRET
94+
valueFrom:
95+
secretKeyRef:
96+
name: my-k8s-secret
97+
key: secret-key
98+
```
99+
100+
101+
### Helm Chart
102+
103+
This repo includes a namespace-scoped Helm chart at `charts/oz-agent-worker`.
104+
105+
The chart deploys:
106+
107+
- a long-lived `Deployment` for `oz-agent-worker`
108+
- a namespaced `ServiceAccount`
109+
- a namespaced `Role` / `RoleBinding`
110+
- a `ConfigMap` containing the worker config
111+
- an optional `Secret` for `WARP_API_KEY` (or a reference to an existing `Secret`)
112+
113+
At runtime, the deployed worker connects outbound to Warp and creates one Kubernetes `Job` per task. The built-in Kubernetes Job controller then manages the task Pod lifecycle.
114+
115+
Recommended install flow:
116+
117+
```bash
118+
kubectl create secret generic oz-agent-worker \
119+
--from-literal=WARP_API_KEY="wk-abc123" \
120+
--namespace agents
121+
122+
helm install oz-agent-worker ./charts/oz-agent-worker \
123+
--namespace agents \
124+
--create-namespace \
125+
--set worker.workerId=my-worker \
126+
--set image.tag=v1.2.3
127+
```
128+
129+
The chart assumes the worker runs inside the target cluster and uses in-cluster Kubernetes auth by default. It does not create CRDs or cluster-scoped RBAC. Set `image.tag` explicitly for each install so the worker image is pinned instead of defaulting to `latest`.
130+
131+
The chart always deploys a single replica for a given `worker.workerId`. If you want multiple workers, deploy multiple releases with distinct worker IDs rather than scaling one release horizontally.
132+
133+
The chart defaults the long-lived worker `Deployment` to a non-root security context and conservative starting resource requests of `100m` CPU and `128Mi` memory. Tune `worker.resources` for your workload and cluster policy.
134+
135+
The Deployment includes a default `exec` liveness probe that checks the worker process is still running (`kill -0 1`). If the worker becomes unresponsive, Kubernetes will restart the pod after three consecutive failures. Override `worker.livenessProbe` in your values to use a custom probe (e.g. `httpGet` if you add a health endpoint), or set it to `null` to disable.
136+
137+
Recommended namespace-scoped permissions for the worker are:
138+
139+
- create, get, list, watch, delete `jobs`
140+
- get, list, watch `pods`
141+
- get `pods/log`
142+
- list `events`
143+
144+
The worker Deployment's `ServiceAccount` is separate from the task Job `serviceAccountName` you may set inside `backend.kubernetes.pod_template` / `kubernetesBackend.podTemplate`. The worker `Deployment` defaults to non-root, but the task namespace must still allow creating Jobs with a root init container, since sidecar materialization currently depends on that pattern. If your cluster restricts image sources for admission or policy reasons, set `kubernetesBackend.preflightImage` in the chart to an allowlisted image for the startup preflight Job, and configure task `imagePullSecrets` inside `podTemplate` when needed.
145+
31146
### Go Install
32147

33148
```bash
@@ -68,6 +183,8 @@ docker run -v /var/run/docker.sock:/var/run/docker.sock \
68183
warpdotdev/oz-agent-worker --worker-id "my-worker" -e MY_SECRET=hunter2
69184
```
70185

186+
When configuring the Kubernetes backend via YAML or Helm, declarative task-container env belongs in `backend.kubernetes.pod_template` / `kubernetesBackend.podTemplate` rather than a separate top-level Kubernetes env list. The `-e` / `--env` flags remain available as backend-agnostic runtime overrides.
187+
71188
## Docker Connectivity
72189

73190
The worker automatically discovers the Docker daemon using standard Docker client mechanisms, in this order:

charts/oz-agent-worker/Chart.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
apiVersion: v2
2+
name: oz-agent-worker
3+
description: Namespace-scoped Helm chart for deploying oz-agent-worker on Kubernetes
4+
type: application
5+
version: 0.1.0
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
{{- define "oz-agent-worker.name" -}}
2+
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
3+
{{- end -}}
4+
5+
{{- define "oz-agent-worker.fullname" -}}
6+
{{- if .Values.fullnameOverride -}}
7+
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
8+
{{- else -}}
9+
{{- $name := include "oz-agent-worker.name" . -}}
10+
{{- if contains $name .Release.Name -}}
11+
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
12+
{{- else -}}
13+
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
14+
{{- end -}}
15+
{{- end -}}
16+
{{- end -}}
17+
18+
{{- define "oz-agent-worker.chart" -}}
19+
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
20+
{{- end -}}
21+
22+
{{- define "oz-agent-worker.selectorLabels" -}}
23+
app.kubernetes.io/name: {{ include "oz-agent-worker.name" . }}
24+
app.kubernetes.io/instance: {{ .Release.Name }}
25+
{{- end -}}
26+
27+
{{- define "oz-agent-worker.labels" -}}
28+
helm.sh/chart: {{ include "oz-agent-worker.chart" . }}
29+
{{ include "oz-agent-worker.selectorLabels" . }}
30+
{{- if .Chart.AppVersion }}
31+
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
32+
{{- end }}
33+
app.kubernetes.io/managed-by: {{ .Release.Service }}
34+
{{- end -}}
35+
36+
{{- define "oz-agent-worker.serviceAccountName" -}}
37+
{{- if .Values.serviceAccount.create -}}
38+
{{- default (include "oz-agent-worker.fullname" .) .Values.serviceAccount.name -}}
39+
{{- else -}}
40+
{{- required "serviceAccount.name is required when serviceAccount.create=false" .Values.serviceAccount.name -}}
41+
{{- end -}}
42+
{{- end -}}
43+
44+
{{- define "oz-agent-worker.apiKeySecretName" -}}
45+
{{- if .Values.warp.apiKeySecret.create -}}
46+
{{- default (printf "%s-api-key" (include "oz-agent-worker.fullname" .)) .Values.warp.apiKeySecret.name -}}
47+
{{- else -}}
48+
{{- required "warp.apiKeySecret.name is required when warp.apiKeySecret.create=false" .Values.warp.apiKeySecret.name -}}
49+
{{- end -}}
50+
{{- end -}}
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: {{ include "oz-agent-worker.fullname" . }}-config
5+
labels:
6+
{{- include "oz-agent-worker.labels" . | nindent 4 }}
7+
data:
8+
config.yaml: |
9+
worker_id: {{ required "worker.workerId is required" .Values.worker.workerId | quote }}
10+
cleanup: {{ .Values.worker.cleanup }}
11+
max_concurrent_tasks: {{ .Values.worker.maxConcurrentTasks }}
12+
{{- if .Values.worker.idleOnComplete }}
13+
idle_on_complete: {{ .Values.worker.idleOnComplete | quote }}
14+
{{- end }}
15+
backend:
16+
kubernetes:
17+
namespace: {{ default .Release.Namespace .Values.kubernetesBackend.namespace | quote }}
18+
image_pull_policy: {{ .Values.kubernetesBackend.imagePullPolicy | quote }}
19+
unschedulable_timeout: {{ .Values.kubernetesBackend.unschedulableTimeout | quote }}
20+
{{- if .Values.kubernetesBackend.preflightImage }}
21+
preflight_image: {{ .Values.kubernetesBackend.preflightImage | quote }}
22+
{{- end }}
23+
{{- if .Values.kubernetesBackend.setupCommand }}
24+
setup_command: |-
25+
{{ .Values.kubernetesBackend.setupCommand | indent 10 }}
26+
{{- end }}
27+
{{- if .Values.kubernetesBackend.teardownCommand }}
28+
teardown_command: |-
29+
{{ .Values.kubernetesBackend.teardownCommand | indent 10 }}
30+
{{- end }}
31+
{{- with .Values.kubernetesBackend.extraLabels }}
32+
extra_labels:
33+
{{ toYaml . | indent 10 }}
34+
{{- end }}
35+
{{- with .Values.kubernetesBackend.extraAnnotations }}
36+
extra_annotations:
37+
{{ toYaml . | indent 10 }}
38+
{{- end }}
39+
{{- if .Values.kubernetesBackend.activeDeadlineSeconds }}
40+
active_deadline_seconds: {{ .Values.kubernetesBackend.activeDeadlineSeconds }}
41+
{{- end }}
42+
{{- if .Values.kubernetesBackend.workspaceSizeLimit }}
43+
workspace_size_limit: {{ .Values.kubernetesBackend.workspaceSizeLimit | quote }}
44+
{{- end }}
45+
{{- if .Values.kubernetesBackend.podTemplate }}
46+
pod_template:
47+
{{ toYaml .Values.kubernetesBackend.podTemplate | indent 10 }}
48+
{{- end }}

0 commit comments

Comments
 (0)