scheduler-kubernetes.md

title	Scheduler — Kubernetes
description	Hybrid scheduler backend: file-backed on developer laptops, Kubernetes CronJobs in production.
order	10

Scheduler — Kubernetes

Forge's scheduler picks one of two backends at startup:

Backend	When used	Persistence	Timing
file	Outside a Kubernetes pod	<WorkDir>/.forge/memory/SCHEDULES.md	30s in-process goroutine ticker
kubernetes	Inside a Kubernetes pod (with scheduler.backend: auto or kubernetes in forge.yaml)	K8s CronJob resources (etcd)	Cluster's CronJob controller

The Kubernetes backend solves three operational problems the file backend has in container deploys: no PVC needed (etcd is durable), horizontally safe (CronJob controller is cluster-singleton), and visible to standard kubectl get cronjobs tooling.

Backend selection

forge.yaml:

scheduler:
  backend: auto              # auto (default) | file | kubernetes
  kubernetes:
    namespace: ""            # defaults to the agent pod's own namespace at runtime
    service_url: ""          # in-cluster URL CronJob trigger pods POST to; auto-derived to `http://<agent_id>.<namespace>.svc:<port>/` when empty (mirrors the build-time default, issue #179)
    allow_dynamic: false     # whether schedule_set (LLM-driven) can create CronJobs at runtime
    trigger_image: ""        # container image the trigger Job runs; default: curlimages/curl:8.10.1
    auth_secret_name: ""     # K8s Secret holding the internal token; default: <agent_id>-internal-token

Resolution at startup:

1. scheduler.backend: file → file backend, always
2. scheduler.backend: kubernetes → kubernetes backend, errors at startup when not in-cluster (unless FORGE_IN_CLUSTER=true is set for testing)
3. scheduler.backend: auto (default) → kubernetes when the projected ServiceAccount token at /var/run/secrets/kubernetes.io/serviceaccount/token exists, file otherwise

The escape hatch FORGE_IN_CLUSTER=true|false overrides the file-presence check — useful for forcing file behavior in a single-replica dev pod, or for running the K8s backend's unit tests on a developer laptop.

service_url defaulting

When scheduler.kubernetes.service_url is empty, the runtime derives http://<agent_id>.<namespace>.svc:<port>/ (matching the in-cluster Service DNS that forge package stamps into the generated CronJob YAML at build time — see forge-cli/build/schedule_manifest_stage.go). <port> is the agent's A2A listen port (--port or forge.yaml default 8080). Operators only need to set service_url explicitly when the agent sits behind an Ingress / Gateway or uses a non-standard hostname. Pinned by TestKubernetesBackend_ServiceURLDefaultDerivation (issue #179).

CronJob manifest shape

Forge generates one CronJob per schedule. The shape is fixed (no Helm templates) so kubectl diff and kubectl apply -k work without runtime substitution:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: forge-aibuilderdemo-daily-summary
  namespace: default
  labels:
    forge.agent.id: aibuilderdemo
    forge.schedule.id: daily-summary
    forge.schedule.source: yaml          # or "llm" for LLM-set schedules
spec:
  schedule: "0 9 * * *"
  concurrencyPolicy: Forbid              # K8s-native equivalent of the file backend's overlap skip
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: trigger
              image: curlimages/curl:8.10.1
              env:
                - name: FORGE_AUTH_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: aibuilderdemo-internal-token
                      key: token
              command: ["sh", "-c"]
              args:
                - |
                  curl -sfX POST http://aibuilderdemo.default.svc:8080/ \
                    -H "Authorization: Bearer $FORGE_AUTH_TOKEN" \
                    -H "X-Forge-Schedule-Id: daily-summary" \
                    -H "Content-Type: application/json" \
                    --data '{"jsonrpc":"2.0",...,"id":"sched-daily-summary-'$(date +%s)'",...}'

concurrencyPolicy: Forbid is the K8s-native equivalent of the file backend's overlap check — same semantic, enforced by the cluster.

The CronJob resource name is deterministic: forge-<agent_id>-<schedule_id>, sanitized for K8s naming rules, hash-suffixed when the natural name exceeds the 63-character limit so distinct schedules sharing a prefix don't collide after truncation.

Token plumbing

CronJob trigger Pods authenticate to the agent's A2A endpoint using the same internal bearer token channel adapters use (runner.go:ResolveAuth). The Secret containing the token is not generated by forge package — see the security model below.

# Bootstrap the Secret from the local runtime.token (or mint one):
forge auth mint-token
forge auth secret-yaml | kubectl apply -f -

When the CronJob fires, the trigger container reads FORGE_AUTH_TOKEN from the mounted Secret, sends it as Authorization: Bearer <token>, and the agent's existing static_token auth provider validates it. The auth_verify audit event lands with Source: "internal" identical to a channel callback — no new auth code path, no surprise.

Security model

• The Secret holding the token is never checked in. forge package (part 3) emits a credential-less Secret template; operators populate it out-of-band via forge auth secret-yaml, ExternalSecrets / Sealed Secrets / SOPS / Vault Agent Injector, or kubectl create secret. Applying a Deployment without first populating the Secret leaves the agent pod NotReady with a clear secret "..." not found event.

• allow_dynamic: false is the default. Static schedules from forge.yaml are materialized by forge package at build time. LLM-driven schedule_set calls do not create new CronJobs unless this flag is flipped on, which requires the agent's ServiceAccount to have create/patch/delete RBAC on batch/cronjobs in its namespace — a meaningful privilege escalation worth gating.

• Same-namespace only. CronJobs run in the agent pod's own namespace. Cross-namespace deploys are out of scope.

RBAC

The agent pod's ServiceAccount needs read access to its own CronJobs in any K8s-backend deploy. CRUD verbs are only required when allow_dynamic: true:

- apiGroups: ["batch"]
  resources: ["cronjobs"]
  verbs:
    - get      # always
    - list     # always — powers schedule_list
    - create   # only when allow_dynamic: true
    - update   # only when allow_dynamic: true
    - delete   # only when allow_dynamic: true OR a yaml schedule was removed

forge package (part 3) emits a Role + RoleBinding scoped to the agent's namespace with the minimum verbs based on allow_dynamic.

Annotations on Forge-owned CronJobs

Beyond the labels above, the runtime KubernetesBackend stamps Forge-specific fields as annotations so they round-trip through kubectl get cronjob -o yaml and back into the schedule_list tool:

Annotation	Source
forge.schedule.task	natural-language task description
forge.schedule.skill	optional skill name
forge.schedule.channel	optional channel adapter
forge.schedule.channel_target	optional channel destination ID
forge.schedule.run_count	execution counter (LLM-set schedules only)
forge.schedule.last_status	last execution outcome

LastRun is read from CronJob.Status.LastScheduleTime — operators don't need to write it.

What's NOT in the K8s backend

• schedule_history: returns empty + logs once. The audit stream's schedule_fire / schedule_complete events are the canonical source of truth.
• Cross-namespace deploys: first cut assumes CronJob and agent live in the same namespace.
• Token auto-rotation: the internal token is long-lived. Operators rotate by re-deploying with a fresh token in the Secret + pod restart picking it up.

Local fallback

Outside a cluster — forge run on a laptop, CI, a non-k8s VM — the file backend resolves automatically. The 30s in-process ticker + <WorkDir>/.forge/memory/SCHEDULES.md continue to work byte-identically to pre-#162.

See also

• forge auth — internal-token operator UX (issue #162 part 1)
• Scheduling — declarative forge.yaml schedules[] syntax
• Audit Logging — schedule_fire / schedule_complete events