Distributed Container Orchestration Platform

A declarative infrastructure control plane that accepts Deployment specs, schedules Pods onto registered Nodes, tracks Node health via heartbeats, reconciles desired vs. actual state, recovers from Node failures, and drives a health-aware reverse proxy — with Prometheus/Grafana observability across the full stack.

                        infractl / curl
                              │
                              ▼
                     ┌─────────────────┐
                     │  control-plane  │  deployments, pods, nodes, services
                     │      :7070      │  scheduler + controller loops
                     └────────┬────────┘
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
         worker-1         worker-2         worker-3   (register, heartbeat,
        (:9100 metrics)  (:9101 metrics)  (:9102 metrics)  poll, execute)
              ▲               ▲               ▲
              └───────────────┼───────────────┘
                     ┌─────────────────┐
       client  ───▶  │      proxy      │  fetches healthy nodes from the
                     │      :8080      │  control plane, routes/fails over
                     └─────────────────┘
                              │
                     ┌────────┴────────┐
                     ▼                 ▼
               Prometheus :9090   Grafana :3000

Four binaries make up the system: control-plane (the API server plus its scheduler and controller loops), worker (a node agent that registers, heartbeats, and executes pods), proxy (a health-aware reverse proxy that discovers backends dynamically from the control plane), and infractl (the CLI client). See Contents below for where each component's design is covered.

Repository Layout

.
├── go.mod                   # single Go module for the whole system
├── docker-compose.yml       # control-plane + 3 workers + proxy + Prometheus/Grafana
├── Dockerfile                # multi-stage build producing all 4 binaries
├── cmd/
│   ├── control-plane/        # API server entrypoint
│   ├── worker/                # node agent entrypoint
│   ├── proxy/                  # reverse proxy entrypoint
│   └── infractl/                # CLI entrypoint
├── pkg/
│   ├── apis/                  # domain types: Deployment, Pod, Node, Service
│   ├── api/                    # REST server: routes, handlers, middleware
│   ├── scheduler/                # assigns PENDING pods to HEALTHY nodes
│   ├── controller/                # reconciliation + node failure detection
│   ├── autoscaler/                 # replica autoscaling
│   ├── agent/                        # node agent: register/heartbeat/poll/execute
│   │   └── metrics/
│   ├── proxy/                          # reverse proxy: balancing, health checks, discovery
│   │   └── balancer/ backend/ health/ admin/ config/ metrics/
│   ├── registry/                        # state store: BoltDB or in-memory
│   ├── metrics/                          # control-plane Prometheus metrics
│   ├── logging/                           # shared structured logger
│   └── config/                             # control-plane env config
├── deployments/              # Prometheus config, proxy config, Grafana dashboard/provisioning
├── docs/images/               # benchmark charts embedded in this README
├── examples/                   # example Deployment/Service manifests
├── scripts/                     # demo and benchmark scripts
├── testing/                      # load/storage/security/chaos benchmarking suites
└── tests/integration/              # black-box integration suite

Quickstart

./scripts/run-local-cluster.sh

Builds and starts the control plane, 3 node agents, the proxy, Prometheus, and Grafana.

Control plane API: http://localhost:7070
Proxy: http://localhost:8080 (backend status: /admin/backends)
Prometheus: http://localhost:9090
Grafana (admin/admin): http://localhost:3000

Then walk through the demo scripts in scripts/:

./scripts/submit-demo-jobs.sh      # submit a 20-pod deployment, watch it schedule across nodes
./scripts/demo-worker-failure.sh   # kill a node mid-pod, watch the controller detect and reschedule
./scripts/demo-proxy-failover.sh   # kill a node, watch the proxy route around it and recover
./scripts/benchmark-scheduler.sh   # throughput: deployments submitted to all pods scheduled

Or drive it directly with infractl (go run ./cmd/infractl ...; INFRACTL_SERVER defaults to http://localhost:7070):

infractl deployment submit examples/batch-job.yaml
infractl deployment status <deployment-id>
infractl node list
infractl cluster status

Known limitation: a node that crashes and never comes back stays UNHEALTHY in the control plane's store forever — the heartbeat timeout only ever flips HEALTHY → UNHEALTHY; there's no automatic garbage-collection of permanently-dead nodes. Its pods are correctly rescheduled onto the remaining healthy nodes, so this is a bookkeeping gap, not a scheduling one. A drain-then-timeout path does fully remove a node (see Controller).

Component overview

Component	Package/binary
API Server	`cmd/control-plane` + `pkg/api`
Persistent state store	`pkg/registry` (BoltDB or in-memory)
Scheduler	`pkg/scheduler`
Controller	`pkg/controller`
Autoscaler	`pkg/autoscaler`
Node agent	`cmd/worker` + `pkg/agent`
Proxy	`cmd/proxy` + `pkg/proxy`
CLI client	`cmd/infractl`

Architecture

Pods execute as OS subprocesses by default; when a Deployment spec includes an image: field, the node agent instead uses the Docker SDK to pull the image and run it as a container.

Logical topology — data flow between components, distinct from the network/port diagram at the top of this README:

infractl (CLI client)
      |
      v
Control Plane API (:7070)      ← API Server
      |
      +--> BoltStore / MemoryStore  ← persistent state (see CTRLPLANE_DB_PATH)
      +--> Scheduler           ← assigns PENDING pods to HEALTHY nodes
      +--> Controller          ← desired-state loop, failure detection
      |
      v
Node Agents (cmd/worker)       ← a separate, independently-registering process
      |  registers, heartbeats, polls /nodes/{id}/pods/poll
      +--> subprocess execution (no image: field) or Docker container (image: set)
      |
      v
Proxy (cmd/proxy)
      |
      +--> polls GET /api/v1/proxy/backends
      +--> routes to HEALTHY nodes only (via Node.Address)

Control plane responsibilities:

Receive Deployment specs (declarative desired state: image or command, replicas, retry policy, resources, namespace, labels).
Register Nodes and track liveness via heartbeats.
Schedule Pods (one execution unit per Deployment replica) onto nodes with available capacity.
Reconcile actual pod/node state against desired state every tick: create missing pods (inheriting the Deployment's namespace and labels), cancel excess ones, detect heartbeat timeouts, reschedule orphaned pods, dead-letter pods that exhaust their retry budget.
Expose Service/backend data so the proxy can discover and health-route to the live node fleet.
Resolve Service.Selector (label-based node matching) via GET /api/v1/services/{id}/backends.

Node agent responsibilities (cmd/worker, see Node agent for the full lifecycle):

Registers with the control plane on startup (capacity, address) → gets a Node ID.
Heartbeats on a fixed interval.
Polls for pods assigned to it (SCHEDULED status), executes them as OS subprocesses or Docker containers (determined by whether the pod carries an image field), and reports status transitions back.
Drains in-flight pods on shutdown rather than dropping them.

Why node agents are a separate process from the control plane. Unlike a single-binary setup (where the worker pool is in-process goroutines), here the node agent is a standalone binary that can run on a different host, register/deregister independently, and be killed without taking the scheduler down with it — that decoupling is what makes node-failure recovery (see Controller) a meaningful thing to demonstrate.

Why the proxy is a separate binary. cmd/proxy (pkg/proxy) is built and deployed as its own process rather than folded into the control plane binary, mirroring how a routing/load-balancing layer is operationally distinct from the scheduling/reconciliation layer in production systems — it can be scaled, restarted, and failed over independently. It only depends on the control plane through one HTTP endpoint (GET /api/v1/proxy/backends, polled by pkg/proxy/backend.ConfigProvider) — the proxy decides what to do with that data (load-balancing strategy, health checks, retries) entirely on its own. See Proxy design for its internal design.

Domain model

A Deployment is a declarative template (image or command, replicas, retry policy, resources, namespace, labels). The controller creates Pods from it — one execution unit per replica — and reconciles their count against Replicas every tick. Pods are scheduled onto Nodes (registered, heartbeating agent processes) by the scheduler, which picks the least-loaded healthy node with enough capacity. A Service describes path-based routing and a label Selector that resolves to a set of Node backends, which the proxy discovers dynamically. All four resource types share the same pattern: a Status field, an allowedTransitions map, and a pure Transition validator.

Labels and namespaces

Deployment, Pod, and Service are namespaced (Namespace field, defaults to default); Nodes are cluster-scoped (no Namespace field — a node can run pods from any namespace). Pods inherit their Deployment's Namespace and Labels at creation time. List endpoints accept label-selector query params (?label=key=value), matched via exact-match Service.Selector-style matching.

Persistent state

CTRLPLANE_DB_PATH toggles the state store: unset uses an in-memory store (data lost on restart), set to a file path uses BoltDB (persists across restarts, single-node, no replication — see Architecture for the topology this sits in).

Scheduler

pkg/scheduler assigns PENDING pods to HEALTHY nodes on a fixed poll interval (CTRLPLANE_SCHEDULER_INTERVAL, default 500ms).

Policy. Each tick (Scheduler.Tick):

List PENDING pods, sorted FIFO by CreatedAt. A pod whose RunAfter is still in the future (set by the controller as a backoff gate after a failure) is skipped until that time passes.
List all nodes; placement.SelectNode filters to HEALTHY ones with enough available CPU/memory and concurrency headroom for the pod's deployment's Resources, then picks the least-loaded (lowest RunningJobs), breaking ties by earliest RegisteredAt for determinism.
On a match: assign NodeID, transition the pod to SCHEDULED, and decrement the node's Available capacity / increment RunningJobs.
On no match (ErrNoCapacity): the pod stays PENDING and is retried next tick.

Processing pods in FIFO order satisfies the spec's "FIFO" requirement; the capacity/least-loaded filter inside SelectNode satisfies "resource-aware" — one function does both, since FIFO is about pod order and resource-awareness is about node choice, and they don't conflict.

What the scheduler does NOT do. Retry/backoff logic for failed pods lives in the controller, not here (see Controller) — the scheduler's only job is "pick up anything that's PENDING right now," regardless of why it became pending (first attempt, or a controller-driven retry).

Capacity bookkeeping. Capacity is reserved at schedule time (Available -= Resources, RunningJobs++) and released at pod-completion time (POST /api/v1/nodes/{id}/pods/{pod_id}/status handler, or by the controller if a node disappears mid-pod) — never adjusted on the intermediate "now running" report, since the slot was already accounted for when the pod was dispatched.

Metrics. ctrlplane_scheduler_queue_depth (gauge, pods still pending after the tick) and ctrlplane_scheduler_latency_seconds (histogram, time from Pod.CreatedAt to being scheduled).

Controller

pkg/controller maintains desired state under failure on a fixed interval (CTRLPLANE_RECONCILE_INTERVAL, default 2s). Each tick runs two independent passes.

Pass 1: replica reconciliation. For every non-CANCELLED deployment:

A PENDING deployment is activated (ACTIVE) the first time the controller sees it.
active = count of pods whose status counts toward a replica slot (Pod.Active(): PENDING/SCHEDULED/RUNNING/RETRYING/FAILED/SUCCEEDED — only DEAD_LETTER and CANCELLED free a slot). This treats Replicas as "run this many pod instances" (batch semantics), not "keep N running forever" — see the comment on Pod.Active() for why, and for the documented gap around restart_policy: always.
If active < Replicas: create the missing pods as PENDING, copying the deployment's Command/Args and inheriting the deployment's Namespace and Labels (pod-template semantics: the Deployment is the template, Pods are instances).
If active > Replicas (scale-down): cancel the newest excess PENDING/SCHEDULED pods first. A RUNNING pod is never cancelled by scale-down — it is left to finish.
If any pod for the deployment is DEAD_LETTER, the deployment is marked DEGRADED (and back to ACTIVE once no dead-lettered pods remain) — a cheap signal that something needed manual attention.

Pass 2: node heartbeat timeout detection. For every node whose now - LastHeartbeatAt exceeds CTRLPLANE_HEARTBEAT_TIMEOUT (default 15s, ~3x the node agent's own 5s heartbeat interval):

HEALTHY → UNHEALTHY, then reschedule its pods (see below).
DRAINING → REMOVED instead — an operator-initiated decommission (infractl node drain) that has gone quiet is treated as a completed removal, not a failure.

A node recovers (UNHEALTHY → HEALTHY) the instant it heartbeats again — that direction is owned by the heartbeat HTTP handler, not the controller, since receiving a heartbeat is definitionally proof of life. Known gap: a node that crashes and never sends another heartbeat stays UNHEALTHY forever; there is no automatic garbage-collection for permanently-dead nodes (only the DRAINING → REMOVED path fully removes one).

Rescheduling a failed node's pods:

A pod that was SCHEDULED but never reached RUNNING (the node died between dispatch and pickup) is sent straight back to PENDING — it never executed, so it does not burn a retry attempt.
A pod that was actually RUNNING is pessimistically assumed lost: Attempt++, and if Attempt <= Deployment.MaxRetries, it is requeued to PENDING with an exponential-backoff RunAfter (2^attempt seconds, capped at 60s). If the retry budget is exhausted, it goes to DEAD_LETTER instead.

This two-case split was found by actually running the node-failure demo against a live cluster — an earlier version only handled the RUNNING case, leaving SCHEDULED pods permanently stuck pointing at a dead node (see TestController_OrphanedScheduledPodRequeuedWithoutBurningRetry).

Chaos-tested finding: under sustained deployment-submission load, Controller.Tick()'s sequential design (reconcile-deployments pass before heartbeat-detection pass, same goroutine) can starve node-failure detection for 60s+ — see Testing & Benchmarking for the full write-up.

Autoscaling

pkg/autoscaler adjusts Replicas for type: service deployments that opt in via an autoscale: policy, on its own fixed interval (CTRLPLANE_AUTOSCALE_INTERVAL, default 5s) — independent of the scheduler's and controller's own tick intervals.

Opting in:

autoscale:
  enabled: true
  min_replicas: 2
  max_replicas: 10
  scale_up_threshold: 5    # cluster-wide PENDING pod count that triggers scale-up
  scale_down_threshold: 0  # cluster-wide PENDING pod count at/below which scale-down is considered
  step_size: 1              # replicas added/removed per decision
  stabilization_window: 20s # cooldown between decisions for this deployment

Omitting autoscale: entirely (the default for every other example manifest) leaves Deployment.Autoscale nil, which the autoscaler treats identically to enabled: false — no behavior change for existing manifests.

Why the signal is cluster-wide PENDING pod count, not real CPU/memory. There is no per-service request-rate telemetry or real CPU/memory utilization measurement anywhere in this system — Node.Capacity/Available are static, operator-set numbers (WORKER_CPU/WORKER_MEMORY_MB), never tied to actual measured host usage (see Documented simplifications). Rather than inventing synthetic per-tenant metrics to imitate a real metrics-server, the autoscaler reuses the already-computed cluster-wide count of PENDING pods (the same number pkg/scheduler exposes as ctrlplane_scheduler_queue_depth) as a shared capacity-pressure signal. Every autoscaling-enabled deployment reacts to this one signal, gated independently by its own thresholds and cooldown.

This is a coarse, cluster-wide signal, not a precise per-tenant one — a backlog caused by an unrelated deployment's pods can trigger another deployment's scale-up. It's also honest about a hard limit: the autoscaler can widen a deployment's desired replica count, but it cannot manufacture cluster capacity — if all nodes are already saturated, more Replicas just means more pods sitting PENDING, not more pods actually running.

Decision policy (pkg/autoscaler's Tick), each tick, once per autoscaling-enabled deployment:

Skip if Type != service, Autoscale is nil/disabled, or the deployment is CANCELLED.
Skip if less than stabilization_window has passed since this deployment's last scale decision (cooldown; state is in-memory, not persisted — a control-plane restart resets it, a documented simplification).
If the cluster-wide PENDING count is >= scale_up_threshold and Replicas < MaxReplicas: increase Replicas by step_size, clamped to MaxReplicas.
Else if the count is <= scale_down_threshold and Replicas > MinReplicas: decrease Replicas by step_size, clamped to MinReplicas.

The autoscaler only ever changes Replicas; it never creates or cancels pods directly — pkg/controller's existing replica reconciliation (see Controller) reconciles actual pod count to whatever Replicas currently is, regardless of who set it.

Why batch deployments are never autoscaled. A type: batch deployment's Pod.Active() already treats a SUCCEEDED pod as permanently filling its replica slot — a PENDING backlog for a batch job means "hasn't finished being scheduled yet," not "demand increased." Scaling a batch job's replica count up would add more work-item instances to a job with a fixed intended count, which is semantically wrong. The autoscaler filters to Type == service explicitly.

Observability:

GET /api/v1/deployments/{id}/autoscaling — current policy, current_replicas, and (if the autoscaler has made a decision since it started) last_scale_at/last_scale_direction/last_scale_backlog.
Metrics: ctrlplane_autoscale_decisions_total{deployment_id,direction} (counter), ctrlplane_autoscale_desired_replicas{deployment_id} (gauge) — the only label-vectored metrics in pkg/metrics, justified since cardinality is bounded by the (small) number of autoscaling-enabled deployments, not by pods.

A caveat in the example manifest. examples/autoscaling-service.yaml uses a deliberately long sleep duration. RestartPolicy is stored on every Deployment but is not actually enforced anywhere in the controller (a separate, pre-existing, out-of-scope gap) — a short-lived sleep pod that exits successfully would SUCCEED and then permanently occupy its replica slot without being replaced, which would look like the autoscaler stalled. The long sleep avoids that confusion for demo/load-test purposes; it doesn't fix the underlying gap. See Testing & Benchmarking for how this policy behaves under real load.

Node agent

There are two distinct things named "node" in this codebase — worth disambiguating up front:

pkg/apis.Node — the control plane's server-side record of a registered node: ID, address, capacity, status, last heartbeat, labels. Pure data, lives in the state store.
pkg/agent.Agent — the client-side process (cmd/worker) that registers as a Node, heartbeats, polls for Pods, and executes them. This section is about the agent.

Lifecycle:

start node agent
  -> register with control plane (capacity, address) -> gets a Node ID
  -> heartbeat loop: POST /api/v1/nodes/{id}/heartbeat every 5s (WORKER_HEARTBEAT_INTERVAL)
  -> poll loop: GET /api/v1/nodes/{id}/pods/poll every 1s (WORKER_POLL_INTERVAL)
       -> on a pod: acquire a concurrency-semaphore slot, run it as a subprocess
          or Docker container (if pod.Image is set), report RUNNING then
          SUCCEEDED/FAILED/CANCELLED via POST .../status
  -> on SIGTERM: stop polling immediately, let in-flight pods finish
       (up to WORKER_SHUTDOWN_TIMEOUT, default 10s), then cancel stragglers

Node identity is not persisted across restarts — a node agent that is killed and restarted registers fresh and gets a brand-new ID. The control plane never reconciles "this is actually the same physical node as before"; it is simply a new entry in the store, and the old ID stays UNHEALTHY forever (see Controller's known gap).

Pod execution. pkg/agent/executor.go dispatches each pod to one of two execution paths based on whether the pod carries an image field:

Subprocess (image is empty): exec.CommandContext runs pod.Command pod.Args... as an OS subprocess. stdout/stderr are captured into a bytes.Buffer; the exit code is extracted via *exec.ExitError.
Docker container (image is set): pkg/agent/container.go uses the official Docker Go SDK (github.com/docker/docker/client) to pull the image, create and start a container, wait for it to exit, collect its logs via stdcopy.StdCopy, and then remove the container. CPU and memory limits from the pod's resources field are mapped to Docker HostConfig.NanoCPUs and HostConfig.Memory. The node agent's Docker daemon socket is configured via WORKER_DOCKER_HOST (default: DOCKER_HOST env var or Docker's default socket).

In both paths the status sequence is identical: RUNNING on start, then SUCCEEDED, FAILED, or CANCELLED on finish.

Graceful shutdown. The agent's pod-execution context (runCtx in pkg/agent/agent.go) is not derived from the process's cancellation context — if it were, an in-flight pod would be killed the instant SIGTERM arrives, defeating the point of draining. Instead, runCtx is only cancelled by the shutdown-timeout escape hatch. This was a real bug caught by TestAgent_GracefulShutdownDrainsInFlightPod during development, not a hypothetical.

Concurrency. A buffered channel (chan struct{}, capacity WORKER_MAX_CONCURRENT_JOBS) gates how many pods run at once — simpler than a full goroutine pool since each node agent self-throttles via polling rather than consuming from a shared dispatch channel.

Metrics and HTTP listener. The node agent isn't a pod-serving HTTP service — pods run as subprocesses, not as requests it handles. Its only HTTP listener (WORKER_METRICS_PORT, default 9100) exists purely for /healthz, /readyz, and /metrics — for Prometheus scraping, container healthchecks, and (in the proxy-failover demo) as the address the proxy forwards traffic to. Node agent metrics (worker_running_jobs, worker_completed_jobs_total, etc.) live in pkg/agent/metrics, a package separate from the control plane's pkg/metrics — splitting it mattered because otherwise every node's /metrics would also expose always-zero control-plane metrics, and vice versa, since both binaries would be importing the same package.

Proxy design

pkg/proxy (built as cmd/proxy) is a reverse proxy and HTTP load balancer written from scratch, with no third-party HTTP or proxy frameworks. It forwards client requests to a pool of backend servers, actively monitors backend health, fails over around broken backends, and exposes Prometheus-style metrics — the same core mechanics found in API gateways and edge proxies like NGINX, HAProxy, and Envoy, implemented in a small, readable codebase. In this system it runs in dynamic-discovery mode: it polls the control plane's GET /api/v1/proxy/backends endpoint (see Architecture) instead of reading a static backend list, so its pool of targets tracks the live, HEALTHY node fleet automatically.

Features:

Reverse proxy forwarding — rewrites and streams requests to backends, preserving path, query parameters, and headers; injects X-Forwarded-For, X-Forwarded-Host, and X-Forwarded-Proto.
Load-balancing strategies — round robin, least connections, and weighted round robin, selected via config.
Active health checking — concurrent per-backend probes on a configurable interval/path, with independent healthy/unhealthy consecutive-count thresholds to avoid flapping.
Retry and failover — idempotent requests (GET/HEAD/OPTIONS) are retried against a different healthy backend on connection failure or timeout, up to a configurable limit; non-idempotent methods are never silently retried.
Per-backend connection tracking — atomic in-flight request counters feed both the least-connections strategy and the metrics endpoint.
Graceful shutdown — stops accepting new connections on SIGINT/SIGTERM while letting in-flight requests finish, bounded by a configurable timeout.
Dynamic backend discovery — pkg/proxy/backend.ConfigProvider polls the control plane on a fixed interval and swaps the pool's backend set to match, reusing existing *Backend entries (so health/connection state survives a refresh) when a backend's URL and weight haven't changed.
Observability endpoints — /healthz, /admin/backends (live per-backend JSON status), and /metrics (Prometheus text exposition format, hand-rolled with no client library dependency).

Package layout:

cmd/proxy/          entrypoint: loads config, wires balancer + health checker + handler, serves, handles shutdown
pkg/proxy/
  config/            YAML parsing, defaults, validation
  backend/           Backend (atomic health/connection/request state), Pool, and ConfigProvider
  balancer/          Balancer interface + round robin, least connections, weighted round robin
  health/            active health checker with consecutive-threshold hysteresis
  admin/             /admin/backends JSON status handler
  metrics/           Prometheus-format counter/gauge registry, no external dependency
  (top-level)        HTTP handler: backend selection, retry/failover, header rewriting, response streaming

Request lifecycle:

Client
  │
  ▼
http.Server  (cmd/proxy)
  │  ServeHTTP
  ▼
proxy.Handler                     pkg/proxy
  │  1. balancer.Next(pool) ─────▶ pkg/proxy/balancer (round robin / least conn / weighted)
  │                                       │ reads pool.Healthy()
  │  2. build outbound request            ▼
  │     (director.go: rewrite URL,   pkg/proxy/backend (Pool, Backend atomics)
  │      inject X-Forwarded-*)            ▲
  │  3. client.Do via shared             │ IsAlive() / ActiveConnections()
  │     keep-alive Transport              │
  │  4. on failure + idempotent      pkg/proxy/health.Checker
  │     method → retry next backend       │ probes /health on interval,
  │  5. stream response, or                │ flips Backend.alive after
  │     502/503/504 if exhausted           │ N consecutive failures/successes
  ▼                                        │
Client response               pkg/proxy/metrics.Metrics ◀── counters updated
                                            │              on every request/check
                               /metrics, /admin/backends, /healthz

Design decisions:

Manual forwarding instead of httputil.ReverseProxy. The standard library's ReverseProxy forwards to a single fixed target per request; retrying against a different backend after a failed attempt needs control over exactly when response headers get committed to the client. Handler.proxyOnce (pkg/proxy/handler.go) only writes the upstream's status line and headers to the client after a response has been successfully received — a dial failure, connection refusal, or timeout happens before anything is written to the client, so the same request can be retried against another backend transparently. This is also why redirects are never followed automatically (CheckRedirect returns http.ErrUseLastResponse): a reverse proxy must hand a 3xx straight to the client, not chase it on the client's behalf.

Retry safety is method-aware. Only GET, HEAD, and OPTIONS are retried automatically. Retrying a POST against a second backend after an ambiguous failure (e.g. the first backend processed the request but the response was lost) could duplicate a side effect, so non-idempotent methods get exactly one attempt and surface the failure to the client instead.

Lock-free hot path. Backend (pkg/proxy/backend/backend.go) stores liveness, active-connection count, and request/error counters as atomic.Bool/atomic.Int64 fields rather than behind a mutex, since these are read and written on every single proxied request. Pool is rebuilt wholesale on each discovery refresh (Pool.Swap) rather than mutated in place — health checks and the balancer only ever mutate the atomic fields inside each Backend, so no locking is needed to read a consistent backend list under concurrent load.

Health is a hysteresis state machine, not a single probe. health.Checker (pkg/proxy/health/checker.go) tracks a signed consecutive-count per backend: positive runs of successes, negative runs of failures. A backend only flips from alive to dead after unhealthy_threshold consecutive failed probes, and only recovers after healthy_threshold consecutive successful ones. This absorbs single transient blips (a dropped packet, a slow GC pause) without removing a backend from rotation, while still reacting decisively to a real outage.

Metrics push, not pull, on every state change. Rather than having the /metrics handler reach into the balancer and health checker at scrape time, each component (the proxy handler, the health checker) pushes counter/gauge updates into a shared metrics.Metrics registry as events happen. Render() then just formats whatever is currently in the registry. This keeps the metrics package fully decoupled from balancing and health-check logic — it has no imports from either.

Why round robin / least connections / weighted round robin specifically. Round robin is the right default when backends are roughly homogeneous and request cost is uniform. Least connections corrects for the common case where request latency varies — it routes around a backend that's currently slow rather than blindly continuing to send it 1-in-N requests. Weighted round robin exists for heterogeneous backend capacity (e.g. a bigger instance that should take proportionally more traffic) and is implemented as a deterministic cumulative-weight walk rather than a "current weight" decay algorithm, trading a small bias under concurrent access for a simpler, easily-tested implementation.

Load-balancing strategies:

Round robin (pkg/proxy/balancer/round_robin.go) — an atomic counter cycles through the currently-healthy backend list. Simple, and fair when backends are equivalent.
Least connections (least_conn.go) — scans healthy backends and picks the one with the fewest in-flight requests, tracked via the per-backend atomic connection counter. Better than round robin when request cost is uneven.
Weighted round robin (weighted.go) — walks a cumulative-weight ring sized to the sum of healthy backend weights, so a backend with weight 3 receives ~3x the traffic of a backend with weight 1.

All three strategies only ever consider pool.Healthy(), so an unhealthy backend is structurally excluded from selection, not just deprioritized.

Health checks. Each backend is probed concurrently on health_check.interval against health_check.path. A probe counts as a success only on a 2xx response within health_check.timeout; anything else — non-2xx status, timeout, or connection error — counts as a failure. The checker tracks consecutive successes/failures per backend and only flips Backend.alive once healthy_threshold or unhealthy_threshold consecutive results are observed, which is what prevents a single slow response from yanking a backend out of rotation. Every transition is logged, and the live status of each backend is always available at /admin/backends and as the proxy_backend_health_status metric.

Retry and failover. When a request fails against the selected backend with a transport-level error (connection refused, timeout) — as opposed to the backend returning a valid HTTP response, even an error one — the proxy retries against a different backend selected fresh from the current healthy pool, up to load_balancer.max_retries additional attempts. Retries are only performed for idempotent methods. If every attempt is exhausted, the client receives 502 Bad Gateway (forwarding/connection failure), 504 Gateway Timeout (backend exceeded backend_timeout), or 503 Service Unavailable (no healthy backend was available to try in the first place).

Configuration. The proxy is configured through a YAML file (deployments/proxy.yaml in this system, override with -config):

Section	Key	Purpose
`server`	`listen_addr`, `read_timeout`, `write_timeout`, `shutdown_timeout`	HTTP server tuning and graceful-shutdown deadline
`load_balancer`	`strategy`, `max_retries`, `backend_timeout`	`round_robin` \| `least_conn` \| `weighted_round_robin`, retry budget, per-attempt timeout
`health_check`	`enabled`, `path`, `interval`, `timeout`, `unhealthy_threshold`, `healthy_threshold`	active probe behavior
`discovery`	`enabled`, `control_plane_url`, `refresh_interval`	dynamic backend discovery against the control plane
`logging`	`level`, `format`	`slog` level and `json`/`text` output
`metrics`	`enabled`, `path`	toggles and locates the Prometheus-format endpoint

API endpoints:

Endpoint	Description
`/*`	Proxied traffic — forwarded to the backend pool per the configured strategy
`/healthz`	Liveness check for the proxy process itself
`/admin/backends`	JSON status (health, weight, active connections, request/error counts) for every backend
`/metrics`	Prometheus text-exposition metrics

Observability. /metrics exposes, per backend where applicable:

proxy_up, proxy_requests_total, proxy_retries_total, proxy_request_duration_seconds_{count,sum}
proxy_backend_requests_total{backend=...}, proxy_backend_errors_total{backend=...}
proxy_backend_active_connections{backend=...}, proxy_backend_health_status{backend=...}

These are emitted in standard Prometheus text format and require no prometheus/client_golang dependency — pkg/proxy/metrics formats them directly.

Failure handling:

Failure	Behavior
Backend connection refused	Retry a different healthy backend (idempotent methods only)
Backend exceeds `backend_timeout`	`504 Gateway Timeout`, retried if idempotent
No healthy backend available	`503 Service Unavailable`
All retries exhausted	`502 Bad Gateway`
Health probe fails `unhealthy_threshold` times	Backend marked unhealthy, excluded from selection
Health probe succeeds `healthy_threshold` times	Backend restored to rotation
`SIGINT`/`SIGTERM` received	Stop accepting new connections, drain in-flight requests, exit

Testing:

Unit tests (pkg/proxy/.../*_test.go) cover backend selection for all three strategies, health-state transitions under consecutive thresholds, config parsing/defaults/validation, and metrics rendering.
Integration tests (tests/integration/proxy_integration_test.go) run the real proxy.Handler and health.Checker against httptest backends — no mocks — and verify traffic distribution, automatic backend removal/recovery, and graceful shutdown end-to-end.
Benchmarks (pkg/proxy/benchmark_test.go) measure proxy handler overhead in isolation.

Security considerations. This component focuses on proxy and load-balancing mechanics, not production hardening. Deploying it beyond a demo would additionally need: TLS termination, request size limits, rate limiting, IP allow/deny lists, header sanitization, authentication on /admin/*, and slowloris-style abuse protection.

Non-goals. This is not a replacement for NGINX, HAProxy, Envoy, or Traefik. It does not implement HTTP/2 or HTTP/3 proxying, service-mesh features, or ingress controller integration — the goal is to make the core mechanics of a load balancer legible in a small codebase, not to compete with mature production systems.

Local development

go build ./...
go vet ./...
go test ./... -race

All four binaries live in one Go module (go.mod at the repo root), so these commands work directly — no per-component build steps needed.

Testing & Benchmarking

This isn't a smoke test — it's five suites built to put real pressure on the system (sustained load, a hard-killed node mid-traffic-surge, an adversarial scan of the container image) and then explain why each number came out the way it did, not just report it. Numbers below are from a single full run of testing/gcp/run-all.sh on a fresh GCP e2-standard-4 (4 vCPU / 16GB) Compute Engine instance, Ubuntu 24.04 LTS, with all five suites run back-to-back against the same VM so the numbers are comparable to each other. Raw output lives in each suite's own gitignored results/ directory when you run it yourself — see testing/README.md for prerequisites and override variables. Charts are generated by testing/plots/ (./testing/plots/generate-all.sh).

TL;DR

Load: 101,320 requests at up to 500 concurrent VUs, 0% failures, p95 900ms. The admission path holds up.
Autoscaling: reacted to load in 14.1s; the climb from 2→10 replicas after that was paced by the policy's own cooldown, not by lag in the system — see below.
Storage: persistence costs ~2.6ms p50 / ~7.5ms p95 versus an in-memory store, for zero throughput cost at this load — a reasonable price for surviving a restart.
Security: 0 OS-level vulnerabilities in the hardened image; one disclosed, unresolved gap (a transitive dependency's CVEs) rather than a clean bill of health papered over.
Chaos — the headline finding: under sustained submission load, node failure detection can be starved for 60s+ by the reconciler's own per-tick cost. Found, root-caused by reading the actual controller code, and written up below rather than quietly re-run until it passed.

Load test & autoscaling

Setup: k6 ramped POST /api/v1/deployments from 10 to 500 VUs against examples/autoscaling-service.yaml (2 min ramp-up, 3 min hold, 1 min ramp-down; min_replicas: 2, max_replicas: 10, step_size: 1, stabilization_window: 20s).

101,320 requests, 0% failure rate, p95 request duration 900ms
Autoscaler reacted in 14.1s to a backlog of 11 pending pods, then climbed Replicas 2 → 10 (its configured ceiling), reaching max at ~154.5s

The top panel is the signal (cluster-wide pending-pod count); the bottom is the response (Replicas). Two things are worth separating here, because they have different causes:

Detection was fast — 14.1s from load starting to the first scale decision. The autoscaler only evaluates on a 5s tick (CTRLPLANE_AUTOSCALE_INTERVAL), so 14.1s is roughly 2-3 ticks: one for the backlog to build past scale_up_threshold: 5, then the confirming tick that actually fires the decision — not an instant reaction, but fast for a coarse, cluster-wide polling signal.
The climb to max was rate-limited on purpose. Going from 2 to 10 replicas at step_size: 1 needs 8 individual scale-up decisions. The first fires at 14.1s; every decision after that is gated by the 20s stabilization_window cooldown, so the earliest the 8th (final) decision can fire is 14.1s + 7 × 20s = 154.1s. Observed: 154.5s — a match close enough to confirm the cooldown, not detection lag or the reconciler, is what paced the climb. That's the intended behavior of a cooldown: it exists to stop a policy from thrashing on a noisy signal, at the cost of a slower response to a real spike. A more aggressive step_size or shorter window trades that safety margin for faster scale-out.

The backlog keeps growing after replicas hit 10 (up to ~3,800 pending pods by the end of the run) because 10 replicas of a 0.1-CPU pod genuinely can't absorb a 500-VU submission rate — that's max_replicas doing its job as a ceiling on a 3-worker demo cluster, not the autoscaler failing to keep up. See Autoscaling for why pending-pod count is the signal in the first place, instead of per-service CPU/memory (this system has no real telemetry for either).

Storage: fio benchmark

Setup: docker compose --profile tools run --rm fio against the same named volume (ctrlplane-data) the control plane's BoltDB file lives on, direct=1 (O_DIRECT, bypasses the page cache) so results reflect the actual block device, not cache-hit speed.

job	IOPS	bandwidth	mean latency
seq-write	72	73.3 MB/s	14.0ms
seq-read	81	83.2 MB/s	12.3ms
rand-write (4k, iodepth32)	306	1.2 MB/s	104.7ms
rand-read (4k, iodepth32)	161	0.6 MB/s	199.3ms
mixed 70/30	129r / 54w	0.5 / 0.2 MB/s	89ms / 84ms

These are Compute Engine's default persistent-disk numbers, not a tuned SSD — useful as a baseline for "is the disk the bottleneck," not a hardware recommendation. One number worth flagging rather than reading at face value: random writes outran random reads (306 IOPS vs. 161 IOPS, roughly half the latency), which looks backwards next to raw local SSD behavior. This is a known characteristic of network-attached block storage (GCE Persistent Disk, AWS EBS, etc.), not a quirk of this system: a write is acknowledged once it's durably accepted into the storage backend's write path, while a read has to actually fetch the data — so on synthetic microbenchmarks against network block storage, writes routinely look faster than reads. It's an artifact of the storage layer being benchmarked, not a claim about this application's I/O pattern, and it's the kind of thing worth naming explicitly rather than letting a chart imply "writes are cheaper than reads" without qualification.

Storage: BoltDB vs in-memory

Setup: substitutes for a storage-class comparison (there's only one real persistent backend here): the same k6 load test run twice — once against the default persistent BoltDB-backed stack, once with CTRLPLANE_DB_PATH unset (falls back to the in-memory store) via testing/storage/docker-compose.memory-override.yml.

metric	bolt (persistent)	in-memory	diff
p50 latency	3.5ms	0.9ms	memory 2.6ms faster
p95 latency	9.4ms	1.9ms	memory 7.5ms faster
throughput	60.35 req/s	60.69 req/s	negligible

The gap is explained directly by what each backend does per write: pkg/registry/bolt_store.go opens the database with BoltDB's default NoSync: false, so every db.Update is a serialized, fsync'd B+tree transaction (Bolt allows one read-write transaction at a time, by design) — versus the in-memory store's sync.Map, which never touches disk at all. That combination (fsync + single-writer serialization) is exactly what buys crash-durability, and exactly what shows up as the extra few milliseconds of tail latency. At this request rate the store was never the throughput bottleneck for either backend — the two req/s numbers are within noise of each other — so the honest read is "persistence has a latency cost, not a capacity cost," at least at this load level.

Security: Trivy

This system has no real kube-apiserver/etcd/kubelet for a tool like kube-bench (which checks the CIS Kubernetes Benchmark against those specific processes' config) to meaningfully target. Trivy does legitimately apply — the Docker image and Go dependencies are real — so it's used instead, alongside a manual review of what a Docker/container-hygiene checklist actually looks like for this project. Run it yourself: ./testing/security/scan-image.sh (see testing/README.md).

Finding	Severity	Before	After	Fix applied
Container ran as root	—	No `USER` directive; confirmed via `docker run ... whoami` → `root`	Non-root `USER ctrlplane` (uid 100); confirmed via the same check → `ctrlplane`	`Dockerfile`
Base image not pinned	—	`FROM alpine:3.20` (tag, mutable)	`FROM alpine:3.20@sha256:d9e853e...` (digest, immutable)	`Dockerfile`
No authentication on the control-plane API	—	Every `/api/v1/*` endpoint open to anyone who could reach port 7070	Opt-in bearer-token auth (`CTRLPLANE_API_TOKEN`); verified unauthenticated calls get `401`, correct token gets `200`	`pkg/api/middleware.go`
No authentication on the proxy's admin endpoint	—	`/admin/backends` open to anyone who could reach port 8080	Opt-in `PROXY_ADMIN_TOKEN`, verified the same way	`cmd/proxy/main.go`
No TLS anywhere	—	Plain HTTP only	Opt-in TLS (`CTRLPLANE_TLS_CERT_FILE`/`KEY_FILE`); verified HTTPS succeeds and plain HTTP is correctly rejected on the same port	`pkg/api/server.go`
Hardcoded Grafana admin password	—	`GF_SECURITY_ADMIN_PASSWORD=admin` committed in `docker-compose.yml`	`.env`-driven (`.env.example` checked in, `.env` gitignored)	`docker-compose.yml`
Dockerfile misconfigurations (Trivy `dockerfile` scanner)	LOW	Not scanned pre-hardening (no tooling existed yet)	26/27 checks pass. One remaining: `DS-0026` — no `HEALTHCHECK` instruction	Not yet fixed — see below
`github.com/docker/docker` dependency (used by `pkg/agent` for container-mode pod execution)	HIGH (×3), MEDIUM (×2)	Same vulnerabilities present (this predates hardening — a dependency issue, not a Dockerfile one)	Still present — `v28.5.2+incompatible`, fixed in `29.3.1`. CVE-2026-34040 (auth bypass, fixed upstream), CVE-2026-41567/CVE-2026-42306 (arbitrary code execution via malicious image, no fix yet), CVE-2026-33997 (plugin privilege bypass, fixed), CVE-2026-41568 (DoS via race in `docker cp`)	Not yet fixed — see below
Secrets accidentally committed	—	Not scanned pre-hardening	None found (Trivy `secret` scanner, clean)	N/A

"Before" rows without a Trivy citation are described from this project's own git history and the live verification already done in the security-hardening commit (fd3fe90) — not re-derived from a second scan, since the change and its correctness were already directly tested (e.g. docker run --entrypoint sh ... whoami before/after). The vulnerability and Dockerfile-misconfig rows are direct Trivy output, captured via ./testing/security/scan-image.sh on a GCP VM (see testing/gcp/) building both control-plane:hardened (current Dockerfile) and control-plane:pre-hardening (the Dockerfile as of fd3fe90~1, git history) against the identical current source tree, so the diff isolates "did hardening help" from "did the app code change since."

Two things are worth being upfront about rather than stopping at the clean numbers. First, the "before" isn't a strawman — it's the state of this repo's own Dockerfile immediately before the hardening commit, scanned with the same tool on the same VM, so the diff isolates what hardening actually changed. Second, hardening didn't fix everything: both the hardened and pre-hardening images still carry 3 HIGH CVEs from github.com/docker/docker, a transitive Go dependency pkg/agent uses for container-mode pod execution. That's a dependency-version problem, not a Dockerfile problem — no amount of USER/pinning/.env changes touches it. A security pass that only reports the things it fixed isn't trustworthy; this one also reports the thing it didn't.

What's still open:

github.com/docker/docker HIGH-severity CVEs. This is a real, unresolved gap — bumping the module (go get github.com/docker/docker@v29.3.1 or later, then go mod tidy and re-verifying pkg/agent/container.go still builds against the new SDK surface) is out of scope for this pass but should be the next thing addressed.
No HEALTHCHECK in the Dockerfile. Low severity, easy fix, not yet applied — docker-compose.yml's own health-check-via-curl-loop in the demo scripts covers the same need operationally today, so this wasn't urgent, but it's a legitimate one-line addition (HEALTHCHECK CMD curl -f http://localhost:$PORT/healthz || exit 1 — needs per-binary handling since one image serves four different roles with four different ports/health semantics).
RBAC, admission webhooks, CRDs: still not implemented (see Documented simplifications) — this project intentionally doesn't attempt to replicate that surface.

Chaos: fault injection during load

Setup: testing/chaos/chaos-load-test.sh hard-kills a node (docker compose kill worker-2) 100s into the same 500-VU k6 ramp used above, then polls for the control plane to mark it UNHEALTHY.

Deployment-submission availability during the fault window: 100% (2,951/2,951 requests succeeded)
UNHEALTHY detection did not fire within 60s (nominal idle-cluster detection time is ~17-20s, confirmed at ~12s via a no-load baseline)

The 100% availability number is not the interesting result — it just reflects that POST /api/v1/deployments accepts submissions regardless of node health, so it was never at risk of dropping in this test. The real finding is the missed detection, and it was chased down to an actual line of code rather than left as "sometimes this test flakes": Controller.Tick() (see Controller) runs reconcileDeployments() — an O(deployments × pods) scan via ListPodsByDeployment, called once per deployment — before detectUnhealthyNodes(), sequentially, on a single ticker goroutine. Under the sustained submission backlog this load test generates, the reconciliation pass grows expensive enough per tick to starve failure detection of its turn entirely.

That explanation was earned, not assumed: BoltDB write contention, VU count, and stale database state were each tested and ruled out individually before landing on tick sequencing as the cause (see the comments in chaos-load-test.sh for the elimination chain, and the no-load baseline above, which detects the same fault in ~12s with zero reconciler backlog to compete with).

This is exactly the scenario that matters most in a real incident — node failure coinciding with a traffic surge — so it's reported here as a legitimate architectural finding from this testing effort, not quietly patched by loosening a timeout until the test passed. The natural fix is splitting node-health detection onto its own ticker, decoupled from deployment reconciliation, so a large backlog can no longer delay it; that change is scoped but intentionally left undone here, since fixing it inside a testing pass would bury a real finding instead of reporting it.

Reproducing these numbers

./testing/gcp/run-all.sh          # create VM, run all 5 suites, collect results, destroy VM
./testing/plots/generate-all.sh   # regenerate the charts above from the pulled results

Or run any suite locally against ./scripts/run-local-cluster.sh — see testing/README.md for each suite's script and override variables.

Documented simplifications

These are intentional design decisions for the scope of this project, not bugs:

Production orchestrators	This project
Distributed consensus store (e.g. etcd with Raft, gRPC)	BoltDB: embedded single-file ACID store, no network
Watch/informer event streams from the state store	Node agent polls on a timer (500 ms); controller ticks on a timer (2 s)
Services select Pods by label via overlay networking	Services select Nodes by label (Pods have no network address; nodes are the routable entities)
Full container runtime (containerd, runc, CNI networking)	Docker SDK — pods run as Docker containers when `image:` is set, or OS subprocesses otherwise
RBAC, admission webhooks, CRDs	Not implemented
Metrics-server-backed autoscaling on real CPU/memory utilization	`pkg/autoscaler` — cluster-wide PENDING-pod-count proxy signal instead (see Autoscaling)
Dedicated load-balancing layer as a separate, mature project	`pkg/proxy` — a from-scratch reverse proxy, not a replacement for NGINX/HAProxy/Envoy/Traefik

References

Go Workspaces — a mechanism this project previously used to tie multiple modules together; the system is now a single module
Prometheus Go client — instrumentation library used by the control plane and node agent
Prometheus — metrics collection and alerting
Grafana — metrics visualization and dashboards
Docker Compose — local multi-service orchestration for the demo stack
Controller / reconcile-loop pattern — the desired-vs-actual-state design that inspired pkg/controller
go.etcd.io/bbolt — embedded single-file ACID key-value store used as the persistent state backend
gopkg.in/yaml.v3 — YAML parsing used for declarative deployment spec files and proxy config
testify — assertion library used across the test suite
Docker Go SDK — official Docker client used by the node agent to pull images and manage container lifecycle when image: is set in a Deployment spec

License

This project is licensed under the MIT License. See LICENSE for details.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Distributed Container Orchestration Platform

Contents

Repository Layout

Quickstart

Component overview

Architecture

Domain model

Labels and namespaces

Persistent state

Scheduler

Controller

Autoscaling

Node agent

Proxy design

Local development

Testing & Benchmarking

Load test & autoscaling

Storage: fio benchmark

Storage: BoltDB vs in-memory

Security: Trivy

Chaos: fault injection during load

Reproducing these numbers

Documented simplifications

References

License

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 46 Commits
cmd		cmd
deployments		deployments
docs/images		docs/images
examples		examples
pkg		pkg
scripts		scripts
testing		testing
tests/integration		tests/integration
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
go.mod		go.mod
go.sum		go.sum

Folders and files

Latest commit

History

Repository files navigation

Distributed Container Orchestration Platform

Contents

Repository Layout

Quickstart

Component overview

Architecture

Domain model

Labels and namespaces

Persistent state

Scheduler

Controller

Autoscaling

Node agent

Proxy design

Local development

Testing & Benchmarking

Load test & autoscaling

Storage: fio benchmark

Storage: BoltDB vs in-memory

Security: Trivy

Chaos: fault injection during load

Reproducing these numbers

Documented simplifications

References

License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages