Skip to content

Commit 91ef958

Browse files
feat(jobs): deploy_idle_scaler — scale-to-zero idle descheduler (#54) (#94)
* feat(jobs): deploy_idle_scaler — scale-to-zero idle descheduler (Task #54) Worker half of scale-to-zero. New periodic job (every 2 min) that patches idle, healthy, not-pinned deployments to replicas=0 (~$0 compute) — reversible via the api wake endpoint. Sibling to deployment_expirer (idle ≠ expired). Flag-gated behind DEPLOY_SCALE_TO_ZERO_ENABLED (default OFF): Work() short-circuits at DEBUG when off — no k8s patch, no DB write (proven by TestDeployIdleScaler_FlagOffNoOp: zero SQL issued, zero scale calls). Fail-open when k8s is unreachable (nil client → WARN per tick, other jobs unaffected). Idle SIGNAL (stated honestly): deployments.last_activity_at (api migration 068) — stamped at create, bumped on deploy/redeploy/wake. v1 idle = "no deploy/redeploy/wake for N min" (default 30, floored at 5), NOT per-HTTP traffic, because the api/worker are not in the request path and no nginx-ingress request scrape is wired yet. Follow-up noted in the job header to lift this to traffic-based idle. CAS double-guard: candidate SELECT and the scaled_to_zero UPDATE share the healthy + not-zeroed + not-always-on predicate, so a row that raced into a woken/pinned/redeployed/expired state between SELECT and UPDATE is skipped (0 rows), never wrongly slept. NotFound Deployment = skip (torn down), not fail. Metric (rule 25): instant_deploy_scaled_to_zero_total{outcome} (scaled_down | woke_up | wake_failed | scale_failed) + instant_deploy_idle_apps gauge, primed in metrics_test.go. Alert+tile+catalog ship in the infra PR. Tests: flag-off no-op, nil-k8s no-op, scale-down happy path (+counter+gauge), CAS-race skip, NotFound skip, scale-error → scale_failed, idle-minutes floor, provider_id→namespace derivation, real k8sDeployScaleClient vs fake clientset. make gate GREEN. Awaiting operator enable of DEPLOY_SCALE_TO_ZERO_ENABLED to verify real scale-down in prod. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(jobs): idle-scaler error-branch + Kind/cluster-ctor coverage (#54) Closes the 100%-patch gaps on deploy_idle_scaler.go: Kind(), the no-config cluster-constructor error path (CI-only, gated like the status client), list-query error, scan error, foreign provider_id skip, db-flip error, gauge-sample error. Work() and the SQL helpers now 100%. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(scale-to-zero): cover idle-scaler cluster-ctor + config env-parse + AddWorker wiring Closes the 6 uncovered changed lines flagged by the patch-coverage gate: - config: DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES parse branch (valid override + sub-5/non-numeric floor-to-30) now exercised directly. - deploy_idle_scaler: introduce newDeployScaleClientset package-var seam so NewK8sDeployScaleClientFromCluster's success return is testable without a reachable cluster; add a success test alongside the NoConfig error test. - workers: extract the idle-scaler k8s-client wiring (the AddWorker else-branch) into a unit-testable buildIdleScaleK8s helper; cover both the success and the fail-open (nil) branches via the seam. Flag remains default-OFF; behaviour unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 695f026 commit 91ef958

7 files changed

Lines changed: 1005 additions & 0 deletions

File tree

internal/config/config.go

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import (
55
"fmt"
66
"log/slog"
77
"os"
8+
"strconv"
89
"strings"
910
)
1011

@@ -178,6 +179,15 @@ type Config struct {
178179
FlowSyntheticTier string // FLOW_SYNTHETIC_TIER — seeded tier (default free)
179180
FlowSyntheticDisabled string // FLOW_SYNTHETIC_DISABLED — comma list of per-flow kill switches
180181
FlowSyntheticJWTSecret string // JWT_SECRET — shared with api; mints the synthetic session JWT
182+
183+
// Scale-to-zero idle-scaler (deploy_idle_scaler.go, Task #54). INERT unless
184+
// DeployScaleToZeroEnabled is true — the master flag (shared name with the
185+
// api's wake-path flag). When off, the idle-scaler sweep is a no-op (no k8s
186+
// patch, no DB write). DeployScaleToZeroIdleMinutes is the no-activity
187+
// threshold before an app is descheduled (default 30; floored at 5 to avoid
188+
// pathological flapping). Enabling is an operator action after a canary.
189+
DeployScaleToZeroEnabled bool // DEPLOY_SCALE_TO_ZERO_ENABLED — master flag (default false)
190+
DeployScaleToZeroIdleMinutes int // DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES — idle threshold (default 30)
181191
}
182192

183193
// ErrMissingConfig is returned when a required env var is absent.
@@ -294,6 +304,20 @@ func Load() *Config {
294304
FlowSyntheticTier: os.Getenv("FLOW_SYNTHETIC_TIER"),
295305
FlowSyntheticDisabled: os.Getenv("FLOW_SYNTHETIC_DISABLED"),
296306
FlowSyntheticJWTSecret: os.Getenv("JWT_SECRET"),
307+
308+
// Scale-to-zero idle-scaler (Task #54). Default OFF; idle threshold
309+
// default 30 min (parsed below).
310+
DeployScaleToZeroEnabled: os.Getenv("DEPLOY_SCALE_TO_ZERO_ENABLED") == "true",
311+
}
312+
313+
// DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES: minutes of no-activity before an app is
314+
// descheduled. Default 30; an unset / unparseable / sub-5 value floors to 30
315+
// so a misconfig can't make the scaler aggressively flap apps to sleep.
316+
cfg.DeployScaleToZeroIdleMinutes = 30
317+
if v := strings.TrimSpace(os.Getenv("DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES")); v != "" {
318+
if n, err := strconv.Atoi(v); err == nil && n >= 5 {
319+
cfg.DeployScaleToZeroIdleMinutes = n
320+
}
297321
}
298322

299323
// Fall back to the shared object-store bucket when the operator hasn't

internal/config/config_test.go

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,36 @@ func TestLoad_Defaults(t *testing.T) {
126126
}
127127
}
128128

129+
// TestLoad_DeployScaleToZeroIdleMinutes exercises the env-parse branch for
130+
// DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES: a valid value is honoured; an invalid /
131+
// sub-5 value floors to the 30-minute default.
132+
func TestLoad_DeployScaleToZeroIdleMinutes(t *testing.T) {
133+
t.Run("valid override", func(t *testing.T) {
134+
clearEnv(t)
135+
t.Setenv("DATABASE_URL", "postgres://localhost/db")
136+
t.Setenv("DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES", "45")
137+
if got := Load().DeployScaleToZeroIdleMinutes; got != 45 {
138+
t.Errorf("DeployScaleToZeroIdleMinutes = %d; want 45", got)
139+
}
140+
})
141+
t.Run("sub-5 floors to default", func(t *testing.T) {
142+
clearEnv(t)
143+
t.Setenv("DATABASE_URL", "postgres://localhost/db")
144+
t.Setenv("DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES", "3")
145+
if got := Load().DeployScaleToZeroIdleMinutes; got != 30 {
146+
t.Errorf("sub-5 DeployScaleToZeroIdleMinutes = %d; want floor 30", got)
147+
}
148+
})
149+
t.Run("non-numeric floors to default", func(t *testing.T) {
150+
clearEnv(t)
151+
t.Setenv("DATABASE_URL", "postgres://localhost/db")
152+
t.Setenv("DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES", "abc")
153+
if got := Load().DeployScaleToZeroIdleMinutes; got != 30 {
154+
t.Errorf("non-numeric DeployScaleToZeroIdleMinutes = %d; want floor 30", got)
155+
}
156+
})
157+
}
158+
129159
func TestLoad_PanicsWithoutDatabaseURL(t *testing.T) {
130160
clearEnv(t)
131161
defer func() {

0 commit comments

Comments
 (0)