[karpenter]: rolling restart consolidation for PDB-blocked nodes

[jeremywgleeson]

Summary

Ports the rolling restart consolidation feature onto the current main API (v1/v1alpha1), replacing the previous implementation that targeted the old v1beta1/v1alpha5 API. Instead of rejecting PDB-blocked nodes outright, single-PDB-blocked candidates whose workloads have opted in via karpenter.sh/allow-rolling-restart are marked pdbBlocked=true and consolidated via rolling restart of owning workloads (Deployments/StatefulSets) instead of pod eviction. Nodes without the annotation are rejected at candidate creation time, identical to pre-rolling-restart behavior.

Key behaviors

• Typed PDBBlockReason on PodBlockEvictionError — ValidatePodsDisruptable now classifies errors as PDBBlockReasonSinglePDB, PDBBlockReasonMultiplePDB, or PDBBlockReasonOther (do-not-disrupt). isSinglePDBBlockError uses state.GetPDBBlockReason() instead of string matching.
• AllowRollingRestartAnnotationKey (karpenter.sh/allow-rolling-restart) opt-in annotation on workloads
• Early annotation check at NewCandidate: podsEligibleForRollingRestart verifies all reschedulable pods have annotated owners before marking a candidate as PDB-blocked. Without the annotation, the node is rejected identically to before — never visible to consolidation
• Dedup at computation time: collectRestartTargets checks kubectl.kubernetes.io/restartedAt within a 10-minute window before building the command
• PDB re-verification: computeRollingRestartCommand re-fetches PDB limits before issuing a rolling restart, guarding against stale pdbBlocked state
• Validation: detects PDB status changes during TTL wait to reject stale commands
• Cleanup on failure: rolling restart trigger failure in StartCommand now clears both the NoSchedule taint and the disruption reason condition on NodeClaims

Files changed

• statenode.go: PDBBlockReason enum, NewPodBlockEvictionErrorWithReason, GetPDBBlockReason; ValidatePodsDisruptable uses typed reasons
• labels.go: AllowRollingRestartAnnotationKey annotation
• types.go: pdbBlocked field on Candidate, isSinglePDBBlockError (typed check), Restarts field on Command, early annotation eligibility check in NewCandidate
• metrics.go: 4 new rolling restart metrics (triggered, skipped/dedup, candidates gauge, errors)
• rolling_restart.go: new file — podsEligibleForRollingRestart, owner resolution, opt-in check, dedup, trigger logic. Both podsEligibleForRollingRestart and collectRestartTargets use IsReschedulable consistently.
• singlenodeconsolidation.go: computeRollingRestartCommand with PDB re-verification; timeout remains at 3 minutes
• multinodeconsolidation.go: restart target collection for PDB-blocked candidates
• queue.go: trigger rolling restarts after cordoning; failure path clears disruption condition via ClearNodeClaimsCondition
• validation.go: PDB status consistency check
• suite_test.go: existing PDB tests unchanged (PDB-blocked nodes without annotation still rejected as errors)
• rolling_restart_test.go: new file — 24 tests covering candidate creation, annotation gating, consolidation, dedup, validation

Updates since last revision

Latest (gofmt fix):

• Fixed gofmt alignment in PDBBlockReason const block that was causing CI make verify to fail

Previous (review fixes):

• Reverted SingleNodeConsolidationTimeoutDuration from 60min back to 3min (20x increase was unintentional)
• Replaced brittle string matching in isSinglePDBBlockError with typed PDBBlockReason enum on PodBlockEvictionError
• Added ClearNodeClaimsCondition to rolling restart failure cleanup in queue.go (was only clearing taint)
• Changed collectRestartTargets to use IsReschedulable for consistency with podsEligibleForRollingRestart
• Documented performance tradeoff for podsEligibleForRollingRestart API calls in hot path

Previous (annotation check at candidate creation):

• Fixed bug where PDB-blocked candidates without allow-rolling-restart annotation were incorrectly included in consolidation
• Added podsEligibleForRollingRestart function to check annotation eligibility at NewCandidate time before marking as pdbBlocked
• Nodes with PDB-blocked pods but no annotated owners are now rejected identically to pre-rolling-restart behavior (never become candidates)
• Uses IsReschedulable (not IsEvictable) to properly handle DaemonSet/mirror pods — nodes with only non-reschedulable PDB-blocked pods are rejected
• Added negative tests for annotation gating in candidate creation and consolidation
• Test count: 244/244 pass (was 242, added 2 new negative tests)

Initial (port to main):

• Rebased onto current main (new API). During the port, three test failures were discovered and fixed:
  1. do-not-disrupt test: annotation must be set on Node (not just NodeClaim) since StateNode.Annotations() returns Node.Annotations when registered
  2. dedup test: annotation key corrected to kubectl.kubernetes.io/restartedAt; dedup check added at command computation time (was only at execution time)
  3. validation test: added PDB re-verification in computeRollingRestartCommand via pdb.NewLimits so deleted PDBs are detected before issuing a rolling restart
  4. lint fix: lowercased 3 error strings in rolling_restart.go to satisfy staticcheck ST1005

CI status: all presubmit checks pass (1.29.x–1.34.x). The 1.35.x failure is pre-existing infrastructure (go-licenses build error + Coveralls 530), unrelated to this PR.

Review & Testing Checklist for Human

• End-to-end cluster testing — Unit tests pass (244/244), but the rolling restart flow hasn't been tested in a real cluster. Deploy to a test cluster with PDB-protected workloads and verify: (a) workloads with the annotation restart correctly, (b) nodes without annotation are rejected at candidate creation, (c) dedup prevents cascading restarts, (d) PDB re-verification catches stale commands, (e) metrics are emitted correctly, (f) failure cleanup properly clears both taint and disruption condition.
• API call performance in hot path — podsEligibleForRollingRestart makes 2 API calls per reschedulable pod in NewCandidate for every single-PDB-blocked node. Monitor candidate creation latency in clusters with many PDB-blocked nodes. Consider adding caching or batch-fetching if this becomes a bottleneck.
• Annotation removal race — If a workload's allow-rolling-restart annotation is removed between candidate creation (podsEligibleForRollingRestart) and command computation (collectRestartTargets), the command will fail. Verify this is handled gracefully and doesn't leave nodes in a bad state.
• 3-minute timeout sufficiency — SingleNodeConsolidationTimeoutDuration is 3 minutes. In clusters with many PDB-blocked candidates, rolling restart operations (owner resolution + PDB re-verification) take longer than standard consolidation. Monitor timeout metrics to ensure 3 minutes is adequate.
• Verify typed PDBBlockReason backward compatibility — The new NewPodBlockEvictionErrorWithReason function is used in ValidatePodsDisruptable, but the old NewPodBlockEvictionError still exists and defaults to PDBBlockReasonOther. Confirm no other callers of NewPodBlockEvictionError are affected by the new PDBReason field.

Notes

All 244 disruption tests pass locally. The typed error approach eliminates the brittle string matching that was flagged in review. The IsReschedulable consistency fix ensures DaemonSet pods are properly excluded from both eligibility checks and target collection.

---

Link to Devin run: https://app.devin.ai/sessions/a822da3f61094a318e3d05a6db48aefd
Requested by: @jeremywgleeson

[coveralls]

Pull Request Test Coverage Report for Build 23100567941

Details

• 435 of 612 (71.08%) changed or added relevant lines in 15 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage decreased (-0.2%) to 80.296%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controllers/disruption/validation.go	6	9	66.67%
pkg/controllers/state/statenode.go	12	15	80.0%
pkg/controllers/disruption/multinodeconsolidation.go	15	19	78.95%
pkg/controllers/state/informer/nodeclaim.go	4	8	50.0%
pkg/controllers/controllers.go	0	5	0.0%
pkg/controllers/disruption/queue.go	5	12	41.67%
pkg/controllers/state/informer/nodepool.go	6	13	46.15%
pkg/controllers/disruption/singlenodeconsolidation.go	34	48	70.83%
pkg/controllers/state/informer/pricing.go	18	59	30.51%
pkg/state/cost/cost.go	170	212	80.19%

Files with Coverage Reduction	New Missed Lines	%
pkg/controllers/state/informer/nodepool.go	1	60.0%

Totals
Change from base Build 22327604045:	-0.2%
Covered Lines:	12311
Relevant Lines:	15332

---

💛 - Coveralls