[CASCL-1386] Address post-merge review fixes for evict-legacy-nodes#3228
Draft
L3n41c wants to merge 2 commits into
Draft
[CASCL-1386] Address post-merge review fixes for evict-legacy-nodes#3228L3n41c wants to merge 2 commits into
L3n41c wants to merge 2 commits into
Conversation
Follow-up fixes from a deep review of the merged evict-legacy-nodes implementation: - Karpenter user NodePools: explicitly delete the NodeClaim backing each drained node (matched by providerID) instead of relying on Karpenter consolidation, which never reaps the empty cordoned node when the user disabled consolidation. This makes step 3 (remove the underlying instance) consistent across all four node-group types. A missing NodeClaim CRD (legacy Karpenter v0.x) falls back to consolidation. - drainNode now refuses a node hosting pods not managed by a controller (bare pods) unless the new --force flag is set, mirroring kubectl drain and avoiding silent, unrecoverable pod loss. - run.go reclaims leaked temporary PDBs before ensureTempPDBs on the eviction path (fail-fast): a leaked temp PDB plus a user-added PDB for the same workload would otherwise deadlock the drain, since the Eviction API refuses pods covered by more than one PDB. - Wire the previously-dead --debug flag (zap.UseDevMode(opts.Debug)), add the evict package doc comment, and label the eviction step 3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
🎯 Code Coverage (details) 🔗 Commit SHA: 3125c7f | Docs | Datadog PR Page | Give us feedback! |
Add unit tests for the new Karpenter NodeClaim deletion branches so the patch is covered: NodeClaim list failure (surfaced, node still drained), delete failure (surfaced), drain failure (NodeClaim left intact), and an absent NodeClaim CRD (falls back to consolidation without error). Brings evictKarpenterUserNodePool, nodeClaimsByProviderID and deleteNodeClaim to 100% statement coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Follow-up correctness and consistency fixes surfaced by a deep post-merge review of the
kubectl datadog autoscaling cluster evict-legacy-nodescommand (originally landed via #3160–#3178):evictKarpenterUserNodePooldeletes theNodeClaimbacking each drained node (matched byproviderID) instead of relying on Karpenter consolidation, which never reaps the empty cordoned node when the user has consolidation disabled. This makes step 3 (remove the underlying instance) consistent across all four node-group types (ASG terminates instances, standalone terminates instances, EKS MNG scales to 0, Karpenter deletes NodeClaims). Matching byproviderIDrather than thekarpenter.sh/nodepoollabel is deliberate:classifyNodeByLabelalso buckets Karpenter nodes bykarpenter.k8s.aws/ec2nodeclassor the legacykarpenter.sh/provisioner-name, so the target entity is not always the NodePool name. A missing NodeClaim CRD (legacy Karpenter v0.x) falls back to consolidation.drainNoderefuses a node hosting pods not managed by a controller unless the new--forceflag is set, mirroringkubectl drain. Without it such a pod would be evicted with no replacement and lost.Runnow reclaims leaked temp PDBs (fail-fast) before creating fresh ones. Otherwise a temp PDB left by an interrupted run plus a user-added PDB for the same workload would double-cover a pod, and the Eviction API refuses pods matched by more than one PDB.--debugflag is wired (zap.UseDevMode(opts.Debug)), theevictpackage gets a doc comment (all sibling packages have one), and the eviction step is labeled "Step 3".Motivation
A thorough review of the merged implementation (comparing the reference PR #3026 against what landed across the ~13 split PRs, and re-checking every review comment) turned up a handful of genuine gaps: the Karpenter type was the odd one out for actively removing capacity, bare pods were evicted unconditionally, an interrupted run could leave a state that deadlocks the next drain, and
--debugdid nothing.Additional Notes
--dry-rundoes not preview the--forcebare-pod refusal, because dry-run short-circuits before listing per-pod state (as it already does for all per-pod details). Low impact — a real run without--forcestill refuses the node safely.IsNotFoundon delete.Minimum Agent Versions
None — this is a
kubectl-datadogplugin change with no Datadog Agent or Cluster Agent version dependency.Describe your test plan
go test -race ./cmd/kubectl-datadog/autoscaling/cluster/evict/...— new subtests cover the bare-pod--forcegate (refuse without, evict with) and NodeClaim deletion (deleted after drain, dry-run leaves it, unrelated claim untouched); existing drain fixtures updated to be controller-managed so they exercise the default no---forcepath.go test ./cmd/kubectl-datadog/autoscaling/cluster/...— full command tree green (clients scheme change included).golangci-lint run— 0 issues.Checklist
bug,enhancement,refactoring,documentation,tooling, and/ordependenciesqa/skip-qalabel