Skip to content

[CASCL-1386] Address post-merge review fixes for evict-legacy-nodes#3228

Draft
L3n41c wants to merge 2 commits into
mainfrom
lenaic/CASCL-1386-evict-review-fixes
Draft

[CASCL-1386] Address post-merge review fixes for evict-legacy-nodes#3228
L3n41c wants to merge 2 commits into
mainfrom
lenaic/CASCL-1386-evict-review-fixes

Conversation

@L3n41c

@L3n41c L3n41c commented Jul 7, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Follow-up correctness and consistency fixes surfaced by a deep post-merge review of the kubectl datadog autoscaling cluster evict-legacy-nodes command (originally landed via #3160#3178):

  • Karpenter user NodePools now actively remove nodes. evictKarpenterUserNodePool deletes the NodeClaim backing each drained node (matched by providerID) instead of relying on Karpenter consolidation, which never reaps the empty cordoned node when the user has consolidation disabled. This makes step 3 (remove the underlying instance) consistent across all four node-group types (ASG terminates instances, standalone terminates instances, EKS MNG scales to 0, Karpenter deletes NodeClaims). Matching by providerID rather than the karpenter.sh/nodepool label is deliberate: classifyNodeByLabel also buckets Karpenter nodes by karpenter.k8s.aws/ec2nodeclass or the legacy karpenter.sh/provisioner-name, so the target entity is not always the NodePool name. A missing NodeClaim CRD (legacy Karpenter v0.x) falls back to consolidation.
  • Bare pods are no longer silently destroyed. drainNode refuses a node hosting pods not managed by a controller unless the new --force flag is set, mirroring kubectl drain. Without it such a pod would be evicted with no replacement and lost.
  • Leaked temporary PDBs can no longer deadlock the drain. On the eviction path, Run now reclaims leaked temp PDBs (fail-fast) before creating fresh ones. Otherwise a temp PDB left by an interrupted run plus a user-added PDB for the same workload would double-cover a pod, and the Eviction API refuses pods matched by more than one PDB.
  • Misc: the previously-dead --debug flag is wired (zap.UseDevMode(opts.Debug)), the evict package gets a doc comment (all sibling packages have one), and the eviction step is labeled "Step 3".

Motivation

A thorough review of the merged implementation (comparing the reference PR #3026 against what landed across the ~13 split PRs, and re-checking every review comment) turned up a handful of genuine gaps: the Karpenter type was the odd one out for actively removing capacity, bare pods were evicted unconditionally, an interrupted run could leave a state that deadlocks the next drain, and --debug did nothing.

Additional Notes

  • Known limitation: --dry-run does not preview the --force bare-pod refusal, because dry-run short-circuits before listing per-pod state (as it already does for all per-pod details). Low impact — a real run without --force still refuses the node safely.
  • Deliberately not changed: a drained Karpenter node with no matching NodeClaim logs a warning and continues rather than erroring. The unmatched case is either a legacy v0.x node this command cannot remove via a v1 NodeClaim anyway, or a node already being torn down (where success is correct); the reap-between-list-and-delete race is already handled by tolerating IsNotFound on delete.

Minimum Agent Versions

None — this is a kubectl-datadog plugin change with no Datadog Agent or Cluster Agent version dependency.

Describe your test plan

  • go test -race ./cmd/kubectl-datadog/autoscaling/cluster/evict/... — new subtests cover the bare-pod --force gate (refuse without, evict with) and NodeClaim deletion (deleted after drain, dry-run leaves it, unrelated claim untouched); existing drain fixtures updated to be controller-managed so they exercise the default no---force path.
  • go test ./cmd/kubectl-datadog/autoscaling/cluster/... — full command tree green (clients scheme change included).
  • golangci-lint run — 0 issues.

Checklist

  • PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
  • PR has a milestone or the qa/skip-qa label
  • All commits are signed (see: signing commits)

Follow-up fixes from a deep review of the merged evict-legacy-nodes
implementation:

- Karpenter user NodePools: explicitly delete the NodeClaim backing each
  drained node (matched by providerID) instead of relying on Karpenter
  consolidation, which never reaps the empty cordoned node when the user
  disabled consolidation. This makes step 3 (remove the underlying
  instance) consistent across all four node-group types. A missing
  NodeClaim CRD (legacy Karpenter v0.x) falls back to consolidation.
- drainNode now refuses a node hosting pods not managed by a controller
  (bare pods) unless the new --force flag is set, mirroring kubectl drain
  and avoiding silent, unrecoverable pod loss.
- run.go reclaims leaked temporary PDBs before ensureTempPDBs on the
  eviction path (fail-fast): a leaked temp PDB plus a user-added PDB for
  the same workload would otherwise deadlock the drain, since the
  Eviction API refuses pods covered by more than one PDB.
- Wire the previously-dead --debug flag (zap.UseDevMode(opts.Debug)),
  add the evict package doc comment, and label the eviction step 3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@L3n41c L3n41c added the bug Something isn't working label Jul 7, 2026
@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Jul 7, 2026

Copy link
Copy Markdown

Code Coverage

🎯 Code Coverage (details)
Patch Coverage: 80.43%
Overall Coverage: 46.35% (+0.06%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3125c7f | Docs | Datadog PR Page | Give us feedback!

Add unit tests for the new Karpenter NodeClaim deletion branches so the
patch is covered: NodeClaim list failure (surfaced, node still drained),
delete failure (surfaced), drain failure (NodeClaim left intact), and an
absent NodeClaim CRD (falls back to consolidation without error). Brings
evictKarpenterUserNodePool, nodeClaimsByProviderID and deleteNodeClaim to
100% statement coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant