🐛 fix: clean up orphaned ENIs before subnet deletion at teardown by damdo · Pull Request #6073 · kubernetes-sigs/cluster-api-provider-aws

damdo · 2026-06-25T09:58:32Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

After NLB deletion during cluster teardown, AWS asynchronously cleans up the NLB's cross-AZ ENIs, a process that can take up to 1 hour. CAPA proceeds immediately to security group and subnet deletion with no ENI cleanup step, causing DependencyViolation errors that block cluster deletion until the e2e test times out.

This PR adds a deleteOrphanedENIs() step in DeleteNetwork(), right before subnet deletion. It discovers ENIs in the VPC that are:

in available state (not attached to any instance or service)
referencing one of the cluster's security groups (scoped to only this cluster's ENIs)

This targets exactly the ENIs blocking SG/subnet deletion without touching other clusters' ENIs sharing the same VPC.

Which issue(s) this PR fixes:
Fixes #6067

Special notes for your reviewer:

The ENI cleanup is non-fatal: if it fails, the error is logged and subnet deletion proceeds (which will fail with DependencyViolation and trigger a requeue, as before). On the next reconcile, the ENIs that have since become available are cleaned up, unblocking both SG and subnet deletion.

No new service factory, interface, or controller wiring is needed, the implementation lives entirely in the existing network service's DeleteNetwork() flow.

Checklist:

Release note:

fix: clean up orphaned network interfaces (ENIs) before subnet deletion during cluster teardown, preventing DependencyViolation errors caused by NLB ENIs that AWS cleans up asynchronously after load balancer deletion.

damdo · 2026-06-29T15:07:10Z

/test pull-cluster-api-provider-aws-e2e

damdo · 2026-06-29T17:03:42Z

/test pull-cluster-api-provider-aws-e2e

damdo · 2026-06-29T20:16:55Z

/test pull-cluster-api-provider-aws-e2e

damdo · 2026-06-29T22:09:44Z

/test pull-cluster-api-provider-aws-e2e

After NLB deletion, AWS asynchronously cleans up the NLB's cross-AZ ENIs, which can take up to 1 hour. CAPA proceeds immediately to SG/subnet deletion with no ENI cleanup step, causing DependencyViolation errors that block cluster deletion until the e2e test times out. Add a deleteOrphanedENIs() step in DeleteNetwork(), before subnet deletion. It discovers ENIs in the VPC that are in "available" state and reference the cluster's security groups, then deletes them. This targets exactly the ENIs blocking SG/subnet deletion without touching other clusters' ENIs. Signed-off-by: Damiano Donati <damiano.donati@gmail.com>

damdo · 2026-06-30T09:30:48Z

full e2es passed 5 times in a row: https://prow.k8s.io/pr-history/?org=kubernetes-sigs&repo=cluster-api-provider-aws&pr=6073

damdo · 2026-06-30T10:33:49Z

/assign @richardcase @dlipovetsky @AndiDog @nrb @clebs

damdo · 2026-07-01T08:38:10Z

/test pull-cluster-api-provider-aws-e2e

damdo · 2026-07-01T10:41:11Z

/test pull-cluster-api-provider-aws-e2e-blocking

damdo · 2026-07-01T10:41:19Z

/test pull-cluster-api-provider-aws-e2e

clebs · 2026-07-01T13:24:55Z

/lgtm

kubernetes-prow · 2026-07-01T13:25:02Z

LGTM label has been added.

Details

Git tree hash: 21c960c01953b683f99f4b10a068790c8e49785b

nrb

One missed context.TODO(), otherwise 👍

nrb · 2026-07-01T19:05:53Z

+		eniID := aws.ToString(eni.NetworkInterfaceId)
+		s.scope.Debug("Deleting orphaned network interface", "eni-id", eniID)
+
+		if _, err := s.EC2Client.DeleteNetworkInterface(context.TODO(), &ec2.DeleteNetworkInterfaceInput{


Suggested change

if _, err := s.EC2Client.DeleteNetworkInterface(context.TODO(), &ec2.DeleteNetworkInterfaceInput{

if _, err := s.EC2Client.DeleteNetworkInterface(ctx, &ec2.DeleteNetworkInterfaceInput{

ENI deletion operations now consistently use the passed reconciliation context instead of context.TODO(), ensuring proper timeout semantics throughout the entire ENI cleanup phase.

kubernetes-prow · 2026-07-02T07:56:49Z

@damdo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-aws-apidiff-main	`151ad7e`	link	false	`/test pull-cluster-api-provider-aws-apidiff-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

damdo · 2026-07-02T08:13:48Z

/test pull-cluster-api-provider-aws-e2e

damdo · 2026-07-02T08:20:25Z

/test pull-cluster-api-provider-aws-build-docker

damdo · 2026-07-02T10:36:15Z

build-docker is ooming, I've created: kubernetes/test-infra#37376

damdo · 2026-07-02T11:35:00Z

/test pull-cluster-api-provider-aws-build-docker

clebs · 2026-07-02T12:07:31Z

/lgtm

Re-tagging after fixing docker OOM issue.

kubernetes-prow · 2026-07-02T12:07:38Z

LGTM label has been added.

Details

Git tree hash: bf3a730ae5f5bb3bfc6be1e7b74f4043be8ee531

damdo · 2026-07-02T14:56:16Z

/test pull-cluster-api-provider-aws-e2e-eks

damdo · 2026-07-02T17:21:21Z

/unhold

damdo · 2026-07-03T13:34:56Z

/cherry-pick release-2.11

k8s-infra-cherrypick-robot · 2026-07-03T13:35:34Z

@damdo: new pull request created: #6091

Details

In response to this:

/cherry-pick release-2.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kubernetes-prow Bot requested review from dlipovetsky and richardcase June 25, 2026 09:58

kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 25, 2026

damdo marked this pull request as draft June 25, 2026 10:12

serngawy mentioned this pull request Jun 23, 2026

🐛 ROSA Fix cross account creation issues-6037 #6038

Merged

6 tasks

kubernetes-prow Bot closed this in #6038 Jun 25, 2026

damdo reopened this Jun 25, 2026

damdo changed the title ~~WIP: 🐛 Clean up orphaned ENIs before subnet deletion during cluster teardown~~ 🐛 Clean up orphaned ENIs before subnet deletion during cluster teardown Jun 30, 2026

damdo marked this pull request as ready for review June 30, 2026 08:29

kubernetes-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 30, 2026

kubernetes-prow Bot requested a review from fiunchinho June 30, 2026 08:29

damdo closed this Jun 30, 2026

damdo reopened this Jun 30, 2026

damdo force-pushed the fix/eni-cleanup-during-deletion branch from fbbe945 to baaa412 Compare June 30, 2026 08:32

kubernetes-prow Bot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Jun 30, 2026

damdo changed the title ~~🐛 Clean up orphaned ENIs before subnet deletion during cluster teardown~~ 🐛 fix: clean up orphaned ENIs before subnet deletion at teardown Jun 30, 2026

kubernetes-prow Bot assigned AndiDog Jun 30, 2026

kubernetes-prow Bot requested review from AndiDog, clebs and nrb July 1, 2026 08:36

refactor: thread context through DeleteNetwork interface

b50a84c

damdo force-pushed the fix/eni-cleanup-during-deletion branch from b6cc2c7 to b50a84c Compare July 1, 2026 09:23

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 1, 2026

nrb reviewed Jul 1, 2026

View reviewed changes

fix: use reconciliation context for ENI deletion operations

151ad7e

ENI deletion operations now consistently use the passed reconciliation context instead of context.TODO(), ensuring proper timeout semantics throughout the entire ENI cleanup phase.

kubernetes-prow Bot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 2, 2026

kubernetes-prow Bot requested a review from nrb July 2, 2026 07:54

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 2, 2026

kubernetes-prow Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 2, 2026

kubernetes-prow Bot merged commit b33921c into kubernetes-sigs:main Jul 2, 2026
18 of 19 checks passed

k8s-infra-cherrypick-robot mentioned this pull request Jul 3, 2026

[release-2.11] 🐛 fix: clean up orphaned ENIs before subnet deletion at teardown #6091

Merged

	if _, err := s.EC2Client.DeleteNetworkInterface(context.TODO(), &ec2.DeleteNetworkInterfaceInput{
	if _, err := s.EC2Client.DeleteNetworkInterface(ctx, &ec2.DeleteNetworkInterfaceInput{

Uh oh!

Conversation

damdo commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

damdo commented Jun 29, 2026

Uh oh!

damdo commented Jun 29, 2026

Uh oh!

damdo commented Jun 29, 2026

Uh oh!

damdo commented Jun 29, 2026

Uh oh!

damdo commented Jun 30, 2026

Uh oh!

damdo commented Jun 30, 2026

Uh oh!

damdo commented Jul 1, 2026

Uh oh!

damdo commented Jul 1, 2026

Uh oh!

damdo commented Jul 1, 2026

Uh oh!

clebs commented Jul 1, 2026

Uh oh!

kubernetes-prow Bot commented Jul 1, 2026

Uh oh!

nrb left a comment

Choose a reason for hiding this comment

Uh oh!

nrb Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

kubernetes-prow Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

damdo commented Jul 2, 2026

Uh oh!

damdo commented Jul 2, 2026

Uh oh!

damdo commented Jul 2, 2026

Uh oh!

damdo commented Jul 2, 2026

Uh oh!

clebs commented Jul 2, 2026

Uh oh!

kubernetes-prow Bot commented Jul 2, 2026

Uh oh!

damdo commented Jul 2, 2026

Uh oh!

damdo commented Jul 2, 2026

Uh oh!

Uh oh!

damdo commented Jul 3, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

damdo commented Jun 25, 2026 •

edited

Loading

kubernetes-prow Bot commented Jul 2, 2026 •

edited

Loading