OCPBUGS-81716: Add retry logic for transient network errors in restartKubeletOnNode

[BhargaviGudi]

Add retry logic for transient network errors in restartKubeletOnNode
Bug: https://redhat.atlassian.net/browse/OCPBUGS-81716

Problem
TC4 CNV swap tests were failing intermittently with "connection refused" errors when running oc debug commands to restart kubelet on nodes.

Solution
Added retry mechanism to restartKubeletOnNode that:

• Retries up to 3 times for network errors
• Uses increasing delays between retries (5s, 10s, 15s)
• Fails immediately for non-network errors
• Logs each retry attempt for debugging

Network errors handled:

• connection refused
• connection reset
• connection timed out
• i/o timeout

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[openshift-ci-robot]

@BhargaviGudi: This pull request references Jira Issue OCPBUGS-81716, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Add retry logic for transient network errors in restartKubeletOnNode
Bug: https://redhat.atlassian.net/browse/OCPBUGS-81716

Problem
TC4 CNV swap tests were failing intermittently with "connection refused" errors when running oc debug commands to restart kubelet on nodes.

Solution
Added retry mechanism to restartKubeletOnNode that:

• Retries up to 3 times for network errors
• Uses increasing delays between retries (5s, 10s, 15s)
• Fails immediately for non-network errors
• Logs each retry attempt for debugging

Network errors handled:

• connection refused
• connection reset
• connection timed out
• i/o timeout

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 04a87c1c-5cd1-4aa8-8272-98371f9192b0

📥 Commits

Reviewing files that changed from the base of the PR and between c3c8d51 and 7e1cff3.

📒 Files selected for processing (2)

• test/extended/node/node_swap_cnv.go
• test/extended/node/node_utils.go

---

Walkthrough

restartKubeletOnNode now takes a context.Context, distinguishes transient vs non-transient failures, and retries systemctl restart kubelet up to 3 times with attempt-based backoff. A helper isTransientNetworkError was added. Call sites were updated to pass ctx; cleanupDropInAndRestartKubelet forwards ctx.

Changes

Cohort / File(s)	Summary
Kubelet restart retry logic test/extended/node/node_utils.go	Changed restartKubeletOnNode signature to func(ctx context.Context, ...), added retry loop (up to 3 attempts) with backoff (attempt+1)*5s, early exit on context cancellation, distinct handling/wrapping for non-transient vs transient errors, and added isTransientNetworkError(err error) bool. Updated cleanupDropInAndRestartKubelet to pass ctx.
Call-site updates test/extended/node/node_swap_cnv.go	All invocations of restartKubeletOnNode updated to pass existing ctx as first argument, including deferred cleanup closures and multi-node loops.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[BhargaviGudi]

/jira refresh

[openshift-ci-robot]

@BhargaviGudi: This pull request references Jira Issue OCPBUGS-81716, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[BhargaviGudi]

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-disruptive-longrunning

[openshift-ci]

@BhargaviGudi: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-disruptive-longrunning

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/343f3690-324b-11f1-94ea-e53d6b882dcb-0

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/extended/node/node_utils.go`:
- Around line 200-215: The retry loop in restartKubeletOnNode is blocking and
sleeps even after the final attempt and isn't cancelable; change the signature
to accept a context.Context (restartKubeletOnNode(ctx context.Context, oc
*exutil.CLI, nodeName string)), remove the unconditional time.Sleep on the final
attempt by only sleeping when attempt < maxAttempts, and replace time.Sleep with
a context-aware wait (select between time.After(backoff) and ctx.Done()) so the
retry/backoff can be canceled; keep using ExecOnNodeWithChroot and
isTransientNetworkError to detect retryable errors and propagate the lastErr
when context is done or retries exhausted.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ed04415d-f102-4abf-86dc-8f0f443aeda3

📥 Commits

Reviewing files that changed from the base of the PR and between c77ff4a and c3c8d51.

📒 Files selected for processing (1)

• test/extended/node/node_utils.go

[openshift-ci-robot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

[BhargaviGudi]

All test cases in the openshift/disruptive-longrunning suite are passing successfully.
CI job link: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-origin-30966-nightly-4.22-e2e-aws-disruptive-longrunning/2041403110437425152

[BhargaviGudi]

/verified by CI

[openshift-ci-robot]

@BhargaviGudi: This PR has been marked as verified by CI.

Details

In response to this:

/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sosiouxme]

/override ci/prow/e2e-aws-ovn-fips
/override ci/prow/e2e-aws-ovn-serial-1of2
/override ci/prow/e2e-gcp-ovn
/hold
feel free to release if you need.

[openshift-ci]

@sosiouxme: Overrode contexts on behalf of sosiouxme: ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-serial-1of2, ci/prow/e2e-gcp-ovn

Details

In response to this:

/override ci/prow/e2e-aws-ovn-fips
/override ci/prow/e2e-aws-ovn-serial-1of2
/override ci/prow/e2e-gcp-ovn
/hold
feel free to release if you need.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[BhargaviGudi]

/unhold

[BhargaviGudi]

@cpmeadors and @ngopalak-redhat Could you please help to review this PR? Thanks

[BhargaviGudi]

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-disruptive-longrunning

[openshift-ci]

@BhargaviGudi: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-disruptive-longrunning

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/7160c060-3337-11f1-9c78-e1bda43894ba-0

[openshift-ci-robot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

[BhargaviGudi]

/test e2e-aws-ovn-fips

[cpmeadors]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BhargaviGudi, cpmeadors

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• test/extended/node/OWNERS [cpmeadors]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[BhargaviGudi]

/test e2e-metal-ipi-ovn-ipv6

[openshift-trt]

Job Failure Risk Analysis for sha: 7e1cff3

Job Name	Failure Risk
pull-ci-openshift-origin-main-e2e-vsphere-ovn	Low [Jira:Node][sig-node] Node non-cnv swap configuration should reject user override of swap settings via KubeletConfig API [OCP-86395] [Suite:openshift/conformance/parallel] This test has passed 12.56% of 613 runs on release 5.0 [Overall] in the last week.
pull-ci-openshift-origin-main-e2e-vsphere-ovn-upi	Low [Jira:Node][sig-node] Node non-cnv swap configuration should reject user override of swap settings via KubeletConfig API [OCP-86395] [Suite:openshift/conformance/parallel] This test has passed 12.56% of 613 runs on release 5.0 [Overall] in the last week.

[BhargaviGudi]

/verified by @BhargaviGudi

[openshift-ci-robot]

@BhargaviGudi: This PR has been marked as verified by @BhargaviGudi.

Details

In response to this:

/verified by @BhargaviGudi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@BhargaviGudi: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@BhargaviGudi: Jira Issue Verification Checks: Jira Issue OCPBUGS-81716
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-81716 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Add retry logic for transient network errors in restartKubeletOnNode
Bug: https://redhat.atlassian.net/browse/OCPBUGS-81716

Problem
TC4 CNV swap tests were failing intermittently with "connection refused" errors when running oc debug commands to restart kubelet on nodes.

Solution
Added retry mechanism to restartKubeletOnNode that:

• Retries up to 3 times for network errors
• Uses increasing delays between retries (5s, 10s, 15s)
• Fails immediately for non-network errors
• Logs each retry attempt for debugging

Network errors handled:

• connection refused
• connection reset
• connection timed out
• i/o timeout

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.