test(e2e): retry transient network errors on idempotent requests

[Aias00]

What type of PR is this?

/kind flake
/area testing

What this PR does

Fixes an intermittent TestEGUpgrade teardown flake where the gateway-api conformance suite's cleanup fails to delete the upgrade GatewayClass with a transient connection error:

Delete ".../gatewayclasses/upgrade": EOF

This is test-infra flake (shows up on unrelated PRs, e.g. #9460), not a regression in the code under test.

Root cause

client-go's rest.Request only retries GET on transient connection errors (IsProbableEOF / IsConnectionReset / IsHTTP2ConnectionLost); DELETE and other write verbs surface the error immediately. So when a teardown DELETE reaches the apiserver but the response is lost (EOF / connection reset / HTTP/2 GOAWAY / timeout), the test fails — even though the resource was already deleted.

The teardown DELETE goes through suite.Client, which the conformance suite rebuilds per test via client.New(suite.RestConfig, ...) in setClientsetForTest. So a client.Client-level wrapper would be bypassed; the only hook inherited by those rebuilt clients is rest.Config.WrapTransport.

Fix

Install a retrying http.RoundTripper via rest.Config.WrapTransport in test/utils/kubernetes's NewClient (composed with any pre-existing wrapper via transport.Wrappers).

Retry policy (conservative):

• Methods: GET, HEAD, DELETE only. Never POST/PATCH/PUT.
• Body: only retry when replayable — req.Body == nil, req.Body == http.NoBody, or req.GetBody != nil. (client-go builds DELETE bodies from bytes.NewReader; the stdlib auto-populates GetBody for *bytes.Reader, so DeleteOptions bodies are replayable. http.NoBody must be checked explicitly because the stdlib substitutes it for a zero-length body instead of nil.)
• Before each retry after the first: re-derive the body via req.GetBody() and assign to a req.Clone(...) (Clone copies GetBody but does not reset Body).
• Errors retried: io.EOF, io.ErrUnexpectedEOF, connection reset, HTTP/2 GOAWAY/connection-lost, net.Error timeouts.
• HTTP status codes are NOT retried (5xx/429/404 returned as-is), so a 404 from a retried DELETE flows through to MustApplyWithCleanup's existing IsNotFound → success handling.
• Close any partial resp.Body before retrying; honor context cancellation in the backoff select.

Tests

Adds test/utils/kubernetes/client_test.go covering:

• EOF retry on GET then succeeds
• DELETE-with-body EOF retry, body fully replayed via GetBody, 404 pass-through (the exact gatewayclasses/upgrade flake scenario)
• POST not retried
• non-replayable body (no GetBody) not retried
• HTTP status codes (500/429/404) not retried
• attempt exhaustion returns last error
• context cancellation short-circuits backoff
• net.Error timeout retried

go test -race ./test/utils/kubernetes/   # PASS
golangci-lint (project .golangci.yml, all build tags)  # clean

Blast radius: NewClient is the shared entrypoint for all e2e suites (e2e, upgrade, multiple_gc, merge_gateways, benchmark, resilience); all compile clean.

Related

Fixes #9467

Historical TestEGUpgrade flake issues (different root causes): #6009, #3311, #3262, #2937

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for cerulean-figolla-1f9435 ready!

Name	Link
🔨 Latest commit	9471bc9
🔍 Latest deploy log	https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6a58e456c8b8d20008e8338e
😎 Deploy Preview	https://deploy-preview-9468--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

This PR hardens the e2e Kubernetes client used by Envoy Gateway test suites by adding a rest.Config transport wrapper that retries transient, connection-level failures for idempotent, replayable requests (notably teardown DELETEs), addressing intermittent conformance cleanup flakes.

Changes:

• Add a retrying http.RoundTripper (retryTransport) and wire it into the shared e2e NewClient path via rest.Config transport wrapping.
• Implement conservative retry rules (GET/HEAD/DELETE only; only replayable bodies; transient network errors only; no HTTP status retries).
• Add focused unit tests covering retry behavior, replaying DELETE bodies via GetBody, attempt exhaustion, and context-cancel behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
test/utils/kubernetes/client.go	Installs a retrying transport wrapper and implements the retry/backoff + transient-error detection logic.
test/utils/kubernetes/client_test.go	Adds unit tests validating retry behavior, body replay semantics, and non-retryable scenarios.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.43%. Comparing base (2df4a96) to head (9471bc9).
⚠️ Report is 15 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9468      +/-   ##
==========================================
+ Coverage   75.41%   75.43%   +0.02%     
==========================================
  Files         252      252              
  Lines       41660    41641      -19     
==========================================
- Hits        31417    31412       -5     
+ Misses       8113     8095      -18     
- Partials     2130     2134       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.