Skip to content

fix/refactor: deep copy status in translator for all resources#9458

Merged
rudrakhp merged 2 commits into
envoyproxy:mainfrom
rudrakhp:translator_coalesce_race_gw
Jul 16, 2026
Merged

fix/refactor: deep copy status in translator for all resources#9458
rudrakhp merged 2 commits into
envoyproxy:mainfrom
rudrakhp:translator_coalesce_race_gw

Conversation

@rudrakhp

@rudrakhp rudrakhp commented Jul 9, 2026

Copy link
Copy Markdown
Member

@rudrakhp
rudrakhp requested a review from a team as a code owner July 9, 2026 13:24
@netlify

netlify Bot commented Jul 9, 2026

Copy link
Copy Markdown

Deploy Preview for cerulean-figolla-1f9435 ready!

Name Link
🔨 Latest commit 9083b96
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6a58341fb791410008e5dd9c
😎 Deploy Preview https://deploy-preview-9458--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@rudrakhp
rudrakhp force-pushed the translator_coalesce_race_gw branch from 0c169cb to a9bef03 Compare July 9, 2026 13:57
@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.10526% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.38%. Comparing base (2df4a96) to head (3247d7d).

Files with missing lines Patch % Lines
internal/gatewayapi/resource/resource.go 91.30% 2 Missing and 2 partials ⚠️
internal/gatewayapi/securitypolicy.go 66.66% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9458      +/-   ##
==========================================
- Coverage   75.41%   75.38%   -0.04%     
==========================================
  Files         252      252              
  Lines       41660    41608      -52     
==========================================
- Hits        31417    31365      -52     
+ Misses       8113     8111       -2     
- Partials     2130     2132       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rudrakhp
rudrakhp force-pushed the translator_coalesce_race_gw branch 3 times, most recently from 864d3a4 to be6f0dd Compare July 9, 2026 17:16
@rudrakhp rudrakhp added this to the v1.9.0-rc.1 Release milestone Jul 9, 2026

@arkodg arkodg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks !

@arkodg
arkodg requested review from a team July 12, 2026 22:33
@arkodg

arkodg commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

hey @rudrakhp gen-check is failng

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
@rudrakhp
rudrakhp force-pushed the translator_coalesce_race_gw branch from 2b66ec8 to 3247d7d Compare July 13, 2026 14:46
@anrs

anrs commented Jul 14, 2026

Copy link
Copy Markdown

StatusDeepCopy() isolates status mutations correctly, but resolveLocalObjectRefsInPolicy mutates Spec.ResponseOverride[*].Response.Body (Type/Inline/ValueRef) in place on the same *CustomResponseBody that the original holds. Since StatusDeepCopy only deep-copies Status and shallow-copies Spec, the Spec pointer graph is still shared with watchable's coalesce goroutine. This leaves the same class of race open for any BTP with a ResponseOverride.Response.Body.ValueRef — it just needs a different trigger than the original Gateway/GatewayClass panic probably.

@rudrakhp

rudrakhp commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

Good catch, and you're right — resolveLocalObjectRefsInPolicy mutates Spec.ResponseOverride[*].Response.Body in place on the same *CustomResponseBody the shared tree holds, and since StatusDeepCopy() only deep-copies Status, that Spec pointer graph is still shared with the coalesce goroutine's reflect.DeepEqual walk. So this is a genuine instance of the same class of race.

Two clarifications on the scope:

  • It's gated on the merge path (mergeBackendTrafficPolicy returns early when routePolicy.Spec.MergeType == nil), so the trigger is a BTP that has both spec.mergeType set and a responseOverride[...].response.body.valueRef — narrower than any BTP with a ValueRef, but still real and reachable.
  • As you note, it won't reproduce the exact slice index out of range (that needs a slice resize mid-walk); reassigning the three body fields would more likely surface as a torn read / different panic under the race detector. Same class, different symptom.

On the fix: I'd like to keep this PR's invariant that only Status is ever deep-copied — broadening the pre-translation copy to cover Spec sub-trees would reintroduce the memory cost we're explicitly avoiding. The right fix here is to stop mutating Spec at all, and instead adopt the field-owner pattern introduced for SecurityPolicy merge in #8538.

I will open a follow up for that fix.

cc @arkodg @kkk777-7

@kkk777-7

Copy link
Copy Markdown
Member

Hi @rudrakhp thanks for working this, and @anrs thanks for raising this issue! :)

To fix same issue in resolveLocalObjectRefsInPolicy, I prefer adopting field-owner pattern.

keep this PR's invariant that only Status

so it sounds good to me.

@kkk777-7

Copy link
Copy Markdown
Member

LGTM, thanks!

@rudrakhp
rudrakhp merged commit e51b417 into envoyproxy:main Jul 16, 2026
59 of 63 checks passed
jukie pushed a commit to jukie/gateway that referenced this pull request Jul 22, 2026
…proxy#9458)

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Co-authored-by: zirain <zirain2009@gmail.com>
(cherry picked from commit e51b417)
Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>
jukie added a commit that referenced this pull request Jul 22, 2026
* detect ipv6 literal hosts in url2cluster and token endpoint check (#9165)

* detect ipv6 literal hosts in url2cluster and token endpoint check

Signed-off-by: dxbjavid <dxbjavid@gmail.com>

* add release note for ipv6 literal endpoint fix

Signed-off-by: dxbjavid <dxbjavid@gmail.com>

---------

Signed-off-by: dxbjavid <dxbjavid@gmail.com>
Signed-off-by: zirain <zirain2009@gmail.com>
Signed-off-by: Javid Khan <dxbjavid@gmail.com>
Co-authored-by: zirain <zirain2009@gmail.com>
(cherry picked from commit 30d456e)
Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>

* Revert LuaPerRoute source materialization for Lua (#9426)

* Revert "[xds] stabilize listener-level Lua XDS filters to avoid listener drain (#8598)"

This reverts commit 80b2762.

Why:
- Back out the LuaPerRoute stabilization layer together with #8355 while
  investigating the Gateway-scoped Lua memory regression from #9355.
- #8598 only stabilizes per-route Lua HCM slot names; it still leaves Lua
  source materialized per route, so it is not sufficient for the memory issue.

Test plan:
- `go test ./internal/xds/translator`

Signed-off-by: Arko Dasgupta <arkodg@gmail.com>

* Revert "refactor/perf: use LuaPerRoute instead of FilterConfig (#8355)"

This reverts commit f31ac4e.

Why:
- #8355 materialized Gateway-targeted Lua as per-route LuaPerRoute source code.
- With large route tables, identical Gateway-scoped scripts are duplicated across
  route entries and worker Lua runtime state, matching the memory regression
  reported in #9355.
- Restoring listener-level Lua source avoids scaling the Gateway-scoped script
  with route count.

Test plan:
- `go test ./internal/xds/translator`

Fixes #9355

Signed-off-by: Arko Dasgupta <arkodg@gmail.com>

* add filterContext in per route level

Signed-off-by: Arko Dasgupta <arkodg@gmail.com>

* [xds] scope Lua revert to translator changes

Remove unrelated Gateway API status-merge and redirect regression files that were accidentally included with the filterContext follow-up.

Add a breaking-change fragment documenting the listener-level Lua source layout and filter-key migration for EnvoyPatchPolicy and extension consumers.

Test: make test
Signed-off-by: Arko Dasgupta <arkodg@gmail.com>

---------

Signed-off-by: Arko Dasgupta <arkodg@gmail.com>
(cherry picked from commit 76bd888)
Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>

* fix: default backend TLS max version to 1.3 (#9412)

Backend (upstream) TLS connections were silently capped at TLS 1.2.
Apply the defaults (min 1.2, max 1.3) in applyBackendTLSSetting when not configured.

Fixes #9395

Signed-off-by: vishwas-bm <b_m.vishwas@nokia.com>
(cherry picked from commit 581a9c5)
Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>

* fix: add default retry policy for Envoy wasm module fetch (#9462)

add default retry policy for Envoy wasm module fetch

Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>
(cherry picked from commit 3e18557)
Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>

* fix/refactor: deep copy status in translator for all resources (#9458)

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Co-authored-by: zirain <zirain2009@gmail.com>
(cherry picked from commit e51b417)
Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>

* fix(logs): restore ISO 8601 timestamp format (#9493)

* fix(logs): restore ISO 8601 timestamp format

Signed-off-by: Florian Wiegand <florian.wiegand@sap.com>

* fix(logs): add release note for ISO 8601 timestamp fix

Signed-off-by: Florian Wiegand <florian.wiegand@sap.com>

---------

Signed-off-by: Florian Wiegand <florian.wiegand@sap.com>
(cherry picked from commit 2621405)
Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>

* fix: unreferenced Secret events trigger reconciliation when HTTPRouteFilter CRD exists (#9488)

* fix: unreferenced Secret events trigger reconciliation when HTTPRouteFilter CRD exists

isHTTPRouteFilterReferencingSecret returned true on both branches, so with
the HTTPRouteFilter CRD installed (the default), every Secret write in the
cluster passed validateSecretForReconcile and enqueued a full reconcile.
Any high-frequency Secret writer (secret sync controllers, cert rotation)
drives the controller to reconcile continuously.

Return false when no HTTPRouteFilter references the Secret, matching the
other is*ReferencingSecret helpers. Enable hrfCRDExists in the predicate
test so the existing unreferenced-Secret case guards this path, and add
positive/negative cases for HTTPRouteFilter credential injection refs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: James Hong <james.hong@zapier.com>
(cherry picked from commit 4d7686c)
Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>

---------

Signed-off-by: dxbjavid <dxbjavid@gmail.com>
Signed-off-by: zirain <zirain2009@gmail.com>
Signed-off-by: Javid Khan <dxbjavid@gmail.com>
Signed-off-by: jukie <10012479+jukie@users.noreply.github.com>
Signed-off-by: Arko Dasgupta <arkodg@gmail.com>
Signed-off-by: vishwas-bm <b_m.vishwas@nokia.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Signed-off-by: Florian Wiegand <florian.wiegand@sap.com>
Signed-off-by: James Hong <james.hong@zapier.com>
Co-authored-by: Javid Khan <dxbjavid@gmail.com>
Co-authored-by: zirain <zirain2009@gmail.com>
Co-authored-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Co-authored-by: vishwas-bm <b_m.vishwas@nokia.com>
Co-authored-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Co-authored-by: Florian Wiegand <36593900+wiegandf@users.noreply.github.com>
Co-authored-by: James Hong <greyeye@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants