fix: eliminate cfg.Logger data race in server()'s reload/shutdown path - #9581
Conversation
✅ Deploy Preview for cerulean-figolla-1f9435 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c0ab8c14be
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Loader.New() aliases the *config.Server passed to it, so the watcher goroutine's cfg.Logger reassignment on each config reload directly mutates the same struct instance server()'s top-level select loop reads from when logging shutdown/hook-error messages. The watcher writes under cfgMu, but server() read cfg.Logger with no synchronization at all, producing a genuine data race (confirmed via the actual CI job log: write at configloader.go:91 vs. read at server.go:112, and reproduced locally under linux/-race). Fix: the loader publishes each new logger over a small "latest value" channel as reloads happen. server() keeps its own local logger reference and swaps it only upon receiving from that channel, so the update is synchronized via channel send/receive instead of a shared, unguarded field. Signed-off-by: Muhammad Waqar <mwaqar@confluent.io>
c0ab8c1 to
ef9f22e
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9581 +/- ##
==========================================
+ Coverage 75.64% 75.67% +0.02%
==========================================
Files 252 254 +2
Lines 41758 41937 +179
==========================================
+ Hits 31589 31735 +146
- Misses 8046 8071 +25
- Partials 2123 2131 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Nice find, but I think this is a bit overengineered. The loader already has func (r *Loader) Logger() logging.Logger {
r.cfgMu.RLock()
defer r.cfgMu.RUnlock()
return r.cfg.Logger
} |
I was under the impression that undergoing a lock contention every time we need to access a logger here was not preferred, hence this version. |
The lock is only taken inside the configloader, and Logger() would be called once per process lifetime, so there's effectively no overhead. I’d prefer the simpler fix here. |
…ssor Per review feedback (github.com/envoyproxy/pull/9581#issuecomment-5077942710): drop the loggerUpdates channel and drain logic in favor of a simple RLock-guarded Logger() getter on Loader, reusing the existing cfgMu rather than introducing new synchronization.
|
@zhaohuabing implemented the simpler version |
…ssor Per review feedback (github.com/envoyproxy/pull/9581#issuecomment-5077942710): drop the loggerUpdates channel and drain logic in favor of a simple RLock-guarded Logger() getter on Loader, reusing the existing cfgMu rather than introducing new synchronization. Signed-off-by: Muhammad Waqar <mwaqar@confluent.io>
d24ed3f to
0814e1c
Compare
|
Thanks! |
|
/retest |
envoyproxy#9581) * fix: eliminate cfg.Logger data race in server()'s reload/shutdown path Loader.New() aliases the *config.Server passed to it, so the watcher goroutine's cfg.Logger reassignment on each config reload directly mutates the same struct instance server()'s top-level select loop reads from when logging shutdown/hook-error messages. The watcher writes under cfgMu, but server() read cfg.Logger with no synchronization at all, producing a genuine data race (confirmed via the actual CI job log: write at configloader.go:91 vs. read at server.go:112, and reproduced locally under linux/-race). Fix: the loader publishes each new logger over a small "latest value" channel as reloads happen. server() keeps its own local logger reference and swaps it only upon receiving from that channel, so the update is synchronized via channel send/receive instead of a shared, unguarded field. Signed-off-by: Muhammad Waqar <mwaqar@confluent.io> * fix: simplify logger race fix to a mutex-guarded Loader.Logger() accessor Per review feedback (github.com/envoyproxy/pull/9581#issuecomment-5077942710): drop the loggerUpdates channel and drain logic in favor of a simple RLock-guarded Logger() getter on Loader, reusing the existing cfgMu rather than introducing new synchronization. Signed-off-by: Muhammad Waqar <mwaqar@confluent.io> --------- Signed-off-by: Muhammad Waqar <mwaqar@confluent.io>
Summary
loader.New()(internal/envoygateway/config/loader/configloader.go) stores the*config.Serverpointer passed to it without copying it, so it's aliased with thecfgvariable held byserver()(internal/cmd/server.go). On a config-file reload, the loader's watcher goroutine reassignscfg.Logger(andcfg.EnvoyGateway) undercfgMu. Meanwhile,server()'s own top-levelselectloop readscfg.Loggerdirectly (cfg.Logger.Error(...)/cfg.Logger.Info(...)) with no synchronization at all — it has no knowledge ofcfgMu. That's an unsynchronized concurrent read/write on the same struct field: a genuine data race.This was caught by
go test -raceinTestCustomProviderCancelWhenConfigReload:Reproduced locally under
-race(needed Linux, since fsnotify's event-duplication behavior differs enough from macOS/kqueue that it didn't reproduce as reliably there).Fix
The loader now publishes each new logger over a small "latest value" channel (
LoggerUpdates()) as reloads happen.server()keeps its own localloggervariable and swaps it only upon receiving from that channel in its select loop, so the update is synchronized via channel send/receive rather than a shared, unguarded struct field.Test plan
go build ./...go vet ./internal/cmd/... ./internal/envoygateway/config/loader/...go test -race -run TestCustomProviderCancelWhenConfigReload -count=100+ ./internal/cmd/...— reproduced the exact race pre-fix (Linux/Docker); 100/100 clean post-fix (both macOS and Linux)