Skip to content

Commit ddf2fc4

Browse files
authored
fix: eliminate cfg.Logger data race in server()'s reload/shutdown path (#9581)
* fix: eliminate cfg.Logger data race in server()'s reload/shutdown path Loader.New() aliases the *config.Server passed to it, so the watcher goroutine's cfg.Logger reassignment on each config reload directly mutates the same struct instance server()'s top-level select loop reads from when logging shutdown/hook-error messages. The watcher writes under cfgMu, but server() read cfg.Logger with no synchronization at all, producing a genuine data race (confirmed via the actual CI job log: write at configloader.go:91 vs. read at server.go:112, and reproduced locally under linux/-race). Fix: the loader publishes each new logger over a small "latest value" channel as reloads happen. server() keeps its own local logger reference and swaps it only upon receiving from that channel, so the update is synchronized via channel send/receive instead of a shared, unguarded field. Signed-off-by: Muhammad Waqar <mwaqar@confluent.io> * fix: simplify logger race fix to a mutex-guarded Loader.Logger() accessor Per review feedback (github.com//pull/9581#issuecomment-5077942710): drop the loggerUpdates channel and drain logic in favor of a simple RLock-guarded Logger() getter on Loader, reusing the existing cfgMu rather than introducing new synchronization. Signed-off-by: Muhammad Waqar <mwaqar@confluent.io> --------- Signed-off-by: Muhammad Waqar <mwaqar@confluent.io>
1 parent 8e32174 commit ddf2fc4

3 files changed

Lines changed: 14 additions & 2 deletions

File tree

internal/cmd/server.go

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -102,14 +102,17 @@ func server(ctx context.Context, stdout, stderr io.Writer, cfgPath string, hook
102102
// Exit if the config loader fails to start the runners.
103103
// Continuing with failed runners would cause EG to function incorrectly.
104104
case err := <-l.Errors():
105-
cfg.Logger.Error(err, "failed to start runners")
105+
// Read the logger through the loader so the read is guarded by the
106+
// same mutex the reload path holds while replacing it. Reading
107+
// cfg.Logger directly here races with that swap.
108+
l.Logger().Error(err, "failed to start runners")
106109
// Wait for runners to finish before shutting down.
107110
// This is to make sure no orphaned runner process is left running in standalone mode.
108111
l.Wait()
109112
return err
110113
// Wait for the context to be done, which usually happens the process receives a SIGTERM or SIGINT.
111114
case <-ctx.Done():
112-
cfg.Logger.Info("shutting down")
115+
l.Logger().Info("shutting down")
113116
// Wait for runners to finish before shutting down.
114117
// This is to make sure no orphaned runner process is left running in standalone mode.
115118
l.Wait()

internal/envoygateway/config/loader/configloader.go

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,14 @@ func (r *Loader) Errors() <-chan error {
142142
return r.hookErr
143143
}
144144

145+
// Logger returns the current logger, safe to call concurrently with a config
146+
// reload replacing it.
147+
func (r *Loader) Logger() logging.Logger {
148+
r.cfgMu.RLock()
149+
defer r.cfgMu.RUnlock()
150+
return r.cfg.Logger
151+
}
152+
145153
// Wait returns when success to acquire mutex, which means no hook is running.
146154
func (r *Loader) Wait() {
147155
r.hookMutex.Lock()
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Fixed a data race between config-reload and the standalone server's shutdown/error logging by reading the logger through a new mutex-guarded `Loader.Logger()` accessor instead of the shared `cfg.Logger` field directly.

0 commit comments

Comments
 (0)