systemd-resolved silently drops its 127.0.0.53:53 stub-listener fd from epoll, wedging all libc-path DNS while the daemon stays "alive"

[charles-teese-telestream]

systemd-resolved silently drops its 127.0.0.53:53 stub-listener fd from epoll, wedging all libc-path DNS while the daemon stays "alive"

tl;dr

On Bottlerocket aws-k8s-1.34 (observed on 1.60.0 and 1.61.0), systemd-resolved occasionally enters a state where the file descriptor backing its 127.0.0.53:53 UDP stub-listener socket is silently missing from its epoll set. The process keeps running, processes timerfd events normally (~1 wakeup/sec → cache TTL housekeeping is fine), but never recv-loops on the wedged socket. The kernel queues incoming DNS queries until the socket buffer overflows, then drops them. From the libc side this presents as dial tcp: lookup <host> on 127.0.0.53:53: ... i/o timeout on every consumer (containerd image pulls, metricdog HTTPS posts, any host process using glibc resolution). Pod-network DNS via the cluster's in-cluster resolver is unaffected, so kubelet NodeReady conditions stay green and the failure is invisible to standard health checks.

Every wedged node we've observed has had a public IPv4 address attached. Same-cluster, same-age, same-Bottlerocket-version private-IP nodes have not been observed to wedge despite carrying comparable host-side DNS load.

We've now hit this several times in ~30 days. We captured the live kernel state of a wedged daemon for the first time, and identified the divergence from a healthy peer.

Environment

• Bottlerocket: 1.61.0 (aws-k8s-1.34) — also reproduced on 1.60.0 (same variant)
• Kernel: 6.12.83
• containerd: 2.1.7+bottlerocket
• systemd: 252-55.el9_7.7 (per resolvectl --version)
• AWS EC2 (mixed x86_64 instance shapes), Karpenter-provisioned
• Cilium CNI; pod-network DNS goes through cluster DNS, host-network DNS through systemd-resolved on 127.0.0.53
• Every wedged node has had a public IPv4 attached (see tl;dr note above)

Symptom

dial tcp: lookup <host> on 127.0.0.53:53: read udp 127.0.0.1:<port>-> 127.0.0.53:53: i/o timeout

Reproduced on multiple independent nodes across May–June 2026. Bottlerocket version varies (1.60.0 and 1.61.0). On every wedged node, journald shows systemd-resolved logging for the first ~10 seconds after boot then going silent. The last log line is consistently the routine "Clock change detected. Flushing caches." emitted around T+10s (presumably triggered by chrony stepping the clock during early boot). After that, nothing — for the entire lifetime of the node.

Live evidence (most recent capture)

Process state on a wedged daemon, captured live via a privileged debug pod (hostPID: true):

Process state

State:                       S (sleeping)
Threads:                     1
VmRSS:                       9928 kB         ← 2.5× a healthy peer
voluntary_ctxt_switches:     34041           ← ~1/sec — timerfds firing normally
nonvoluntary_ctxt_switches:  8892

Kernel stack

/proc/<pid>/wchan: ep_poll

/proc/<pid>/stack:
  [<0>] ep_poll+0x34a/0x3a0
  [<0>] do_epoll_wait+0xc1/0xe0
  [<0>] __x64_sys_epoll_wait+0x5d/0xf0
  [<0>] do_syscall_64+0x79/0x150
  [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

Syscall args (single thread)

/proc/<pid>/syscall: 232 0x4 0x55f7958fb050 0x2c 0xffffffffffffffff ...
                     ↑   ↑                  ↑        ↑
                    nr  epfd            maxevents=44  timeout=-1 (infinite)

Wedged listener socket

/proc/net/udp row for 127.0.0.53:53:
  sl    local_address  rem_address st tx_queue          rx_queue ... uid timeout inode ... drops
  5296: 3500007F:0035  00000000:0000 07 00000000:00000000   00000000:00034100 ... 980 0 6159 ... 3840
                                                            ↑                                       ↑
                                                          213 KB unread                3,840 kernel drops

Fd → socket inode (4 listener sockets: IPv4/IPv6 × 127.0.0.53/.54)

readlink /proc/<pid>/fd/15  →  socket:[6159]   (127.0.0.53:53, the wedged one)
readlink /proc/<pid>/fd/16  →  socket:[6160]   (IPv6 sibling)
readlink /proc/<pid>/fd/17  →  socket:[6161]   (127.0.0.54:53, internal DNSSEC validator)
readlink /proc/<pid>/fd/18  →  socket:[6162]   (IPv6 sibling)

Epoll set membership — the smoking gun

$ cat /proc/<pid>/fdinfo/4   (the epoll instance)

tfd:  5 events: 19 ... ino:29   sdev:10    (timerfd)
tfd:  6 events: 19 ... ino:29   sdev:10    (timerfd)
tfd:  7 events: 18 ... ino:aae  sdev:30
tfd:  8 events: 19 ... ino:29   sdev:10
tfd:  9 events: 19 ... ino:ab1  sdev:9
tfd: 10 events: 19 ... ino:29   sdev:10
tfd: 12 events: 19 ... ino:ab3  sdev:9
tfd: 13 events: 1a ... ino:935  sdev:1b    (signalfd)
tfd: 16 events: 19 ... ino:1810 sdev:9     ← UDP listener fd 16
tfd: 17 events: 19 ... ino:1811 sdev:9     ← UDP listener fd 17
tfd: 18 events: 19 ... ino:1812 sdev:9     ← UDP listener fd 18
tfd: 19 events: 19 ... ino:ab9  sdev:9
tfd: 20 events: 19 ... ino:aba  sdev:9
tfd: 21 events: 19 ... ino:29   sdev:10
tfd: 22 events: 19 ... ino:29   sdev:10

15 fds total — fd 15 (→ socket:[6159] → 127.0.0.53:53) is conspicuously absent.

The other three listener sockets (fds 16, 17, 18) are present. The kernel-level architecture is identical; just one EPOLL_CTL_ADD seems to have been missed (or unregistered without re-add).

Side-by-side with a healthy peer

We captured the same data from a sibling node: identical Bottlerocket build, identical age, identical instance class, both with public IPv4.

Field	Healthy	Wedged
Bottlerocket	1.61.0 (aws-k8s-1.34)	1.61.0 (aws-k8s-1.34)
Node age	identical	identical
Process state	S (sleeping)	S (sleeping)
Threads	1	1
wchan	ep_poll	ep_poll
Stack	ep_poll → do_epoll_wait → __x64_sys_epoll_wait	identical
Syscall	epoll_wait(4, …, 44, -1)	identical
voluntary_ctxt_switches	35,187	34,041
VmRSS	3,860 kB	9,928 kB
Epoll set size	16 fds	15 fds
fd 15 (→ 127.0.0.53:53)	in epoll set ✓	NOT in epoll set ✗
fd 16/17/18 (sibling listeners)	in epoll set	in epoll set
/proc/net/udp drops on 127.0.0.53:53	0	4,160 (at capture)
/proc/net/udp rx_queue on 127.0.0.53:53	0	213 KB
resolvectl statistics	16,097 transactions, 0 failures	(daemon never sees queries, so unmeasurable from inside)

Across every observable field except the missing epoll registration and its downstream consequences (queue backlog, RSS bloat from buffered state) the two processes are byte-for-byte equivalent.

Hypothesized bug class

The wedge correlates 1:1 with the "Clock change detected. Flushing caches." log line at T+10s. That message originates from systemd-resolved's clock-step handler, which flushes caches and (we suspect) closes & reopens listener sockets to clear in-flight resolver state. The most plausible bug:

• The clock-step handler closes the old 127.0.0.53:53 socket fd (which auto-removes it from epoll on close), opens a new one, but misses or races on the epoll_ctl(EPOLL_CTL_ADD, new_fd, EPOLLIN) for the new socket. The sibling listeners go through the same code path successfully — so it's a race / off-by-one in one branch rather than a complete missing call.

We don't have a source-level diagnosis. Open questions for maintainers:

1. Does the cache-flush path indeed touch listener-socket FDs, or are we misreading the trigger?
2. Is the clock-step handler in systemd-resolved a known hot spot for this?
3. Has anyone seen this pattern in containerized/cloud-VM environments where chrony does an initial clock step during boot? Particularly on instances with public IPv4 attached — could the extra systemd-networkd configuration around the public interface delay/reorder the resolved startup in a way that exposes the race?

Trigger / reproducer status

Not deterministic. Empirically:

• A small fraction of newly-provisioned Bottlerocket nodes hit this within seconds of boot
• Multiple same-age, same-OS-version Bottlerocket 1.61 nodes side-by-side: a subset wedged, the rest healthy
• Wedge persists for the entire lifetime of the node — no spontaneous recovery in any observed case
• Every observed wedge is on a node with a public IPv4 attached. Private-IP nodes with comparable DNS load have not wedged.

The clock-step at T+10s is plausibly the trigger (chrony correcting an initial RTC offset during early boot), but we haven't reproduced it on demand.

Recovery procedure

SIGTERM the wedged systemd-resolved PID from a privileged pod with hostPID: true. systemd's Restart=on-failure immediately starts a fresh instance with a healthy epoll set. Verified end-to-end: old PID exits, new PID spawned within 1 second, /proc/net/udp drops counter reset to 0, the previously-missing fd now correctly registered in the new epoll instance, resolvectl statistics accumulating transactions, every libc-path DNS consumer immediately functional. Downtime <1s.

Detection (for anyone else hitting this)

Cheap test, runnable from a hostPID pod or via nsenter --target 1 --net --mount:

awk '$2 ~ /:0035$/ && $NF != "0"' /proc/net/udp

Non-zero drops on 127.0.0.53:53 is the signal. The definitive confirmation is /proc/<resolved-pid>/fdinfo/<epoll-fd> missing the watch entry for the 127.0.0.53:53 socket fd.

Asks for Bottlerocket maintainers

1. Confirm whether this is a Bottlerocket-layer bug (systemd integration / Bottlerocket-specific resolved config) or strictly upstream systemd, so we know whether to file in parallel against systemd/systemd.
2. Pointers on related issues or known patches we may have missed (none turned up in our search of bottlerocket-os/bottlerocket and systemd/systemd).
3. Guidance on a Bottlerocket-shipped mitigation. Two plausible options:
   • Drop-in WatchdogSec=30s + Restart=on-watchdog for systemd-resolved.service. Requires systemd-resolved to actually issue sd_notify(WATCHDOG=1) periodically — needs confirmation in the 252.x build shipped.
   • Expose a Bottlerocket setting to disable systemd-resolved entirely and fall back to a simple stub (we've seen this requested in Set DNSStubListenerExtra #4732 and Ability to set ndots in /etc/resolv.conf #3543, but the demand is broader than just our wedge).

Happy to share additional captures, run further forensic instrumentation on the next occurrence, or test patches against this configuration.