Skip to content

fix(server): close inherited fds after daemonizing to avoid pipe deadlock (supersedes #2314)#2771

Open
brianmacy wants to merge 1 commit into
mozilla:mainfrom
brianmacy:fd-hygiene-daemonize
Open

fix(server): close inherited fds after daemonizing to avoid pipe deadlock (supersedes #2314)#2771
brianmacy wants to merge 1 commit into
mozilla:mainfrom
brianmacy:fd-hygiene-daemonize

Conversation

@brianmacy

Copy link
Copy Markdown

Summary

Fixes the 0%-CPU daemon deadlock in #1011.

The persistent cache server is lazily fork+exec'd by a compile-job client
and inherits that client's open file descriptors — including ninja's jobserver
pipe and the cmake ... | tee stdout pipe. Because the long-lived daemon keeps
those write-ends open, the writer end never observes EOF, and the build wedges
at 0% CPU (reliably reproducible under a ninja build that pipes through tee,
e.g. an ONNX Runtime link step).

Fix

After daemonizing — and before the server binds its socket — close all inherited
file descriptors >= 3 via the close_fds
crate.

  • Scoped to the cache server (Command::InternalStartServer). sccache-dist
    has already constructed a reqwest client that owns runtime/epoll fds by the
    time it daemonizes, so blindly closing fds there would be unsound — it opts out.
  • daemonize() now takes (Option<File>, bool): the error-log stderr redirect
    is folded into daemonize (configured via daemonix's Stdio) instead of a
    separate post-daemonize step, and the bool selects whether to run the sweep.
  • Opt-outs: SCCACHE_NO_DAEMON=1 (foreground debug) skips daemonizing and
    the sweep; SCCACHE_NO_FD_HYGIENE=1 is an escape hatch that skips only the
    sweep. On Windows there is no daemon fork, so daemonize() just redirects stderr.

Test

Adds a Unix regression test that forks, runs close_open_fds(3, &[]) in the
child, and asserts an inherited pipe fd is closed (fcntl(F_GETFD)EBADF)
while stderr (fd 2) survives the sweep.

This has also been running in our own CI at scale (a fork build carrying this
change) across all six release targets with no regressions, and it eliminates
the ONNX-link deadlock we hit consistently without it.

Credit

This finishes off @ntrrgc's #2314, rebased onto current main and adapted to
the daemonize → daemonix migration (hence the daemonize(Option<File>, bool)
shape and the close_fds 0.3.2 dependency). Co-authored accordingly.

Closes #1011. Supersedes #2314.

…lock

The persistent cache server is lazily fork+exec'd by a compile-job
client and inherits the client's open file descriptors, including
ninja's jobserver pipe and the `cmake ... | tee` stdout pipe. Because
the long-lived daemon keeps those write-ends open, the writer never
observes EOF, producing a 0%-CPU deadlock (mozilla#1011).

After daemonizing, close all inherited file descriptors >= 3 (before the
server binds its socket) via the close_fds crate. The sweep is scoped to
the cache server (Command::InternalStartServer): sccache-dist has already
built a reqwest client that owns runtime/epoll fds at daemonize time, so
closing arbitrary fds there would be unsound -- it opts out.

daemonize() now takes (Option<File>, bool): the error-log stderr redirect
is folded into daemonize (configured through daemonix's Stdio) instead of
a separate post-daemonize step, and the bool selects whether to run the
fd sweep. SCCACHE_NO_DAEMON=1 (foreground debug mode) skips daemonizing
and the sweep; SCCACHE_NO_FD_HYGIENE=1 is an escape hatch that skips only
the sweep. On Windows there is no daemon fork, so daemonize() just
redirects stderr.

Adds a Unix regression test that forks, runs close_open_fds(3, &[]) in
the child, and asserts an inherited pipe fd is closed (fcntl F_GETFD
returns -1 with errno EBADF) while stderr (fd 2) survives the sweep.

Closes mozilla#1011. Supersedes mozilla#2314.

Co-authored-by: Alicia Boya García <aboya@igalia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hang / deadlock while compiling?

1 participant