compiler: fall back to local on distributed-compile failure#2777
Open
jetm wants to merge 6 commits into
Open
Conversation
sccache logged the distribute-vs-local decision only at debug ("Compiling
locally", "Attempting distributed compilation"), while an infrastructure
fallback warned. At info a successful distribution and a local compile were
both silent, so the only visible dist signal was the failure path - leaving
no way to see the distribute/fallback ratio without the full debug firehose.
Diagnosing why a distributed build under-distributes meant guessing.
Promote the decision points to info and add a log on successful
distribution naming the server and exit code, so SCCACHE_LOG=info gives a
per-compile dist trace (attempt then distributed-on-server, compiled-locally,
or falling-back-with-reason). sccache only emits logs when SCCACHE_LOG is
set, so default runs are unaffected.
Signed-off-by: Javier Tia <javier@peridio.com>
(cherry picked from commit 950ddfdb3e801ecda2d07ade4f6651ffacc750cb)
A distributed compile the build-server rejects is often a distribution artifact, not a genuine compiler error: an object that .incbin's a binary the inputs packager does not ship - the kernel's vdso, embedded-config, and dtb wrappers - cannot be assembled remotely and returns non-zero, which failed the whole build with no recourse and forced the kernel to be excluded wholesale. Treat a non-zero remote result as a fallback trigger rather than a terminal error: recompile locally, which either succeeds (confirming a dist-only artifact) or reproduces the genuine error. A remote failure can no longer break a build that would compile locally. Only failing dist compiles are affected - successful ones are returned unchanged - so the kernel now distributes (1124/1128 compiles), with its handful of .incbin objects falling back to local. Signed-off-by: Javier Tia <javier@peridio.com> (cherry picked from commit b53367ce3083f7ca52861f84a89bbe530c5e80df)
A distributed compile can return a successful (exit 0) result yet omit a
declared output. glibc's ldconfig.o and sprof.o compile fine on the
build-server, but it does not return their `.o.dt` dependency file, so zipping
the compiler outputs fails fatally ("failed to open file ...o.dt: No such file")
with no recourse. One dropped output among 6427 forced glibc to be excluded from
distribution wholesale, even though 6425 of its compiles distribute cleanly.
Capture the declared output paths before the compilation is moved into the
packagers, and after a successful remote compile verify each one exists on disk.
A missing output is a distribution artifact, not a compiler error, so bail into
the existing local-recompile fallback (the same one that already salvages
non-zero remote results), which reproduces the full output set. Only compiles
whose dist output set is incomplete are affected; complete ones are returned
unchanged. glibc now distributes, with ldconfig.o and sprof.o falling back to
local.
Signed-off-by: Javier Tia <javier@peridio.com>
(cherry picked from commit 3b365f3dfa596c4b50815ac4424ee796503232f6)
When sccache runs as a compiler wrapper (`sccache <compiler> ...`) it
shares stderr with the wrapped compiler. A build's feature probes treat
any unexpected stderr as a compiler failure: libtool's -fPIC check
(_LT_COMPILER_OPTION) sets pic_flag="" whenever the probe compile emits
stderr, even on exit 0. With SCCACHE_LOG set, the client's own
env_logger records ("Attempting to read config file", "Server sent
CompileStarted") landed on that shared stderr, so an autotools package
configured with pic_flag="" and produced non-PIC objects that failed to
link into a shared library (relocation R_X86_64_32 ... recompile with
-fPIC). It looked concurrency-dependent because the client's startup
logging varies with server-connection timing, so the probe stderr only
sometimes diverged from libtool's expected boilerplate.
Initialize logging after parsing the command so the compile-wrapper case
can choose its target: route sccache's own records to SCCACHE_ERROR_LOG
when set, keep stderr only when it is an interactive terminal (so
`SCCACHE_LOG=debug sccache gcc ...` at a prompt still shows logs), and
discard them when stderr is captured by a build. The wrapped compiler's
forwarded stderr is untouched.
Signed-off-by: Javier Tia <javier@peridio.com>
(cherry picked from commit 63df688bd154dcc97a8b1a8f809d8cc92a952042)
Softening the rust dep-info rewriter (previous commit) was not enough to make rust distribute: a rust compile still fell back to local, now caught by the second guard. After a distributed compile the client verifies every declared output exists and, if one is missing, discards the result and recompiles locally. That guard ignored the `optional` flag, so a missing optional output forced a local recompile even though the output was declared droppable - stranding every rust compile on the client because the rust dep-info (.d) file, which the build server does not reliably return, was declared non-optional. Make the guard honor `optional`: only non-optional outputs must exist. The C dep file and glibc's dropped .o.dt stay non-optional, so their local-fallback (the reason that guard exists) is unchanged; a `-gsplit-dwarf` .dwo, already declared optional, no longer wrongly forces fallback. Then mark the rust .d optional, since it only feeds cargo rebuild tracking a from-scratch bitbake do_compile never consumes. With both, a distributed rust compile whose .d is absent keeps its object and rlib instead of recompiling locally. Signed-off-by: Javier Tia <floss@jetm.me> (cherry picked from commit 0ac0fe17badabb93fd8b22e348bb3f5995780877)
When a distributed compile exits non-zero the client logs only the exit code and falls back to a local recompile, discarding the remote compiler's stdout and stderr. That output is the only place the actual failure is visible - a missing target std, an input the packager did not ship, a sandbox path that does not resolve - so diagnosing a dist regression meant patching in a temporary dump and rebuilding the binary each time. Emit the remote stdout/stderr at debug on the non-zero path. SCCACHE_LOG=debug now surfaces the real remote error with no rebuild, while a normal SCCACHE_LOG=info run stays quiet (the one-line fallback reason is still logged at warn by the caller). Signed-off-by: Javier Tia <floss@jetm.me> (cherry picked from commit 1a4182c1db95b777f6e0ffcac11120e0f2730b0d)
16 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #2775 (splitting the OpenEmbedded/Yocto sccache-dist work into focused
PRs; replaces the closed #2750).
Problem
Any remote failure during a distributed compile - a non-zero result, a dropped
output, or unmatched metadata - failed the whole build instead of just the
distribution, so a single distributable input that misbehaved remotely broke the
build. Separately, sccache's own log output on stderr corrupted autoconf/libtool
probe output (for example a libtool
-fPICprobe reading wrapper logs offstderr).
Approach
output that did not materialize) as a trigger to recompile locally.
compile.
the fallback path is diagnosable.
Evidence
Local fallback is exercised repeatedly per cold Yocto build with zero build
failures; the libtool
-fPICprobe reproduction is fixed.