Symptom
During a make bar run (in-VM bazel test //... --remote_cache=grpc://100.121.199.61:9092 --remote_upload_local_results=false), the foreign_cc LLVM cmake targets rebuild locally in the VM instead of hitting the remote cache:
[316 / 352] ... Foreign Cc - CMake: Building llvm; 667s disk-cache, remote-cache, darwin-sandbox
This added ~11 min (llvm + llvm_tblgen) and pushed the bar from ~17 min to ~27 min. Other 358 actions got 133 remote cache hits, so the cache works in general — //:llvm specifically misses.
Context
This surfaced right after #287-era work made zstd hermetic (LLVM_ENABLE_ZSTD=OFF, commit on mergequeue-vz-kit). The host (macbook) rebuilt //:llvm with the new cache_entries and uploaded with --remote_upload_local_results=true, but the VM did not get a cache hit for that target and rebuilt from source. Before the change, libLLVM.dylib was a remote cache hit (which is how the cache writer's Homebrew zstd path leaked into the VM in the first place), so host↔VM sharing for this target worked at least once previously.
Likely cause
rules_foreign_cc cmake() actions are notoriously non-deterministic across machines — absolute paths, CMake/Ninja discovery, compiler wrapper paths, and sandbox layout can all perturb the action key, so the host's uploaded key and the VM's computed key diverge.
Impact
Not a correctness issue. The bar still reports the right verdict. It just rebuilds LLVM (~11 min) on every VM run that doesn't already have a warm local cache, which slows the merge-queue gate.
Options to investigate
- Compare the
//:llvm action key host vs VM (bazel aquery) to find the diverging input.
- Make the mac mini (the cache server host) the canonical
//:llvm writer so its key matches future consumers, or pre-warm the VM's bazel disk-cache in the post-ios snapshot.
- Accept the per-run LLVM rebuild and bake a warm bazel cache into the VM image.
Found while validating the Bazel merge-queue bar (chunk D).
Symptom
During a
make barrun (in-VMbazel test //... --remote_cache=grpc://100.121.199.61:9092 --remote_upload_local_results=false), theforeign_ccLLVM cmake targets rebuild locally in the VM instead of hitting the remote cache:This added ~11 min (llvm + llvm_tblgen) and pushed the bar from ~17 min to ~27 min. Other 358 actions got 133 remote cache hits, so the cache works in general —
//:llvmspecifically misses.Context
This surfaced right after #287-era work made zstd hermetic (
LLVM_ENABLE_ZSTD=OFF, commit onmergequeue-vz-kit). The host (macbook) rebuilt//:llvmwith the new cache_entries and uploaded with--remote_upload_local_results=true, but the VM did not get a cache hit for that target and rebuilt from source. Before the change,libLLVM.dylibwas a remote cache hit (which is how the cache writer's Homebrew zstd path leaked into the VM in the first place), so host↔VM sharing for this target worked at least once previously.Likely cause
rules_foreign_cccmake() actions are notoriously non-deterministic across machines — absolute paths, CMake/Ninja discovery, compiler wrapper paths, and sandbox layout can all perturb the action key, so the host's uploaded key and the VM's computed key diverge.Impact
Not a correctness issue. The bar still reports the right verdict. It just rebuilds LLVM (~11 min) on every VM run that doesn't already have a warm local cache, which slows the merge-queue gate.
Options to investigate
//:llvmaction key host vs VM (bazel aquery) to find the diverging input.//:llvmwriter so its key matches future consumers, or pre-warm the VM's bazel disk-cache in thepost-iossnapshot.Found while validating the Bazel merge-queue bar (chunk D).