ci(docker): AUT-921 use COPY --link to keep uv sync layer cacheable#4973
Closed
svcnemo-autobot wants to merge 2 commits into
Closed
ci(docker): AUT-921 use COPY --link to keep uv sync layer cacheable#4973svcnemo-autobot wants to merge 2 commits into
svcnemo-autobot wants to merge 2 commits into
Conversation
The CICD NeMo container build re-runs the ~40 min `uv sync` layer on every build — including no-op main builds — because the six dependency-manifest COPY steps that gate it do not cache against the registry buildcache. Without --link a COPY is keyed to the exact parent-layer chain and forces BuildKit to pull and extract the large DeepEP RUN layer above it just to run the copy. Across the ephemeral per-run buildx builders that CI creates, those COPY layers reliably cache-miss, which invalidates the expensive `uv sync` layer below them. Mark the manifest COPYs and the final source COPY with --link so each is an independent, content-addressed layer rebased onto its parent. The copies (and the uv sync layer beneath them) now hit the registry cache whenever their inputs are unchanged, and BuildKit no longer has to extract the DeepEP layer to run them. Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Collaborator
Author
|
/ok to test bb47b41 |
nemo-autobot-origin Bot
pushed a commit
to svcnemo-autobot/Megatron-Bridge
that referenced
this pull request
Jul 20, 2026
The CICD NeMo container build re-runs the ~40 min uv sync layer on every build because the dependency-manifest COPY steps that gate it do not cache against the registry buildcache. Mark the manifest COPYs and the final source COPY with --link so each is an independent, content-addressed layer rebased onto its parent, keeping the uv sync layer cacheable across builds. Cherry-picked from NVIDIA-NeMo#4973 (bb47b41). Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
ko3n1g
previously approved these changes
Jul 20, 2026
ko3n1g
enabled auto-merge (squash)
July 20, 2026 22:45
ko3n1g
disabled auto-merge
July 20, 2026 22:45
Collaborator
Author
|
CI note: the The red The test hard-codes an expected perf-recipe-builder count of 396, but the repo now has 397 (a recipe builder was added without bumping the count). The same two tests fail identically on |
Collaborator
Author
|
/ok to test bb47b41 |
nemo-autobot-origin
Bot
force-pushed
the
fix/ci-container-copy-link-cache
branch
from
July 21, 2026 00:28
1d8c470 to
e8d764f
Compare
Collaborator
Author
|
/ok to test e8d764f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The
CICD NeMocontainer build (cicd-container-build) re-runs the ~40 minuv synclayer on every build — including no-opmainbuilds (observed:#16 DONE ~2400son main→main with the registry cache present and the DeepEP layer#9cached with identical digests).Two compounding causes:
COPYsteps that gateuv syncdo not cache-hit against themode=maxregistry cache even with identical content and a cached parent — a known BuildKit registry-cache/COPYinteraction.COPY --linkmakes each an independent, content-addressed layer (parent- and metadata-independent), the standard remedy.Compute cache keysstep seeds every PR frommain-buildcacheand writes<PR>-buildcache, so a PR inherits only what main last built. Any Dockerfile-level cache fix therefore only takes effect after it lands onmainand main rebuilds its cache — it is NOT observable on the PR build itself.What changed
--linkto the six dependency-manifestCOPYsteps and the final sourceCOPYindocker/Dockerfile.ci.Details
Without
--link, aCOPYis keyed to the exact parent-layer chain and forces BuildKit to pull and extract the large DeepEP RUN layer just to run the copy; across CI's ephemeral per-run buildx builders these COPY layers reliably cache-miss, invalidating the expensiveuv synclayer beneath them. Output semantics are unchanged (overlay-merge into/opt/Megatron-Bridge).Tested
Both
cicd-container-buildjobs (aws+gcp) built successfully with--linkon the prior head, proving the change is build-correct. The cache SPEEDUP is not observable on the PR build (seeds frommain-buildcache); it must be validated onmain's 2nd build after merge. No controlled two-build reproduction was run on this infra.