Skip to content

ci(docker): AUT-921 use COPY --link to keep uv sync layer cacheable#4973

Closed
svcnemo-autobot wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/ci-container-copy-link-cache
Closed

ci(docker): AUT-921 use COPY --link to keep uv sync layer cacheable#4973
svcnemo-autobot wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/ci-container-copy-link-cache

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Background

The CICD NeMo container build (cicd-container-build) re-runs the ~40 min uv sync layer on every build — including no-op main builds (observed: #16 DONE ~2400s on main→main with the registry cache present and the DeepEP layer #9 cached with identical digests).

Two compounding causes:

  1. COPY cache miss: the dependency-manifest COPY steps that gate uv sync do not cache-hit against the mode=max registry cache even with identical content and a cached parent — a known BuildKit registry-cache/COPY interaction. COPY --link makes each an independent, content-addressed layer (parent- and metadata-independent), the standard remedy.
  2. PR builds can't show it: the Compute cache keys step seeds every PR from main-buildcache and writes <PR>-buildcache, so a PR inherits only what main last built. Any Dockerfile-level cache fix therefore only takes effect after it lands on main and main rebuilds its cache — it is NOT observable on the PR build itself.

What changed

  • Add --link to the six dependency-manifest COPY steps and the final source COPY in docker/Dockerfile.ci.

Details

Without --link, a COPY is keyed to the exact parent-layer chain and forces BuildKit to pull and extract the large DeepEP RUN layer just to run the copy; across CI's ephemeral per-run buildx builders these COPY layers reliably cache-miss, invalidating the expensive uv sync layer beneath them. Output semantics are unchanged (overlay-merge into /opt/Megatron-Bridge).

Tested

Both cicd-container-build jobs (aws+gcp) built successfully with --link on the prior head, proving the change is build-correct. The cache SPEEDUP is not observable on the PR build (seeds from main-buildcache); it must be validated on main's 2nd build after merge. No controlled two-build reproduction was run on this infra.

The CICD NeMo container build re-runs the ~40 min `uv sync` layer on
every build — including no-op main builds — because the six
dependency-manifest COPY steps that gate it do not cache against the
registry buildcache.

Without --link a COPY is keyed to the exact parent-layer chain and
forces BuildKit to pull and extract the large DeepEP RUN layer above it
just to run the copy. Across the ephemeral per-run buildx builders that
CI creates, those COPY layers reliably cache-miss, which invalidates the
expensive `uv sync` layer below them.

Mark the manifest COPYs and the final source COPY with --link so each is
an independent, content-addressed layer rebased onto its parent. The
copies (and the uv sync layer beneath them) now hit the registry cache
whenever their inputs are unchanged, and BuildKit no longer has to
extract the DeepEP layer to run them.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
svcnemo-autobot requested a review from a team as a code owner July 20, 2026 22:21
@svcnemo-autobot svcnemo-autobot added area:build Dependencies, packaging, images, and environment setup ci CI, automation, test queue, or workflow infrastructure work labels Jul 20, 2026
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test bb47b41

nemo-autobot-origin Bot pushed a commit to svcnemo-autobot/Megatron-Bridge that referenced this pull request Jul 20, 2026
The CICD NeMo container build re-runs the ~40 min uv sync layer on every
build because the dependency-manifest COPY steps that gate it do not cache
against the registry buildcache. Mark the manifest COPYs and the final
source COPY with --link so each is an independent, content-addressed layer
rebased onto its parent, keeping the uv sync layer cacheable across builds.

Cherry-picked from NVIDIA-NeMo#4973 (bb47b41).

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
ko3n1g
ko3n1g previously approved these changes Jul 20, 2026
@ko3n1g
ko3n1g enabled auto-merge (squash) July 20, 2026 22:45
@ko3n1g
ko3n1g disabled auto-merge July 20, 2026 22:45
@yaoyu-33 yaoyu-33 added the ready-to-merge PR is approved, current, and only waiting for CI to pass before merge label Jul 20, 2026
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

CI note: the cicd-container-build jobs (aws + gcp) — the gate this COPY --link change targets — are green, so the fix builds correctly.

The red Launch_Unit_Tests_Core / Nemo_CICD_Test are a pre-existing failure on main, unrelated to this PR (which touches only docker/Dockerfile.ci):

tests/unit_tests/recipes/test_perf_recipe_environment.py::test_every_flat_recipe_builder_declares_its_environment_inline
tests/unit_tests/recipes/test_perf_recipe_environment.py::test_explicit_environment_invariants_across_all_flat_recipes
E   AssertionError: assert 397 == 396   (test_perf_recipe_environment.py:147 & :188)

The test hard-codes an expected perf-recipe-builder count of 396, but the repo now has 397 (a recipe builder was added without bumping the count). The same two tests fail identically on main's own CI (jobs 88494647557, 88466552646), so a CI re-run will not clear it and it is not caused by this change.

@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test bb47b41

@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test e8d764f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build Dependencies, packaging, images, and environment setup ci CI, automation, test queue, or workflow infrastructure work ready-to-merge PR is approved, current, and only waiting for CI to pass before merge Run CICD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants