feat(container): restrict recipe stage to manifest-only COPY to prevent spurious cook cache invalidation#1863
Merged
josecelano merged 4 commits intoJun 2, 2026
Conversation
There was a problem hiding this comment.
Pull request overview
This PR optimizes the container build cache behavior for Rust dependency compilation by changing the recipe stage to copy only Cargo manifests (and Cargo.lock) before running cargo chef prepare, reducing unnecessary downstream cache invalidation on source-only edits.
Changes:
- Replaced
COPY . /build/srcwith explicit manifest-onlyCOPYlines in the Containerfilerecipestage. - Added an inline maintenance comment describing how to keep the manifest list updated.
- Updated the issue spec to mark the implementation tasks / acceptance criteria as completed.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
Containerfile |
Restricts the recipe stage inputs to Cargo manifests + lockfile to stabilize cargo chef cook caching. |
docs/issues/open/1852-1840-workflow-performance-recipe-stage-manifest-only-copy/ISSUE.md |
Updates task/AC status and progress log to reflect the implementation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1863 +/- ##
===========================================
- Coverage 77.79% 77.79% -0.01%
===========================================
Files 382 382
Lines 28648 28648
Branches 28648 28648
===========================================
- Hits 22288 22287 -1
Misses 6051 6051
- Partials 309 310 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
josecelano
added a commit
to josecelano/torrust-tracker
that referenced
this pull request
Jun 2, 2026
… stage comment The maintenance comment and ISSUE.md AC1 evidence referred to 'workspace members in the root Cargo.toml', which implies the [workspace].members key (only 4 entries). The COPY list actually covers all in-repo path crates auto-discovered by Cargo via path dependencies — i.e. every package returned by 'cargo metadata --no-deps' whose manifest is inside this repo. Clarify wording to 'in-repo path crates (packages/, console/, contrib/)' and explain that [workspace].members is a much smaller set and should not be used as the authoritative list. Addresses Copilot review comments on PR torrust#1863.
…nt spurious cook cache invalidation Replaces the full-tree `COPY . /build/src` in the `recipe` stage with individual `COPY <manifest> <dest>/` lines for every workspace member. This prevents Docker from invalidating the expensive `cargo chef cook` dependency layers on every source-code change. Cook layers are now only invalidated when `Cargo.toml` or `Cargo.lock` actually changes. Closes torrust#1852
…ata validation cargo chef prepare calls cargo metadata internally, which validates that all explicitly-declared [lib] and [[bench]] targets have their source files present on disk. The manifest-only copy leaves those files absent, causing cargo metadata to fail with 'can't find library'. Add a RUN step that creates minimal empty stub files for the 7 targets that are explicitly declared in workspace Cargo.toml files: - src/lib.rs (root [lib]) - packages/tracker-client/src/lib.rs ([lib]) - console/tracker-client/src/lib.rs ([lib]) - packages/http-tracker-core/benches/http_tracker_core_benchmark.rs ([[bench]]) - packages/udp-tracker-core/benches/udp_tracker_core_benchmark.rs ([[bench]]) - packages/torrent-repository-benchmarking/benches/repository_benchmark.rs ([[bench]]) - contrib/bencode/benches/bencode_benchmark.rs ([[bench]]) Auto-detected targets (no explicit Cargo.toml section) are intentionally excluded: cargo metadata silently omits undiscovered targets without failing, and creating stubs for them risks leaving phantom source files in the build stage after COPY overwrites the directory (COPY does not delete files already in the base image). Closes torrust#1852 (partial - CI verification pending)
… stage comment The maintenance comment and ISSUE.md AC1 evidence referred to 'workspace members in the root Cargo.toml', which implies the [workspace].members key (only 4 entries). The COPY list actually covers all in-repo path crates auto-discovered by Cargo via path dependencies — i.e. every package returned by 'cargo metadata --no-deps' whose manifest is inside this repo. Clarify wording to 'in-repo path crates (packages/, console/, contrib/)' and explain that [workspace].members is a much smaller set and should not be used as the authoritative list. Addresses Copilot review comments on PR torrust#1863.
cargo metadata requires every package to have at least one resolvable target file on disk, regardless of whether the target is explicitly declared in Cargo.toml or auto-detected (src/lib.rs, src/main.rs, src/bin/*.rs, examples/, benches/). The previous stub step only covered 7 explicitly-declared [lib] and [[bench]] targets. Auto-detected targets were incorrectly assumed to be silently skipped — in practice cargo metadata aborts with 'no targets specified in the manifest' for any package whose auto-detected source files are absent. Additional failures discovered by local reproduction: - root default-run = 'torrust-tracker' → needs src/main.rs - root src/bin/*.rs binaries (4 files) - all packages/*/src/lib.rs (auto-detected libs, 20 files) - console/tracker-client/src/bin/*.rs (4 binaries) - contrib/bencode/src/lib.rs - contrib/dev-tools/analysis/workspace-coupling/src/main.rs - packages/axum-http-server/examples/http_only_public_tracker.rs - packages/tracker-core/src/bin/persistence_benchmark_runner.rs - packages/udp-server/examples/udp_only_public_tracker.rs The canonical list is derived from: cargo metadata --no-deps --format-version 1 | jq -r '.packages[].targets[].src_path' filtered to paths inside this repository.
josecelano
force-pushed
the
1852-recipe-stage-manifest-only-copy
branch
from
2345dca to
2af6803
Compare
Member
Author
|
ACK 2af6803 |
josecelano
added a commit
to josecelano/torrust-tracker
that referenced
this pull request
Jun 2, 2026
…oses torrust#1853) Remove `--benches --examples --all-targets` from all 6 cargo commands in the Containerfile (`cargo chef cook` × 2, warmup `cargo nextest archive` × 2, final `cargo nextest archive` × 2). Baseline analysis (torrust#1841) confirmed 27/30 top compile units are unrelated to the runtime image; benchmarks and examples alone account for ~250–300 s of avoidable link cost per profile. Cold no-cache release build succeeds in 5m23s with narrowed scope. Required executables (`torrust-tracker`, `http_health_check`) are present in the 173 MB release image. Also fixes a pre-existing `.dockerignore` bug: the audit in torrust#1851 excluded all of `contrib/dev-tools/` except `su-exec/`, but PR torrust#1863 (issue torrust#1852) later added a recipe-stage `COPY` for `contrib/dev-tools/analysis/workspace-coupling/Cargo.toml`. That path is a workspace member so `cargo chef prepare` requires its manifest. Add `!/contrib/dev-tools/analysis/workspace-coupling/Cargo.toml` to unblock local cold builds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1852
Replaces the full-tree
COPY . /build/srcin therecipestage of theContainerfilewith individualCOPY <manifest> <dest>/lines for every workspace member.Problem
The
cargo chef preparecommand only needsCargo.tomlmanifests andCargo.lockto producerecipe.json. However, the previousCOPY . /build/srccopied the entire source tree into the recipe stage, causing Docker to invalidate that layer — and the expensive downstreamcargo chef cookdependency layers — on every source-code change.Solution
Replaced the single full-tree copy with manifest-only COPY instructions for all 28 workspace members plus root
Cargo.tomlandCargo.lock. Added a maintenance comment explaining the sync requirement.Effect
dependenciesanddependencies_debug) now stay cached for source-only edits (noCargo.toml/Cargo.lockchanges).Maintenance
The manifest list must be kept in sync with the workspace members in
Cargo.toml. A comment block in the Containerfile documents this requirement.Checklist
linter allexits0Cargo.tomlfiles are listed (verified withcargo metadata)