Goal
Determine whether lto = "fat" in [profile.dev] is still necessary, and remove or replace it with an appropriate setting that does not unnecessarily slow down development builds.
Background
Commit 3c715fbb (fix: [#898] docker build error: failed to load bitcode of module criterion) changed lto = "thin" to lto = "fat" in [profile.dev] as a workaround for an LLVM bitcode compatibility error that occurred when building benchmarks (criterion) inside a Docker container with Rust 1.79/1.81-nightly (mid-2024):
error: failed to load bitcode of module "criterion-...": failed to load bitcode
The root cause was an LLVM cross-module bitcode version mismatch triggered by lto = "thin" when mixing crates compiled with different LLVM/rustc versions inside a container build. Switching to "fat" forced all bitcode into a single monolithic unit, eliminating the cross-module issue.
This was a legitimate workaround at the time but carries a significant cost: lto = "fat" in [profile.dev] applies full-program LTO to every incremental development build, substantially increasing compile times with no benefit for day-to-day development iteration.
The project now targets MSRV 1.88 (as of 2026-06). The LLVM version bundled with Rust 1.88 is well past the version where this bug was observed, and the Containerfile now builds with the stable toolchain. The original triggering conditions may no longer exist.
This also affects CI: testing.yaml runs tests without --release (using [profile.dev]), and the Containerfile has a cargo chef cook debug step — both carry the fat LTO cost unnecessarily. This is a child issue of the PR Workflow Performance EPIC (#1840).
Scope
In Scope
- Investigate whether removing
lto = "fat" from [profile.dev] still causes the Docker build to fail with current Rust/LLVM versions
- If the bug is gone: remove
lto = "fat" from [profile.dev] (restore lto = "thin" or remove the key to use the Cargo default of false)
- If the bug persists: document exactly why, pin the minimum fix to the narrowest possible scope (e.g. only the benchmark crate, only the release profile, or via a per-crate override), and open a follow-up tracking upstream resolution
- Keep
lto = "fat" in [profile.release] — it is appropriate there for production binary optimization
Out of Scope
- Changing
[profile.release] LTO settings
- Restructuring the Containerfile beyond what is required to verify the fix
Implementation Plan
| ID |
Task |
Notes |
| T1 |
Reproduce the original failure (optional, low priority) |
Confirm what Rust/LLVM version combination triggers the bitcode error, if reproducible at all |
| T2 |
Remove lto = "fat" from [profile.dev] (or restore lto = "thin") |
Cargo.toml [profile.dev] no longer carries fat LTO |
| T3 |
Run the full local test suite with the updated dev profile |
cargo test --tests --benches --examples --workspace --all-targets --all-features exits with code 0 |
| T4 |
Run the Docker build with the updated dev profile to verify no bitcode error |
docker build --target release ... completes without failed to load bitcode error |
| T5 |
If T4 fails: scope the workaround narrowly and document the upstream tracking issue |
Narrowest fix applied; comment in Cargo.toml explains why with a link |
| T6 |
Run pre-commit checks |
./contrib/dev-tools/git/hooks/pre-commit.sh exits with code 0 |
References
Goal
Determine whether
lto = "fat"in[profile.dev]is still necessary, and remove or replace it with an appropriate setting that does not unnecessarily slow down development builds.Background
Commit
3c715fbb(fix: [#898] docker build error: failed to load bitcode of module criterion) changedlto = "thin"tolto = "fat"in[profile.dev]as a workaround for an LLVM bitcode compatibility error that occurred when building benchmarks (criterion) inside a Docker container with Rust 1.79/1.81-nightly (mid-2024):The root cause was an LLVM cross-module bitcode version mismatch triggered by
lto = "thin"when mixing crates compiled with different LLVM/rustc versions inside a container build. Switching to"fat"forced all bitcode into a single monolithic unit, eliminating the cross-module issue.This was a legitimate workaround at the time but carries a significant cost:
lto = "fat"in[profile.dev]applies full-program LTO to every incremental development build, substantially increasing compile times with no benefit for day-to-day development iteration.The project now targets MSRV 1.88 (as of 2026-06). The LLVM version bundled with Rust 1.88 is well past the version where this bug was observed, and the
Containerfilenow builds with the stable toolchain. The original triggering conditions may no longer exist.This also affects CI:
testing.yamlruns tests without--release(using[profile.dev]), and theContainerfilehas acargo chef cookdebug step — both carry the fat LTO cost unnecessarily. This is a child issue of the PR Workflow Performance EPIC (#1840).Scope
In Scope
lto = "fat"from[profile.dev]still causes the Docker build to fail with current Rust/LLVM versionslto = "fat"from[profile.dev](restorelto = "thin"or remove the key to use the Cargo default offalse)lto = "fat"in[profile.release]— it is appropriate there for production binary optimizationOut of Scope
[profile.release]LTO settingsImplementation Plan
lto = "fat"from[profile.dev](or restorelto = "thin")Cargo.toml[profile.dev]no longer carries fat LTOcargo test --tests --benches --examples --workspace --all-targets --all-featuresexits with code 0docker build --target release ...completes withoutfailed to load bitcodeerrorCargo.tomlexplains why with a link./contrib/dev-tools/git/hooks/pre-commit.shexits with code 0References
3c715fbb— original workaround: "fix: [Docker build is failing:failed to load bitcode of module "criterion-af9a3f7183f1573d.criterion.b69900c842eb33fa-cgu.08.rcgu.o"#898] docker build error: failed to load bitcode of module criterion"