fix(build): dependency-aware cache key so dependents rebuild on dep change#5425
Merged
Conversation
…hange
The build's rebuild-vs-reuse decision hashed only a package's OWN source
(`getPackageSourceHash`) and compared to the stored hash. It ignored
dependencies — so when a package's exports changed, any dependent whose own
source was untouched stayed a cache hit and was never rebuilt, leaving its dist
compiled against the old dependency API (dangling imports / stubs). Whack-a-mole
after every merge.
Introduce an effective build key:
key(pkg) = hash(own source hash + sorted keys of its workspace deps)
Computed once over the toposort (deps before dependents), stored in build meta.
Any transitive dependency change flips a dependent's key → it becomes a cache
miss and rebuilds — on a plain `yarn build`, no `--rebuild-dependents` needed.
Relies on the workspace dependency graph (package.json), which `adio` keeps in
sync with actual imports.
Verified: editing @webiny/error rebuilds it + its 108 dependents (was 1);
147 keys compute in ~1s (no measurable overhead).
`--rebuild-dependents` is now redundant but left in place; it (and the CI flag)
can be removed in a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The effective key hashed own source + workspace dep keys, and captured third-party deps only via the package.json range (part of the own-source hash). That missed the case where a range resolves to new code without any package.json edit — a floating range picking up a release, or a `yarn.lock` update. Dependents of that resolution weren't rebuilt. Parse `yarn.lock` and fold each package's *resolved* third-party dependency tokens (checksums) into its key. A lockfile change invalidates only the packages that actually declare the changed dependency — plus their dependents — not the whole repo. Verified: bumping a niche dep's checksum flips only its declarer + dependents (67/147); bumping `typescript` (a near-universal devDep) flips 142/147. Key compute stays ~1.1s. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the dependency-aware effective key opt-in so it can merge without affecting anyone by default, and be validated locally first. - default (flag off): build key = own-source hash — byte-identical to the previous behavior, so merging forces no rebuild; `--rebuild-dependents` remains the mechanism for rebuilding dependents. - WEBINY_EXPERIMENTAL_DEP_AWARE_CACHE=true: build key folds in transitive workspace dep keys + resolved third-party deps, so a plain `yarn build` rebuilds dependents of any changed package. Verified: flag-off own hashes equal getPackageSourceHash (no forced rebuild); both modes run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The build decides rebuild-vs-reuse per package by hashing only that package's own
src(getPackageSourceHash) and comparing to the hash inmeta.json. It ignores dependencies. So when package P changes its exports, any dependent Q whose own source is untouched stays a cache hit → never rebuilt, and itsdistremains compiled against P's old API → dangling imports / stubs. Whack-a-mole after every merge.--rebuild-dependentsonly patches this if run on the exact first build after the change (it anchors off transient cache misses, and a plain build records the new hash, erasing the signal).Fix — dependency-aware effective key
Computed once over the toposort (deps first), stored in build meta. Any transitive workspace dep change — or a resolved third-party dep change (parsed from
yarn.lock, so floating ranges / lockfile updates count too) — flips a dependent's key → cache miss → rebuilds, on a plainyarn build, no flag.Relies on the workspace dependency graph (
package.json), whichadiokeeps in sync with real imports.⚑ Behind an experimental flag — OFF by default
To merge safely and let it be validated locally first, the dep-aware key is opt-in:
--rebuild-dependentsremains the mechanism for rebuilding dependents.WEBINY_EXPERIMENTAL_DEP_AWARE_CACHE=true(or1): the dependency-aware key above is used.Try it locally:
Verified
@webiny/error→ plainyarn buildrebuilds 109 packages (error + 108 dependents); own-hash rebuilt 1, leaving 108 stale.typescript(near-universal devDep) → 142.getOwnHashesequalsgetPackageSourceHash(byte-identical → no forced rebuild); both modes run.Scope / follow-up
--rebuild-dependents(getBatches step 1.5 + CI flag) stays while the flag is off — it's the safety net. Removing it (was refactor(build): remove --rebuild-dependents (redundant after dep-aware key) #5427, now closed) happens once the flag is promoted to default-on.yarn.locktokens.🤖 Generated with Claude Code