You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#367 made the Tiered cache probe deeper tiers when a request's If-Match/If-Range validator names a version the local tier does not hold. A replica whose disk tier has diverged from the shared tier (e.g. S3) now serves version-pinned range requests — client.ParallelGet chunks — from the deeper tier instead of rejecting them or serving the wrong version.
Gap
Partial bodies are never backfilled (correct: a range must not be stored as if it were the whole object), so serving pinned chunks does not repair the divergent tier-0 entry. An unranged pinned read heals it via the normal backfill (see the IfMatchFullReadBackfillsLower test in internal/cache/tiered_test.go), but the parallel-download path is all ranged reads, so:
every chunk pays a wasted tier-0 open plus a deeper-tier fetch, for the whole download, for every client, and
the stale entry lingers until TTL expiry, eviction, or the next write-through — unpinned full reads still serve the stale tier-0 copy.
For large snapshot downloads against a stale replica this is sustained probe amplification with no convergence.
Proposal
When Tiered.Open detects divergence (tier 0 fails the validator while a deeper tier satisfies it), trigger an asynchronous full-object refresh of the lowest tier from the deeper tier. Points to get right:
Only refresh when the deeper tier's current version differs from tier 0's — never replace a local version that is newer than the client's pin (the pin may itself be stale; a lagging shared-tier write can make tier 0 legitimately ahead).
Coalesce concurrent triggers per key (singleflight) and bound refresh concurrency/bandwidth so a popular stale object doesn't stampede the shared tier.
Context
#367 made the
Tieredcache probe deeper tiers when a request'sIf-Match/If-Rangevalidator names a version the local tier does not hold. A replica whose disk tier has diverged from the shared tier (e.g. S3) now serves version-pinned range requests —client.ParallelGetchunks — from the deeper tier instead of rejecting them or serving the wrong version.Gap
Partial bodies are never backfilled (correct: a range must not be stored as if it were the whole object), so serving pinned chunks does not repair the divergent tier-0 entry. An unranged pinned read heals it via the normal backfill (see the
IfMatchFullReadBackfillsLowertest ininternal/cache/tiered_test.go), but the parallel-download path is all ranged reads, so:For large snapshot downloads against a stale replica this is sustained probe amplification with no convergence.
Proposal
When
Tiered.Opendetects divergence (tier 0 fails the validator while a deeper tier satisfies it), trigger an asynchronous full-object refresh of the lowest tier from the deeper tier. Points to get right:🤖 Filed by Alec's AI agent.