Skip to content

Tiered: heal divergent lower tier detected by pinned range reads #368

Description

@alecthomas

Context

#367 made the Tiered cache probe deeper tiers when a request's If-Match/If-Range validator names a version the local tier does not hold. A replica whose disk tier has diverged from the shared tier (e.g. S3) now serves version-pinned range requests — client.ParallelGet chunks — from the deeper tier instead of rejecting them or serving the wrong version.

Gap

Partial bodies are never backfilled (correct: a range must not be stored as if it were the whole object), so serving pinned chunks does not repair the divergent tier-0 entry. An unranged pinned read heals it via the normal backfill (see the IfMatchFullReadBackfillsLower test in internal/cache/tiered_test.go), but the parallel-download path is all ranged reads, so:

  • every chunk pays a wasted tier-0 open plus a deeper-tier fetch, for the whole download, for every client, and
  • the stale entry lingers until TTL expiry, eviction, or the next write-through — unpinned full reads still serve the stale tier-0 copy.

For large snapshot downloads against a stale replica this is sustained probe amplification with no convergence.

Proposal

When Tiered.Open detects divergence (tier 0 fails the validator while a deeper tier satisfies it), trigger an asynchronous full-object refresh of the lowest tier from the deeper tier. Points to get right:

  • Only refresh when the deeper tier's current version differs from tier 0's — never replace a local version that is newer than the client's pin (the pin may itself be stale; a lagging shared-tier write can make tier 0 legitimately ahead).
  • Coalesce concurrent triggers per key (singleflight) and bound refresh concurrency/bandwidth so a popular stale object doesn't stampede the shared tier.
  • Deleting the stale tier-0 entry instead of refreshing was considered in fix(cache): consult deeper tiers on ETag mismatch #367 and rejected for the same tier-0-may-be-newer reason.

🤖 Filed by Alec's AI agent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions