Summary

This change fixes cache updates being missed after cache application moved off the request hot path. It replaces the lossy async updater with a lossless queue, skips truly empty async cache updates, moves endpoint lifecycle reconciliation off the cache-apply critical path

Problem

After moving cache application off the request hot path, cache updates could either be lost or applied too slowly. In production this showed up as:

• cache update backlog growing continuously
• pending cache update objects increasing memory usage
• long delays in applying updates on the executor path
• logs showing many cache updates with little or no change in cache size

We also observed that a large number of responses carried a cache_update field that was present but effectively empty, which added queue pressure without changing routing state.

Root Cause

There were a few separate issues:

• async cache update handling used a lossy pending-update model, so intermediate updates could be overwritten before being applied
• databaseId=0 updates could clear an already initialized finder state even when they did not carry meaningful cache contents
• endpoint lifecycle reconciliation still ran in the cache update path, adding unnecessary work to cache application
• many empty cache updates were still being enqueued and drained even though they were no-ops

What This PR Changes

• replace the lossy async cache updater with a lossless FIFO queue so all material cache updates are preserved
• keep zero-database-id updates from clearing existing cache state
• skip truly empty async cache updates while still preserving:
  • all non-empty updates
  • database-id-only transitions that can invalidate old cache state
• batch queue draining to reduce scheduling and lock churn
• decouple lifecycle address reconciliation from the cache update critical section

Result

Based on production logs after the fix:

• empty cache updates are being skipped in large numbers
• material pending cache updates are shrinking instead of growing
• cache apply time remains very small
• routing cache size remains stable and traffic converges away from the default endpoint quickly

This indicates the main issue was queue pressure from lossy handling and empty/no-op updates, not slow cache map mutation itself.