Commit 961ed7c
[IcebergIO] Fix stale openWriters count in RecordWriterManager (#39157)
RecordWriterManager uses a Guava Cache with expireAfterAccess(1 min)
for writer eviction. However, Guava Cache eviction is lazy — expired
entries are only removed on subsequent access or explicit cleanUp().
The write() method checks `openWriters >= maxNumWriters` without first
calling cleanUp(), so expired writers remain in the cache and
openWriters stays stale.
With DEFAULT_MAX_WRITERS_PER_BUNDLE = 20 and more than 20 partitions,
this causes 100% false spill — every record for the 21st+ partition
is rejected as if the writer pool is full.
Add writers.cleanUp() before the capacity check to evict expired
entries and update openWriters accurately.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 402b41c commit 961ed7c
1 file changed
Lines changed: 4 additions & 1 deletion
Lines changed: 4 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
163 | 163 | | |
164 | 164 | | |
165 | 165 | | |
166 | | - | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
167 | 170 | | |
168 | 171 | | |
169 | 172 | | |
| |||
0 commit comments