Commit 9428487
authored
* GH-3484 Eliminate per-page heap allocation for CRC32 checksums when using direct `ByteBufferAllocator`
Why this is safe
- CRC32.update(ByteBuffer) exists since Java 9, processes bytes from position to limit, advancing position.
- toByteBuffer(releaser) returns either a slice() of the internal buffer (independent position) or a freshly allocated copy. Either way, the original BytesInput is unaffected for the subsequent buf.collect() call, because ByteBufferBytesInput.writeInto() uses buffer.duplicate().
- When the allocator is direct, toByteBuffer(releaser) returns the direct buffer directly -- zero heap copy. When the allocator is heap-based, behavior is functionally equivalent to the old toByteArray() path.
- The releaser field already exists on ColumnChunkPageWriter (line 124) and manages buffer lifecycle.
* GH-3484 Release CRC32 ByteBuffers per page to avoid leak across column chunk
The previous change reused the ColumnChunkPageWriter's long-lived
ByteBufferReleaser for the CRC32 update buffers. That releaser only
closes when the column chunk writer closes (end of row group), so
every per-page allocation made by BytesInput.toByteBuffer(releaser)
accumulated until then. With many pages in a single large column
chunk this exhausted the heap (TestLargeColumnChunk OOM in CI).
Scope each CRC update to a try-with-resources ByteBufferReleaser so
allocated buffers are returned to the allocator immediately after
crc.update(). writePageV2 shares one per-page releaser across the
three updates.
1 parent 3e05546 commit 9428487
1 file changed
Lines changed: 15 additions & 9 deletions
Lines changed: 15 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
| 97 | + | |
97 | 98 | | |
98 | 99 | | |
99 | 100 | | |
| |||
121 | 122 | | |
122 | 123 | | |
123 | 124 | | |
| 125 | + | |
124 | 126 | | |
125 | 127 | | |
126 | 128 | | |
| |||
217 | 219 | | |
218 | 220 | | |
219 | 221 | | |
220 | | - | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
221 | 225 | | |
222 | 226 | | |
223 | 227 | | |
| |||
321 | 325 | | |
322 | 326 | | |
323 | 327 | | |
324 | | - | |
325 | | - | |
326 | | - | |
327 | | - | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
332 | 338 | | |
333 | 339 | | |
334 | 340 | | |
| |||
0 commit comments