Commit ded1a47
committed
apacheGH-3522: Use pack32Values and cache packers in delta writers
Delta binary packing writers encoded each 32-value mini block with four
pack8Values calls despite the generated packers exposing pack32Values.
Switch both int and long writers to the 32-value entry point so each
mini block is packed in a single call.
Also cache BytePacker/BytePackerForLong instances in per-instance arrays
indexed by bit width, avoiding the factory dispatch per miniblock flush.
This resolves the TODO at DeltaBinaryPackingValuesWriterForLong.java:119.
Combined benchmark for both par12 commits (decode batch unpack32 + encode
pack32Values + cached packers):
IntEncodingBenchmark (100k INT32 values, JMH -wi 3 -i 5 -f 1):
Benchmark Pattern Before (ops/s) After (ops/s) Improvement
decodeDelta SEQUENTIAL 903,892,292 1,096,895,285 +21% (1.21x)
decodeDelta RANDOM 364,659,977 410,632,530 +13% (1.13x)
decodeDelta LOW_CARDINALITY 581,649,861 676,449,008 +16% (1.16x)
decodeDelta HIGH_CARDINALITY 370,718,831 506,116,434 +37% (1.37x)
encodeDelta SEQUENTIAL 556,155,088 558,868,426 flat
encodeDelta RANDOM 360,327,834 376,594,239 +5%
encodeDelta LOW_CARDINALITY 412,396,181 434,569,306 +5%
encodeDelta HIGH_CARDINALITY 335,702,852 345,528,410 +3%
The decode path shows the larger gains because it eliminates per-miniblock
ByteBuffer.slice() allocations and uses the faster byte[] unpack path.
Encode gains are modest because pack32Values is structurally similar to 4x
pack8Values at this optimization level.
All 573 parquet-column tests pass.1 parent 47bcad3 commit ded1a47
2 files changed
Lines changed: 30 additions & 24 deletions
File tree
- parquet-column/src/main/java/org/apache/parquet/column/values/delta
Lines changed: 15 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
63 | 69 | | |
64 | 70 | | |
65 | 71 | | |
| |||
116 | 122 | | |
117 | 123 | | |
118 | 124 | | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
129 | 129 | | |
130 | | - | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
131 | 135 | | |
132 | 136 | | |
133 | 137 | | |
| |||
Lines changed: 15 additions & 13 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
63 | 69 | | |
64 | 70 | | |
65 | 71 | | |
| |||
116 | 122 | | |
117 | 123 | | |
118 | 124 | | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
131 | 129 | | |
132 | | - | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
133 | 135 | | |
134 | 136 | | |
135 | 137 | | |
| |||
0 commit comments