Commit 3ad38d0
authored
fix(parquet/file): use adaptive batch sizing to avoid panic (#690)
### Rationale for this change
Issue reported in
#622 (comment)
where accumulated data on a given page exceeds the DataPageSize.
### What changes are included in this PR?
Removing a broken mid-batch flush in `writeValues`/`writeValuesSpaced`,
instead relying back on `encoder.Put()/encoder.PutSpaced()`.
Updated `WriteBatch` to use an adaptive batch sizing approach for
ByteArray/FLBA writing to properly handle v2 data page row-boundary
alignment without breaking on very large individual values.
### Are these changes tested?
New tests are added to cover this scenario to ensure test coverage.
### Are there any user-facing changes?
No1 parent 3194e44 commit 3ad38d0
4 files changed
Lines changed: 532 additions & 330 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
| 24 | + | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
| |||
303 | 305 | | |
304 | 306 | | |
305 | 307 | | |
306 | | - | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
307 | 314 | | |
308 | 315 | | |
309 | 316 | | |
| |||
378 | 385 | | |
379 | 386 | | |
380 | 387 | | |
381 | | - | |
| 388 | + | |
382 | 389 | | |
383 | 390 | | |
384 | 391 | | |
| |||
0 commit comments