Commit 7e68126
fix(variant): support nested variant access in ROW/ARRAY/MAP reads
A variant-access projection carries the variant extension marker, so
IsVariantField matched it and BuildNestedVariantPlan planned a nested
projection as a full variant: it reassembled struct<value, metadata> into a
column declared with the access struct type, and on an unshredded file it
built no plan at all. Recognise the access type at every nesting level and
share one leaf-plan factory between the top level and nested positions, as
Java does in VariantShreddingReadPlanFactory.
Descend through LIST and MAP as well. The parquet reader rejects partial
projection inside a repeated group, so the file subtree is pushed down
verbatim there and only reassembled back; ChildrenLineUp keeps that safe by
requiring the read and file children to line up one to one.
PruneDataType then has to let such a read through: it fails fast on any
projection inside a repeated group, which blocked the whole table read path.
Replacing a variant by its access projection drops no field, so it passes
through while a real partial projection keeps failing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 4e55a24 commit 7e68126
5 files changed
Lines changed: 955 additions & 132 deletions
File tree
- src/paimon
- common/data/variant
- core/utils
- format/parquet
- test/inte
0 commit comments