Commit 3006be6
authored
Updated Variant array and the new VariantGet expression (#7877)
## Summary
This PR includes two big changes as Variant moves closer to readiness.
1. Potentially holding the `shredded` child of a variant array in the
canonical VariantArray
2. A `VariantGet` expression that can pull extract data out of variant
arrays, either in a typed way or as a more opaque `Variant`.
For reviewers, some relevant context might be:
1. The [VariantGet](vortex-data/rfcs#58) RFC:
this RFC takes some lessons I've learned working on this into account
and reflects my updated view of this problem.
2. The original [Variant
type](https://vortex-data.github.io/rfcs/rfc/0015.html) RFC
I think the Parquet spec is also a pretty good read and a very heavy
influence of this work -
[`Shredding`](https://parquet.apache.org/docs/file-format/types/variantshredding/)
and the [`Variant
type`](https://parquet.apache.org/docs/file-format/types/variantencoding/).
---------
Signed-off-by: "Adam Gutglick" <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>1 parent b71de62 commit 3006be6
34 files changed
Lines changed: 3984 additions & 427 deletions
File tree
- encodings/parquet-variant
- src
- vortex-array
- src
- aggregate_fn/fns/uncompressed_size_in_bytes
- arrays
- chunked/vtable
- constant/vtable
- dict
- filter/execute
- masked
- struct_/vtable
- variant
- compute
- vtable
- expr
- scalar_fn
- fns
- variant_get
- vortex-proto
- proto
- src/generated
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
37 | 38 | | |
38 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
0 commit comments