Skip to content

Commit 3006be6

Browse files
authored
Updated Variant array and the new VariantGet expression (#7877)
## Summary This PR includes two big changes as Variant moves closer to readiness. 1. Potentially holding the `shredded` child of a variant array in the canonical VariantArray 2. A `VariantGet` expression that can pull extract data out of variant arrays, either in a typed way or as a more opaque `Variant`. For reviewers, some relevant context might be: 1. The [VariantGet](vortex-data/rfcs#58) RFC: this RFC takes some lessons I've learned working on this into account and reflects my updated view of this problem. 2. The original [Variant type](https://vortex-data.github.io/rfcs/rfc/0015.html) RFC I think the Parquet spec is also a pretty good read and a very heavy influence of this work - [`Shredding`](https://parquet.apache.org/docs/file-format/types/variantshredding/) and the [`Variant type`](https://parquet.apache.org/docs/file-format/types/variantencoding/). --------- Signed-off-by: "Adam Gutglick" <adam@spiraldb.com> Signed-off-by: Adam Gutglick <adam@spiraldb.com>
1 parent b71de62 commit 3006be6

34 files changed

Lines changed: 3984 additions & 427 deletions

File tree

Cargo.lock

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

encodings/parquet-variant/Cargo.toml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,5 +34,8 @@ vortex-session = { workspace = true }
3434

3535
[dev-dependencies]
3636
rstest = { workspace = true }
37+
tokio = { workspace = true, features = ["full"] }
3738
vortex-array = { workspace = true, features = ["_test-harness"] }
38-
39+
vortex-file = { workspace = true, features = ["tokio"] }
40+
vortex-io = { workspace = true, features = ["tokio"] }
41+
vortex-layout = { workspace = true }

0 commit comments

Comments
 (0)