Skip to content

Commit 1ece694

Browse files
authored
Hack to support forcing normalization (#7517)
## Summary We need to support normalization of vector arrays to perform faster compute. However, this will essentially always create larger arrays on disk. This is a hack specifically for `L2Denorm` that will allow us write normalized vector arrays to disk. Given that writing scalar fn arrays to disk is already gated by an environment variable and the unstable encodings feature flag, I think this hack is fine for now. ## Testing This just allows some arrays that are larger than canonical. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
1 parent 47f1158 commit 1ece694

File tree

1 file changed

+11
-0
lines changed

1 file changed

+11
-0
lines changed

vortex-compressor/src/compressor.rs

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ use vortex_array::arrays::list::ListArrayExt;
2828
use vortex_array::arrays::listview::ListViewArrayExt;
2929
use vortex_array::arrays::listview::list_from_list_view;
3030
use vortex_array::arrays::primitive::PrimitiveArrayExt;
31+
use vortex_array::arrays::scalar_fn::AnyScalarFn;
3132
use vortex_array::arrays::struct_::StructArrayExt;
3233
use vortex_array::dtype::DType;
3334
use vortex_array::dtype::Nullability;
@@ -254,6 +255,11 @@ impl CascadingCompressor {
254255
return Ok(result);
255256
}
256257

258+
// TODO(connor): HACK TO SUPPORT L2 DENORMALIZATION!!!
259+
if result.is::<AnyScalarFn>() {
260+
return Ok(result);
261+
}
262+
257263
// Otherwise, fall back to compressing the underlying storage array.
258264
let compressed_storage = self.compress(ext_array.storage_array())?;
259265

@@ -337,6 +343,11 @@ impl CascadingCompressor {
337343
// TODO(connor): Add a tracing warning here too.
338344
return Ok(compressed);
339345
}
346+
347+
// TODO(connor): HACK TO SUPPORT L2 DENORMALIZATION!!!
348+
if compressed.is::<AnyScalarFn>() {
349+
return Ok(compressed);
350+
}
340351
}
341352

342353
// No scheme improved on the original.

0 commit comments

Comments
 (0)