Conversation
Merging this PR will improve performance by 29.94%
Performance Changes
Comparing Footnotes
|
…lone kernel panics The dynamic dispatch plan builder silently dropped validity (null bitmaps) from nullable arrays, hardcoding Validity::NonNullable on the output. This caused silent data corruption when nullable arrays were fused, and especially when the output fed into downstream kernels like CUB filter. Dynamic dispatch changes: - Thread root array validity through FusedPlan -> MaterializedPlan -> execute_typed, replacing the hardcoded Validity::NonNullable. - Guard against Dict with nullable codes (garbage code values could cause OOB shared memory reads in the DICT gather scalar op). - Guard against RunEnd with nullable ends (garbage end values could cause unpredictable binary search / forward-scan behavior). - Skip kernel launch entirely for Validity::AllInvalid arrays. - Respect nullability in the len==0 early-return path. Standalone kernel fixes: - RunEnd: replace unreachable!() with vortex_bail!() when values have per-element validity (Validity::Array), allowing graceful CPU fallback instead of a panic. - Zstd: replace unimplemented!() with vortex_bail!() when decompressed data contains nulls, allowing graceful CPU fallback instead of a panic. - ALP: clarify that patch validity does not need scattering (the encoder strips null positions from the exception list), remove stale TODO. Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
joseph-isaacs
left a comment
There was a problem hiding this comment.
do you actually do any work in the plan. is that in a different pr
the plan doesn't really have to do sth in addition here other than attaching the validity to the output array. the kernels are fine operating on garbage data. the ones that aren't require non-nullable |
All GPU kernels should now handle validity or fall back to the CPU otherwise in cases they don't.
Dynamic dispatch:
Standalone kernels:
for unsupported nullable cases, enabling graceful CPU fallback.
encoder already strips null positions from the exception list.