Commit bde8e5b
authored
IN LIST: add UInt16 bitmap filter (apache#23012)
## Which issue does this PR close?
- Part of apache#19241.
- Stacked on apache#23011.
- Next in stack: apache#23035.
- Extracted from apache#19390.
## Rationale for this change
apache#23011 uses a bitmap checklist for `UInt8`, where there are 256 possible
values. `UInt16` is the same idea with a larger value range: 0 through
65,535.
That is still small enough to represent directly. A `UInt16` bitmap
needs one bit for each possible value:
- 65,536 possible values
- 65,536 bits total
- 8 KB of memory
Then a lookup is still simple: use the input value as the bit position
and check whether that bit is set. For example, if the list contains
`42`, bit `42` is set, and every input row with value `42` can be
recognized with one bit test.
This PR keeps the scope narrow: it adds the unsigned 2-byte bitmap path
as a concrete `UInt16` filter. apache#23035 then unifies the `UInt8` and
`UInt16` implementations, and apache#23013 uses that shared shape for signed
same-width reinterpretation.
## What changes are included in this PR?
- Adds `UInt16BitmapFilter`, backed by a heap-allocated 65,536-bit
bitmap.
- Routes `UInt16` constant-list filtering to that bitmap path.
- Keeps the same `IN` / `NOT IN` null behavior as the generic path.
- Adds focused coverage for `UInt16` boundary values, nulls, and `NOT
IN`.
## Are these changes tested?
Yes.
- `cargo fmt --all`
- `cargo test -p datafusion-physical-expr bitmap_filter_u16 --lib`
- `cargo test -p datafusion-physical-expr in_list_int_types --lib`
- `cargo test -p datafusion-physical-expr
test_in_list_from_array_type_combinations --lib`
- `cargo test -p datafusion-physical-expr test_in_list_dictionary_types
--lib`
- `cargo clippy -p datafusion-physical-expr --all-targets --all-features
-- -D warnings`
## Are there any user-facing changes?
No. This is an internal performance optimization only.
<!-- codex-benchmark-start -->
## Benchmark note
No local `in_list_strategy` numbers are included for this PR because the
benchmark harness does not currently include a direct `UInt16` case. The
available `i16` rows measure the signed reinterpretation path added in
apache#23013 after the bitmap unification in apache#23035, not this PR's unsigned
`UInt16` bitmap filter.
<!-- codex-benchmark-end -->1 parent ff677c4 commit bde8e5b
2 files changed
Lines changed: 103 additions & 4 deletions
Lines changed: 102 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
107 | 182 | | |
108 | 183 | | |
109 | 184 | | |
| |||
294 | 369 | | |
295 | 370 | | |
296 | 371 | | |
297 | | - | |
298 | 372 | | |
299 | 373 | | |
300 | 374 | | |
| |||
315 | 389 | | |
316 | 390 | | |
317 | 391 | | |
318 | | - | |
| 392 | + | |
319 | 393 | | |
320 | 394 | | |
321 | | - | |
| 395 | + | |
322 | 396 | | |
323 | 397 | | |
324 | 398 | | |
| |||
355 | 429 | | |
356 | 430 | | |
357 | 431 | | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
358 | 457 | | |
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
46 | | - | |
| 46 | + | |
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| |||
0 commit comments