fix: NULL bitmap corruption in InplaceSort and roaring filter by XuPeng-SH · Pull Request #24057 · matrixorigin/matrixone

XuPeng-SH · 2026-04-03T19:54:34Z

What type of PR is this?

Which issue(s) this PR fixes:

Fixes #24055 — hashbuild InplaceSort NULL bitmap corruption in runtime filter
Fixes #24056 — FuzzyFilter roaring_filter NULL handling for UNIQUE INDEX

What this PR does / why we need it:

Fixes three related NULL-handling bugs discovered during the investigation of PR #24054:

1. hashbuild runtime filter InplaceSort (build.go)

handleRuntimeFilter() calls InplaceSort() on UniqueJoinKeys vectors before serializing them as runtime IN-filters. However, InplaceSort() reorders the data array but never updates the null bitmap (nsp). After sorting, null positions in the bitmap no longer correspond to actual null values, producing corrupted serialized filter data.

Fix: Call GetNulls().Reset() before InplaceSort(). NULLs are irrelevant for IN-filter matching (probe side should never match NULLs), so clearing the bitmap is the correct and cheapest fix.

2. fuzzyfilter roaring bitmap NULL handling (roaring_filter.go)

addFunc, testFunc, and testAndAddFunc iterate all vector values without checking the null bitmap. NULL values (which appear as zero in the raw data) get added to the roaring bitmap as if they were real values.

Fix: Add nsp.Contains(uint64(i)) check to skip NULL rows in all three functions. Per SQL standard, NULL != NULL.

3. fuzzyfilter pass2RuntimeFilter InplaceSort (filter.go)

Defensive fix: Reset null bitmap before InplaceSort() on pass2RuntimeFilter, same pattern as fix #1.

4. indexbuild InplaceSort (indexbuild/build.go)

Same InplaceSort null bitmap corruption pattern as hashbuild. indexBuild.handleRuntimeFilter() sorts vectors without resetting the null bitmap.

Test coverage

roaring_filter_test.go (new, 30 sub-tests)

TestRoaringAddFuncSkipsNulls / TestRoaringTestFuncSkipsNulls / TestRoaringTestAndAddFuncSkipsNulls
TestRoaringMultipleTypes / TestRoaringTestAndAddMultipleBatches
TestValueToString / TestIfCanUseRoaringFilter / TestNewRoaringFilterAllTypes

build_test.go (2 new tests)

TestHashBuildRuntimeFilterWithNulls / TestHashBuildRuntimeFilterWithNullsHashOnPK

Function	Coverage
addFunc	100%
testFunc	100%
testAndAddFunc	100%
handleRuntimeFilter	52.7% (fix lines covered)

qodo-code-review · 2026-04-03T19:55:14Z

Review Summary by Qodo

Fix NULL bitmap corruption in InplaceSort and roaring filter

🐞 Bug fix

Walkthroughs

Description

• Fix NULL bitmap corruption in InplaceSort for runtime filters
  - Reset null bitmap before InplaceSort in hashbuild and fuzzyfilter
  - InplaceSort reorders data but not null bitmap, causing corruption
• Skip NULL values in roaring filter duplicate detection
  - Add null bitmap checks in addFunc, testFunc, testAndAddFunc
  - Prevents false "Duplicate entry" errors with nullable UNIQUE INDEX
• Add comprehensive test coverage for NULL handling
  - 30+ test cases for roaring filter NULL scenarios
  - 2 new tests for hashbuild runtime filter with NULLs

Diagram

flowchart LR
  A["InplaceSort<br/>reorders data"] -->|"corrupts bitmap"| B["NULL bitmap<br/>misaligned"]
  B -->|"fix: Reset()"| C["Cleared bitmap<br/>before sort"]
  C -->|"correct"| D["Valid serialized<br/>filter"]
  
  E["Roaring filter<br/>processes values"] -->|"includes NULLs"| F["False duplicates<br/>detected"]
  F -->|"fix: nsp.Contains check"| G["NULLs skipped<br/>per SQL standard"]
  G -->|"correct"| H["Accurate duplicate<br/>detection"]

File Changes

1. pkg/sql/colexec/fuzzyfilter/filter.go 🐞 Bug fix +3/-0

Reset null bitmap before InplaceSort

• Add null bitmap reset before InplaceSort in pass2RuntimeFilter path
• Defensive fix to prevent null bitmap corruption during serialization
• Matches pattern applied in hashbuild and roaring_filter

pkg/sql/colexec/fuzzyfilter/filter.go

2. pkg/sql/colexec/fuzzyfilter/roaring_filter.go 🐞 Bug fix +14/-1

Skip NULLs in roaring filter functions

• Add null bitmap checks in addFunc to skip NULL values during add
• Add null bitmap checks in testFunc to skip NULL values during test
• Add null bitmap checks in testAndAddFunc to skip NULL values during duplicate detection
• Implements SQL standard: NULL != NULL, preventing false duplicate errors

pkg/sql/colexec/fuzzyfilter/roaring_filter.go

3. pkg/sql/colexec/fuzzyfilter/roaring_filter_test.go 🧪 Tests +347/-0

Add 30+ test cases for roaring filter NULL handling

• Add TestRoaringAddFuncSkipsNulls with 5 test cases covering no nulls, all nulls, mixed,
 null-at-zero, real-zero-vs-null
• Add TestRoaringTestFuncSkipsNulls with 4 test cases for match, null-only, null-before-match
 scenarios
• Add TestRoaringTestAndAddFuncSkipsNulls with 5 test cases for duplicate detection with NULLs
• Add TestRoaringMultipleTypes covering all 5 supported types (int8/16, uint8/16/32)
• Add TestRoaringTestAndAddMultipleBatches for cross-batch NULL handling
• Add TestValueToString, TestIfCanUseRoaringFilter, TestNewRoaringFilterAllTypes for comprehensive
 coverage

pkg/sql/colexec/fuzzyfilter/roaring_filter_test.go

View more (2)

4. pkg/sql/colexec/hashbuild/build.go 🐞 Bug fix +4/-0

Reset null bitmap before InplaceSort in runtime filter

• Add null bitmap reset before InplaceSort in handleRuntimeFilter for expression path
• Add null bitmap reset before InplaceSort in hashOnPK path for UniqueJoinKeys
• Prevents NULL bitmap corruption when serializing runtime IN-filters
• NULLs are irrelevant for IN-filter matching, so clearing bitmap is safe and efficient

pkg/sql/colexec/hashbuild/build.go

5. pkg/sql/colexec/hashbuild/build_test.go 🧪 Tests +111/-0

Add tests for hashbuild runtime filter with NULLs

• Add TestHashBuildRuntimeFilterWithNulls to verify NULL handling in standard runtime filter path
• Add TestHashBuildRuntimeFilterWithNullsHashOnPK to verify NULL handling in hashOnPK path
• Both tests create batches with NULLs and verify runtime filter execution completes without
 corruption

pkg/sql/colexec/hashbuild/build_test.go

qodo-code-review · 2026-04-03T19:55:16Z

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX Issues (0)

1. NULLs become IN values 🐞 Bug ➹ Performance

Description

Resetting the null bitmap before sorting/serializing runtime IN-filters makes previously-NULL rows
indistinguishable from non-NULL rows, so their underlying data values get included in the IN set and
can significantly increase false positives (reduced pruning). This is observable because the IN
operator excludes list entries solely via the tuple’s null bitmap.

Code

pkg/sql/colexec/hashbuild/build.go[R375-384]

+			// InplaceSort reorders data but NOT the null bitmap.
+			// NULLs are irrelevant for IN-filter: clear bitmap before sort.
+			vec.GetNulls().Reset()
			vec.InplaceSort()
			data, err = vec.MarshalBinary()
			free()
		} else {
+			ctr.hashmapBuilder.UniqueJoinKeys[0].GetNulls().Reset()
			ctr.hashmapBuilder.UniqueJoinKeys[0].InplaceSort()
			data, err = ctr.hashmapBuilder.UniqueJoinKeys[0].MarshalBinary()

Evidence

HashBuild clears the null bitmap right before InplaceSort+MarshalBinary, but does not remove NULL
rows from the vector. Vector.InplaceSort sorts only the data slice and does not adjust the null
bitmap; and the IN operator’s initialization uses the tuple’s null bitmap (via GetValue’s nullMap)
to skip NULL entries—so after Reset(), those formerly-NULL entries become included as normal values.
Test utilities also demonstrate that NULL rows can retain populated underlying values (NULL marking
is separate from data population).

pkg/sql/colexec/hashbuild/build.go[361-385]
pkg/container/vector/vector.go[4655-4686]
pkg/sql/plan/function/operator_in.go[51-74]
pkg/testutil/util_new.go[219-231]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`GetNulls().Reset()` clears the NULL bitmap but does not remove NULL rows. After serialization, the IN operator will treat previously-NULL rows as non-NULL and include their underlying data values in the IN set, increasing false positives and reducing runtime-filter pruning.

### Issue Context
- `vector.Vector.InplaceSort()` sorts data but does not reorder/update `v.nsp` (null bitmap), so a fix is needed before sorting.
- The IN operator builds its membership map by iterating the tuple vector and skipping entries only when the tuple’s null bitmap marks them NULL.

### Fix Focus Areas
- Filter out NULL rows from the IN-list vector before sorting/serializing (e.g., `vec.ShrinkByMask(vec.GetNulls().GetBitmap(), true, 0)`), then sort and marshal.
- Update `rowCount`/`runtimeFilter.Card` to reflect the new vector length after NULL removal.

- pkg/sql/colexec/hashbuild/build.go[361-395]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

ⓘ The new review experience is currently in Beta. Learn more

Fix four related NULL-handling bugs: 1. hashbuild/build.go: Reset null bitmap before InplaceSort in handleRuntimeFilter. InplaceSort reorders data but NOT the null bitmap, causing bitmap corruption in serialized runtime filters for JOIN queries with nullable join keys. (Fixes matrixorigin#24055) 2. fuzzyfilter/roaring_filter.go: Skip NULL values in addFunc, testFunc, and testAndAddFunc. SQL standard: NULL != NULL, so NULLs must not participate in duplicate detection. Without this fix, multiple NULL values in a UNIQUE INDEX column could false- trigger 'Duplicate entry' errors. (Fixes matrixorigin#24056) 3. fuzzyfilter/filter.go: Defensive null bitmap reset before InplaceSort in pass2RuntimeFilter path. 4. indexbuild/build.go: Same InplaceSort null bitmap corruption as hashbuild — reset bitmap before sort in handleRuntimeFilter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

XuPeng-SH · 2026-04-04T08:34:07Z

@mergify refresh

- Rename newroaringFilter to newRoaringFilter (Go naming convention) - Simplify constructor by assigning directly to struct fields - Add default panic for unsupported types - Use MustFixedColNoTypeCheck in testAndAddFunc for consistency

mergify · 2026-04-04T21:17:33Z

refresh

✅ Pull request refreshed

mergify · 2026-04-04T21:30:20Z

Merge Queue Status

✅ Entered queue — 2026-04-04 21:30 UTC · Rule: main
✅ Checks skipped · PR is already up-to-date
✅ Merged — 2026-04-04 21:43 UTC · at 9a2661575bf4bcad8f532956baca89f40be93c50

This pull request spent 13 minutes 41 seconds in the queue, including 9 seconds running CI.

Required conditions to merge

XuPeng-SH requested review from aunjgr and ouyuanning as code owners April 3, 2026 19:54

XuPeng-SH had a problem deploying to ci April 3, 2026 19:54 — with GitHub Actions Error

XuPeng-SH temporarily deployed to ci April 3, 2026 19:54 — with GitHub Actions Inactive

XuPeng-SH had a problem deploying to ci April 3, 2026 19:54 — with GitHub Actions Error

XuPeng-SH temporarily deployed to ci April 3, 2026 19:54 — with GitHub Actions Inactive

matrix-meow added the size/M Denotes a PR that changes [100,499] lines label Apr 3, 2026

mergify Bot added the kind/bug Something isn't working label Apr 3, 2026

XuPeng-SH force-pushed the fix-null-bitmap-inplacesort-and-roaring branch from ed7bea7 to b0b6a51 Compare April 3, 2026 20:02

XuPeng-SH temporarily deployed to ci April 3, 2026 20:03 — with GitHub Actions Inactive

XuPeng-SH had a problem deploying to ci April 3, 2026 20:03 — with GitHub Actions Failure

XuPeng-SH temporarily deployed to ci April 3, 2026 20:03 — with GitHub Actions Inactive

aunjgr approved these changes Apr 4, 2026

View reviewed changes

XuPeng-SH temporarily deployed to ci April 4, 2026 13:58 — with GitHub Actions Inactive

fuzzyfilter: improve roaringFilter code quality

9a26615

- Rename newroaringFilter to newRoaringFilter (Go naming convention) - Simplify constructor by assigning directly to struct fields - Add default panic for unsupported types - Use MustFixedColNoTypeCheck in testAndAddFunc for consistency

XuPeng-SH temporarily deployed to ci April 4, 2026 14:24 — with GitHub Actions Inactive

matrix-meow added size/L Denotes a PR that changes [500,999] lines and removed size/M Denotes a PR that changes [100,499] lines labels Apr 4, 2026

mergify Bot added the queued label Apr 4, 2026

mergify Bot merged commit 1e65087 into matrixorigin:main Apr 4, 2026
24 of 25 checks passed

mergify Bot had a problem deploying to ci April 4, 2026 21:43 Failure

mergify Bot temporarily deployed to ci April 4, 2026 21:43 Inactive

mergify Bot removed the queued label Apr 4, 2026

mergify Bot temporarily deployed to ci April 4, 2026 21:51 Inactive

mergify Bot temporarily deployed to ci April 4, 2026 22:06 Inactive

mergify Bot temporarily deployed to ci April 4, 2026 22:40 Inactive

mergify Bot temporarily deployed to ci April 4, 2026 23:05 Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: NULL bitmap corruption in InplaceSort and roaring filter#24057

fix: NULL bitmap corruption in InplaceSort and roaring filter#24057
mergify[bot] merged 2 commits intomatrixorigin:mainfrom
XuPeng-SH:fix-null-bitmap-inplacesort-and-roaring

XuPeng-SH commented Apr 3, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot commented Apr 3, 2026

Uh oh!

qodo-code-review Bot commented Apr 3, 2026 •

edited

Loading

Uh oh!

XuPeng-SH commented Apr 4, 2026

Uh oh!

mergify Bot commented Apr 4, 2026

Uh oh!

mergify Bot commented Apr 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

XuPeng-SH commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

1. hashbuild runtime filter InplaceSort (build.go)

2. fuzzyfilter roaring bitmap NULL handling (roaring_filter.go)

3. fuzzyfilter pass2RuntimeFilter InplaceSort (filter.go)

4. indexbuild InplaceSort (indexbuild/build.go)

Test coverage

roaring_filter_test.go (new, 30 sub-tests)

build_test.go (2 new tests)

Uh oh!

qodo-code-review Bot commented Apr 3, 2026

Review Summary by Qodo

Walkthroughs

File Changes

Uh oh!

qodo-code-review Bot commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

XuPeng-SH commented Apr 4, 2026

Uh oh!

mergify Bot commented Apr 4, 2026

✅ Pull request refreshed

Uh oh!

mergify Bot commented Apr 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

XuPeng-SH commented Apr 3, 2026 •

edited

Loading

qodo-code-review Bot commented Apr 3, 2026 •

edited

Loading

mergify Bot commented Apr 4, 2026 •

edited

Loading