Optimize roaring64 BSI BatchEqual and int64 comparisons

[guymolinari]

Description

Optimizes roaring64.BSI equality, batch equality, and int64-width comparison paths while preserving the existing public API. The BigValue APIs now internally use the optimized int64 paths when safe, while true wider-than-64-bit values continue to use the existing generic behavior.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement
• Code refactoring
• Documentation update
• Test improvements
• Build/CI changes

Changes Made

What was changed?

• Added bitmap-native BatchEqual execution for int64-width roaring64.BSI values.
• Added a cube-pattern fast path for complete bit-pattern query sets, such as scattered odd-value lists.
• Routed CompareValue through bitmap-native signed int64 comparison logic for LT, LE, EQ, GE, GT, and RANGE.
• Updated CompareBigValue and BatchEqualBig to delegate to optimized int64 paths when the BSI and query values fit in signed 64-bit space.
• Preserved generic fallback behavior for true wider-than-64-bit values.
• Fixed BatchEqualBig value matching so positive and negative values with the same magnitude, such as 5 and -5, do not collide.
• Added BSI64 correctness tests, BigValue compatibility tests, benchmark coverage, and benchmark notes.
• Cleaned roaring64 golint comments for touched package readiness.

Why was this changed?

The existing BSI64 equality and comparison paths materialized or inspected values row-by-row, which was very expensive for common analytical query shapes. These changes keep the public API stable while allowing int64-width operations to use bitmap algebra directly.

Compatibility

• No public method signatures were changed.
• Existing CompareValue, CompareBigValue, BatchEqual, and BatchEqualBig callers continue to work.
• CompareBigValue and BatchEqualBig only delegate to optimized int64 logic when safe.
• True big values still use the existing generic paths.

Verification

Run:

go test ./... -count=1
golint ./roaring64
go test ./roaring64 -run '^$' -bench 'BenchmarkBSI64BatchEqual|BenchmarkBSI64Compare(Big)?Value' -benchmem -count 5

Benchmark details are documented in roaring64/BSI_BENCHMARKS.md.

Why was it changed?

How was it changed?

Testing

Please add a unit test if you are fixing a bug.

You must run

go test

Formatting

Please run

go fmt

Fuzzing

Please run our fuzzer on your changes.

1. Generate initial smat corpus:

go test -tags=gofuzz -run=TestGenerateSmatCorpus

You should see a directory workdir created with initial corpus files.

2. Run the fuzz test:

go test -run='^$' -fuzz=FuzzSmat -fuzztime=300s -timeout=60s

Adjust -fuzztime as needed for longer or shorter runs. If crashes are found,
check the test output and the reproducer files in the workdir directory.
You may copy the reproducers to roaring_tests.go

Performance Impact

If applicable, describe any performance implications of these changes and include benchmark results.

Running Benchmarks

This project includes comprehensive benchmarks. Please run relevant benchmarks before and after your changes:

Basic Benchmarks

# Run all benchmarks
go test -bench=. -run=^$

# Run with memory allocation statistics
go test -bench=. -benchmem -run=^$

# Run specific benchmark
go test -bench=BenchmarkIteratorAlloc -run=^$

Parallel Benchmarks

# Run parallel processing benchmarks
go test -bench=BenchmarkIntersectionLargeParallel -run=^$

Real Data Benchmarks

# Requires real-roaring-datasets (run: go get github.com/RoaringBitmap/real-roaring-datasets)
BENCH_REAL_DATA=1 go test -bench=BenchmarkRealData -run=^$

Benchmark Results Format

Please include before/after results in the following format:

BenchmarkName-8    1000000    1234 ns/op    567 B/op    12 allocs/op

Performance Analysis

• Compare benchmark results before and after changes
• Note any significant improvements or regressions
• Include memory usage changes if relevant
• Mention any trade-offs (e.g., memory vs speed)

Breaking Changes

If this PR introduces breaking changes, please describe them and the migration path:

Related Issues

Fixes # (issue number)
Related to # (issue number)

Additional Notes

Any additional information or context about this pull request.