Binary search unequal-length suffixes in leaf term blocks. by vsop-479 · Pull Request #16407 · apache/lucene

vsop-479 · 2026-07-20T09:03:20Z

Description

This change builds a suffixOffsets table by reading term suffix lengths, to implement binary search unequal-length suffixes in leaf term blocks.

Although this adds one suffixOffsets int array, leaf blocks no longer need to keep a separate suffixLengthBytes array for this path.

Compared to #13192, this version only adds one suffixOffsets array, avoiding separate arrays for suffix lengths, byte offsets, and reader positions.

vsop-479 · 2026-07-20T09:20:22Z

@mikemccand Please take a look when you get a chance.

If the additional suffixOffsets table is acceptable, we can use Simple64 to batch decode suffix lengths, maybe in a separate PR.

vsop-479 · 2026-07-20T09:36:26Z

I ran luceneutil on wikimediumall.
Overall the change looks neutral, most tasks are within noise:

round1's iter19

TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                         LowTerm     2293.73     (16.6%)     2204.72     (16.9%)   -3.9% ( -32% -   35%) 0.464
                       OrHighMed      658.38      (9.9%)      636.02      (9.7%)   -3.4% ( -20% -   18%) 0.274
                   OrHighNotHigh      556.13      (7.2%)      540.14      (7.7%)   -2.9% ( -16% -   12%) 0.224
                         Prefix3      800.90     (13.6%)      779.01     (16.4%)   -2.7% ( -28% -   31%) 0.566
           BrowseMonthSSDVFacets        7.36     (18.7%)        7.17     (16.8%)   -2.6% ( -32% -   40%) 0.641
                    OrNotHighMed      389.36      (6.0%)      380.92      (7.3%)   -2.2% ( -14% -   11%) 0.305
                      OrHighHigh      305.52     (12.2%)      299.13     (13.6%)   -2.1% ( -24% -   27%) 0.609
             LowIntervalsOrdered       42.27      (4.9%)       41.52      (4.7%)   -1.8% ( -10% -    8%) 0.245
           HighTermDayOfYearSort      274.72      (7.8%)      270.70      (6.9%)   -1.5% ( -14% -   14%) 0.529
     BrowseRandomLabelSSDVFacets        6.43     (18.0%)        6.35     (17.9%)   -1.2% ( -31% -   42%) 0.838
            BrowseDateSSDVFacets        1.38      (6.3%)        1.36      (6.8%)   -1.1% ( -13% -   12%) 0.596
                       OrHighLow     1390.58      (7.9%)     1376.47      (8.5%)   -1.0% ( -16% -   16%) 0.694
            BrowseDateTaxoFacets        5.26     (14.1%)        5.21     (11.1%)   -1.0% ( -22% -   28%) 0.806
                        HighTerm     1337.94     (13.1%)     1326.02     (15.8%)   -0.9% ( -26% -   32%) 0.846
                     AndHighHigh      305.10     (14.8%)      302.48     (13.8%)   -0.9% ( -25% -   32%) 0.849
               HighTermMonthSort     1813.99      (9.9%)     1799.13     (12.6%)   -0.8% ( -21% -   23%) 0.819
                     MedSpanNear      102.43      (6.5%)      101.60      (6.9%)   -0.8% ( -13% -   13%) 0.701
                      TermDTSort      290.76     (10.5%)      288.57      (9.6%)   -0.8% ( -18% -   21%) 0.813
                          IntNRQ      180.16     (11.4%)      178.90      (9.9%)   -0.7% ( -19% -   23%) 0.836
                      HighPhrase      237.60      (5.1%)      236.23      (4.5%)   -0.6% (  -9% -    9%) 0.704
       BrowseDayOfYearTaxoFacets        5.28     (13.6%)        5.25     (10.9%)   -0.6% ( -22% -   27%) 0.887
         AndHighMedDayTaxoFacets       40.09      (5.0%)       39.89      (4.9%)   -0.5% (  -9% -    9%) 0.756
                       MedPhrase       94.54      (5.3%)       94.11      (4.8%)   -0.4% (  -9% -   10%) 0.778
                           range     9400.95     (10.4%)     9366.96     (10.4%)   -0.4% ( -19% -   22%) 0.912
                     LowSpanNear       44.20      (7.2%)       44.06      (7.5%)   -0.3% ( -13% -   15%) 0.893
                    OrNotHighLow     1532.51      (7.1%)     1528.23      (8.0%)   -0.3% ( -14% -   16%) 0.907
                   OrNotHighHigh      500.00     (10.3%)      498.69      (8.5%)   -0.3% ( -17% -   20%) 0.930
        AndHighHighDayTaxoFacets       14.04      (4.0%)       14.03      (4.3%)   -0.1% (  -8% -    8%) 0.925
                         Respell       78.61      (3.1%)       78.57      (3.1%)   -0.0% (  -6% -    6%) 0.963
                          Fuzzy2       86.25      (4.8%)       86.27      (3.9%)    0.0% (  -8% -    9%) 0.983
                      AndHighMed     1284.44      (7.5%)     1285.39      (8.7%)    0.1% ( -14% -   17%) 0.977
       BrowseDayOfYearSSDVFacets        6.80      (7.0%)        6.81      (6.9%)    0.2% ( -12% -   15%) 0.942
                        PKLookup      435.30      (4.3%)      436.30      (3.2%)    0.2% (  -7% -    8%) 0.849
            MedTermDayTaxoFacets       31.32      (2.9%)       31.39      (2.8%)    0.2% (  -5% -    6%) 0.791
                          Fuzzy1      108.76      (4.4%)      109.03      (4.6%)    0.3% (  -8% -    9%) 0.861
            HighTermTitleBDVSort       29.51      (7.4%)       29.61      (7.3%)    0.3% ( -13% -   16%) 0.888
                    OrHighNotLow     1244.70     (12.0%)     1248.90     (13.3%)    0.3% ( -22% -   29%) 0.933
                        BM25MSM2       30.64      (5.8%)       30.75      (5.0%)    0.4% (  -9% -   11%) 0.836
     BrowseRandomLabelTaxoFacets        4.50     (11.6%)        4.52      (9.4%)    0.4% ( -18% -   24%) 0.912
           BrowseMonthTaxoFacets        5.18     (10.3%)        5.20      (5.0%)    0.4% ( -13% -   17%) 0.877
                    OrHighNotMed     1203.38     (10.2%)     1208.33     (11.1%)    0.4% ( -18% -   24%) 0.903
               HighTermTitleSort      131.31     (11.8%)      131.92      (9.5%)    0.5% ( -18% -   24%) 0.891
                HighSloppyPhrase       40.48      (6.8%)       40.69      (6.3%)    0.5% ( -11% -   14%) 0.803
                      AndHighLow     2133.90      (7.5%)     2144.98      (8.0%)    0.5% ( -13% -   17%) 0.833
                    HighSpanNear       31.18      (6.6%)       31.41      (4.9%)    0.7% ( -10% -   13%) 0.693
                        Wildcard      123.43      (3.8%)      124.38      (5.3%)    0.8% (  -8% -   10%) 0.601
                         MedTerm     1411.90     (12.9%)     1422.93     (14.3%)    0.8% ( -23% -   32%) 0.856
                 MedSloppyPhrase       23.74      (6.2%)       23.94      (7.2%)    0.9% ( -11% -   15%) 0.679
             MedIntervalsOrdered       75.88      (5.5%)       76.71      (6.2%)    1.1% ( -10% -   13%) 0.559
                       LowPhrase      401.72      (8.0%)      406.46      (6.9%)    1.2% ( -12% -   17%) 0.619
                  AndMissingHigh     4925.48     (11.8%)     5006.12     (10.5%)    1.6% ( -18% -   27%) 0.643
            HighIntervalsOrdered       29.67      (8.0%)       30.26      (8.4%)    2.0% ( -13% -   19%) 0.442
                 LowSloppyPhrase       22.91      (4.3%)       23.39      (4.3%)    2.1% (  -6% -   11%) 0.123
          OrHighMedDayTaxoFacets        3.77      (5.5%)        3.87      (4.3%)    2.8% (  -6% -   13%) 0.068
                          IntSet     1043.97     (14.6%)     1080.40     (17.7%)    3.5% ( -25% -   41%) 0.497
                       ConstMSM2     1858.12     (14.7%)     1930.76     (12.0%)    3.9% ( -19% -   35%) 0.357

round2's iter19

TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                          IntSet     1203.70      (6.9%)     1157.34     (11.9%)   -3.9% ( -21% -   16%) 0.212
                     MedSpanNear      158.03      (5.6%)      154.19      (5.4%)   -2.4% ( -12% -    9%) 0.164
             LowIntervalsOrdered      270.08      (5.6%)      263.88      (5.1%)   -2.3% ( -12% -    8%) 0.174
                           range    10267.60      (8.1%)    10053.28      (9.2%)   -2.1% ( -17% -   16%) 0.446
                    OrNotHighMed      497.61      (5.7%)      487.42      (6.6%)   -2.0% ( -13% -   10%) 0.297
                   OrNotHighHigh      618.82      (7.0%)      608.17      (6.0%)   -1.7% ( -13% -   12%) 0.402
            HighIntervalsOrdered       70.50      (5.6%)       69.37      (5.9%)   -1.6% ( -12% -   10%) 0.379
           BrowseMonthSSDVFacets        7.90     (20.1%)        7.79     (18.1%)   -1.4% ( -33% -   46%) 0.812
               HighTermMonthSort     1989.67      (4.7%)     1961.80      (5.3%)   -1.4% ( -10% -    9%) 0.377
       BrowseDayOfYearSSDVFacets        7.28     (15.6%)        7.18     (12.4%)   -1.3% ( -25% -   31%) 0.765
                      AndHighMed      815.20      (5.5%)      806.11      (3.9%)   -1.1% (  -9% -    8%) 0.457
         AndHighMedDayTaxoFacets      108.99      (2.6%)      108.01      (2.8%)   -0.9% (  -6% -    4%) 0.290
                   OrHighNotHigh      546.59      (4.6%)      542.50      (6.3%)   -0.7% ( -11% -   10%) 0.667
                          IntNRQ      171.75      (4.6%)      170.51      (4.4%)   -0.7% (  -9% -    8%) 0.611
                  AndMissingHigh     5458.69      (7.9%)     5420.60      (7.0%)   -0.7% ( -14% -   15%) 0.768
            BrowseDateSSDVFacets        1.41      (7.4%)        1.40      (8.2%)   -0.7% ( -15% -   16%) 0.781
                      HighPhrase       85.01      (2.9%)       84.44      (3.0%)   -0.7% (  -6% -    5%) 0.476
                       LowPhrase      126.06      (2.6%)      125.29      (2.7%)   -0.6% (  -5% -    4%) 0.462
                       MedPhrase      258.86      (4.9%)      257.49      (6.1%)   -0.5% ( -10% -   10%) 0.762
                 MedSloppyPhrase       71.66      (3.6%)       71.32      (3.6%)   -0.5% (  -7% -    6%) 0.680
                      AndHighLow     2142.38      (7.8%)     2132.53      (7.2%)   -0.5% ( -14% -   15%) 0.846
            HighTermTitleBDVSort       50.84      (3.1%)       50.63      (2.9%)   -0.4% (  -6% -    5%) 0.662
                         MedTerm     1743.12      (9.6%)     1735.96      (9.4%)   -0.4% ( -17% -   20%) 0.891
           HighTermDayOfYearSort      287.13      (4.1%)      286.11      (2.4%)   -0.4% (  -6% -    6%) 0.742
           BrowseMonthTaxoFacets        5.33      (5.0%)        5.32      (5.6%)   -0.3% ( -10% -   10%) 0.843
                 LowSloppyPhrase      551.79      (3.4%)      550.14      (3.4%)   -0.3% (  -6% -    6%) 0.778
               HighTermTitleSort      137.32      (3.9%)      136.97      (5.7%)   -0.2% (  -9% -    9%) 0.872
        AndHighHighDayTaxoFacets       44.93      (2.4%)       44.94      (2.3%)    0.0% (  -4% -    4%) 0.971
                     LowSpanNear      391.53      (1.5%)      391.79      (2.5%)    0.1% (  -3% -    4%) 0.917
            MedTermDayTaxoFacets       57.78      (2.3%)       57.85      (2.6%)    0.1% (  -4% -    5%) 0.882
     BrowseRandomLabelSSDVFacets        6.35     (16.1%)        6.36     (14.3%)    0.3% ( -25% -   36%) 0.951
                       OrHighMed      818.54      (6.8%)      821.03      (7.1%)    0.3% ( -12% -   15%) 0.890
                    OrHighNotMed      950.95      (7.2%)      954.04      (6.3%)    0.3% ( -12% -   14%) 0.879
                        BM25MSM2       31.69      (3.8%)       31.81      (3.3%)    0.4% (  -6% -    7%) 0.741
                       ConstMSM2     2191.83      (6.5%)     2201.40      (7.0%)    0.4% ( -12% -   14%) 0.838
                          Fuzzy2       96.35      (3.1%)       96.78      (2.0%)    0.4% (  -4% -    5%) 0.585
                HighSloppyPhrase       48.18      (3.8%)       48.46      (3.2%)    0.6% (  -6% -    7%) 0.605
                    OrHighNotLow     1926.94      (6.7%)     1938.17      (6.2%)    0.6% ( -11% -   14%) 0.775
                       OrHighLow     1528.64      (4.7%)     1541.29      (4.9%)    0.8% (  -8% -   11%) 0.588
             MedIntervalsOrdered       50.05      (1.8%)       50.47      (1.3%)    0.8% (  -2% -    4%) 0.091
                         Respell       50.19      (1.9%)       50.69      (1.4%)    1.0% (  -2% -    4%) 0.057
                    HighSpanNear       32.36      (3.6%)       32.69      (3.1%)    1.0% (  -5% -    7%) 0.335
                         LowTerm     2776.09     (14.5%)     2804.64     (14.6%)    1.0% ( -24% -   35%) 0.823
                     AndHighHigh      261.35      (9.8%)      264.13      (9.7%)    1.1% ( -16% -   22%) 0.730
          OrHighMedDayTaxoFacets       15.24      (3.1%)       15.40      (2.9%)    1.1% (  -4% -    7%) 0.263
                          Fuzzy1      101.35      (2.3%)      102.49      (1.7%)    1.1% (  -2% -    5%) 0.079
                      TermDTSort      298.23      (6.1%)      301.70      (5.2%)    1.2% (  -9% -   13%) 0.518
                        HighTerm     1625.43     (16.1%)     1648.94     (17.6%)    1.4% ( -27% -   41%) 0.787
                      OrHighHigh      392.52     (11.4%)      399.54     (11.2%)    1.8% ( -18% -   27%) 0.617
                         Prefix3      539.79      (3.7%)      549.77      (3.6%)    1.9% (  -5% -    9%) 0.110
                        PKLookup      433.18      (2.9%)      441.28      (2.3%)    1.9% (  -3% -    7%) 0.024
                    OrNotHighLow     2193.36      (7.6%)     2237.64      (6.4%)    2.0% ( -11% -   17%) 0.362
                        Wildcard       38.24      (3.0%)       39.13      (4.0%)    2.3% (  -4% -    9%) 0.035
     BrowseRandomLabelTaxoFacets        4.70      (9.5%)        4.82     (10.7%)    2.5% ( -16% -   25%) 0.426
       BrowseDayOfYearTaxoFacets        5.29     (12.2%)        5.49     (13.9%)    3.9% ( -19% -   34%) 0.351
            BrowseDateTaxoFacets        5.26     (13.1%)        5.47     (14.8%)    4.1% ( -21% -   36%) 0.355

gc stats

Metric	Baseline	Candidate
Elapsed time	15,663 ms	15,896 ms
JIT compilation time	11,430 ms	11,238 ms
Young generation GC time	13 ms	14 ms
Young generation GC collections	10	10
Old generation GC time	19 ms	18 ms
Old generation GC collections	1	1
Garbage generated in young generation	0.0 MiB	0.0 MiB
Garbage generated in survivor generation	0.0 MiB	0.0 MiB
Garbage generated in old generation	0.0 MiB	0.0 MiB
Average CPU load	637.3126 / 1200	657.18945 / 1200

vsop-479 added 2 commits July 20, 2026 11:32

Impl binarySearchTermLeafUnEqual.

1494ef0

Add changes entry.

b5ce92a

github-actions Bot added the module:core/codecs label Jul 20, 2026

vsop-479 requested a review from mikemccand July 20, 2026 09:37

vsop-479 mentioned this pull request Jul 20, 2026

Simple64 #16245

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Binary search unequal-length suffixes in leaf term blocks.#16407

Binary search unequal-length suffixes in leaf term blocks.#16407
vsop-479 wants to merge 2 commits into
apache:mainfrom
vsop-479:binary_search_unequal_leaf

vsop-479 commented Jul 20, 2026 •

edited

Loading

Uh oh!

vsop-479 commented Jul 20, 2026

Uh oh!

vsop-479 commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

vsop-479 commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

vsop-479 commented Jul 20, 2026

Uh oh!

vsop-479 commented Jul 20, 2026

round1's iter19

round2's iter19

gc stats

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vsop-479 commented Jul 20, 2026 •

edited

Loading