Make doc values skip index extensible and add DocValuesField/SkipStat<T> API

[sgup432]

Description

As of today, if someone needs to add a new metadata or stats in the skip index, it requires a version bump every time, changing the skip index jump table value to the updated value which takes into account of the size from new changes, and then accordingly during read, add version checks to read those extra values. This is because the skip index uses a fixed schema where every byte position is predetermined, so the reader navigates purely by precomputed offsets and any new field invalidates them. For example, PR #15737 adds pre-aggregated sum and value count per interval to the skip index, which required a backward compatibility commit, recalculating the fixed byte offsets, and gating reads behind version checks.

I feel this is pretty cumbersome and error prone if we need to add new metadata every time. With this PR, I have replaced the fixed-schema layout with length-prefixed, type-tagged entries. Each stat is identified by a type byte, and the reader uses stored lengths to skip over anything it doesn't recognize. This means future stats can be appended by simply writing a new type tag in the writer and adding a case in the reader's switch, with no version bump, no jump table recalculation, and older readers that don't know about the new stat will just skip past it harmlessly.

For example, adding a new stats(eg: sum like mentioned above) after this PR:

Adding a new stat (e.g. sum) after this PR:
  
  Writer (Consumer):
  // Just append after existing stats in the entry buffer
  entryBuffer.writeByte(Lucene90DocValuesFormat.SKIP_STAT_SUM);
  entryBuffer.writeLong(accumulator.sumValue);

  Reader (Producer):
  switch (statType) {
      case SKIP_STAT_RANGE -> { /* existing */ }
      case SKIP_STAT_SUM -> { 
          sumValue[level] = input.readLong();
      }   
      default -> input.seek(entryEnd);
  }

  Format constant:
  static final byte SKIP_STAT_SUM = 0x02;

[sgup432]

Example of how the data layout on the disk changes after this PR:

Before (fixed schema, version ≤ 2):

  
  Per entry:
    [1B: numLevels]
    per level (high → low):
      [4B: maxDocID] [4B: minDocID] [8B: maxValue] [8B: minValue] [4B: docCount]
      = exactly 28 bytes, always

  Single-level entry = 1 + 28 = 29 bytes
  Two-level entry    = 1 + 56 = 57 bytes

  Reader skips by precomputed offset: skipBytes(SKIP_INDEX_JUMP_LENGTH_PER_LEVEL[level])

---

After (stats-based, version 3):

  
  Per entry:
    [1B: numLevels]
    [4B: intervalLength]
    per level (high → low):
      [4B: maxDocID]
      [4B: entryLength]
      [1B: SKIP_STAT_RANGE (0x01)]
      [4B: minDocID] [8B: maxValue] [8B: minValue] [4B: docCount]

  Single-level entry = 1 + 4 + (4 + 4 + 1 + 24) = 38 bytes
  Two-level entry    = 1 + 4 + 2×(4 + 4 + 1 + 24) = 71 bytes

  Reader skips by stored length: seek(intervalEnd)

---

After, with a future stat (sum) added:

Per entry:
  [1B: numLevels]
  [4B: intervalLength]
  per level (high → low):
    [4B: maxDocID]
    [4B: entryLength]
    [1B: SKIP_STAT_RANGE (0x01)]
    [4B: minDocID] [8B: maxValue] [8B: minValue] [4B: docCount]
    [1B: SKIP_STAT_SUM (0x02)]    ← just appended
    [8B: sumValue]                 ← no other changes needed

[sgup432]

@romseygeek Looking for your suggestion on this. In a nutshell, this change is to allow anyone to add bits of information in skip index with more ease without having to bump version everytime, recalculate the jump table and add more version checks during read (all of which looks very painful).

[romseygeek]

Thanks for opening @sgup432!

My main worry around extending skip indexes is actually on the read side - it's really not obvious to me yet how we can make it clear which stats are available on an index, and indeed how those stats should be accessed without polluting the API. I sent a message to the dev list a few days ago about making Codec-specific data structures available on LeafReader with a possible solution.

[sgup432]

@romseygeek Yeah I agree on the read side of things and that we have been polluting doc values skipper interface. I do remember the email you sent out. I did think about it, and have had few ideas to solve it, maybe we will follow up with another PR.
But while doing that I also realized that even how the data is written today and how we skip through entries is also very rigid, which also requires some rethinking. Which is what this PR focusses on. I believe this also additionally make it easier to add stats and complement read of things. As with this PR we are explicitly writing stats tags on disk, so we build an API to fetch desired stats available for a field.

[sgup432]

@romseygeek I have opened a draft PR here - sgup432#1 on top of this PR. Basically showcasing how we can read the stats available for a field, building on top of this PR nicely.

Essentially it exposes two API:

• DocValuesField which stores the default stats which exist(min, max etc) and then provides a way to know and then fetch other stats available. Decoupling it from DocValuesSkipper.
• SkipStat: This can be used to add new stats. Generic type

Also did some did integration in some places(for reference) to fetch stats from DocValuesField without opening the skip index file.

[sgup432]

@romseygeek I have added above mentioned change here ie for the read side. Namely the APIs - DocValueField, SkipStat. Makes it easier for anyone to extend the current lucene codecs, and write their own pretty easily without changing lucene core.
I have only added the API/infra changes, actual integration is still pending. Can do it in followup PR if needed as don't want to bloat this up.