Skip to content

Commit e1f3e96

Browse files
docs(vortex-onpair): clarify bits() is minimum addressing width
bits() returns ceil(log2(dict_size)), the minimum code width to address the dictionary, not the physical per-code storage width. The codes child is stored as a U8/U16 primitive array and may be re-encoded by the cascading compressor, so "Bits per token code" was misleading. Signed-off-by: Francesco Gargiulo <francesco@spiraldb.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 3b1c590 commit e1f3e96

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

  • encodings/experimental/onpair/src

encodings/experimental/onpair/src/array.rs

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,11 @@ impl OnPairData {
152152
self.len == 0
153153
}
154154

155-
/// Bits per token code, derived from the dictionary size (not stored).
155+
/// Minimum bits needed to address the dictionary (`ceil(log2(dict_size))`),
156+
/// derived from the dictionary size rather than stored. This is a logical
157+
/// lower bound, not the physical width of the `codes` child — that child is
158+
/// stored as a U8/U16 primitive array and may be independently re-encoded by
159+
/// the cascading compressor.
156160
pub fn bits(&self) -> u32 {
157161
u32::from(onpair::code_bits_for_num_tokens(self.dict_size as usize))
158162
}

0 commit comments

Comments
 (0)