Skip to content

Commit 230711f

Browse files
authored
PARQUET-2241: Update wording of BYTE_STREAM_SPLIT encoding (#192)
1 parent 5205dc7 commit 230711f

2 files changed

Lines changed: 6 additions & 2 deletions

File tree

Encodings.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -319,10 +319,14 @@ This encoding does not reduce the size of the data but can lead to a significant
319319
compression ratio and speed when a compression algorithm is used afterwards.
320320

321321
This encoding creates K byte-streams of length N where K is the size in bytes of the data
322-
type and N is the number of elements in the data sequence.
322+
type and N is the number of elements in the data sequence. Specifically, K is 4 for FLOAT
323+
type and 8 for DOUBLE type.
323324
The bytes of each value are scattered to the corresponding streams. The 0-th byte goes to the
324325
0-th stream, the 1-st byte goes to the 1-st stream and so on.
325326
The streams are concatenated in the following order: 0-th stream, 1-st stream, etc.
327+
The total length of encoded streams is K * N bytes. Because it does not have any metadata
328+
to indicate the total length, the end of the streams is also the end of data page. No padding
329+
is allowed inside the data page.
326330

327331
Example:
328332
Original data is three 32-bit floats and for simplicity we look at their raw representation.

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ nothing else.
199199

200200
## Data Pages
201201
For data pages, the 3 pieces of information are encoded back to back, after the page
202-
header.
202+
header. No padding is allowed in the data page.
203203
In order we have:
204204
1. repetition levels data
205205
1. definition levels data

0 commit comments

Comments
 (0)