Skip to content

Commit f782a6e

Browse files
a10ya10y
authored
fix comment (#7272)
#7215 dropped PatchedArray and left this very descriptive doc comment hanging. I moved it to be a module level comment. Signed-off-by: Andrew Duffy <andrew@a10y.dev>
1 parent 3ed9492 commit f782a6e

2 files changed

Lines changed: 66 additions & 65 deletions

File tree

‎vortex-array/src/arrays/patched/array.rs‎

Lines changed: 0 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -27,71 +27,6 @@ use crate::patches::Patches;
2727
use crate::stats::ArrayStats;
2828
use crate::validity::Validity;
2929

30-
/// An array that partially "patches" another array with new values.
31-
///
32-
/// # Background
33-
///
34-
/// This is meant to be the foundation of a fully data-parallel patching strategy, based on the
35-
/// work published in ["G-ALP" from Hepkema et al.](https://ir.cwi.nl/pub/35205/35205.pdf)
36-
///
37-
/// Patching is common when an encoding almost completely covers an array save a few exceptions.
38-
/// In that case, rather than avoid the encoding entirely, it's preferable to
39-
///
40-
/// * Replace unencodable values with fillers (zeros, frequent values, nulls, etc.)
41-
/// * Wrap the array with a `PatchedArray` signaling that when the original array is executed,
42-
/// some of the decoded values must be overwritten.
43-
///
44-
/// In Vortex, the FastLanes bit-packing encoding is often the terminal node in an encoding tree,
45-
/// and FastLanes has an intrinsic chunking of 1024 elements. Thus, 1024 elements is pervasively
46-
/// a useful unit of chunking throughout Vortex, and so we use 1024 as a chunk size here
47-
/// as well.
48-
///
49-
/// # Details
50-
///
51-
/// To patch an array, we first divide it into a set of chunks of length 1024, and then within
52-
/// each chunk, we assign each position to a lane. The number of lanes depends on the width of
53-
/// the underlying type.
54-
///
55-
/// Thus, rather than sorting patch indices and values by their global offset, they are sorted
56-
/// primarily by their chunk, and then subsequently by their lanes.
57-
///
58-
/// The Patched array layout has 4 children
59-
///
60-
/// * `inner`: the inner array is the one containing encoded values, including the filler values
61-
/// that need to be patched over at execution time
62-
/// * `lane_offsets`: this is an indexing buffer that allows you to see into ranges of the other
63-
/// two children
64-
/// * `indices`: An array of `u16` chunk indices, indicating where within the chunk should the value
65-
/// be overwritten by the patch value
66-
/// * `values`: The child array containing the patch values, which should be inserted over
67-
/// the values of the `inner` at the locations provided by `indices`
68-
///
69-
/// `indices` and `values` are aligned and accessed together.
70-
///
71-
/// ```text
72-
///
73-
/// chunk 0 chunk 0 chunk 0 chunk 0 chunk 0 chunk 0
74-
/// lane 0 lane 1 lane 2 lane 3 lane 4 lane 5
75-
/// ┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
76-
/// lane_offsets │ 0 │ 0 │ 2 │ 2 │ 3 │ 5 │ ...
77-
/// └─────┬──────┴─────┬──────┴─────┬──────┴──────┬─────┴──────┬─────┴──────┬─────┘
78-
/// │ │ │ │ │ │
79-
/// │ │ │ │ │ │
80-
/// ┌─────┴────────────┘ └──────┬──────┘ ┌──────┘ └─────┐
81-
/// │ │ │ │
82-
/// │ │ │ │
83-
/// │ │ │ │
84-
/// ▼────────────┬────────────┬────────────▼────────────▼────────────┬────────────▼
85-
/// indices │ │ │ │ │ │ │
86-
/// │ │ │ │ │ │ │
87-
/// ├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
88-
/// values │ │ │ │ │ │ │
89-
/// │ │ │ │ │ │ │
90-
/// └────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
91-
/// ```
92-
///
93-
/// It turns out that this layout is optimal for executing patching on GPUs, because the
94-
/// `lane_offsets` allows each thread in a warp to seek to its patches in constant time.
9530
/// The inner array containing the base unpatched values.
9631
pub(super) const INNER_SLOT: usize = 0;
9732
/// The lane offsets array for locating patches within lanes.

‎vortex-array/src/arrays/patched/mod.rs‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,72 @@
11
// SPDX-License-Identifier: Apache-2.0
22
// SPDX-FileCopyrightText: Copyright the Vortex contributors
33

4+
//! An array that partially "patches" another array with new values.
5+
//!
6+
//! # Background
7+
//!
8+
//! This is meant to be the foundation of a fully data-parallel patching strategy, based on the
9+
//! work published in ["G-ALP" from Hepkema et al.](https://ir.cwi.nl/pub/35205/35205.pdf)
10+
//!
11+
//! Patching is common when an encoding almost completely covers an array save a few exceptions.
12+
//! In that case, rather than avoid the encoding entirely, it's preferable to
13+
//!
14+
//! * Replace unencodable values with fillers (zeros, frequent values, nulls, etc.)
15+
//! * Wrap the array with a `PatchedArray` signaling that when the original array is executed,
16+
//! some of the decoded values must be overwritten.
17+
//!
18+
//! In Vortex, the FastLanes bit-packing encoding is often the terminal node in an encoding tree,
19+
//! and FastLanes has an intrinsic chunking of 1024 elements. Thus, 1024 elements is pervasively
20+
//! a useful unit of chunking throughout Vortex, and so we use 1024 as a chunk size here
21+
//! as well.
22+
//!
23+
//! # Details
24+
//!
25+
//! To patch an array, we first divide it into a set of chunks of length 1024, and then within
26+
//! each chunk, we assign each position to a lane. The number of lanes depends on the width of
27+
//! the underlying type.
28+
//!
29+
//! Thus, rather than sorting patch indices and values by their global offset, they are sorted
30+
//! primarily by their chunk, and then subsequently by their lanes.
31+
//!
32+
//! The Patched array layout has 4 children
33+
//!
34+
//! * `inner`: the inner array is the one containing encoded values, including the filler values
35+
//! that need to be patched over at execution time
36+
//! * `lane_offsets`: this is an indexing buffer that allows you to see into ranges of the other
37+
//! two children
38+
//! * `indices`: An array of `u16` chunk indices, indicating where within the chunk should the value
39+
//! be overwritten by the patch value
40+
//! * `values`: The child array containing the patch values, which should be inserted over
41+
//! the values of the `inner` at the locations provided by `indices`
42+
//!
43+
//! `indices` and `values` are aligned and accessed together.
44+
//!
45+
//! ```text
46+
//!
47+
//! chunk 0 chunk 0 chunk 0 chunk 0 chunk 0 chunk 0
48+
//! lane 0 lane 1 lane 2 lane 3 lane 4 lane 5
49+
//! ┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
50+
//! lane_offsets │ 0 │ 0 │ 2 │ 2 │ 3 │ 5 │ ...
51+
//! └─────┬──────┴─────┬──────┴─────┬──────┴──────┬─────┴──────┬─────┴──────┬─────┘
52+
//! │ │ │ │ │ │
53+
//! │ │ │ │ │ │
54+
//! ┌─────┴────────────┘ └──────┬──────┘ ┌──────┘ └─────┐
55+
//! │ │ │ │
56+
//! │ │ │ │
57+
//! │ │ │ │
58+
//! ▼────────────┬────────────┬────────────▼────────────▼────────────┬────────────▼
59+
//! indices │ │ │ │ │ │ │
60+
//! │ │ │ │ │ │ │
61+
//! ├────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
62+
//! values │ │ │ │ │ │ │
63+
//! │ │ │ │ │ │ │
64+
//! └────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
65+
//! ```
66+
//!
67+
//! It turns out that this layout is optimal for executing patching on GPUs, because the
68+
//! `lane_offsets` allows each thread in a warp to seek to its patches in constant time.
69+
470
mod array;
571
mod compute;
672
mod vtable;

0 commit comments

Comments
 (0)