Skip to content

Commit 6485db8

Browse files
dfa1claude
andcommitted
docs(adr-0013): reconcile with the shipped fused-kernel design
The refactor removed Mask and the FilterKernel/MapKernel/ReduceKernel interfaces (§1-§3) in favor of the fused single-pass kernels Compute.filteredSum / filteredAggregate — no intermediate selection bitmap. Update ADR 0013's Status to say so and add an Implementation note documenting the pivot (§1-§3 preserved below as the original proposal, read as history). §4/§5/§6 are unchanged and implemented. Slim the TODO item to match: the residual/boundary tier landed; what remains are the deferred follow-ups (benchmark, encoded-domain specialization, transducer façade ADR 0019). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent d6b50d4 commit 6485db8

2 files changed

Lines changed: 44 additions & 26 deletions

File tree

TODO.md

Lines changed: 12 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -85,23 +85,18 @@ Per-encoding gotchas:
8585

8686
## Compute
8787

88-
- [ ] **Compute primitives — masks, kernels, no-materialize** — pushdown filter/compare/aggregate
89-
kernels operating on Lazy arrays without materializing. See [ADR-0013](docs/adr/0013-compute-primitives.md)
90-
(Proposed). Gate: a concrete downstream consumer (e.g. the vortex-arrow bridge or filter pushdown).
91-
Done: §6 read-side surface — `ScanIterator.columnZoneStats(col)` exposes per-zone
92-
min/max/sum/null count, decoding sum from the `vortex.stats` zone-map table (matches files from
93-
Rust, whose flat writer omits per-flat sum). Calcite `VortexAggregates.SUM`/`AVG` now fold those
94-
per-zone sums (metadata-only), falling back to a full scan only when a column has no zone map.
95-
The fold is a reusable `reader.compute.ZoneReducer.sum(col)` (the seam a future `vortex-compute`
96-
extracts), consumed by the planner: `VortexAggregatePushDownRule` rewrites a whole-table
97-
`MIN`/`MAX`/`COUNT`/`SUM`/`AVG` to a single-row `Values`, abandoning to the scan only when a zone
98-
carries no usable sum (an all-null column answers SQL `NULL`; `AVG` reduces to `SUM`/`COUNT`). The
99-
rule auto-registers over a bare `jdbc:calcite:` connection via `VortexTableScan.register()`, so
100-
SQL over JDBC is rewritten with no caller wiring. A `SUM` with a `WHERE` still abandons (whole-zone
101-
stats can't answer a filtered aggregate) — that is the residual tier below.
102-
Next: the residual tier — give `ZoneReducer` predicate support (whole-zone fold for fully-selected
103-
zones + boundary-zone streaming for partially-selected ones), then let the rule push `SUM` with a
104-
`WHERE`. `Mask`/`Predicate`/kernel vocab on top.
88+
- [ ] **Compute primitives — encoded-domain specialization & façade** — the remaining ADR-0013
89+
follow-ups now the fused kernels have shipped. See [ADR-0013](docs/adr/0013-compute-primitives.md).
90+
Done: §4 `Predicate`; §5 `RowFilter` unified over `Predicate`; §6 zone-map aggregate push-down in
91+
both tiers — the whole-zone `ZoneReducer` fold wired into `VortexAggregatePushDownRule` (rewrites a
92+
whole-table `MIN`/`MAX`/`COUNT`/`SUM`/`AVG` to a single-row `Values`, auto-registered over a bare
93+
`jdbc:calcite:` connection), plus the boundary/residual tier so a `SUM`/`MIN`/`MAX`/`COUNT` with a
94+
`WHERE` folds fully-selected zones from stats and decodes only the straddling boundary chunks via
95+
the fused `Compute.filteredAggregate`. §1–§3 (`Mask`, the `FilterKernel`/`MapKernel`/`ReduceKernel`
96+
interfaces) were built then removed for the fused single-pass kernels — no intermediate bitmap.
97+
Next: (1) benchmark the fused-kernel speedup (`ComputeKernelBenchmark`, 100M rows) to record §6's
98+
payoff; (2) encoded-domain kernel specialization — push the predicate into the ALP/FoR/Dict integer
99+
domain without decoding (its own perf ADR); (3) the columnar transducer façade (ADR 0019).
105100

106101
## Encodings
107102

docs/adr/0013-compute-primitives.md

Lines changed: 32 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
11
# ADR 0013: Compute primitives — masks, kernels, no-materialize contract
22

3-
- **Status:** Accepted — §1 (Mask), §4 (Predicate), §2/§3 (filter/reduce kernels, a generic
4-
streaming baseline plus a type-specialized boxing-free fast lane, behind a minimal `Compute`
5-
entry point), §5 (`RowFilter` unified over `Predicate` — a `RowFilter.Column` binds a column to a
6-
shared public `Predicate`; the same predicate is compiled against zone-map stats for pruning and
7-
against the decoded array for the boundary fold), and §6 (zone-map aggregate push-down, both the
8-
whole-zone and boundary-zone tiers) are implemented in `reader.compute`. Deferred: encoded-domain
9-
kernel specialization (perf escalation — pushing the predicate into the ALP/FoR/Dict integer
10-
domain without decoding), and the ergonomic façade (a columnar transducer — its own ADR).
11-
`MapKernel` is unbuilt (no consumer yet).
3+
- **Status:** Accepted, implemented — with §1–§3 superseded in implementation (see the
4+
*Implementation note* below). Built in `reader.compute`: §4 (`Predicate`), §5 (`RowFilter`
5+
unified over `Predicate` — a `RowFilter.Column` binds a column to a shared public `Predicate`,
6+
the same predicate compiled against zone-map stats for pruning and against the decoded array for
7+
the boundary fold), and §6 (zone-map aggregate push-down, both the whole-zone and boundary-zone
8+
tiers). §1 (`Mask`) and §2/§3 (the `FilterKernel`/`MapKernel`/`ReduceKernel` interfaces and the
9+
mask-based no-materialize contract) were built and then **removed**: the shipped design is the
10+
fused single-pass kernels `Compute.filteredSum` / `filteredAggregate`, which fold filter and
11+
reduce in one scan with **no intermediate selection bitmap** — a stronger no-materialize
12+
guarantee than a `Mask` could give. Deferred to their own ADRs: encoded-domain kernel
13+
specialization (perf escalation — pushing the predicate into the ALP/FoR/Dict integer domain
14+
without decoding) and the ergonomic façade (a columnar transducer, ADR 0019).
1215
- **Date:** 2026-06-15
1316
- **Deciders:** project maintainer
1417
- **Supersedes:**
@@ -18,6 +21,26 @@
1821
[ADR 0010 — Lazy decode](0010-lazy-decode.md),
1922
[ADR 0012 — Zero-copy layout decoding](0012-zero-copy-layout-decoding.md)
2023

24+
## Implementation note (2026-07-01)
25+
26+
The primitives shipped, but §1–§3 as drawn below did not survive the perf work. The `Mask` sealed
27+
type, the `FilterKernel` / `MapKernel` / `ReduceKernel` interfaces, and the mask-based
28+
no-materialize contract were implemented, benchmarked, and then removed. A materialized selection
29+
`Mask` is a positional bitmap the reduce must re-scan — strictly slower than fusing the filter and
30+
the reduce into one pass. The shipped surface is two fused kernels behind the minimal public
31+
`Compute` entry point, with everything else package-private:
32+
33+
- `Compute.filteredSum(filterColumn, predicate, aggColumn)` — one filter column, one summed column.
34+
- `Compute.filteredAggregate(chunk, filter, aggColumn)` — an n-ary `AND` `RowFilter` and a
35+
`SUM` / `MIN` / `MAX` / non-null-count fold over the rows it selects.
36+
37+
Both fold filter and reduce in a single scan with no intermediate bitmap, honoring §3's
38+
no-materialize intent more strictly than a `Mask` pipeline would; each is type-specialized into
39+
boxing-free long / double lanes with a generic boxing fallback. `MapKernel` was never built (no
40+
consumer). The `Predicate` (§4) and `RowFilter` unification (§5) vocabulary is unchanged and is what
41+
these kernels consume. The sections below are preserved as the original proposal; read §1–§3 as
42+
history, not as the built design.
43+
2144
## Context
2245

2346
ADR 0010 establishes that the reader defers transform work until access

0 commit comments

Comments
 (0)