You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/api/lazy-frame.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# SpatialLazyFrame
2
2
3
-
`SpatialLazyFrame` is an immutable plan builder. Operations can be declared in any order — the optimizer reorders, fuses, and pushes them down before execution. Nothing runs until `.collect()` is called.
3
+
`SpatialLazyFrame` is an immutable plan builder. Operations can be declared in any order. The optimizer reorders, fuses, and pushes them down before execution. Nothing runs until `.collect()` is called.
4
4
5
5
`SpatialGroupBy` is returned by `.group_by()` and holds the keys for a fused aggregate-join.
Copy file name to clipboardExpand all lines: docs/how-it-works/execution-paths.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@ The EXPR path keeps the spatial engine inside Polars' processing loop, allowing
16
16
17
17
## IO path
18
18
19
-
Used when few results are expected (low selectivity — e.g. a tight bounding box or a point-in-polygon query on a sparse dataset). The index is queried directly, and the result row indices are used to slice the DataFrame:
19
+
Used when few results are expected (low selectivity, e.g. a tight bounding box or a point-in-polygon query on a sparse dataset). The index is queried directly, and the result row indices are used to slice the DataFrame:
@@ -26,7 +26,7 @@ No Polars expression pipeline is involved. This avoids the overhead of batch pro
26
26
27
27
## Polars / PyO3 integration
28
28
29
-
The Rust engine is compiled as a PyO3 extension (`pycanopy._core`). Coordinate arrays are passed as zero-copy numpy views at the Python/Rust boundary — no allocation occurs for the handoff. The index structures themselves (KD-tree, R-tree, grid) are packed immutable Rust structs that live for the lifetime of the `Engine` object.
29
+
The Rust engine is compiled as a PyO3 extension (`pycanopy._core`). Coordinate arrays are passed as zero-copy numpy views at the Python/Rust boundary. No allocation occurs for the handoff. The index structures themselves (KD-tree, R-tree, grid) are packed immutable Rust structs that live for the lifetime of the `Engine` object.
30
30
31
31
For the EXPR path, the plugin is registered via Polars' `register_plugin_function` API, which lets the Rust closure participate in Polars' lazy evaluation graph natively.
Copy file name to clipboardExpand all lines: docs/how-it-works/query-planner.md
+4-6Lines changed: 4 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,15 +13,15 @@ The optimizer runs a fixed sequence of passes over the plan:
13
13
14
14
**1. Selectivity estimation**
15
15
16
-
Each node is annotated with an estimated selectivity — the fraction of the dataset expected to survive. Scalar Polars filters get an estimate from column statistics; spatial predicates use the cost model's histogram or kNN ratio.
16
+
Each node is annotated with an estimated selectivity, the fraction of the dataset expected to survive. Scalar Polars filters get an estimate from column statistics; spatial predicates use the cost model's histogram or kNN ratio.
17
17
18
18
**2. Predicate pushdown**
19
19
20
20
Scalar `filter()` nodes are sunk to the bottom of the plan so they run first, reducing the row count before any spatial work begins. A scalar filter that eliminates 90% of rows makes every subsequent spatial operation 10x cheaper.
21
21
22
22
**3. Cost-sort**
23
23
24
-
Spatial predicates are reordered by ascending estimated output size. Cheaper, more selective predicates run earlier. kNN and join nodes act as barriers — no reordering crosses them.
24
+
Spatial predicates are reordered by ascending estimated output size. Cheaper, more selective predicates run earlier. kNN and join nodes act as barriers. No reordering crosses them.
25
25
26
26
**4. Filter fusion**
27
27
@@ -39,7 +39,5 @@ A terminal `.select(cols)` is pinned as the last node and its column set is prop
39
39
40
40
After optimization, the executor picks one of two execution strategies per node:
41
41
42
-
-**IO path** — used when selectivity is low (few results expected). The index is queried directly and the result is returned as a slice of the DataFrame. No Polars expression pipeline is involved.
43
-
-**EXPR path** — used when selectivity is high. The spatial closure runs as a Polars `map_batches` plugin, processing the DataFrame in batches. Scalar filters run first inside the batch, then the spatial query runs on the surviving rows.
44
-
45
-
The optimizer annotates each node with the chosen path based on the selectivity threshold from the cost model.
42
+
-**IO path**: used when selectivity is low (few results expected). The index is queried directly and the result is returned as a slice of the DataFrame. No Polars expression pipeline is involved.
43
+
-**EXPR path**: used when selectivity is high. The spatial closure runs as a Polars `map_batches` plugin, processing the DataFrame in batches. Scalar filters run first inside the batch, then the spatial query runs on the surviving rows.
Copy file name to clipboardExpand all lines: docs/how-it-works/streaming.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,7 @@ The probe side of any join is sliced into fixed-size chunks called morsels:
10
10
MORSEL_ROWS = 262_144 # 256K rows per morsel
11
11
```
12
12
13
-
Morsels are produced via `iter_slices` — a zero-copy operation that yields views into the probe DataFrame without copying data. Each morsel is joined independently against the full index, and its result is either yielded, accumulated, or written to disk depending on the collection method.
13
+
Morsels are produced via `iter_slices`, a zero-copy operation that yields views into the probe DataFrame without copying data. Each morsel is joined independently against the full index, and its result is either yielded, accumulated, or written to disk depending on the collection method.
Exposes the join result as a native Polars `LazyFrame` source. This lets you fuse spatial join output with downstream Polars operations — sorts, sinks, further filters — into a single spilling pipeline:
42
+
Exposes the join result as a native Polars `LazyFrame` source. This lets you fuse spatial join output with downstream Polars operations (sorts, sinks, further filters) into a single spilling pipeline:
43
43
44
44
```python
45
45
(
@@ -52,8 +52,8 @@ Exposes the join result as a native Polars `LazyFrame` source. This lets you fus
52
52
)
53
53
```
54
54
55
-
Polars handles spilling to disk for the sort if the result exceeds memory, so the entire pipeline — join, select, sort, write — never requires the full result to be in RAM at once.
55
+
Polars handles spilling to disk for the sort if the result exceeds memory, so the entire pipeline (join, select, sort, write) never requires the full result to be in RAM at once.
56
56
57
57
## Aggregate-join streaming
58
58
59
-
`.group_by(keys).agg(...)` reduces over the morsel stream using associative partial aggregations. Each morsel produces per-group partials (counts, sums, etc.) that are combined across morsels at the end. The full pair frame never materialises — only the per-group accumulators are held in memory, which are bounded by the number of unique groups rather than the number of join pairs.
59
+
`.group_by(keys).agg(...)` reduces over the morsel stream using associative partial aggregations. Each morsel produces per-group partials (counts, sums, etc.) that are combined across morsels at the end. The full pair frame never materialises. Only the per-group accumulators are held in memory, bounded by the number of unique groups rather than the number of join pairs.
Copy file name to clipboardExpand all lines: docs/index.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ A declarative spatial query layer for Polars. Rust core, Python API.
4
4
5
5
## What is PyCanopy
6
6
7
-
PyCanopy brings spatial queries — range, kNN, joins, polygon containment — into the Polars ecosystem without leaving Python. You declare operations in any order; the query planner reorders, fuses, and pushes them down before execution. The index type (KD-tree, R-tree, grid, or brute force) is selected automatically by a cost model calibrated to your hardware.
7
+
PyCanopy brings spatial queries (range, kNN, joins, polygon containment) into the Polars ecosystem without leaving Python. You declare operations in any order; the query planner reorders, fuses, and pushes them down before execution. The index type (KD-tree, R-tree, grid, or brute force) is selected automatically by a cost model calibrated to your hardware.
0 commit comments