Skip to content

Commit 007e5b2

Browse files
tweaks to docs
1 parent 5f16029 commit 007e5b2

7 files changed

Lines changed: 19 additions & 23 deletions

File tree

README.md

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -278,18 +278,16 @@ The empirical constants ($c_{\text{scan}}$, $c_{\text{tree}}$, $c_{\text{grid}}$
278278
`select_index` is a rule-based pre-filter that picks a candidate index type:
279279

280280
```mermaid
281-
flowchart LR
281+
flowchart TD
282282
A[Query arrives] --> B{N < 500\nor sel > 50%?}
283283
B -- yes --> BF[Brute force]
284-
B -- no --> C{kNN and\nk/N > 10%?}
284+
B -- no --> C{kNN with\nk/N > 10%?}
285285
C -- yes --> BF
286286
C -- no --> D{Polygon\ndataset?}
287287
D -- yes --> RT[R-tree]
288-
D -- no --> E{Query type}
289-
E -- kNN / contains --> KD[KD-tree]
290-
E -- range --> F{Uniform?}
291-
F -- yes --> GR[Grid]
292-
F -- no --> KD
288+
D -- no --> E{Range query\nand uniform?}
289+
E -- yes --> GR[Grid]
290+
E -- no --> KD[KD-tree]
293291
```
294292

295293
All index types share the same coordinate arrays with no duplication.

docs/api/lazy-frame.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# SpatialLazyFrame
22

3-
`SpatialLazyFrame` is an immutable plan builder. Operations can be declared in any order — the optimizer reorders, fuses, and pushes them down before execution. Nothing runs until `.collect()` is called.
3+
`SpatialLazyFrame` is an immutable plan builder. Operations can be declared in any order. The optimizer reorders, fuses, and pushes them down before execution. Nothing runs until `.collect()` is called.
44

55
`SpatialGroupBy` is returned by `.group_by()` and holds the keys for a fused aggregate-join.
66

docs/how-it-works/execution-paths.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ The EXPR path keeps the spatial engine inside Polars' processing loop, allowing
1616

1717
## IO path
1818

19-
Used when few results are expected (low selectivity e.g. a tight bounding box or a point-in-polygon query on a sparse dataset). The index is queried directly, and the result row indices are used to slice the DataFrame:
19+
Used when few results are expected (low selectivity, e.g. a tight bounding box or a point-in-polygon query on a sparse dataset). The index is queried directly, and the result row indices are used to slice the DataFrame:
2020

2121
```
2222
index.range_query(bbox) → [i, j, k, ...] → df[i, j, k, ...]
@@ -26,7 +26,7 @@ No Polars expression pipeline is involved. This avoids the overhead of batch pro
2626

2727
## Polars / PyO3 integration
2828

29-
The Rust engine is compiled as a PyO3 extension (`pycanopy._core`). Coordinate arrays are passed as zero-copy numpy views at the Python/Rust boundary — no allocation occurs for the handoff. The index structures themselves (KD-tree, R-tree, grid) are packed immutable Rust structs that live for the lifetime of the `Engine` object.
29+
The Rust engine is compiled as a PyO3 extension (`pycanopy._core`). Coordinate arrays are passed as zero-copy numpy views at the Python/Rust boundary. No allocation occurs for the handoff. The index structures themselves (KD-tree, R-tree, grid) are packed immutable Rust structs that live for the lifetime of the `Engine` object.
3030

3131
For the EXPR path, the plugin is registered via Polars' `register_plugin_function` API, which lets the Rust closure participate in Polars' lazy evaluation graph natively.
3232

docs/how-it-works/query-planner.md

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,15 @@ The optimizer runs a fixed sequence of passes over the plan:
1313

1414
**1. Selectivity estimation**
1515

16-
Each node is annotated with an estimated selectivity the fraction of the dataset expected to survive. Scalar Polars filters get an estimate from column statistics; spatial predicates use the cost model's histogram or kNN ratio.
16+
Each node is annotated with an estimated selectivity, the fraction of the dataset expected to survive. Scalar Polars filters get an estimate from column statistics; spatial predicates use the cost model's histogram or kNN ratio.
1717

1818
**2. Predicate pushdown**
1919

2020
Scalar `filter()` nodes are sunk to the bottom of the plan so they run first, reducing the row count before any spatial work begins. A scalar filter that eliminates 90% of rows makes every subsequent spatial operation 10x cheaper.
2121

2222
**3. Cost-sort**
2323

24-
Spatial predicates are reordered by ascending estimated output size. Cheaper, more selective predicates run earlier. kNN and join nodes act as barriers — no reordering crosses them.
24+
Spatial predicates are reordered by ascending estimated output size. Cheaper, more selective predicates run earlier. kNN and join nodes act as barriers. No reordering crosses them.
2525

2626
**4. Filter fusion**
2727

@@ -39,7 +39,5 @@ A terminal `.select(cols)` is pinned as the last node and its column set is prop
3939

4040
After optimization, the executor picks one of two execution strategies per node:
4141

42-
- **IO path** — used when selectivity is low (few results expected). The index is queried directly and the result is returned as a slice of the DataFrame. No Polars expression pipeline is involved.
43-
- **EXPR path** — used when selectivity is high. The spatial closure runs as a Polars `map_batches` plugin, processing the DataFrame in batches. Scalar filters run first inside the batch, then the spatial query runs on the surviving rows.
44-
45-
The optimizer annotates each node with the chosen path based on the selectivity threshold from the cost model.
42+
- **IO path**: used when selectivity is low (few results expected). The index is queried directly and the result is returned as a slice of the DataFrame. No Polars expression pipeline is involved.
43+
- **EXPR path**: used when selectivity is high. The spatial closure runs as a Polars `map_batches` plugin, processing the DataFrame in batches. Scalar filters run first inside the batch, then the spatial query runs on the surviving rows.

docs/how-it-works/streaming.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ The probe side of any join is sliced into fixed-size chunks called morsels:
1010
MORSEL_ROWS = 262_144 # 256K rows per morsel
1111
```
1212

13-
Morsels are produced via `iter_slices` a zero-copy operation that yields views into the probe DataFrame without copying data. Each morsel is joined independently against the full index, and its result is either yielded, accumulated, or written to disk depending on the collection method.
13+
Morsels are produced via `iter_slices`, a zero-copy operation that yields views into the probe DataFrame without copying data. Each morsel is joined independently against the full index, and its result is either yielded, accumulated, or written to disk depending on the collection method.
1414

1515
## collect()
1616

@@ -39,7 +39,7 @@ sf.lazy().polygon_knn_join(trips, "lon", "lat", k=5).sink_parquet("result.parque
3939

4040
## lazy_source()
4141

42-
Exposes the join result as a native Polars `LazyFrame` source. This lets you fuse spatial join output with downstream Polars operations sorts, sinks, further filters into a single spilling pipeline:
42+
Exposes the join result as a native Polars `LazyFrame` source. This lets you fuse spatial join output with downstream Polars operations (sorts, sinks, further filters) into a single spilling pipeline:
4343

4444
```python
4545
(
@@ -52,8 +52,8 @@ Exposes the join result as a native Polars `LazyFrame` source. This lets you fus
5252
)
5353
```
5454

55-
Polars handles spilling to disk for the sort if the result exceeds memory, so the entire pipeline join, select, sort, write never requires the full result to be in RAM at once.
55+
Polars handles spilling to disk for the sort if the result exceeds memory, so the entire pipeline (join, select, sort, write) never requires the full result to be in RAM at once.
5656

5757
## Aggregate-join streaming
5858

59-
`.group_by(keys).agg(...)` reduces over the morsel stream using associative partial aggregations. Each morsel produces per-group partials (counts, sums, etc.) that are combined across morsels at the end. The full pair frame never materialises — only the per-group accumulators are held in memory, which are bounded by the number of unique groups rather than the number of join pairs.
59+
`.group_by(keys).agg(...)` reduces over the morsel stream using associative partial aggregations. Each morsel produces per-group partials (counts, sums, etc.) that are combined across morsels at the end. The full pair frame never materialises. Only the per-group accumulators are held in memory, bounded by the number of unique groups rather than the number of join pairs.

docs/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ A declarative spatial query layer for Polars. Rust core, Python API.
44

55
## What is PyCanopy
66

7-
PyCanopy brings spatial queries range, kNN, joins, polygon containment into the Polars ecosystem without leaving Python. You declare operations in any order; the query planner reorders, fuses, and pushes them down before execution. The index type (KD-tree, R-tree, grid, or brute force) is selected automatically by a cost model calibrated to your hardware.
7+
PyCanopy brings spatial queries (range, kNN, joins, polygon containment) into the Polars ecosystem without leaving Python. You declare operations in any order; the query planner reorders, fuses, and pushes them down before execution. The index type (KD-tree, R-tree, grid, or brute force) is selected automatically by a cost model calibrated to your hardware.
88

99
## Why PyCanopy
1010

docs/quickstart.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ print(lf.explain())
8383
# DF [N=100,000; path: EXPR]
8484
```
8585

86-
The optimizer flipped the declaration order scalar filter runs first, spatial query runs on the smaller survivor set.
86+
The optimizer flipped the declaration order: scalar filter runs first, spatial query runs on the smaller survivor set.
8787

8888
## Streaming large results
8989

@@ -115,7 +115,7 @@ sf = SpatialFrame(df, x_col="lon", y_col="lat", index_mode="auto")
115115
```python
116116
import numpy as np
117117

118-
# Append new pointsvisible to queries immediately, no index rebuild
118+
# Append new points, visible to queries immediately with no index rebuild
119119
sf.engine.append_delta(np.array([2.5]), np.array([48.9]))
120120
result = sf.lazy().range_query(-10.0, 35.0, 40.0, 70.0).collect()
121121

0 commit comments

Comments
 (0)