Skip to content

Commit c09bc02

Browse files
pranav-walimbeclaude
pranav-walimbe
and
claude
committed
trim README content, reference docs site for full API reference
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 61e1d0b commit c09bc02

1 file changed

Lines changed: 2 additions & 288 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 288 deletions
Original file line numberDiff line numberDiff line change
@@ -56,280 +56,9 @@ The only spatial engine with a Polars-native API, cost-model-driven index select
5656

5757
---
5858

59-
## Operations
60-
61-
**Point datasets**
62-
63-
| Operation | Call | Returns |
64-
|:-----------------------|:------------------------------------------------------|:-------------------------------------------------|
65-
| Range query | `.range_query(min_x, min_y, max_x, max_y)` | Rows inside the bounding box |
66-
| Range filter | `.range_filter(min_x, min_y, max_x, max_y)` | New SpatialFrame with only rows inside the bounding box |
67-
| k-nearest neighbours | `.knn(x, y, k)` | The `k` rows nearest a point |
68-
| kNN join | `.knn_join(df, x_col, y_col, k)` | The `k` nearest rows for each query point |
69-
| Within-distance join | `.within_distance_join(df, x_col, y_col, distance)` | Rows within `distance` of each query point |
70-
| Convex-hull area | `SpatialFrame.convex_hull_area(xs, ys)` | Area of the convex hull of a point set |
71-
| Batch convex-hull area | `Engine.group_convex_hull_areas(xs_series, ys_series)` | Convex hull area for each group, given Polars `List[Float64]` columns |
72-
| WKB point distance | `wkb_point_distance(series_a, series_b)` | Euclidean distance between two WKB point columns |
73-
74-
**Polygon datasets**
75-
76-
| Operation | Call | Returns |
77-
|:------------------------------|:-------------------------------------------------------------|:--------------------------------------------------------|
78-
| Point in polygon | `.contains(x, y)` | Polygons that contain the point |
79-
| MBR range | `.range_query(min_x, min_y, max_x, max_y)` | Polygons whose bounding box meets the query box |
80-
| Range filter | `.range_filter(min_x, min_y, max_x, max_y)` | New SpatialFrame with only polygons intersecting the bounding box |
81-
| Within join | `.within_join(df, x_col, y_col)` | Polygons that contain each query point |
82-
| Point-to-polygon distance join | `.polygon_within_distance_join(df, x_col, y_col, distance)` | Polygons within `distance` of each query point |
83-
| Point-to-polygon kNN join | `.polygon_knn_join(df, x_col, y_col, k)` | The `k` nearest polygons for each query point |
84-
| Intersects self-join | `.intersects_pairs(key_col=None)` | Intersecting polygon pairs with overlap area and IoU; `key_col` replaces positional indices with values from that column |
85-
| Area | `.polygon_areas()` | Area of each polygon |
86-
| Points near a polygon | `.points_within_distance_of_polygon(polygon, distance)` | Points within `distance` of a single polygon |
87-
88-
**Reductions and streaming** (compose with any join)
89-
90-
| Operation | Call | Returns |
91-
|:-----------------------|:-----------------------------------------------------------|:-------------------------------------------------------------|
92-
| Aggregate-join | `.group_by(keys).agg(pc.agg.count/sum/mean/min/max(...))` | One row per group, reduced over the join with no pair frame |
93-
| Projection pushdown | `.select(cols)` | Narrows both join sides before the gather |
94-
| Stream in batches | `.collect_batched()` | An iterator of result morsels, bounded memory |
95-
| Stream to Parquet | `.sink_parquet(path)` | Writes the result to disk in bounded memory |
96-
| Out-of-core pipeline | `.lazy_source()` | A Polars source that fuses join + sort + sink, spilling to disk |
59+
## API Reference Docs
9760

98-
---
99-
100-
## Usage
101-
102-
### Point dataset: range and KNN
103-
104-
```python
105-
import polars as pl
106-
from pycanopy import SpatialFrame
107-
108-
df = pl.read_parquet("cities.parquet")
109-
sf = SpatialFrame(df, x_col="lon", y_col="lat")
110-
111-
# Bounding-box filter combined with a scalar predicate.
112-
# Optimizer places the scalar filter first, then runs the range query
113-
# on the reduced row set.
114-
result = (
115-
sf.lazy()
116-
.filter(pl.col("population") > 100_000)
117-
.range_query(min_x=-10.0, min_y=35.0, max_x=40.0, max_y=70.0)
118-
.collect()
119-
)
120-
121-
# k-nearest neighbours
122-
nearest = sf.lazy().knn(x=2.35, y=48.85, k=5).collect()
123-
```
124-
125-
### Inspecting the plan
126-
127-
```python
128-
# Declare ops in any order. explain() shows what the optimizer will actually run.
129-
lf = (
130-
sf.lazy()
131-
.range_query(min_x=-10.0, min_y=35.0, max_x=40.0, max_y=70.0)
132-
.filter(pl.col("population") > 100_000)
133-
)
134-
135-
print(lf.explain())
136-
# RANGE_QUERY [(-10, 35) → (40, 70)]
137-
# FROM
138-
# FILTER [(col("population")) > (dyn int: 100000)]
139-
# FROM
140-
# DF [N=100,000; path: EXPR]
141-
```
142-
143-
The optimizer flipped the declaration order. The scalar filter runs first on all rows, then the spatial query runs on the smaller survivor set. Plans follow Polars' FROM-chain convention, so the bottom runs first and the top is the final result.
144-
145-
### Aggregate over a join
146-
147-
```python
148-
import pycanopy as pc
149-
150-
# Count trips per zone and average their fare, reduced over a streamed
151-
# point-in-polygon join. The full pair frame is never materialised: each
152-
# morsel reduces to per-group partials that combine into the final result.
153-
stats = (
154-
zones.lazy()
155-
.within_join(trips, x_col="lon", y_col="lat")
156-
.group_by(["zone_id", "zone_name"])
157-
.agg(trip_count=pc.agg.count(), avg_fare=pc.agg.mean("fare"))
158-
)
159-
```
160-
161-
### Out-of-core joins (larger than RAM)
162-
163-
```python
164-
# A join whose result exceeds memory: stream it straight to Parquet,
165-
# bounded to one morsel at a time.
166-
sf.lazy().polygon_knn_join(trips, "lon", "lat", k=5).sink_parquet("nearest.parquet")
167-
168-
# Or fuse the join with a sort and sink into a single spilling Polars
169-
# pipeline, so even an ordered result larger than RAM never materialises.
170-
(
171-
sf.lazy()
172-
.polygon_knn_join(trips, "lon", "lat", k=5)
173-
.select(["trip_id", "building_id", "distance_to_polygon"])
174-
.lazy_source()
175-
.sort("distance_to_polygon")
176-
.sink_parquet("nearest_sorted.parquet")
177-
)
178-
```
179-
180-
<details>
181-
<summary>More examples: point and polygon joins, aggregations, branching, delta buffer, index modes</summary>
182-
183-
### Chaining multiple spatial predicates
184-
185-
```python
186-
# Two range predicates are fused into a single index build on large datasets.
187-
result = (
188-
sf.lazy()
189-
.range_query(0.0, 0.0, 50.0, 50.0)
190-
.range_query(10.0, 10.0, 40.0, 40.0)
191-
.collect()
192-
)
193-
```
194-
195-
### KNN join
196-
197-
```python
198-
query_df = pl.DataFrame({"qx": [2.35, 13.4], "qy": [48.85, 52.5]})
199-
200-
# For each row in query_df, find the 3 nearest rows in sf.
201-
result = sf.lazy().knn_join(query_df, x_col="qx", y_col="qy", k=3).collect()
202-
```
203-
204-
### Polygon dataset: contains and range
205-
206-
```python
207-
from shapely.geometry import box
208-
from pycanopy import SpatialFrame
209-
210-
polygons = [box(i, 0, i + 0.9, 0.9) for i in range(100_000)]
211-
df = pl.DataFrame({"id": list(range(100_000)), "geom": polygons})
212-
sf = SpatialFrame.from_polygons(df, geometry_col="geom")
213-
214-
# Which polygons contain this point?
215-
containing = sf.lazy().contains(x=5.5, y=0.5).collect()
216-
217-
# Which polygon MBRs intersect this bbox?
218-
intersecting = sf.lazy().range_query(0.0, 0.0, 10.0, 1.0).collect()
219-
```
220-
221-
### Polygon holes
222-
223-
```python
224-
from shapely.geometry import Polygon
225-
226-
# Interior rings (holes) are fully supported.
227-
outer = [(0, 0), (10, 0), (10, 10), (0, 10)]
228-
hole = [(2, 2), (8, 2), (8, 8), (2, 8)]
229-
donut = Polygon(outer, [hole])
230-
231-
sf = SpatialFrame.from_polygons(pl.DataFrame({"id": [0], "geom": [donut]}), geometry_col="geom")
232-
233-
# Point inside the hole is NOT contained.
234-
sf.lazy().contains(x=5.0, y=5.0).collect() # empty
235-
236-
# Point outside the hole but inside the outer ring IS contained.
237-
sf.lazy().contains(x=1.0, y=1.0).collect() # returns the polygon row
238-
```
239-
240-
### Within join
241-
242-
```python
243-
# For each query point, find which polygons in sf contain it.
244-
query_df = pl.DataFrame({"qx": [5.5, 12.3], "qy": [0.5, 0.5]})
245-
result = sf.lazy().within_join(query_df, x_col="qx", y_col="qy").collect()
246-
```
247-
248-
### Within-distance join
249-
250-
```python
251-
# For each query point, find all sf points within 50 km.
252-
query_df = pl.DataFrame({"qx": [2.35, 13.4], "qy": [48.85, 52.5]})
253-
result = sf.lazy().within_distance_join(query_df, x_col="qx", y_col="qy", distance=50.0).collect()
254-
```
255-
256-
### Point-to-polygon joins
257-
258-
```python
259-
# (polygon SpatialFrame) For each query point, the polygons within a distance
260-
# of it. Distance is to the polygon boundary, and zero when the point is inside.
261-
query_df = pl.DataFrame({"qx": [5.5, 12.3], "qy": [0.5, 0.5]})
262-
near = sf.lazy().polygon_within_distance_join(query_df, x_col="qx", y_col="qy", distance=2.0).collect()
263-
264-
# For each query point, its k nearest polygons (adds a distance_to_polygon column).
265-
nearest = sf.lazy().polygon_knn_join(query_df, x_col="qx", y_col="qy", k=3).collect()
266-
```
267-
268-
### Polygon aggregations
269-
270-
```python
271-
# Area of every polygon (appends an 'area' column).
272-
areas = sf.polygon_areas()
273-
274-
# All intersecting polygon pairs, with overlap area and IoU.
275-
overlaps = sf.intersects_pairs()
276-
277-
# (point SpatialFrame) rows whose point lies within a distance of one polygon.
278-
from shapely.geometry import box
279-
pts = point_sf.points_within_distance_of_polygon(box(0.0, 0.0, 1.0, 1.0), distance=0.5)
280-
```
281-
282-
### Convex-hull area
283-
284-
```python
285-
import numpy as np
286-
287-
# Area of the convex hull of a standalone point set (no frame needed).
288-
area = SpatialFrame.convex_hull_area(np.array([0.0, 1.0, 0.5]), np.array([0.0, 0.0, 1.0]))
289-
```
290-
291-
### Index mode
292-
293-
```python
294-
# Fixed per frame. "auto" lets the cost model choose index vs scan per query;
295-
# "auto" (default) builds when justified and reuses for free after. "eager" always builds. "none" always scans.
296-
sf = SpatialFrame(df, x_col="lon", y_col="lat", index_mode="auto")
297-
```
298-
299-
### Branching from a shared base
300-
301-
```python
302-
from pycanopy import SpatialFrame, SpatialLazyFrame
303-
304-
# Expensive filter applied once; two queries branch from the result.
305-
base = sf.lazy().filter(pl.col("population") > 100_000).range_query(-10.0, 35.0, 40.0, 70.0)
306-
307-
major = base.filter(pl.col("population") > 1_000_000)
308-
minor = base.filter(pl.col("population") <= 1_000_000)
309-
310-
# collect_all detects the shared prefix, caches it in Polars,
311-
# and executes both branches in a single pass.
312-
results = SpatialLazyFrame.collect_all([major, minor])
313-
df_major, df_minor = results
314-
```
315-
316-
### Live updates via delta buffer
317-
318-
```python
319-
# Append new points -- visible to queries immediately, no index rebuild yet.
320-
import numpy as np
321-
sf.engine.append_delta(np.array([2.5]), np.array([48.9]))
322-
323-
# Queries probe the main index and scan the delta in parallel.
324-
result = sf.lazy().range_query(-10.0, 35.0, 40.0, 70.0).collect()
325-
326-
# The buffer flushes automatically when accumulated query cost exceeds
327-
# the estimated index rebuild cost, or when it exceeds 10% of N.
328-
# Force a flush manually if needed.
329-
sf.engine.flush()
330-
```
331-
332-
</details>
61+
The full API reference, operation catalogue, and technical deep dive into the planner and cost model live at **[pranav-walimbe.github.io/PyCanopy](https://pranav-walimbe.github.io/PyCanopy)**.
33362

33463
---
33564

@@ -509,21 +238,6 @@ The hot paths need packed immutable index structures, zero-copy array slices at
509238

510239
---
511240

512-
## Accepted input formats
513-
514-
| Format | Example |
515-
|:-----------------------------------|:-------------------------------------------|
516-
| numpy `(N, 2)` array | `np.array([[x, y], ...])` |
517-
| GeoArrow PyArrow array | `pa.StructArray` or `FixedSizeList<2>` |
518-
| geopandas `GeoSeries` | `gdf.geometry` |
519-
| shapely Points / Polygons / MultiPolygons | `[Point(x, y), ...]` |
520-
| list of `(x, y)` tuples | `[(x, y), ...]` |
521-
| Separate coordinate sequences | `Engine.from_coords(xs, ys)` |
522-
| WKB point column (Binary) | `SpatialFrame.from_wkb_points(df, "geom")` |
523-
| WKB polygon column (Binary) | `SpatialFrame.from_wkb_polygons(df, "geom")` |
524-
525-
---
526-
527241
## Acknowledgements
528242

529243
Some works that inspired this project:

0 commit comments

Comments
 (0)