Skip to content

Commit 86d129d

Browse files
giswqspre-commit-ci[bot]
giswqs
and
pre-commit-ci[bot]
authored
Add first-class Polars DataFrame support (#1283)
* Add first-class Polars DataFrame support Fixes #1276 This commit adds native support for Polars-based geospatial workflows in leafmap, addressing the feature request for first-class Polars-ST and polars-h3 integration. ## Changes: ### New Method: add_polars() - Accepts Polars DataFrames with geometry columns (WKB or WKT format) - Supports Polars-ST spatial DataFrames - Automatic CRS detection with fallback to EPSG:4326 - Full compatibility with existing leafmap styling and options - Internally converts to GeoPandas for rendering ### Features: - Works with WKB (Well-Known Binary) geometries from Polars-ST - Works with WKT (Well-Known Text) geometry strings - Supports all geometry types (Point, LineString, Polygon, etc.) - Preserves non-geometry columns as attributes - Compatible with polars-h3 H3-indexed workflows - Comprehensive error handling and validation ### Documentation: - Added example notebook: examples/notebooks/polars_support.ipynb - Includes examples for: - Simple WKT point data - Converting GeoDataFrame to Polars - Polars-first data processing pipelines - Filtering and visualization ## Benefits: 1. Users can stay in Polars-native workflows end-to-end 2. No manual conversion code required 3. Better performance for large datasets 4. Seamless integration with Polars-ST and polars-h3 5. Natural visualization frontend for Arrow-native geospatial stacks ## Implementation: The add_polars() method: 1. Validates input is a Polars DataFrame 2. Checks geometry column exists 3. Converts Polars to pandas 4. Parses WKB/WKT geometries using shapely 5. Creates GeoPandas GeoDataFrame 6. Calls existing add_gdf() method This approach ensures compatibility with all existing leafmap backends (folium, ipyleaflet, maplibre, etc.) without requiring changes to the rendering pipeline. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add polars as optional dependency and update documentation - Added 'polars' optional dependency in pyproject.toml - Updated installation.md with Polars support section - Documented pip install "leafmap[polars]" command - Listed other optional dependencies for reference - Tested with geo conda environment - all tests passed * Add add_polars() to folium and maplibre backends - Added add_polars() method to leafmap/foliumap.py - Added add_polars() method to leafmap/maplibregl.py - Both backends now support Polars DataFrame visualization - Tested with WKT and WKB geometries - All backends (ipyleaflet, folium, maplibre) working correctly The method signatures match each backend's existing add_gdf() API: - foliumap: Simpler API with layer_name, zoom_to_layer, info_mode, opacity - maplibre: Full API with layer_type, filter, paint, fit_bounds, visible, etc. All tests passed with geo conda environment. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix installation.md: restore missing bash code fences Accidentally removed bash code markers when adding Optional Dependencies section. All bash code blocks now properly formatted with code fences. * Update notebook example * Address GitHub Copilot review comments for add_polars() Fixes based on 10 Copilot comments: 1. Type consistency: Changed Optional[list[str]] to Optional[List[str]] 2. Added missing 'encoding' parameter to match add_gdf() API 3. Improved exception handling: Use specific TypeError/ValueError instead of bare except 4. Replace print() with warnings.warn() for UserWarning 5. Added datetime column handling (same as add_gdf()) 6. Fixed notebook imports: Added wkb to top-level imports, reordered Example 3 7. Simplified CRS detection: Removed speculative Polars-ST metadata code 8. Improved performance: Use vectorized gpd.GeoSeries.from_wkb/from_wkt 9. Added null geometry validation before processing 10. Better error messages with specific parse failures Changes: - Use warnings.warn() with UserWarning and stacklevel=2 - Vectorized geometry parsing with gpd.GeoSeries.from_wkb() and from_wkt() - Early validation for null/empty geometry columns - Datetime handling to prevent GeoJSON serialization issues - Clear CRS warning message directing users to 'crs' parameter * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 67f0da4 commit 86d129d

7 files changed

Lines changed: 754 additions & 0 deletions

File tree

‎docs/installation.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,33 @@ Optionally, you can install some [Jupyter notebook extensions](https://github.co
3030
conda install jupyter_contrib_nbextensions -c conda-forge
3131
```
3232

33+
## Optional Dependencies
34+
35+
Leafmap has various optional dependencies to enable additional features. You can install them as needed:
36+
37+
### Polars Support
38+
39+
To enable Polars DataFrame support for modern, high-performance geospatial workflows:
40+
41+
```bash
42+
pip install "leafmap[polars]"
43+
```
44+
45+
This enables the `add_polars()` method for visualizing Polars DataFrames with geometry columns (e.g., from Polars-ST or polars-h3).
46+
47+
### Other Optional Dependencies
48+
49+
Install other optional features:
50+
51+
```bash
52+
pip install "leafmap[vector]" # Vector analysis tools
53+
pip install "leafmap[raster]" # Raster data support
54+
pip install "leafmap[lidar]" # LiDAR data analysis
55+
pip install "leafmap[backends]" # All mapping backends
56+
pip install "leafmap[maplibre]" # MapLibre GL backend
57+
pip install "leafmap[ai]" # AI/ML features
58+
```
59+
3360
## Install from GitHub
3461

3562
To install the development version from GitHub using [Git](https://git-scm.com/), run the following command in your terminal:

‎docs/notebooks/110_polars.ipynb‎

Lines changed: 236 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,236 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Polars DataFrame Support in Leafmap\n",
8+
"\n",
9+
"This notebook demonstrates first-class support for Polars-based geospatial workflows in leafmap.\n",
10+
"\n",
11+
"## Installation\n",
12+
"\n",
13+
"```bash\n",
14+
"pip install leafmap polars geopandas\n",
15+
"```\n",
16+
"\n",
17+
"For Polars-ST support:\n",
18+
"```bash\n",
19+
"pip install polars-st\n",
20+
"```"
21+
]
22+
},
23+
{
24+
"cell_type": "code",
25+
"execution_count": null,
26+
"metadata": {},
27+
"outputs": [],
28+
"source": [
29+
"import leafmap\n",
30+
"import polars as pl\n",
31+
"import geopandas as gpd\n",
32+
"from shapely.geometry import Point, Polygon, box\n",
33+
"from shapely import wkb"
34+
]
35+
},
36+
{
37+
"cell_type": "markdown",
38+
"metadata": {},
39+
"source": [
40+
"## Example 1: Simple Point Data\n",
41+
"\n",
42+
"Create a Polars DataFrame with point geometries using WKT (Well-Known Text)"
43+
]
44+
},
45+
{
46+
"cell_type": "code",
47+
"execution_count": null,
48+
"metadata": {},
49+
"outputs": [],
50+
"source": [
51+
"# Create sample data with WKT geometries\n",
52+
"data = {\n",
53+
" \"city\": [\"New York\", \"Los Angeles\", \"Chicago\", \"Houston\", \"Phoenix\"],\n",
54+
" \"population\": [8336817, 3979576, 2693976, 2320268, 1680992],\n",
55+
" \"geometry\": [\n",
56+
" \"POINT(-74.0060 40.7128)\", # New York\n",
57+
" \"POINT(-118.2437 34.0522)\", # Los Angeles\n",
58+
" \"POINT(-87.6298 41.8781)\", # Chicago\n",
59+
" \"POINT(-95.3698 29.7604)\", # Houston\n",
60+
" \"POINT(-112.0740 33.4484)\", # Phoenix\n",
61+
" ],\n",
62+
"}\n",
63+
"\n",
64+
"df_polars = pl.DataFrame(data)\n",
65+
"print(df_polars)"
66+
]
67+
},
68+
{
69+
"cell_type": "code",
70+
"execution_count": null,
71+
"metadata": {},
72+
"outputs": [],
73+
"source": [
74+
"# Visualize with leafmap\n",
75+
"m = leafmap.Map(center=[37.0902, -95.7129], zoom=4)\n",
76+
"m.add_polars(\n",
77+
" df_polars,\n",
78+
" geometry=\"geometry\",\n",
79+
" crs=\"EPSG:4326\",\n",
80+
" layer_name=\"US Cities\",\n",
81+
" zoom_to_layer=True,\n",
82+
")\n",
83+
"m"
84+
]
85+
},
86+
{
87+
"cell_type": "markdown",
88+
"metadata": {},
89+
"source": [
90+
"## Example 2: Converting from GeoDataFrame to Polars\n",
91+
"\n",
92+
"Read GeoJSON/GeoParquet and work with it in Polars"
93+
]
94+
},
95+
{
96+
"cell_type": "code",
97+
"execution_count": null,
98+
"metadata": {},
99+
"outputs": [],
100+
"source": [
101+
"# Start with a GeoDataFrame\n",
102+
"url = \"https://github.com/opengeos/datasets/releases/download/vector/cables.geojson\"\n",
103+
"gdf = gpd.read_file(url)\n",
104+
"\n",
105+
"# Convert to Polars with WKB geometry\n",
106+
"from shapely import wkb\n",
107+
"\n",
108+
"# Convert geometries to WKB bytes\n",
109+
"gdf[\"geometry_wkb\"] = gdf[\"geometry\"].apply(lambda x: wkb.dumps(x))\n",
110+
"df_polars = pl.DataFrame(gdf.drop(columns=[\"geometry\"]))\n",
111+
"\n",
112+
"print(df_polars.head())"
113+
]
114+
},
115+
{
116+
"cell_type": "code",
117+
"execution_count": null,
118+
"metadata": {},
119+
"outputs": [],
120+
"source": [
121+
"# Visualize the Polars DataFrame\n",
122+
"m = leafmap.Map()\n",
123+
"m.add_polars(\n",
124+
" df_polars,\n",
125+
" geometry=\"geometry_wkb\",\n",
126+
" crs=\"EPSG:4326\",\n",
127+
" layer_name=\"Submarine Cables\",\n",
128+
" zoom_to_layer=True,\n",
129+
" style={\"color\": \"blue\", \"weight\": 2},\n",
130+
")\n",
131+
"m"
132+
]
133+
},
134+
{
135+
"cell_type": "markdown",
136+
"metadata": {},
137+
"source": [
138+
"## Example 3: Polars Data Processing Pipeline\n",
139+
"\n",
140+
"Demonstrate Polars-first workflow with filtering and aggregation"
141+
]
142+
},
143+
{
144+
"cell_type": "code",
145+
"execution_count": null,
146+
"metadata": {},
147+
"outputs": [],
148+
"source": [
149+
"from shapely.geometry import box\n",
150+
"import numpy as np\n",
151+
"\n",
152+
"# Create sample polygon data\n",
153+
"\n",
154+
"# Create grid of rectangles\n",
155+
"geometries = []\n",
156+
"names = []\n",
157+
"values = []\n",
158+
"\n",
159+
"\n",
160+
"for i in range(-5, 5):\n",
161+
" for j in range(-5, 5):\n",
162+
" geometries.append(wkb.dumps(box(i, j, i + 0.8, j + 0.8)))\n",
163+
" names.append(f\"Cell_{i}_{j}\")\n",
164+
" values.append(np.random.randint(0, 100))\n",
165+
"\n",
166+
"df_grid = pl.DataFrame({\"name\": names, \"value\": values, \"geometry\": geometries})\n",
167+
"\n",
168+
"print(df_grid.head())\n",
169+
"print(f\"\\nTotal cells: {len(df_grid)}\")"
170+
]
171+
},
172+
{
173+
"cell_type": "code",
174+
"execution_count": null,
175+
"metadata": {},
176+
"outputs": [],
177+
"source": [
178+
"# Filter high-value cells using Polars\n",
179+
"df_filtered = df_grid.filter(pl.col(\"value\") > 75)\n",
180+
"print(f\"High-value cells: {len(df_filtered)}\")\n",
181+
"\n",
182+
"# Visualize\n",
183+
"m = leafmap.Map(center=[0, 0], zoom=6)\n",
184+
"m.add_polars(\n",
185+
" df_filtered,\n",
186+
" geometry=\"geometry\",\n",
187+
" crs=\"EPSG:4326\",\n",
188+
" layer_name=\"High Value Cells\",\n",
189+
" zoom_to_layer=True,\n",
190+
" style={\"fillColor\": \"red\", \"fillOpacity\": 0.5, \"color\": \"darkred\"},\n",
191+
")\n",
192+
"m"
193+
]
194+
},
195+
{
196+
"cell_type": "markdown",
197+
"metadata": {},
198+
"source": [
199+
"## Benefits of Polars Integration\n",
200+
"\n",
201+
"1. **Performance**: Polars is significantly faster than Pandas for large datasets\n",
202+
"2. **Memory Efficiency**: Better memory management with Apache Arrow backend\n",
203+
"3. **Native Pipeline**: Stay in Polars-native workflow end-to-end\n",
204+
"4. **Modern API**: Expressive and intuitive query syntax\n",
205+
"5. **GeoParquet Support**: Native support for reading/writing GeoParquet files\n",
206+
"\n",
207+
"## Next Steps\n",
208+
"\n",
209+
"- Explore [Polars-ST](https://github.com/Oreilles/polars-st) for advanced spatial operations\n",
210+
"- Try [polars-h3](https://github.com/Filimoa/polars-h3) for H3 indexing workflows\n",
211+
"- Read GeoParquet files directly with Polars for maximum performance"
212+
]
213+
}
214+
],
215+
"metadata": {
216+
"kernelspec": {
217+
"display_name": "geo",
218+
"language": "python",
219+
"name": "python3"
220+
},
221+
"language_info": {
222+
"codemirror_mode": {
223+
"name": "ipython",
224+
"version": 3
225+
},
226+
"file_extension": ".py",
227+
"mimetype": "text/x-python",
228+
"name": "python",
229+
"nbconvert_exporter": "python",
230+
"pygments_lexer": "ipython3",
231+
"version": "3.12.12"
232+
}
233+
},
234+
"nbformat": 4,
235+
"nbformat_minor": 4
236+
}

0 commit comments

Comments
 (0)