-
-
Notifications
You must be signed in to change notification settings - Fork 465
Add first-class Polars DataFrame support #1283
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
f9517fd
6951472
7672555
f7db421
06ac685
d522680
ef7844f
66951c0
c2f15dc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,236 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Polars DataFrame Support in Leafmap\n", | ||
| "\n", | ||
| "This notebook demonstrates first-class support for Polars-based geospatial workflows in leafmap.\n", | ||
| "\n", | ||
| "## Installation\n", | ||
| "\n", | ||
| "```bash\n", | ||
| "pip install leafmap polars geopandas\n", | ||
| "```\n", | ||
| "\n", | ||
| "For Polars-ST support:\n", | ||
| "```bash\n", | ||
| "pip install polars-st\n", | ||
| "```" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import leafmap\n", | ||
| "import polars as pl\n", | ||
| "import geopandas as gpd\n", | ||
| "from shapely.geometry import Point, Polygon, box\n", | ||
| "from shapely import wkb" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Example 1: Simple Point Data\n", | ||
| "\n", | ||
| "Create a Polars DataFrame with point geometries using WKT (Well-Known Text)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Create sample data with WKT geometries\n", | ||
| "data = {\n", | ||
| " \"city\": [\"New York\", \"Los Angeles\", \"Chicago\", \"Houston\", \"Phoenix\"],\n", | ||
| " \"population\": [8336817, 3979576, 2693976, 2320268, 1680992],\n", | ||
| " \"geometry\": [\n", | ||
| " \"POINT(-74.0060 40.7128)\", # New York\n", | ||
| " \"POINT(-118.2437 34.0522)\", # Los Angeles\n", | ||
| " \"POINT(-87.6298 41.8781)\", # Chicago\n", | ||
| " \"POINT(-95.3698 29.7604)\", # Houston\n", | ||
| " \"POINT(-112.0740 33.4484)\", # Phoenix\n", | ||
| " ],\n", | ||
| "}\n", | ||
| "\n", | ||
| "df_polars = pl.DataFrame(data)\n", | ||
| "print(df_polars)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Visualize with leafmap\n", | ||
| "m = leafmap.Map(center=[37.0902, -95.7129], zoom=4)\n", | ||
| "m.add_polars(\n", | ||
| " df_polars,\n", | ||
| " geometry=\"geometry\",\n", | ||
| " crs=\"EPSG:4326\",\n", | ||
| " layer_name=\"US Cities\",\n", | ||
| " zoom_to_layer=True,\n", | ||
| ")\n", | ||
| "m" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Example 2: Converting from GeoDataFrame to Polars\n", | ||
| "\n", | ||
| "Read GeoJSON/GeoParquet and work with it in Polars" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Start with a GeoDataFrame\n", | ||
| "url = \"https://github.com/opengeos/datasets/releases/download/vector/cables.geojson\"\n", | ||
| "gdf = gpd.read_file(url)\n", | ||
| "\n", | ||
| "# Convert to Polars with WKB geometry\n", | ||
| "from shapely import wkb\n", | ||
| "\n", | ||
| "# Convert geometries to WKB bytes\n", | ||
| "gdf[\"geometry_wkb\"] = gdf[\"geometry\"].apply(lambda x: wkb.dumps(x))\n", | ||
| "df_polars = pl.DataFrame(gdf.drop(columns=[\"geometry\"]))\n", | ||
| "\n", | ||
| "print(df_polars.head())" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Visualize the Polars DataFrame\n", | ||
| "m = leafmap.Map()\n", | ||
| "m.add_polars(\n", | ||
| " df_polars,\n", | ||
| " geometry=\"geometry_wkb\",\n", | ||
| " crs=\"EPSG:4326\",\n", | ||
| " layer_name=\"Submarine Cables\",\n", | ||
| " zoom_to_layer=True,\n", | ||
| " style={\"color\": \"blue\", \"weight\": 2},\n", | ||
| ")\n", | ||
| "m" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Example 3: Polars Data Processing Pipeline\n", | ||
| "\n", | ||
| "Demonstrate Polars-first workflow with filtering and aggregation" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from shapely.geometry import box\n", | ||
| "import numpy as np\n", | ||
| "\n", | ||
| "# Create sample polygon data\n", | ||
| "\n", | ||
| "# Create grid of rectangles\n", | ||
| "geometries = []\n", | ||
| "names = []\n", | ||
| "values = []\n", | ||
| "\n", | ||
| "\n", | ||
| "for i in range(-5, 5):\n", | ||
| " for j in range(-5, 5):\n", | ||
| " geometries.append(wkb.dumps(box(i, j, i + 0.8, j + 0.8)))\n", | ||
| " names.append(f\"Cell_{i}_{j}\")\n", | ||
| " values.append(np.random.randint(0, 100))\n", | ||
|
Comment on lines
+144
to
+164
|
||
| "\n", | ||
| "df_grid = pl.DataFrame({\"name\": names, \"value\": values, \"geometry\": geometries})\n", | ||
| "\n", | ||
| "print(df_grid.head())\n", | ||
| "print(f\"\\nTotal cells: {len(df_grid)}\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Filter high-value cells using Polars\n", | ||
| "df_filtered = df_grid.filter(pl.col(\"value\") > 75)\n", | ||
| "print(f\"High-value cells: {len(df_filtered)}\")\n", | ||
| "\n", | ||
| "# Visualize\n", | ||
| "m = leafmap.Map(center=[0, 0], zoom=6)\n", | ||
| "m.add_polars(\n", | ||
| " df_filtered,\n", | ||
| " geometry=\"geometry\",\n", | ||
| " crs=\"EPSG:4326\",\n", | ||
| " layer_name=\"High Value Cells\",\n", | ||
| " zoom_to_layer=True,\n", | ||
| " style={\"fillColor\": \"red\", \"fillOpacity\": 0.5, \"color\": \"darkred\"},\n", | ||
| ")\n", | ||
| "m" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Benefits of Polars Integration\n", | ||
| "\n", | ||
| "1. **Performance**: Polars is significantly faster than Pandas for large datasets\n", | ||
| "2. **Memory Efficiency**: Better memory management with Apache Arrow backend\n", | ||
| "3. **Native Pipeline**: Stay in Polars-native workflow end-to-end\n", | ||
| "4. **Modern API**: Expressive and intuitive query syntax\n", | ||
| "5. **GeoParquet Support**: Native support for reading/writing GeoParquet files\n", | ||
| "\n", | ||
| "## Next Steps\n", | ||
| "\n", | ||
| "- Explore [Polars-ST](https://github.com/Oreilles/polars-st) for advanced spatial operations\n", | ||
| "- Try [polars-h3](https://github.com/Filimoa/polars-h3) for H3 indexing workflows\n", | ||
| "- Read GeoParquet files directly with Polars for maximum performance" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "geo", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.12.12" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 4 | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The import
from shapely import wkbis done in line 105 (Example 2 cell) but then used in line 160 (Example 3 cell). If users run Example 3 without first running Example 2, they will encounter a NameError. Consider either moving the wkb import to the initial imports cell (lines 29-32) or adding it to the Example 3 cell where it's used.