Merge pull request #26 from googlemaps-samples/feat/full-dataset-support

feat: Enable Full Dataset Support & Improve Documentation

Author: henrikvalv3

places_insights/notebooks/sample_data_demo/places_insights_sample_data_demo.ipynb

@@ -76,9 +76,9 @@
       "source": [
         "### Overall Goal\n",
         "\n",
-        "This notebook serves as a technical introduction for developers and data analysts who have subscribed to the [Places Insights Sample Datasets](https://developers.google.com/maps/documentation/placesinsights/cloud-setup#sample_data).\n",
+        "This notebook serves as a technical introduction for developers and data analysts who have subscribed to the **[Places Insights Datasets](https://developers.google.com/maps/documentation/placesinsights)**.\n",
         "\n",
-        "Its primary purpose is to demonstrate how to query, aggregate, and visualize Google Maps Platform [Places Insights](https://developers.google.com/maps/documentation/placesinsights) data within a **BigQuery** environment. By running this notebook, you will learn how to transition from raw dataset subscriptions to actionable geospatial insights using Standard SQL and Python.\n",
+        "Its primary purpose is to demonstrate how to query, aggregate, and visualize Google Maps Platform Places Insights data within a **BigQuery** environment. Whether you are evaluating the product using **Sample Data** or building production workflows with **Full Data**, this notebook will help you transition from raw dataset subscriptions to actionable geospatial insights using Standard SQL and Python.\n",
         "\n",
         "### Key Technologies Used\n",
         "\n",
@@ -94,7 +94,7 @@
         "\n",
         "### The Step-by-Step Workflow\n",
         "\n",
-        "1.  **Configuration:** The notebook initializes the environment based on your selection of a Sample City (e.g., \"New York City\", \"Tokyo\", \"London\"). It automatically maps your selection to the correct **Analytics Hub Dataset ID** and geographic coordinates.\n",
+        "1.  **Configuration:** The notebook initializes the environment based on your selection of a City (e.g., \"New York City\", \"Tokyo\") and your **Data Source** (Sample or Full). It automatically maps your selection to the correct Analytics Hub Dataset ID and geographic coordinates.\n",
         "\n",
         "2.  **Direct Query Analysis (Radius Search):** We demonstrate how to count places that match specific criteria, such as `primary_type`, `business_status`, and boolean attributes like `allows_dogs`, within a set radius. We utilize the `WITH AGGREGATION_THRESHOLD` clause to compare amenity density across multiple neighborhoods programmatically.\n",
         "\n",
@@ -108,7 +108,7 @@
         "\n",
         "1.  **Prerequisites:**\n",
         "    *   A Google Cloud Project with the **BigQuery API** and **Map Tiles API** enabled.\n",
-        "    *   A subscription to at least one of the [Places Insights Sample Datasets](https://developers.google.com/maps/documentation/placesinsights/cloud-setup#sample_data) via Analytics Hub.\n",
+        "    *   A subscription to either the [Sample Datasets](https://developers.google.com/maps/documentation/placesinsights/cloud-setup#sample_data) or the [Full Datasets](https://developers.google.com/maps/documentation/placesinsights/cloud-setup#full_data) via Analytics Hub.\n",
         "\n",
         "2.  **Set Up Secrets:** Configure the following keys in the Colab \"Secrets\" tab (the **key icon** on the left menu):\n",
         "    *   `GCP_PROJECT_ID`: Your Google Cloud Project ID.\n",
@@ -162,17 +162,22 @@
       "cell_type": "code",
       "source": [
         "# @title 2. Configuration\n",
-        "# @markdown Select a Sample City to demo. This configures the dataset target and map center.\n",
-        "# @markdown\n",
-        "# @markdown **Important:** You must be subscribed to the [Sample Dataset](https://developers.google.com/maps/documentation/placesinsights/cloud-setup#sample_data) in your GCP project.\n",
-        "# @markdown\n",
-        "# @markdown If you renamed the dataset during the Analytics Hub subscription process, enter the full table name (e.g., `my_project.my_dataset.places_sample`) in the **Custom Table Name** field below. Leave it blank to use the default.\n",
+        "# @markdown Select a Sample City to demo. This configures the map center and the target BigQuery dataset.\n",
         "\n",
+        "# @markdown ### 1. Select Location\n",
         "SAMPLE_LOCATION = \"New York City, USA\" # @param [\"Sydney, Australia\", \"Sao Paulo, Brazil\", \"Toronto, Canada\", \"Paris, France\", \"Berlin, Germany\", \"Mumbai, India\", \"Jakarta, Indonesia\", \"Rome, Italy\", \"Tokyo, Japan\", \"Mexico City, Mexico\", \"Madrid, Spain\", \"Zurich, Switzerland\", \"London, UK\", \"New York City, USA\"]\n",
+        "\n",
+        "# @markdown ### 2. Select Data Source\n",
+        "# @markdown Choose **Sample Data** for evaluation (requires [\"Sample\" subscription](https://developers.google.com/maps/documentation/placesinsights/cloud-setup#sample_data)) or **Full Data** for production (requires [\"Full\" subscription](https://developers.google.com/maps/documentation/placesinsights/cloud-setup#full_data)).\n",
+        "DATASET_MODE = \"Sample Data\" # @param [\"Sample Data\", \"Full Data\"]\n",
+        "\n",
+        "# @markdown ### 3. Custom Override (Optional)\n",
+        "# @markdown Enter a specific table name (e.g., `my_project.my_dataset.my_table`) to override the default.\n",
+        "# @markdown\n",
+        "# @markdown Only use this field if you renamed the dataset or table during the Analytics Hub subscription process. Leave blank to use the default naming convention.\n",
         "CUSTOM_TABLE_NAME = \"\" # @param {type:\"string\"}\n",
         "\n",
-        "# Configuration Dictionary mapping selection to Dataset and Center Coordinates\n",
-        "# Note: These table names assume the default naming convention from Analytics Hub.\n",
+        "# Configuration Dictionary mapping selection to Country Code and Center Coordinates\n",
         "config_map = {\n",
         "    \"Sydney, Australia\":    {\"code\": \"au\", \"center\": (-33.8688, 151.2093)},\n",
         "    \"Sao Paulo, Brazil\":    {\"code\": \"br\", \"center\": (-23.5505, -46.6333)},\n",
@@ -190,17 +195,26 @@
         "    \"New York City, USA\":   {\"code\": \"us\", \"center\": (40.7580, -73.9855)},\n",
         "}\n",
         "\n",
+        "# 1. Retrieve Config\n",
         "selected_config = config_map[SAMPLE_LOCATION]\n",
+        "code = selected_config['code']\n",
         "CITY_CENTER_LAT, CITY_CENTER_LNG = selected_config['center']\n",
         "\n",
-        "# Determine Table Name\n",
+        "# 2. Determine Table Name\n",
         "if CUSTOM_TABLE_NAME and CUSTOM_TABLE_NAME.strip():\n",
+        "    # Priority 1: User Custom Override\n",
         "    DATASET_TABLE = CUSTOM_TABLE_NAME.strip()\n",
         "    print(f\"⚠️ Using Custom Table Name: {DATASET_TABLE}\")\n",
         "else:\n",
-        "    # Default construction\n",
-        "    DATASET_TABLE = f\"places_insights___{selected_config['code']}___sample.places_sample\"\n",
-        "    print(f\"🎯 Target Table (Default): {DATASET_TABLE}\")\n",
+        "    # Priority 2: Auto-Construct based on Mode\n",
+        "    if DATASET_MODE == \"Sample Data\":\n",
+        "        # Pattern: places_insights___us___sample.places_sample\n",
+        "        DATASET_TABLE = f\"places_insights___{code}___sample.places_sample\"\n",
+        "    else:\n",
+        "        # Pattern: places_insights___us.places\n",
+        "        DATASET_TABLE = f\"places_insights___{code}.places\"\n",
+        "\n",
+        "    print(f\"🎯 Target Table ({DATASET_MODE}): {DATASET_TABLE}\")\n",
         "\n",
         "print(f\"🌍 Selected Location: {SAMPLE_LOCATION}\")\n",
         "print(f\"📍 Center Point: {CITY_CENTER_LAT}, {CITY_CENTER_LNG}\")"
@@ -292,8 +306,14 @@
     {
       "cell_type": "code",
       "source": [
-        "# @title 4. Direct Query: Analyzing \"Al Fresco & Dog-Friendly\" Hubs\n",
-        "# @markdown The following cell runs the BigQuery analysis to find hotspots.\n",
+        "# @title 4. Direct Query: Identifying \"Al Fresco\" Dining Hubs\n",
+        "# @markdown This cell demonstrates how to perform a radius search to find specific business types with distinct attributes.\n",
+        "# @markdown\n",
+        "# @markdown **The Analysis Workflow:**\n",
+        "# @markdown 1.  **Generate Probe Locations:** Python calculates coordinates for 5 \"neighborhoods\" around the city center.\n",
+        "# @markdown 2.  **Spatial Join (`ST_DWITHIN`):** We query BigQuery to find places within **1,000 meters** of these points.\n",
+        "# @markdown 3.  **Attribute Filtering:** We filter the results for **Operational Restaurants** that specifically feature `outdoor_seating` **OR** `allows_dogs`.\n",
+        "# @markdown 4.  **Privacy Compliance:** The query uses `WITH AGGREGATION_THRESHOLD` to ensure counts are only returned if they meet the privacy threshold (minimum 5).\n",
         "\n",
         "# --- 1. Generate Candidate Locations (Python) ---\n",
         "def get_offset_point(lat, lng, lat_offset, lng_offset):\n",
@@ -405,7 +425,14 @@
       "cell_type": "code",
       "source": [
         "# @title 5. Visualization: Marker Map\n",
-        "# @markdown Renders the query results using the initialized map assets.\n",
+        "# @markdown This cell renders the analysis results on an interactive map.\n",
+        "# @markdown\n",
+        "# @markdown **Visualization Details:**\n",
+        "# @markdown *   **Map Engine:** We use [Folium](https://python-visualization.github.io/folium/latest/) to generate the map interface.\n",
+        "# @markdown *   **Base Layer:** The background uses Google Maps Tiles(initialized in Cell 3).\n",
+        "# @markdown *   **Data Overlay:** Markers represent the 5 distinct neighborhoods analyzed in the previous step. They are **color-coded** to highlight the relative density of dining options (Red = Hotspot, Blue = Low Activity).\n",
+        "# @markdown *   **Interactivity:** Click on any marker to see the exact count of matching places.\n",
+        "\n",
         "import folium\n",
         "from folium import plugins, Element\n",
         "\n",
@@ -468,7 +495,8 @@
         "display(m)"
       ],
       "metadata": {
-        "id": "F4Zt7xqqJjWB"
+        "id": "F4Zt7xqqJjWB",
+        "cellView": "form"
       },
       "execution_count": null,
       "outputs": []
@@ -477,7 +505,16 @@
       "cell_type": "code",
       "source": [
         "# @title 6. Function Query: Fetch H3 Density Data\n",
-        "# @markdown We use the [PLACES_COUNT_PER_H3](https://developers.google.com/maps/documentation/placesinsights/place-count-functions/places-count-per-h3) function to retrieve aggregation data.\n",
+        "# @markdown This cell shows a Grid Search using the [PLACES_COUNT_PER_H3](https://developers.google.com/maps/documentation/placesinsights/place-count-functions/places-count-per-h3) SQL function.\n",
+        "# @markdown\n",
+        "# @markdown **The Analysis Workflow:**\n",
+        "# @markdown 1.  **H3 Grid System:** We use [H3](https://h3geo.org/) (Resolution 8) to normalize the city map into uniform hexagonal cells. This is ideal for comparing density across irregular city layouts.\n",
+        "# @markdown 2.  **The Query:** We call the pre-defined BigQuery function `PLACES_COUNT_PER_H3`. This function automatically aggregates places into the grid cells.\n",
+        "# @markdown 3.  **Search Parameters:**\n",
+        "# @markdown     *   **Area:** A **5km radius** around the city center.\n",
+        "# @markdown     *   **Types:** Broad commercial activity (`restaurant`, `store`, `point_of_interest`).\n",
+        "# @markdown     *   **Status:** Only `OPERATIONAL` places.\n",
+        "# @markdown 4.  **Geometry Processing:** The query returns GeoJSON strings for each hexagon, which we parse into a `GeoDataFrame` for visualization.\n",
         "\n",
         "import json\n",
         "from shapely.geometry import shape\n",
@@ -539,8 +576,15 @@
     {
       "cell_type": "code",
       "source": [
-        "# @title 7. Visualization: H3 Heatmap\n",
-        "# @markdown We visualize the density data as a Choropleth layer on the Google Maps basemap, reusing the backend assets.\n",
+        "# @title 7. Visualization: H3 Density Heatmap\n",
+        "# @markdown This cell renders the H3 grid data as a Choropleth map overlay.\n",
+        "# @markdown\n",
+        "# @markdown **Visualization Details:**\n",
+        "# @markdown *   **Technique (Choropleth):** We map the `count` of places in each H3 hexagon to a color scale.\n",
+        "# @markdown *   **Color Scale:** We use a **Yellow-Orange-Red (`YlOrRd`)** gradient.\n",
+        "# @markdown     *   🟨 **Yellow:** Lower density of places.\n",
+        "# @markdown     *   🟥 **Red:** High density/commercial saturation.\n",
+        "# @markdown *   **Context:** By overlaying this on the map, you can correlate high-density hexagons with specific neighborhoods, transit hubs, or shopping districts.\n",
         "\n",
         "if 'gdf_h3' not in locals() or gdf_h3 is None or gdf_h3.empty:\n",
         "    raise ValueError(\"No H3 data found! Please run Cell 6 successfully first.\")\n",
@@ -574,7 +618,8 @@
         "display(m_hex)"
       ],
       "metadata": {
-        "id": "H_6oq9nrJpev"
+        "id": "H_6oq9nrJpev",
+        "cellView": "form"
       },
       "execution_count": null,
       "outputs": []