Add discussion and limitations

Author: nicolasreichardt

tutorial.ipynb

@@ -29,8 +29,10 @@
     "* [Software Requirements](#software-requirements)\n",
     "* [Data Description](#data-description)\n",
     "* [Methodology](#methodology)\n",
-    "* [Results & Discussion](#results-and-discussion)\n",
+    "* [Results](#results)\n",
+    "* [Discussion & Limitations](#discussion-limitations)\n",
     "* [⭐ Challenge ⭐](#challenge)\n",
+    "* [Further Ressources](#further-ressources)\n",
     "* [References](#references)"
    ]
   },
@@ -1436,19 +1438,157 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "<a name=\"discussion\"></a>\n",
-    "## Discussion\n",
+    "In this tutorial, we evaluate whether a **Prototypical Network** can learn to segment rooftops using only a few labeled examples per task. Because this is a **few-shot learning** setting, traditional large-scale training metrics are not directly applicable. Instead, we monitor two key indicators.\n",
     "\n",
-    "tbd"
+    "### **1. Meta-Training Loss**\n",
+    "\n",
+    "During training, the model optimizes over **episodes**, each composed of:\n",
+    "\n",
+    "- a small **support set**  \n",
+    "- a **query image**\n",
+    "\n",
+    "For each epoch, we compute the **average episode loss**, defined as the mean cross-entropy error across all sampled support–query tasks.  \n",
+    "Across epochs, this loss **decreases steadily**, indicating that:\n",
+    "\n",
+    "- the encoder learns a progressively more meaningful **feature representation**  \n",
+    "- rooftop vs. non-rooftop pixels become **more separable** in feature space  \n",
+    "- prototype-based segmentation improves throughout meta-training  \n",
+    "\n",
+    "Even with very limited supervision, the model internalizes **rooftop characteristics** and reduces prediction errors effectively.\n",
+    "\n",
+    "### **2. Predicted Masks & Quantitative Performance**\n",
+    "\n",
+    "To evaluate the model, we use **5-shot segmentation**:  \n",
+    "five labeled support examples define the rooftop prototype for each test episode.\n",
+    "\n",
+    "On a set of **102 test tiles** drawn from a geographically distinct region of Geneva, the model achieves:\n",
+    "\n",
+    "➡️ Mean IoU: ~0.49\n",
+    "\n",
+    "While modest, this result is encouraging given:\n",
+    "\n",
+    "- the strong label constraints  \n",
+    "- the complexity of urban rooftop structures  \n",
+    "- the geographic domain shift between training and testing  \n",
+    "\n",
+    "A qualitative example highlights this behavior:\n",
+    "\n",
+    "- Support images & masks define the relevant rooftop characteristics  \n",
+    "- On the query image, the predicted mask captures **major rooftop shapes**  \n",
+    "- Large rooftop surfaces are generally identified  \n",
+    "- Fine-grained details remain imperfect, but the model generalizes to **textures and geometries not seen during training**\n",
+    "\n",
+    "These results suggest that the Prototypical Network successfully learns a **useful and transferable feature embedding**.  \n",
+    "With only minimal additional supervision, it is able to extend rooftop segmentation knowledge to **new geographic areas**—demonstrating the strength of metric-based few-shot learning for remote sensing tasks."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Limitations\n",
+    "<a name=\"discussion-limitations\"></a>\n",
+    "\n",
+    "## **Discussion**\n",
+    "\n",
+    "The results show that Prototypical Networks can learn meaningful rooftop representations even with very limited supervision. However, there remains substantial room to improve performance and explore alternative design choices within the few-shot learning setup. Several aspects of the training procedure could be refined to enhance segmentation accuracy:\n",
+    "\n",
+    "### **Model Tuning and Regularization**\n",
+    "\n",
+    "Incorporating techniques such as **weight decay**, **dropout**, **data augmentation**, or **early stopping** could stabilize feature learning and reduce overfitting to the small support sets typically used in few-shot learning.\n",
+    "\n",
+    "### **Training for More Epochs**\n",
+    "\n",
+    "For demonstration purposes, the model was trained for only a limited number of epochs.  \n",
+    "Extending training duration or increasing the number of sampled episodes per epoch could help the encoder converge toward a **more discriminative embedding space**, potentially improving segmentation performance.\n",
+    "\n",
+    "### **Extending the Task Toward Policy Relevance**\n",
+    "\n",
+    "A rough solar potential approximation could be built on top of the segmentation task.  \n",
+    "For example:\n",
+    "\n",
+    "- predicted rooftop area  \n",
+    "- combined with IoU-based uncertainty estimates  \n",
     "\n",
-    "tbd"
+    "could provide a **first-order indicator of solar suitability**, connecting the model’s outputs to real-world energy planning applications.\n",
+    "\n",
+    "### **Trying Different Encoder Backbones**\n",
+    "\n",
+    "The current prototype uses a lightweight CNN encoder for simplicity.  \n",
+    "Replacing it with stronger architectures—such as **ResNet-50** or a **Vision Transformer (ViT)**—may yield more robust and generalizable feature representations, though at increased computational cost.\n",
+    "\n",
+    "### **Train/Test Split Strategy**\n",
+    "\n",
+    "To simulate domain shift across neighborhoods, the dataset was split by geographic region.  \n",
+    "An alternative would be a **random shuffle across all tiles**, which usually produces higher accuracy but does **not** evaluate generalization to new geographic areas.\n",
+    "\n",
+    "Exploring both strategies would highlight the trade-offs between:\n",
+    "\n",
+    "- benchmark performance  \n",
+    "- and real-world deployment robustness under geographic variation.\n",
+    "\n",
+    "\n",
+    "---\n",
+    "\n",
+    "\n",
+    "## Limitations\n",
+    "\n",
+    "While this tutorial successfully demonstrates the core ideas behind few-shot segmentation with **Prototypical Networks**, several important simplifications limit its applicability. Many of these choices were made intentionally to ensure the tutorial remains computationally lightweight and easy to reproduce.\n",
+    "\n",
+    "### **Simplified Experimental Setup**\n",
+    "\n",
+    "To keep the workflow accessible, we used:\n",
+    "\n",
+    "- a **very small training set**, both in the number of tiles and support examples per episode  \n",
+    "- a **lightweight encoder**, rather than higher-capacity backbones common in remote sensing (e.g., ResNet-50, Swin Transformer)  \n",
+    "- a **short training schedule**, with few epochs and limited episode sampling  \n",
+    "\n",
+    "These design choices improve reproducibility but also **restrict the achievable segmentation performance**.  \n",
+    "In practical applications—such as large-scale rooftop or solar mapping—substantially **more data**, **stronger feature extractors**, and **longer training** would be necessary.\n",
+    "\n",
+    "### **Modelling Choices Intentionally Kept Simple**\n",
+    "\n",
+    "Several simplifications reduce the robustness of the resulting predictions:\n",
+    "\n",
+    "- **Binary segmentation** (roof vs. non-roof) ignores roof type, material, shadows, and occlusions—all of which matter for accurate solar potential estimation.  \n",
+    "- **No post-processing** was applied (e.g., morphological filters, CRFs), even though such steps typically improve mask quality.  \n",
+    "- **No uncertainty estimation** was included, despite being crucial for planning and policy-relevant applications.\n",
+    "\n",
+    "These omissions help focus on core concepts but limit real-world applicability.\n",
+    "\n",
+    "### **Dataset Biases and Generalization Limits**\n",
+    "\n",
+    "The dataset itself introduces structural biases:\n",
+    "\n",
+    "- Imagery is taken exclusively from **Geneva**, a wealthy European city with relatively homogeneous architectural styles.  \n",
+    "- Rooftop morphology varies globally—informal housing, climate-adapted roof shapes, and diverse materials are **not represented** here.  \n",
+    "- The geographic split (North–Center–South) creates a **stylized domain shift**, but does not reflect true global variation.\n",
+    "\n",
+    "If applied uncritically in policy contexts, these limitations could **reinforce geographic inequities**—for example, overestimating solar potential in well-represented neighborhoods and underestimating it in underrepresented ones.\n",
+    "\n",
+    "### **Addressing These Challenges**\n",
+    "\n",
+    "To improve real-world deployment, data analysts and practitioners should consider:\n",
+    "\n",
+    "- **Expanding dataset diversity** (more cities, varied roof types, lighting conditions, seasons).  \n",
+    "- **Evaluating fairness and generalization** across socioeconomic and geographic groups.  \n",
+    "- **Incorporating uncertainty estimation**, especially when predictions support infrastructure or planning decisions.  \n",
+    "- **Validating model outputs** with domain experts (urban planners, energy modelers, local authorities).\n",
+    "\n",
+    "By recognizing these limitations, we can better understand the conditions under which few-shot rooftop segmentation performs well—and the steps required to make such models reliable for operational or policy-driven use.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---"
    ]
   },
   {
@@ -1493,7 +1633,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "<a name=\"Further Ressources\"></a>\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"further-ressources\"></a>\n",
     "## Further Ressources\n",
     "\n",
     "### **Foundational Papers**\n",
@@ -1536,6 +1683,13 @@
     "  https://github.com/facebookresearch/segment-anything\n"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},