CAT-1457 Update cell populations template to use scellop

BUILD

@@ -1 +1 @@
-f3faf77
+4bbb195

src/user_templates_api/templates/jupyter_lab/templates/scalable_cell_populations/metadata.json

@@ -1,6 +1,6 @@
 {
   "title": "Scalable Cell Population Explorer",
-  "description": "This notebook shows how to load the scalable cell population viewer CellPop. It loads in datasets with annotated cell types and shows these in an interactive visualization.",
+  "description": "This notebook shows how to load the scalable cell population viewer scellop. It loads in datasets with annotated cell types and shows these in an interactive visualization.",
   "tags": [
     "visualization"
   ], 
@@ -14,5 +14,5 @@
     }
   ],
   "is_hidden": false,
-  "last_modified_unix_timestamp": 1729805397
+  "last_modified_unix_timestamp": 1767696252
 }

src/user_templates_api/templates/jupyter_lab/templates/scalable_cell_populations/template.ipynb

@@ -1,119 +1,154 @@
 {
- "cells": [
-  {
-    "cell_type": "markdown",
-    "metadata": {},
-    "source": [
-      "# Scalable Cell Population Viewer\n",
-      "This notebook shows how to load the scalable cell population viewer CellPop. It loads in datasets with annotated cell types and shows these in an interactive visualization.\n",
-      "\n",
-      "> This work is under active development and will be subject to change.\n",
-      "\n",
-      "The source code can be found [here](https://github.com/hms-dbmi/cellpop). Feel free to submit issues and feature requests [here](https://github.com/hms-dbmi/cellpop/issues). "
-    ]
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Scalable Cell Population Viewer\n",
+        "This notebook shows how to load the scalable cell population viewer _scellop_. It loads in datasets with annotated cell types and shows these in an interactive visualization.\n",
+        "\n",
+        "The source code can be found [here](https://github.com/hms-dbmi/scellop). Feel free to submit issues and feature requests [here](https://github.com/hms-dbmi/scellop/issues). "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install scellop==1.1.0\n",
+        "!pip install hubmap-template-helper"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import scellop\n",
+        "from hubmap_template_helper import uuids as hth_uuids\n",
+        "from hubmap_template_helper import compatibility as hth_comp"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# linked datasets\n",
+        "uuids = {{ uuids | safe }}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# This tool has been tested with datasets from the assays listed below. Other assays could potentially also work. \n",
+        "accepted_assay_display_names=[\"snRNAseq [Salmon]\", \"snRNAseq (SNARE-seq2) [Salmon]\", \"Slide-seq [Salmon]\", \"snRNAseq (10x Genomics) [Salmon]\"]\n",
+        "uuids = hth_comp.check_template_compatibility(uuids, accepted_assay_display_names=accepted_assay_display_names)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# we can get some metadata about these datasets\n",
+        "import requests\n",
+        "import json\n",
+        "\n",
+        "search_api = \"https://search.api.hubmapconsortium.org/v3/portal/search\"\n",
+        "hits = json.loads(requests.post(search_api,json={\"size\": 10000, \"query\": {\"ids\": {\"values\": uuids}}, \"_source\": [\"hubmap_id\", \"title\", \"assay_display_name\", \"donor.mapped_metadata\"],},).text)[\"hits\"][\"hits\"]\n",
+        "first = lambda x: (x or [None])[0]\n",
+        "dmm_keys = [\"age_value\",\"sex\",\"height_value\",\"weight_value\",\"race\",\"body_mass_index_value\",\"abo_blood_group_system\",\"medical_history\",\"cause_of_death\",\"death_event\",\"mechanism_of_injury\"]\n",
+        "row_metadata = {\n",
+        "    ls[\"hubmap_id\"]: {\n",
+        "        \"title\": ls.get(\"title\"),\n",
+        "        \"assay_display_name\": ls.get(\"assay_display_name\"),\n",
+        "        \"anatomy\": first(ls.get(\"anatomy_2\")) or first(ls.get(\"anatomy_1\")),\n",
+        "        **{k: first(dmm.get(k)) for k in dmm_keys},\n",
+        "    } for l in hits if (ls := l.get(\"_source\", {})).get(\"hubmap_id\") for dmm in [ls.get(\"donor\", {}).get(\"mapped_metadata\", {})]\n",
+        "}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## `scellop` visualizes dataframes in an interactive Jupyter notebook widget. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "mapping = hth_uuids.get_uuid_to_hubmap_mapping(uuids)\n",
+        "hubmap_ids = [mapping[uuid] for uuid in uuids]\n",
+        "\n",
+        "anndata_locations_list = [f\"./datasets/{uuid}/secondary_analysis.h5ad\" for uuid in uuids]\n",
+        "\n",
+        "scd = scellop.load_data_multiple(anndata_locations_list, hubmap_ids, \"predicted_CLID\", c_cols_meta=[\"predicted_label\"])\n",
+        "scd.row_metadata.update(row_metadata)\n",
+        "\n",
+        "scellop.ScellopWidget(data=scd)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "\n",
+        "The visualization offers the following controls:\n",
+        "* Dragging heatmap cells or side bar plots to change data order\n",
+        "* Pressing buttons next to heatmap rows to turn them into embedded bar plots\n",
+        "* The magnifying glass icon near the row/column labels enables zooming in to a scrollable view in each dimension.\n",
+        "* The borders between each panel in the visualization are draggable to allow independent resizing of each sector in the grid.\n",
+        "\n",
+        "\n",
+        "Clicking the Settings button (gear icon) at the bottom right for finegrained control of settings:\n",
+        "* Plot Controls\n",
+        "  * **View Type**: Allows toggling to view the data as a traditional stacked bar chart to highlight the advantages of `scellop`'s heatmap-augmented approach.\n",
+        "  * **Heatmap Themes**: Allows selecting from various continuous d3 color schemes or inverting the color scheme as needed.\n",
+        "  * **Heatmap Normalization**: Allows adjustments of how values are displayed on the heatmap:\n",
+        "    * Normalizing by row/column aids in identifying outlier cells which make up a large proportion of small cells.\n",
+        "    * Normalizing by log counts helps work with high-variance data.\n",
+        "  * **Left/Top Graph Type** selections allow configuration of the side plots:\n",
+        "    * **Bars** displays bar charts colored with categorical values corresponding to the row/column they represent. These colros are configurable in the row/column controls.\n",
+        "    * **Continuous Stacked Bars** displays stacked bar charts with segments colored by heatmap scale values.\n",
+        "    * **Categorical Stacked Bars** displays stacked bar charts with segments colored by categorical values corresponding to the row/column they represent. These colors are configurable in the row/column controls.\n",
+        "    * **Violins** displays violin plots that represent the distribution of values for each category.\n",
+        "  * **Transpose Rows and Columns** enables swapping rows and columns in the visualization.\n",
+        "  * **Zoom Controls** allow enabling zoom for the row and column axes and setting the width allotted for each row/column.\n",
+        "* Row and Column Controls\n",
+        "  * **Sorts** allow ordering rows and columns by loaded metadata values. When values besides count/alphabetical sort are selected, bars appear on the opposite side of the heatmap from the side plots with colored metadata bars for categorization.\n",
+        "  * **Filters** allow selection of displayed rows and columns based on selected metadata values.\n",
+        "  * The **Display Options** table contains a list of row/column values, along with fine-grained controls allowing the individual setting of their visibility and color.\n",
+        "* Export Controls\n",
+        "  * The Export tab enables exporting the current state of the visualization as an upscaled PNG image.\n",
+        "* Undo/Redo/Reset buttons enable resetting back to previous visualization states. The Color column's header allows for setting a single color for all rows/columns.\n",
+        "* Right clicking on the heatmap or side plots opens a context menu with relevant controls for sorting/reordering/filtering."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "You can see more examples [here](https://github.com/hms-dbmi/scellop/blob/main/python/example.ipynb). "
+      ]
+    }
+  ],
+  "metadata": {
+    "language_info": {
+      "name": "python"
+    }
   },
-  {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "!pip install cellpop==0.1.0\n",
-      "!pip install hubmap-template-helper"
-    ]
-  },
-  {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "import cellpop\n",
-      "from hubmap_template_helper import uuids as hth_uuids\n",
-      "from hubmap_template_helper import compatibility as hth_comp"
-    ]
-  },
-  {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "# linked datasets\n",
-      "uuids = {{ uuids | safe }}\n",
-      "\n",
-      "# accepted assay_display_names\n",
-      "accepted_assay_display_names=[\"snRNAseq [Salmon]\"]"
-    ]
-  },
-  {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "# # For now, this tool has only been tested with snRNAseq [Salmon] datasets. Other assays may not yet work. If you only want to work with snRNAseq [Salmon], uncomment this cell.\n",
-      "# uuids = hth_comp.check_template_compatibility(uuids, accepted_assay_display_names=accepted_assay_display_names)"
-    ]
-  },
-  {
-    "cell_type": "markdown",
-    "metadata": {},
-    "source": [
-      "## There are multiple ways of loading CellPop. One is directly through the HuBMAP UUIDs. "
-    ]
-  },
-  {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "cellpop.CpWidget(uuids = uuids)"
-    ]
-  },
-  {
-    "cell_type": "markdown",
-    "metadata": {},
-    "source": [
-      "You can interact with the widget, such as changing the order, sorting by metadata and changing the visual encoding. Check the `plot controls` for finegrained control."
-    ]
-  },
-  {
-    "cell_type": "markdown",
-    "metadata": {},
-    "source": [
-      "## There are also other ways to load data, such as with the anndata locations."
-    ]
-  },
-  {
-    "cell_type": "code",
-    "execution_count": null,
-    "metadata": {},
-    "outputs": [],
-    "source": [
-      "mapping = hth_uuids.get_uuid_to_hubmap_mapping(uuids)\n",
-      "hubmap_ids = [mapping[uuid] for uuid in uuids]\n",
-      "\n",
-      "anndataLocationList = [f\"./datasets/{uuid}/secondary_analysis.h5ad\" for uuid in uuids]\n",
-      "\n",
-      "df = cellpop.cpAnnDataList(anndataLocationList, hubmap_ids, \"predicted_CLID\")\n",
-      "cellpop.CpWidget(df=df)"
-    ]
-  },
-  {
-    "cell_type": "markdown",
-    "metadata": {},
-    "source": [
-      "You can also load the data with the obs locations, or from a dataframe directly. You can see more examples [here](https://github.com/hms-dbmi/cellpop/blob/main/python/example.ipynb). "
-    ]
-  }
-],
- "metadata": {
-  "language_info": {
-   "name": "python"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+  "nbformat": 4,
+  "nbformat_minor": 2
+}