fix(notebook,pipeline): B1/B2 wrapper ordering + safe list-column parser

Reorder all four teaching-notebook wrappers (run_heatmap_pipeline,
run_tsne_pipeline, run_heatmap_network_plain_pipeline,
run_heatmap_network_pipeline) to match the core pipeline contract:
exclude codes before sentence extraction (B1), and rebuild
filtered_sentences after group-label replacement so the co-occurrence /
PMI / TF-IDF methods embed the replaced text (B2).

Replace eval() on stringified codes/data_group cells with a bounded,
list[str]-only parse_list_column_cell in vis_tool_core.py and all four
notebook wrappers; it fails loud on tuples, malformed literals,
non-string items, and oversized cells instead of executing arbitrary code.

Add discriminating wrapper-ordering tests and list-column parser tests.
Document the supported clustering method/metric pairs (PMI = cosine only)
and the Ward-linkage clustered-heatmap design note in the README.

Author: Corey-Abramson

README.md

@@ -334,6 +334,23 @@ This section also includes validation checks to:
 
 These definitions are critical for interpretability in downstream visualization and clustering.
 
+### Clustering Methods and Supported Distance Metrics
+
+Word-to-word similarity can be computed with one of four methods, chosen with `clustering_method` and paired with a `distance_metric`:
+
+- **1 — RoBERTa embeddings:** contextual embeddings compared with cosine similarity.
+- **2 — Co-occurrence:** `cosine` uses frequency-sensitive context vectors; the default uses the Jaccard index (set-based, binary overlap).
+- **3 — PMI (Positive Pointwise Mutual Information):** supported with `distance_metric = "cosine"` **only**. PMI builds context vectors whose similarity is meaningful under cosine; other metrics are not defined for this method, so pairing PMI with a non-cosine metric is an unsupported configuration rather than a defect. Use `"cosine"` with method 3.
+- **4 — TF-IDF weighted co-occurrence:** `cosine` (or `default`) uses cosine similarity; other values select an experimental unnormalized-sum variant.
+
+Combining a method and metric outside these supported pairings — most notably PMI with a non-cosine metric — will not produce a usable similarity matrix. This is expected behavior: select a supported method/metric pair.
+
+### Hierarchical Clustering (Ward Linkage) in the Clustered Heatmap
+
+When `clustered = True`, the heatmap also shows a dendrogram-ordered version that groups words using hierarchical clustering (Ward linkage) applied to the rows and columns of the word-by-word similarity matrix. This places similar words next to one another and reveals nested groupings.
+
+Ward linkage here operates on each word's full similarity *profile* (its row/column in the matrix). It is a deliberate choice for producing interpretable dendrograms and word groupings, not a transformation of the individual similarity values. The values shown in the standard (unclustered) heatmap are unchanged; the clustered view only reorders and annotates them.
+
 ## Troubleshooting
 
 Here are solutions to common issues you might encounter:

function/vis_tool_core.py

@@ -1732,6 +1732,69 @@ def repel_nodes(p,iterations=150,k=0.2):
                bbox_to_anchor=(1.0,0.0),frameon=False, fontsize=16)
     plt.axis('off'); plt.tight_layout(); return fig
 
+#---------------------------------------------------------------------------------
+# LIST-COLUMN PARSING (safe replacement for eval())
+#---------------------------------------------------------------------------------
+
+def _validate_str_items(items, max_items):
+    """Validate a parsed sequence is a bounded list of strings; raise otherwise."""
+    if len(items) > max_items:
+        raise ValueError(f"List-column cell has {len(items)} items (max {max_items})")
+    for item in items:
+        if not isinstance(item, str):
+            raise ValueError(
+                f"List-column items must be strings, got {type(item).__name__}: {item!r}"
+            )
+    return list(items)
+
+
+def parse_list_column_cell(value, max_items=1000, max_chars=100000):
+    """Parse a stringified list-column cell (``codes`` / ``data_group``) into list[str].
+
+    Safe, bounded replacement for the previous ``eval()`` on list-column cells:
+    - missing / empty values become ``[]``;
+    - a bracketed list literal is parsed with ``ast.literal_eval`` and must be a
+      flat list of strings within the size bounds;
+    - a bare scalar string becomes a single-item list;
+    - a tuple / dict / set literal, a malformed literal, non-string items, or a
+      cell exceeding the size bounds raise ``ValueError`` (fail loud) rather than
+      silently mis-parsing or executing arbitrary code.
+    """
+    if value is None:
+        return []
+    if isinstance(value, list):
+        return _validate_str_items(value, max_items)
+    if isinstance(value, str):
+        text = value.strip()
+        if not text:
+            return []
+        if text[0] == '[':
+            if len(text) > max_chars:
+                raise ValueError(
+                    f"List-column cell exceeds {max_chars} chars ({len(text)})"
+                )
+            try:
+                parsed = ast.literal_eval(text)
+            except (ValueError, SyntaxError) as exc:
+                raise ValueError(f"Malformed list-column cell: {text[:80]!r}") from exc
+            if not isinstance(parsed, list):
+                raise ValueError(
+                    f"List-column cell must be a list, got {type(parsed).__name__}: {text[:80]!r}"
+                )
+            return _validate_str_items(parsed, max_items)
+        if text[0] in '({':
+            raise ValueError(
+                f"List-column cell must be a list, not a {text[0]!r}-literal: {text[:80]!r}"
+            )
+        return [text]
+    try:
+        if pd.isna(value):
+            return []
+    except (TypeError, ValueError):
+        pass
+    return [str(value)]
+
+
 #---------------------------------------------------------------------------------
 # MAIN PIPELINE FUNCTION
 #---------------------------------------------------------------------------------
@@ -1786,8 +1849,7 @@ def run_visuals_pipeline(input_data):
             # Check if first non-null value is a string that looks like a list
             sample = df[col].dropna().iloc[0] if not df[col].dropna().empty else None
             if isinstance(sample, str) and (sample.startswith('[') or ',' in sample):
-                df[col] = df[col].apply(lambda x: eval(x) if isinstance(x, str) and x.strip() else 
-                                       ([] if pd.isna(x) else [x]))
+                df[col] = df[col].apply(parse_list_column_cell)
 
     # Apply Metadata Filters
     if input_data.projects and 'project' in df.columns:

tests/test_list_column_parser.py

@@ -0,0 +1,68 @@
+"""Bounded list-column parser (GR-EVAL): safe replacement for eval() on codes/data_group.
+
+``parse_list_column_cell`` replaces an ``eval()`` call that previously ran on every
+stringified ``codes`` / ``data_group`` cell.  These tests pin the contract: it parses a
+bracketed list of strings, treats missing/scalar values sensibly, and fails loud on
+tuples, malformed literals, non-string items, and oversized cells.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+import pytest
+
+
+def test_parses_bracketed_string_list(vis_module) -> None:
+    assert vis_module.parse_list_column_cell("['nebeker', 'background']") == [
+        "nebeker",
+        "background",
+    ]
+
+
+def test_parses_single_item_list(vis_module) -> None:
+    assert vis_module.parse_list_column_cell("['interview']") == ["interview"]
+
+
+def test_passes_through_existing_list(vis_module) -> None:
+    assert vis_module.parse_list_column_cell(["a", "b"]) == ["a", "b"]
+
+
+def test_bare_scalar_string_becomes_single_item(vis_module) -> None:
+    assert vis_module.parse_list_column_cell("interview") == ["interview"]
+
+
+@pytest.mark.parametrize("empty", [None, "", "   ", np.nan, float("nan")])
+def test_missing_values_become_empty_list(vis_module, empty) -> None:
+    assert vis_module.parse_list_column_cell(empty) == []
+
+
+def test_tuple_literal_is_rejected(vis_module) -> None:
+    with pytest.raises(ValueError):
+        vis_module.parse_list_column_cell("('a', 'b')")
+
+
+def test_dict_literal_is_rejected(vis_module) -> None:
+    with pytest.raises(ValueError):
+        vis_module.parse_list_column_cell("{'a': 1}")
+
+
+def test_non_string_items_are_rejected(vis_module) -> None:
+    with pytest.raises(ValueError):
+        vis_module.parse_list_column_cell("[1, 2, 3]")
+
+
+def test_malformed_literal_fails_loud(vis_module) -> None:
+    with pytest.raises(ValueError):
+        vis_module.parse_list_column_cell("['unclosed")
+
+
+def test_oversized_cell_is_rejected(vis_module) -> None:
+    oversized = str([f"code_{i}" for i in range(1001)])
+    with pytest.raises(ValueError):
+        vis_module.parse_list_column_cell(oversized)
+
+
+def test_no_arbitrary_code_execution(vis_module) -> None:
+    """A code-injection payload must not execute; it is not a list literal."""
+    with pytest.raises(ValueError):
+        vis_module.parse_list_column_cell("[__import__('os').getcwd()]")