docs: streamline toolkit to tool-only; link schema/workflow/bibliography to teaching

Corey-Abramson · Corey-Abramson · commit ef5ed7ddc9e3 · 2026-07-16T22:15:40.000-07:00
- README Data Structure -&gt; runtime-contract table (matches vis_tool_core.py loader); full schema now canonical in teaching

- Add Design Notes (env-config + word-family rationale); Overview links to teaching workflow

- References point to teaching bibliography

- paper.md: canonical-schema note -&gt; teaching (paper stays self-contained)

- Notebook: markdown lean pass (rationale prose -&gt; README Design Notes; empty artifact cells removed); code cells unchanged
diff --git a/README.md b/README.md
@@ -19,6 +19,7 @@ An easy-to-use toolkit for visualizing patterns in qualitative data, helping res
   - [Regular Usage](#regular-usage)
 - [Using Your Own Data](#using-your-own-data)
   - [Data Structure](#data-structure)
+- [Design Notes](#design-notes)
 - [Troubleshooting](#troubleshooting)
 - [Uninstallation](#uninstallation)
 - [Training Resources](#training-resources)
@@ -35,6 +36,8 @@ The CMAP Visualization Toolkit offers a free suite of open-source tools to analy
 
 The CMAP (Cultural Mapping and Pattern Analysis) tool is free, open-source and produced by the [Computational Ethnography Lab](https://github.com/Computational-Ethnography-Lab) at Rice University.
 
+This toolkit implements the visualization step of the lab's end-to-end qualitative workflow — see [Teaching → Workflow Steps](https://github.com/Computational-Ethnography-Lab/teaching#workflow-steps-end-to-end) for the general workflow and data schema.
+
 ---
 
 ### Notebook Version & Local Installation
@@ -276,58 +279,58 @@ To analyze your own text data:
 
 ### Data Structure
 
-For technically minded users, here's the complete schema for data files (See Abramson et al. 2025):
-
-```python
-# Updated schema with Python typing
-schema = {
-    "project": str,         # List project 
-    "number": str,          # Position information
-    "reference": int,       # Position information
-    "text": str,            # Content, critical field: must not be empty
-    "document": str,        # Data source, Critical field: must not be empty
-    "old_codes": list[str], # Optional: codings, must be a list of strings
-    "start_position": int,  # Position information
-    "end_position": int,    # Position information
-    "data_group": list[str],# Optional, to differentiate document sets: Must be a list of strings
-    "text_length": int,     # Optional: NLP info
-    "word_count": int,      # Optional: NLP info
-    "doc_id": str,          # Optional: NLP info, unique paragrah level identifier
-    "codes": list[str]      # Critical for analyses with codes, Must be a list of strings
-}
-```
-```python
-  example (simulated, not actual data):
-  {
-    "project": "engineer_interviews",  
-    "number": "675:113" #For reconstructing with QDA software  
-    "reference": 244 #For reconstructing with QDA software  
-    "text": "EG002: So the thing is..." # Actual paragraph level text,
-    "document": "INTV_EG002_20250801.txt" # name of document, in which text is found
-    "start_position": 244 #For reconstructing with QDA software  
-    "end_position": 248 #For reconstructing with QDA software  
-    "data_group": ["data_type_cg_interviews","interview"] # For subsetting by data type, or characteristic
-    "text_length": 441 # nlp
-    "word_count": 82 # nlp
-    "doc_id": "EG002_76426", #unique ID for text segment in file, to reconstruct or link to raw data, sequential
-    "codes": ["narrative_life", "education_perceptions"] # qualitative codes, for concepts, themes, variables
-    "old_codes": [
-      "science_belief",
-      "subject_speech_all"
-    ] # qualitative codes, for concepts, themes, variables; archived to silo and reduce clutter
-
-  },//
-```
+This is the **runtime contract** — the columns the notebook's loader (`function/vis_tool_core.py`) actually reads from your CSV. For the full conceptual schema (all fields, types, and rationale), see **[Teaching → Data Schema Example (CMAP)](https://github.com/Computational-Ethnography-Lab/teaching#data-schema-example-cmap)**.
+
+| Column | Required? | Type | Notes |
+|--------|-----------|------|-------|
+| `text` | **Required** | string | Paragraph-level content to analyze. If no `text` column exists, the loader falls back to the first column whose name contains `text`, `content`, or `body`; if none is found it stops. |
+| `codes` | Optional | list of strings | Qualitative codes. Enables code-based analyses and code filtering. |
+| `data_group` | Optional | list of strings | Document-set labels for subsetting (e.g. by data type or characteristic). |
+| `project` | Optional | string | Project label, used to subset the data to one or more projects. |
 
-**Critical Fields**:
-- `text`: Main content field - cannot be empty
-- `document`: Source information - cannot be empty
-- `codes`: Required for code-based analyses - must be a list of strings if used
+Other columns from the full schema (e.g. `document`, position fields, `word_count`, `doc_id`) are preserved if present but are not required by the loader.
 
 **Important Notes**:
-- Lists (like `codes` and `data_group`) must be proper Python lists, not strings that look like lists
-- If you're exporting from qualitative data analysis software, ensure you convert any code fields to proper lists
-- The toolkit will validate your data structure and provide error messages for common issues
+- Lists (like `codes` and `data_group`) must be proper Python lists, not strings that look like lists. The loader will attempt to convert list-looking strings, but real lists are safest.
+- If you're exporting from qualitative data analysis software, ensure you convert any code fields to proper lists.
+- The toolkit will validate your data structure and provide error messages for common issues.
+
+## Design Notes
+
+Rationale for two setup steps in the notebook (`visualization_toolkit_final.ipynb`). The notebook keeps a short key-decision note at each of these steps and points here for the full explanation.
+
+### Environment Configuration
+
+In this section, we define the core directory structure used throughout the replication project. These paths help organize:
+
+- **raw data**,  
+- **trained models**,  
+- **clustering results**, and  
+- **final outputs**.
+
+This setup also makes it easier for users to customize where intermediate results and final outputs will be saved. For example, by changing these directory names, users can **create their own versions of model runs or clustering outputs without overwriting previous results.**
+
+All directories will be automatically created if they don't already exist.
+
+### Stopword Expansion and Semantic Word Family Definitions
+
+This section defines a comprehensive list of stopwords, extending NLTK’s default stopword set with:
+- **punctuation**, 
+- **contractions**, 
+- **common filler words**, and 
+- **project-specific conversational terms** that are semantically uninformative in analysis.
+
+We also build a custom `WORD_FAMILIES` dictionary, which groups related words into unified concepts (e.g., "death", "caregiver", "memory"). This allows the model to:
+- **compress synonyms and variations** into semantically meaningful units,
+- **reduce noise** in the embedding space,
+- and **support cultural/qualitative interpretation** of the results.
+
+This section also includes validation checks to:
+- Ensure **no accidental overlaps** between stopwords and key analytical terms,
+- Detect **redundant words across families**, 
+- And print summaries for user verification.
+
+These definitions are critical for interpretability in downstream visualization and clustering.
 
 ## Troubleshooting
 
@@ -403,7 +406,7 @@ For more detailed information, refer to the [Anaconda Documentation](https://doc
 
 ## References
 
-This toolkit builds on academic work combining computational text analysis with qualitative research methods (Abramson et al. 2018, 2025). Please see the [lab repo](https://github.com/Computational-Ethnography-Lab) for additional resources and related research papers.
+This toolkit builds on academic work combining computational text analysis with qualitative research methods (Abramson et al. 2018, 2025). For a curated topical bibliography with DOIs, see [Teaching → Bibliography](https://github.com/Computational-Ethnography-Lab/teaching#v-bibliography). Please see the [lab repo](https://github.com/Computational-Ethnography-Lab) for additional resources and related research papers.
 
 # Policies
 ## License
diff --git a/documentation/paper.md b/documentation/paper.md
@@ -319,6 +319,8 @@ Reproducible Jupyter notebooks (see workshop repo), CMAP visualizations, codeboo
 
 For structured analysis and visualization (e.g., using the CMAP toolkit), data should be organized into a consistent tabular format (e.g., CSV or DataFrame). Below is an example schema:
 
+> **Note:** The canonical data schema lives in the [teaching repo](https://github.com/Computational-Ethnography-Lab/teaching#data-schema-example-cmap). This paper keeps a self-contained copy so it renders standalone.
+
 ```python
 # Updated schema with Python typing
 schema = {
diff --git a/visualization_toolkit_final.ipynb b/visualization_toolkit_final.ipynb
@@ -7,73 +7,20 @@
    "source": [
     "# Visualization Toolkit Usage Guide\n",
     "\n",
-    "This notebook provides a suite of semantic analysis and visualization tools to explore word-level patterns and relationships in qualitative text data. It supports multiple clustering and embedding strategies for t-SNE, semantic networks, and word clouds. You can review the version with explanations here https://colab.research.google.com/drive/1n90EDMSiXhIaOULUMPJ4W4hqdZCh1NQw?usp=sharing\n",
+    "A suite of semantic-analysis and visualization tools for word-level patterns in qualitative text data — word clouds, t-SNE semantic maps, word/code heatmaps, and semantic networks. An annotated online version is on [Colab](https://colab.research.google.com/drive/1n90EDMSiXhIaOULUMPJ4W4hqdZCh1NQw?usp=sharing) (not for sensitive data); see also the [video tutorial](https://vimeo.com/1122226315) and the [working-paper description](https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit/blob/main/documentation/paper.md).\n",
     "\n",
-    "### What You Can Do with This Toolkit\n",
+    "**Full documentation — install, usage, data schema, design notes, license — is in the [README](https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit#readme).** This notebook is meant to be run top-to-bottom: the **Setup** and **Helper Function** cells are background setup (run them once, in order); then use the **Analytics Tool** sections to generate and customize visuals.\n",
     "\n",
-    "- **Generate Word Clouds**  \n",
-    "  Visualize the most frequent and salient terms across your dataset or within filtered subsets based on keywords or project name.\n",
+    "### How to run\n",
+    "1. **Prepare your data** — a `.csv`/DataFrame with at least a `text` column (optional `project`, `codes`, `data_group` for subsetting). Full schema: [Teaching → Data Schema Example (CMAP)](https://github.com/Computational-Ethnography-Lab/teaching#data-schema-example-cmap).\n",
+    "2. **Run Setup + Helper Functions top-to-bottom** — these load packages, define directories, stopwords / word-families, validators, and the plotting functions.\n",
+    "3. **Run a tool section** — Basic (word cloud), Intermediate (heatmaps, t-SNE, networks), or Advanced (full / interactive). Edit the user-configuration block at the top of each section.\n",
     "\n",
-    "- **Plot t-SNE Semantic Maps**  \n",
-    "  Reduce high-dimensional similarity or co-occurrence matrices into 2D using t-SNE to highlight semantic proximity between words. Seed words are emphasized to anchor interpretation.\n",
-    "\n",
-    "- **Create Word-Based Heatmaps and Semantic Networks**  \n",
-    "  Explore how terms relate to each other in both visual space and co-occurrence structure:\n",
-    "  \n",
-    "  - **Basic Heatmap**: Highlights semantically clustered keywords based on selected embeddings or similarity matrices.\n",
-    "  - **Heatmap + Network**: Adds a basic network graph on top of the heatmap using default node/edge colors (no bundling, no category styling).\n",
-    "  - **Heatmap + Network (Colored Nodes & Edges)**: Fully stylized version including colored clusters, semantic links, and optional edge styling.\n",
-    "\n",
-    "- **Visualize Code Co-Occurrence Heatmaps**  \n",
-    "  Plot the frequency with which qualitative codes appear together in the same entries. \n",
-    "\n",
-    "These tools help you visually investigate language patterns, conceptual clustering, and topic proximity—whether you’re doing grounded theory, thematic analysis, or exploratory semantic mapping.\n",
-    "\n",
-    "### How to Use\n",
-    "1. **Prepare Your Dataset**  \n",
-    "   Make sure your dataset is a `.csv` or DataFrame with at least:\n",
-    "   - A `text` column (raw text)\n",
-    "   - Optionally, a `project` column (for subsetting)\n",
-    "   - optionally a `codes' column` (for subsetting, and analysis)\n",
-    "   - Please see READ.md for the complete schema and more possibilities.\n",
-    "\n",
-    "2. **Set Key Parameters**  \n",
-    "   Most functions accept:\n",
-    "   - `stopwords_path`: path to extra stopwords (optional)\n",
-    "   - `clustering_method`: 1 = RoBERTa, 2 = Jaccard, 3 = PMI, 4 = TF-IDF\n",
-    "   - `distance_metric`: \"cosine\" or \"default\" (used for similarity matrix choice)\n",
-    "\n",
-    "3. **Run Visualizations**\n",
-    "You can explore and generate a variety of visual outputs using the execution blocks provided in the **Analytics Tools** and **Advanced Analytics Tools** sections.\n",
-    "\n",
-    "These include:\n",
-    "- **Word Clouds** for highlighting high-frequency terms in selected texts or projects.\n",
-    "- **t-SNE Semantic Maps** to project word relationships into 2D space for visualizing proximity and clusters.\n",
-    "- **Word-Based Heatmaps** to show how frequently words co-occur.\n",
-    "- **Semantic Network Graphs** (with optional heatmaps), including:\n",
-    "  - show relationships in text\n",
-    "  - basic  node/seed networks,\n",
-    "  - networks with customized node colors,\n",
-    "  - and networks with or without edge bundling.\n",
-    "- **Code-based Heatmap** to visualize co-occurrence patterns among qualitative codes\n",
-    "\n",
-    "📍 Simply scroll down to the relevant execution cells to run and customize these visualizations based on your dataset.\n",
-    "\n",
-    "### File Structure Notes\n",
-    "- Define constants like `DATA_DIR`, `OUTPUT_DIR`, and stopword files before running.\n",
-    "- Ensure required files and embeddings are preloaded or generated using prior pipeline steps.\n",
-    "\n",
-    "> Tip: Start by testing on one project or topic before scaling to all data.\n",
-    "\n",
-    "# TESTING\n",
+    "**Key parameters (set per tool):** `clustering_method` 1 = RoBERTa, 2 = Jaccard, 3 = PMI, 4 = TF-IDF; `distance_metric` `\"cosine\"` or `\"default\"`. Design rationale for the environment-configuration and word-family setup steps is in [README → Design Notes](https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit#design-notes).\n",
     "\n",
     "## License & Citation\n",
     "\n",
-    "**License:** [BSD 3-Clause](https://opensource.org/license/bsd-3-clause/) · “AS IS” — no warranties or liability.\n",
-    "\n",
-    "**Citation:** *Abramson, Corey M. and Yuhan (Victoria) Nian. 2025. CMAP Visualization Toolkit.* <https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit>\n",
-    "\n",
-    "For questions or use requests, please email <corey.abramson@rice.edu>."
+    "BSD 3-Clause (\"AS IS\", no warranty). **Cite as:** Abramson, Corey M. and Yuhan (Victoria) Nian. 2025. *CMAP Visualization Toolkit.* Full citation with the Zenodo DOI and contributor list is in the [README](https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit#policies). Questions: <corey.abramson@rice.edu>."
    ]
   },
   {
@@ -241,16 +188,7 @@
    "source": [
     "### Environment Configuration\n",
     "\n",
-    "In this section, we define the core directory structure used throughout the replication project. These paths help organize:\n",
-    "\n",
-    "- **raw data**,  \n",
-    "- **trained models**,  \n",
-    "- **clustering results**, and  \n",
-    "- **final outputs**.\n",
-    "\n",
-    "This setup also makes it easier for users to customize where intermediate results and final outputs will be saved. For example, by changing these directory names, users can **create their own versions of model runs or clustering outputs without overwriting previous results.**\n",
-    "\n",
-    "All directories will be automatically created if they don't already exist."
+    "Defines the core directory structure (raw data, models, clustering results, outputs); all directories are auto-created. Change these names to keep separate model/clustering runs without overwriting previous results. **Full rationale:** [README → Design Notes](https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit#design-notes)."
    ]
   },
   {
@@ -357,23 +295,7 @@
    "source": [
     "### Stopword Expansion and Semantic Word Family Definitions\n",
     "\n",
-    "This section defines a comprehensive list of stopwords, extending NLTK’s default stopword set with:\n",
-    "- **punctuation**, \n",
-    "- **contractions**, \n",
-    "- **common filler words**, and \n",
-    "- **project-specific conversational terms** that are semantically uninformative in analysis.\n",
-    "\n",
-    "We also build a custom `WORD_FAMILIES` dictionary, which groups related words into unified concepts (e.g., \"death\", \"caregiver\", \"memory\"). This allows the model to:\n",
-    "- **compress synonyms and variations** into semantically meaningful units,\n",
-    "- **reduce noise** in the embedding space,\n",
-    "- and **support cultural/qualitative interpretation** of the results.\n",
-    "\n",
-    "This section also includes validation checks to:\n",
-    "- Ensure **no accidental overlaps** between stopwords and key analytical terms,\n",
-    "- Detect **redundant words across families**, \n",
-    "- And print summaries for user verification.\n",
-    "\n",
-    "These definitions are critical for interpretability in downstream visualization and clustering."
+    "Extends NLTK's stopwords (punctuation, contractions, filler, project-specific terms) and builds the `WORD_FAMILIES` dictionary that groups related words into unified concepts — compressing synonyms, reducing embedding noise, and supporting qualitative interpretation. Validation checks flag stopword/analytical-term overlaps and redundant families. **Full rationale:** [README → Design Notes](https://github.com/Computational-Ethnography-Lab/cmap_visualization_toolkit#design-notes)."
    ]
   },
   {
@@ -3139,12 +3061,6 @@
     "This section contains advanced heatmap and network combined execution blocks."
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "bcbc21be",
-   "metadata": {},
-   "source": []
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -3686,30 +3602,6 @@
     "if __name__ == \"__main__\":\n",
     "    main()"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "536568e4-3b86-4507-852a-9bee92d6d18e",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "1ddb3204-ce24-4895-8c23-ed640eb6e407",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "d443a3f1-86ae-4d39-bd87-9781538cd388",
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
