feat(viz): implement noise reduction, hub gating, and simulation milestones

Author: DerekGooding

docs/roadmap.md

@@ -1,20 +1,19 @@
-# Mythril Project Roadmap: Simulation & Architecture Maturity
+# Visualization Noise Reduction Roadmap
 
-This roadmap focuses on refining the game's economic balance, optimizing the core simulation engines, and enhancing the UI with predictive data.
+## Goal
+Transform the Mythril Lattice visualization from a cluttered "hairball" into a clear progression and economic map by gating high-frequency resource nodes and highlighting simulation milestones.
 
-## Phase 1: Economic & Simulation Hardening
-- [x] **Economic Equilibrium Refinement**: Achieve 100% sustainability for all reachable recurring activities.
-- [x] **Dependency-Tracked Fixpoint Solver**: Implement a worklist-based `LatticeSimulator` for $O(I \cdot \text{avg\_out\_degree})$ performance.
-- [x] **Simulation-Driven Regression Assertions**: Integrate end-game pacing checks into `check_health.py`.
+## Phase 1: Data Architecture & Gating
+- [ ] Implement hub detection in `data_processor.py` (Frequency thresholding).
+- [ ] Categorize edges into `Core` (Progression) vs `Auxiliary` (Economy).
+- [ ] Integrate simulation results for `Sustainability` and `Milestone` flagging.
 
-## Phase 2: UI & UX Predictive Intelligence
-- [x] **Eliminate "Magic String" Metadata**: Refactor ability/quest effects into a typed `EffectDefinition` schema.
-- [x] **Predictive Resource Highlighting**: Visual indicators for unaffordable quest requirements in real-time.
-- [x] **Effect-Driven UI Logic**: Passive stat boosts and magic capacity now driven by typed effects.
-- [x] **Predictive Dependency Overlay**: Implement "Prerequisite Path" highlighting in the `CadenceTree` and `QuestPanel`.
+## Phase 2: Visual Logic & Interactivity
+- [ ] Add filtering logic to `lattice_data.js`.
+- [ ] Implement milestone-specific rendering styles in `lattice_rendering.js`.
+- [ ] Add hover-driven edge visibility for auxiliary links.
 
-## Phase 3: State & Runtime Consolidation
-- [x] **Unified Deterministic State Store**: Implement a central Reducer-based `GameStateStore` for full simulation/runtime parity.
-
----
-*Last Updated: 2026-04-08*
+## Phase 3: UI/UX Controls
+- [ ] Add Dashboard controls for "Progression Only" and "Hide Resource Hubs".
+- [ ] Implement sustainability overlays based on health check metrics.
+- [ ] Final validation with `scripts/check_health.py`.

docs/roadmap_detailed.md

@@ -1,53 +1,31 @@
-# Detailed Technical Roadmap: Simulation & Architecture Maturity
-
-Detailed breakdown of current roadmap initiatives.
-
-## 1. Economic Equilibrium Refinement
-Achieving 100% sustainability for all reachable recurring content.
-
-- [x] **Data Audit**: Identify all "Unsustainable Activities" using `FlowSimulator` report.
-- [x] **Logistics/Resource Injection**: Update `content_graph.json` to ensure bottleneck resources like `Log`, `Mana Leaf`, and `Ancient Bark` have sustainable production loops.
-- [x] **Cost-Benefit Balance**: Adjust refinement costs and rewards for late-game content to prevent resource depletion.
-- [x] **Verification**: Run `check_health.py` and confirm `shield_sustainability` reaches 100%.
-
-## 2. Dependency-Tracked Fixpoint Solver
-Optimizing `LatticeSimulator` performance for large-scale content expansion.
-
-- [x] **Worklist Architecture**: Implement a queue-based `Solve()` loop in `LatticeSimulator.cs`.
-- [x] **Dependency Graph Build**: Create an internal map of `Node -> [DependentNodes]` during initialization.
-- [x] **Delta Propagation**: Only add nodes to the worklist when their inputs (in-edges) change state.
-- [x] **Validation**: Ensure results are bit-for-bit identical to the exhaustive solver.
-
-## 3. Simulation-Driven Regression Assertions
-Automated pacing and balance checks for CI.
-
-- [x] **Baseline Establishment**: Record current `Routed Completion Time` as the performance baseline.
-- [x] **Pacing Logic**: Add logic to `check_health.py` to compare current sim results with baseline.
-- [x] **Regression Thresholds**: Define acceptable variance (e.g., <15% increase without new content).
-- [x] **CI Integration**: Fail the health check if pacing regressions are detected.
-
-## 4. UI: Predictive Dependency Overlay
-Graph-based navigation assistance for the player.
-
-- [ ] **Pathfinding Logic**: Implement BFS/DFS on the client-side content graph to find the shortest path to a locked node.
-- [ ] **UI State Integration**: Update `CadenceTree` and `QuestPanel` to accept a `HighightedNodes` parameter.
-- [ ] **Hover Interactions**: Implement hover-triggers that activate the pathfinding and highlight prerequisite nodes.
-
-## 5. Eliminate "Magic String" Metadata
-Type-safe ability and quest effects.
-
-- [ ] **Schema Definition**: Create a JSON schema/C# record for `EffectDefinition` (e.g., `StatBoost`, `MagicCapacity`, `AutoQuest`).
-- [ ] **Content Migration**: Update all `Ability` and `Quest` nodes in `content_graph.json` to use the new `effects` field.
-- [ ] **Loader Refactor**: Update `ContentLoader.cs` to deserialize the typed effects.
-- [ ] **Code Cleanup**: Remove `Metadata` dictionary lookups and replace with typed property checks.
-
-## 6. Unified Deterministic State Store
-Full parity between simulation and Blazor runtime.
-
-- [x] **State Record**: Create a single `GameStoreState` immutable record.
-- [x] **Actions & Reducers**: Define formal `Actions` (e.g., `CompleteQuest`, `SpendResource`) and a pure `Reducer` function.
-- [x] **Manager Refactor**: Convert `ResourceManager`, `JunctionManager`, and `InventoryManager` to be view-only subscribers to the `GameStateStore`.
-- [x] **Snapshot Support**: Implement a one-click snapshot/restore feature for testing and save-games.
-
----
-*Last Updated: 2026-04-06*
+# Detailed Roadmap: Visualization Noise Reduction
+
+## 1. Data Processor Gating (`modules/visualization/data_processor.py`)
+- **Hub Detection:** Any node with > 10 incoming or outgoing edges will be marked as a `ResourceHub`.
+- **Edge Metadata:** Each edge in the JSON will be tagged with a `category`:
+    - `progression`: `unlocks_cadence`, `requires_quest`, `contains`.
+    - `economy`: `consumes`, `produces`, `rewards`.
+- **Milestone Identification:**
+    - Quests that unlock locations or cadences.
+    - Items that are "Sustainable" per `simulation_report.md`.
+
+## 2. Visual Filtering (`modules/visualization/lattice_rendering.js` & `lattice_data.js`)
+- **Filtered Edge Loading:** `lattice_data.js` will maintain two sets of edges: `activeEdges` and `allEdges`.
+- **Style Overrides:**
+    - Milestones: Gold stroke/glow, larger scale.
+    - Hubs: Dimmed by default, no labels unless hovered.
+    - Progression Edges: Thicker lines with distinct colors.
+
+## 3. UI/UX Enhancements (`modules/visualization/template_engine.py`)
+- **Control Panel:**
+    - Toggle: `Show Resource Hubs` (Default: Off)
+    - Toggle: `Progression Only` (Default: On)
+    - Toggle: `Simulation Overlay` (Default: Off)
+- **Tooltip Upgrades:** Show net flow rates (from simulation) in item tooltips.
+
+## 4. Simulation Integration
+- **Parser:** A new utility to scrape `simulation_report.md` for:
+    - Sustainable vs Unsustainable list.
+    - Completion times.
+    - Critical bottlenecks.
+- **Visual Link:** Nodes in the graph will color-code based on their sustainability status.

modules/visualization/data_processor.py

@@ -1,6 +1,7 @@
 import json
 import os
 import sys
+import re
 from collections import deque
 from .constants import GRAPH_FILE
 
@@ -16,24 +17,81 @@ def load_graph():
     with open(GRAPH_FILE, 'r', encoding='utf-8') as f:
         return json.load(f)
 
+def parse_simulation_report():
+    report_path = "simulation_report.md"
+    if not os.path.exists(report_path):
+        return {}
+    
+    data = {
+        "sustainable": set(),
+        "unsustainable": set(),
+        "rates": {}
+    }
+    
+    try:
+        with open(report_path, "r", encoding="utf-8") as f:
+            content = f.read()
+            
+            # Parse sustainable
+            sust_match = re.search(r"### Sustainable Recurring Activities\n(.*?)\n\n", content, re.DOTALL)
+            if sust_match:
+                for line in sust_match.group(1).split("\n"):
+                    if line.strip().startswith("- "):
+                        data["sustainable"].add(line.strip()[2:])
+            
+            # Parse unsustainable
+            unsust_match = re.search(r"### ⚠️ Unsustainable Activities.*?\n(.*?)\n\n", content, re.DOTALL)
+            if unsust_match:
+                for line in unsust_match.group(1).split("\n"):
+                    if line.strip().startswith("- "):
+                        data["unsustainable"].add(line.strip()[2:])
+            
+            # Parse rates
+            rates_match = re.search(r"### Net Resource Rates \(per second\)\n(.*?)\n\n", content, re.DOTALL)
+            if rates_match:
+                for line in rates_match.group(1).split("\n"):
+                    m = re.match(r"- \*\*(.*?)\*\*: ([\d.]+)/s", line.strip())
+                    if m:
+                        data["rates"][m.group(1)] = float(m.group(2))
+    except Exception as e:
+        print(f"Warning: Failed to parse simulation report: {e}")
+        
+    return data
+
+def _matches_activity(node_name, activity_names):
+    if node_name in activity_names:
+        return True
+    
+    # Try normalization for refinements
+    # node: "Refine Wood - Herb"
+    # activity: "Refine Wood:Log->Herb"
+    norm_node = node_name.lower().replace(" ", "").replace("-", "")
+    for act in activity_names:
+        norm_act = act.lower().replace(" ", "").replace(":", "").replace("->", "")
+        if norm_node in norm_act or norm_act in norm_node:
+            return True
+    return False
+
 def enrich_data(nodes):
     """Calculate tiers and clusters in Python for better performance."""
     node_map = {n["id"]: n for n in nodes}
+    sim_data = parse_simulation_report()
 
-    # 1. Build Adjacency
+    # 1. Build Adjacency and Edge Counts
     adj = {n["id"]: [] for n in nodes}
+    edge_counts = {n["id"]: 0 for n in nodes}
+    
     for n in nodes:
-        if "out_edges" in n:
-            for target_list in n["out_edges"].values():
-                for target in target_list:
-                    target_id = target if isinstance(target, str) else target.get("targetId")
-                    if target_id in adj:
-                        adj[n["id"]].append(target_id)
-        if "in_edges" in n:
-            for source_list in n["in_edges"].values():
-                for source_id in source_list:
-                    if source_id in adj:
-                        adj[source_id].append(n["id"])
+        for direction in ["out_edges", "in_edges"]:
+            if direction in n:
+                for rel_type, targets in n[direction].items():
+                    for target in targets:
+                        target_id = target if isinstance(target, str) else target.get("targetId")
+                        if target_id in node_map:
+                            edge_counts[n["id"]] += 1
+                            edge_counts[target_id] += 1
+                            if direction == "out_edges":
+                                adj[n["id"]].append(target_id)
 
     # 2. BFS for Tiers
     tiers = {n["id"]: 0 for n in nodes}
@@ -61,9 +119,34 @@ def enrich_data(nodes):
     # 3. Clusters
     clusters, cluster_names = _identify_clusters(nodes)
 
+    # Hub Detection Threshold
+    HUB_THRESHOLD = 10
+
+    sust_names = sim_data.get("sustainable", set())
+    unsust_names = sim_data.get("unsustainable", set())
+
     for n in nodes:
         n["tier"] = tiers.get(n["id"], 0)
         n["cluster_id"] = clusters.get(n["id"], "cluster_none")
+        n["is_hub"] = edge_counts[n["id"]] > HUB_THRESHOLD
+        
+        # Milestone Detection
+        is_milestone = False
+        if n["type"] == "Quest":
+            # Unlocks something important
+            out = n.get("out_edges", {})
+            if "unlocks_cadence" in out or "unlocks_location" in out:
+                is_milestone = True
+            if n["id"] == "quest_prologue":
+                is_milestone = True
+        n["is_milestone"] = is_milestone
+
+        # Simulation Integration
+        n["simulation"] = {
+            "sustainable": _matches_activity(n["name"], sust_names),
+            "unsustainable": _matches_activity(n["name"], unsust_names),
+            "net_rate": sim_data.get("rates", {}).get(n["name"], 0)
+        }
 
     return nodes, cluster_names
 

modules/visualization/lattice_data.js

@@ -7,16 +7,21 @@ function processData() {
     }));
 
     nodeMap = new Map(nodes.map(n => [n.id, n]));
+    allEdges = [];
+
+    const progressionTypes = ['unlocks_cadence', 'unlocks_location', 'requires_quest', 'contains', 'provides_ability', 'requires_ability'];
 
     nodes.forEach(node => {
         if (node.out_edges) {
             Object.entries(node.out_edges).forEach(([type, targetList]) => {
                 targetList.forEach(target => {
                     const targetId = typeof target === 'string' ? target : target.targetId;
                     if (nodeMap.has(targetId)) {
-                        edges.push({
+                        allEdges.push({
                             id: `edge-${node.id}-${targetId}`,
-                            source: node.id, target: targetId, type: type
+                            source: node.id, target: targetId, 
+                            type: type,
+                            category: progressionTypes.includes(type) ? 'progression' : 'economy'
                         });
                     }
                 });
@@ -26,9 +31,11 @@ function processData() {
             Object.entries(node.in_edges).forEach(([type, sourceList]) => {
                 sourceList.forEach(sourceId => {
                     if (nodeMap.has(sourceId)) {
-                        edges.push({
+                        allEdges.push({
                             id: `edge-${sourceId}-${node.id}`,
-                            source: sourceId, target: node.id, type: type
+                            source: sourceId, target: node.id, 
+                            type: type,
+                            category: progressionTypes.includes(type) ? 'progression' : 'economy'
                         });
                     }
                 });
@@ -37,14 +44,41 @@ function processData() {
     });
 
     const seenEdges = new Set();
-    edges = edges.filter(e => {
-        const key = `${e.source}-${e.target}`;
+    allEdges = allEdges.filter(e => {
+        const key = `${e.source}-${e.target}-${e.type}`;
         if (seenEdges.has(key)) return false;
         seenEdges.add(key);
         return true;
     });
+
+    filterEdges();
+}
+
+function filterEdges() {
+    edges = allEdges.filter(e => {
+        const sourceNode = nodeMap.get(e.source);
+        const targetNode = nodeMap.get(e.target);
+
+        // Hub filtering
+        if (!showHubs) {
+            if (sourceNode.is_hub || targetNode.is_hub) return false;
+        }
+
+        // Progression Only filtering
+        if (showProgressionOnly) {
+            if (e.category !== 'progression') return false;
+        }
+
+        return true;
+    });
+
+    // Handle node visibility (dimming)
+    nodes.forEach(n => {
+        n.visible = true;
+        if (!showHubs && n.is_hub) n.visible = false;
+    });
 }
 
 function updateStats() {
-    document.getElementById('stats').innerText = `NODES: ${nodes.length} | EDGES: ${edges.length} | TIERS: ${Math.max(...nodes.map(n=>n.tier))+1}`;
+    document.getElementById('stats').innerText = `NODES: ${nodes.filter(n=>n.visible).length}/${nodes.length} | EDGES: ${edges.length} | TIERS: ${Math.max(...nodes.map(n=>n.tier))+1}`;
 }