feat(viz): server-tile + Datashader path for >1M-node graphs (CPU-only, no CUDA)

The d3-force/canvas viz tops out around 5-10K nodes; even with
batched loading and pinning it cannot scale to the 300K+ Cortex now
emits, let alone the 1M-10M target. This commit lays in a CPU-only
tile-streaming pipeline alongside the existing renderer, gated by
``?viz=tilemap`` so the legacy path stays the default until the new
one is hardened.

Architecture (3 plans in tasks/*.md):
  Layout    — tasks/layout-cache-plan.md       (igraph DrL, CPU)
  Tiles     — tasks/tile-server-plan.md        (Datashader, CPU)
  Frontend  — tasks/tilemap-frontend-plan.md   (deck.gl, no CUDA)

Backend additions
─────────────────
* core/layout_engine.py        — igraph DrL wrapper. Pure logic.
* core/tile_renderer.py        — Datashader points→PNG with the
                                 fixed kind palette + XYZ tile bbox
                                 mapping over [-1, 1] world coords.
* infrastructure/layout_pg_store.py — bulk insert + bbox-range read
                                       against the new
                                       ``workflow_graph_layout`` table.
* handlers/recompute_layout.py — /api/recompute_layout: pulls the
                                  cached graph, runs DrL, persists.
* handlers/tile_handler.py     — /api/tile/{z}/{x}/{y}.png: pulls
                                  positions in tile bbox, renders.
* handlers/quadtree_handler.py — /api/quadtree: gzipped Arrow IPC of
                                  every (id, x, y, kind) for client
                                  flatbush picking.
* server/http_standalone.py    — three new GET routes wired in.
* infrastructure/pg_schema.py  — new ``workflow_graph_layout`` table
                                  with B-tree indexes on
                                  (layout_version, kind, x, y).

Frontend additions
──────────────────
* ui/unified/js/workflow_graph_tilemap.js — deck.gl TileLayer +
    OrthographicView (Cartesian, no projection); flatbush index
    built from the Arrow quadtree; hover label + click → existing
    side panel.
* ui/unified/js/workflow_graph.js — ``?viz=tilemap`` gate at the
    top of renderWorkflowGraph; legacy d3-force is the default.
* ui/unified-viz.html — load the tilemap module.

Optional dependencies
─────────────────────
New ``viz-tile`` extra in pyproject.toml:
  * igraph     — CPU layout (MIT, prebuilt wheels)
  * datashader — tile rasterization (BSD, O(viewport pixels))
  * pyarrow    — Arrow IPC for the quadtree endpoint
  * pandas     — Datashader's native frame format
  * cachetools — L1 in-memory tile LRU (PR 2 will use it)
  * Pillow     — PNG encode

NOT installed by default — keeps the package install size unchanged
for users who don't need the >1M-node viz.

Frontend pulls deck.gl 9, Apache Arrow JS 17, and Flatbush 4 from
unpkg on first load; vendoring under deps/ deferred to PR 2.

Smallest-defensible-v1 scope
────────────────────────────
* No tile cache (L1 LRU + L2 disk) — both planned, deferred.
* No edge rendering in tiles — points only; edges become noise at
  1M+ scale and are better surfaced via on-click context.
* No domain/kind filter on tile or quadtree URLs — single global
  view for v1.
* Synchronous /api/recompute_layout (long request) — async job
  deferred to PR 2.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: cdeust

mcp_server/core/layout_engine.py

@@ -0,0 +1,113 @@
+"""CPU layout engine for the workflow graph.
+
+Pure logic: takes a list of node ids + an edge list, returns a list of
+``(node_id, x, y)`` triples. Calls ``igraph`` (MIT, prebuilt PyPI
+wheels for macOS/Linux/Windows) for the actual layout. No I/O, no
+PostgreSQL imports — this module is testable with synthetic graphs.
+
+Algorithm choice — DrL (Distributed Recursive Layout):
+  * O(N log N) per iteration, scales linearly with edge count.
+  * Tuned for force-directed exploratory views; produces well-separated
+    clusters even on 1M-node graphs in under 3 minutes on a modern CPU.
+  * Falls back to Fruchterman-Reingold for tiny graphs (<200 nodes)
+    where DrL's bookkeeping overhead is wasted.
+
+Reference: Martin et al. "OpenOrd: An Open-Source Toolbox for Large
+Graph Layout", SPIE 2011 — DrL is the OpenOrd algorithm under its
+original name.
+"""
+
+from __future__ import annotations
+
+import hashlib
+from typing import Iterable
+
+
+def topology_fingerprint(node_ids: Iterable[str], edges: Iterable[tuple]) -> str:
+    """Stable fingerprint of the graph's topology, used as a cache key.
+
+    A graph's layout is valid as long as the same set of node ids and
+    the same set of (source, target) pairs are present. The
+    fingerprint is a SHA-256 over the sorted concatenation; two builds
+    with the same topology — even with different memory contents —
+    share a fingerprint and reuse the same coords.
+    """
+    h = hashlib.sha256()
+    for nid in sorted(node_ids):
+        h.update(nid.encode("utf-8"))
+        h.update(b"\n")
+    h.update(b"--edges--\n")
+    edge_strs = sorted(f"{s}\x00{t}" for s, t in edges)
+    for e in edge_strs:
+        h.update(e.encode("utf-8"))
+        h.update(b"\n")
+    return h.hexdigest()[:16]
+
+
+def layout(
+    node_ids: list[str],
+    edges: list[tuple[str, str]],
+    *,
+    algorithm: str = "drl",
+    seed: int = 0,
+) -> list[tuple[str, float, float]]:
+    """Compute (x, y) per node and return ``[(id, x, y), ...]``.
+
+    Raises:
+        ImportError: if ``igraph`` is not installed (the optional
+            ``viz-tile`` extra). The caller is expected to surface this
+            as a 503 on the HTTP endpoint.
+        ValueError: if ``node_ids`` is empty.
+    """
+    try:
+        import igraph as ig
+    except ImportError as exc:  # pragma: no cover
+        raise ImportError(
+            "igraph is required for layout — install the 'viz-tile' extra: "
+            "pip install neuro-cortex-memory[viz-tile]"
+        ) from exc
+
+    if not node_ids:
+        raise ValueError("layout requires at least one node id")
+
+    # Build an integer-indexed igraph from the string ids. ``igraph``'s
+    # vertex API accepts string names but the layout algorithms operate
+    # on contiguous integer indices, so we pre-translate.
+    idx_of = {nid: i for i, nid in enumerate(node_ids)}
+    edge_pairs = [
+        (idx_of[s], idx_of[t])
+        for s, t in edges
+        if s in idx_of and t in idx_of and s != t
+    ]
+    g = ig.Graph(n=len(node_ids), edges=edge_pairs, directed=False)
+    g.simplify()
+
+    if algorithm == "drl" and len(node_ids) >= 200:
+        coords = g.layout("drl")
+    elif algorithm == "fr" or len(node_ids) < 200:
+        coords = g.layout_fruchterman_reingold(niter=200, seed=seed)
+    else:
+        # Defensive default: DrL is the expected path; fall back to FR
+        # for any unknown algorithm name rather than raising.
+        coords = g.layout("drl")
+
+    raw = list(coords.coords)
+    if not raw:
+        return []
+
+    # Normalise into [-1, 1] world coords (the tile renderer + the
+    # client coordinate system both assume this range).
+    xs = [p[0] for p in raw]
+    ys = [p[1] for p in raw]
+    min_x, max_x = min(xs), max(xs)
+    min_y, max_y = min(ys), max(ys)
+    span_x = (max_x - min_x) or 1.0
+    span_y = (max_y - min_y) or 1.0
+    span = max(span_x, span_y) * 0.55  # slight padding inside [-1, 1]
+    cx = (min_x + max_x) / 2
+    cy = (min_y + max_y) / 2
+
+    return [
+        (node_ids[i], (raw[i][0] - cx) / span, (raw[i][1] - cy) / span)
+        for i in range(len(node_ids))
+    ]

mcp_server/core/tile_renderer.py

@@ -0,0 +1,118 @@
+"""Datashader-backed tile rendering for the workflow graph.
+
+Pure rendering logic: takes a list of ``(node_id, x, y, kind)`` tuples
+plus tile coordinates and produces PNG bytes. No PostgreSQL imports —
+the handler layer composes this with ``layout_pg_store.read_positions_in_bbox``.
+
+World coordinate system: ``[-1, 1] × [-1, 1]``. The layout engine
+normalises into this range; the tile pyramid maps each ``(z, x, y)`` to
+a sub-rectangle of the world.
+"""
+
+from __future__ import annotations
+
+import io
+
+# Fixed palette. Palette changes invalidate the tile cache via
+# ``PALETTE_VERSION`` — bump it whenever any colour below changes.
+PALETTE_VERSION = 1
+KIND_COLOR_HEX = {
+    "domain": "#FCD34D",
+    "tool_hub": "#F97316",
+    "skill": "#FB923C",
+    "command": "#FACC15",
+    "hook": "#A855F7",
+    "agent": "#EC4899",
+    "mcp": "#6366F1",
+    "memory": "#10B981",
+    "discussion": "#EF4444",
+    "entity": "#50B0C8",
+    "file": "#06B6D4",
+    "symbol": "#64748B",
+}
+DEFAULT_HEX = "#888888"
+
+WORLD_MIN = -1.0
+WORLD_MAX = 1.0
+
+
+def tile_world_bbox(z: int, x: int, y: int) -> tuple[float, float, float, float]:
+    """Map XYZ tile coordinates to ``(min_x, min_y, max_x, max_y)``.
+
+    z=0 covers the whole [-1, 1] world in a single tile. Each level
+    halves the tile span so z=10 has ~1M tiles and a per-tile span of
+    2/1024 ≈ 0.002 in world units (sub-pixel at any practical zoom).
+    """
+    span = (WORLD_MAX - WORLD_MIN) / (1 << z)
+    min_x = WORLD_MIN + x * span
+    max_x = min_x + span
+    # Tile y axis runs top-down (XYZ convention); flip so the world's
+    # +y is the top of tile (0,0) at every zoom.
+    max_y_world = WORLD_MAX - y * span
+    min_y_world = max_y_world - span
+    return (min_x, min_y_world, max_x, max_y_world)
+
+
+def render_tile_png(
+    rows: list[tuple[str, float, float, str]],
+    *,
+    z: int,
+    x: int,
+    y: int,
+    tile_size: int = 512,
+) -> bytes:
+    """Rasterise ``rows`` (already prefiltered to the tile's world bbox)
+    into a ``tile_size × tile_size`` PNG.
+
+    Returns PNG bytes ready for HTTP transport. Empty input renders a
+    fully-transparent tile (still valid PNG; client compositors expect
+    that).
+    """
+    try:
+        import datashader as ds
+        import datashader.transfer_functions as tf
+        import pandas as pd
+    except ImportError as exc:  # pragma: no cover
+        raise ImportError(
+            "datashader + pandas are required for tile rendering — install "
+            "the 'viz-tile' extra: pip install neuro-cortex-memory[viz-tile]"
+        ) from exc
+
+    min_x, min_y, max_x, max_y = tile_world_bbox(z, x, y)
+    if not rows:
+        # Empty frame still produces a valid 512×512 transparent PNG
+        # via Datashader's spread([]) → to_pil().
+        return _empty_tile_png(tile_size)
+
+    # Datashader requires categorical kind to be a pandas Categorical
+    # for ``count_cat`` aggregation. Pre-encode here.
+    df = pd.DataFrame(rows, columns=["id", "x", "y", "kind"])
+    kinds_present = sorted(df["kind"].unique())
+    df["kind"] = pd.Categorical(df["kind"], categories=kinds_present)
+
+    canvas = ds.Canvas(
+        plot_width=tile_size,
+        plot_height=tile_size,
+        x_range=(min_x, max_x),
+        y_range=(min_y, max_y),
+    )
+    agg = canvas.points(df, x="x", y="y", agg=ds.count_cat("kind"))
+    color_key = {k: KIND_COLOR_HEX.get(k, DEFAULT_HEX) for k in kinds_present}
+    img = tf.shade(agg, color_key=color_key, how="eq_hist")
+    img = tf.spread(img, px=1, shape="circle")
+    pil = img.to_pil()
+    buf = io.BytesIO()
+    pil.save(buf, format="PNG", optimize=False)
+    return buf.getvalue()
+
+
+def _empty_tile_png(tile_size: int) -> bytes:
+    """Cheap transparent PNG, used when the bbox query returns no rows."""
+    try:
+        from PIL import Image
+    except ImportError as exc:  # pragma: no cover
+        raise ImportError("Pillow is required for tile rendering") from exc
+    img = Image.new("RGBA", (tile_size, tile_size), (0, 0, 0, 0))
+    buf = io.BytesIO()
+    img.save(buf, format="PNG", optimize=False)
+    return buf.getvalue()

mcp_server/handlers/quadtree_handler.py

@@ -0,0 +1,71 @@
+"""GET /api/quadtree — gzipped Arrow IPC of every node's (id, x, y, kind).
+
+The client builds a quadtree (e.g. flatbush) from this payload to
+resolve hover/click locally in O(log N) without a server roundtrip.
+``id`` and ``kind`` are dictionary-encoded so the wire size is
+dominated by two Float32 columns at 1M nodes ≈ 8 MB raw / ~3-4 MB
+gzipped.
+"""
+
+from __future__ import annotations
+
+import gzip
+import json
+
+
+def serve(handler, store) -> None:
+    try:
+        import pyarrow as pa
+        import pyarrow.ipc as ipc
+        from mcp_server.infrastructure import layout_pg_store
+    except ImportError as exc:
+        body = (
+            f'{{"status":"error","reason":"viz_tile_extra_missing","detail":"{exc}"}}'
+        ).encode("utf-8")
+        handler.send_response(503)
+        handler.send_header("Content-Type", "application/json")
+        handler.send_header("Content-Length", str(len(body)))
+        handler.end_headers()
+        handler.wfile.write(body)
+        return
+
+    rows = layout_pg_store.read_all_positions(store)
+    if not rows:
+        body = json.dumps({"status": "error", "reason": "no_layout"}).encode("utf-8")
+        handler.send_response(503)
+        handler.send_header("Content-Type", "application/json")
+        handler.send_header("Content-Length", str(len(body)))
+        handler.end_headers()
+        handler.wfile.write(body)
+        return
+
+    ids = [r[0] for r in rows]
+    xs = [r[1] for r in rows]
+    ys = [r[2] for r in rows]
+    kinds = [r[3] for r in rows]
+
+    # Dict-encoded id + kind shrink the wire substantially: ``id`` is
+    # high-cardinality but the dict encoding still beats UTF-8 for
+    # lookup; ``kind`` collapses to ~12 distinct values.
+    table = pa.table(
+        {
+            "id": pa.array(ids).dictionary_encode(),
+            "x": pa.array(xs, type=pa.float32()),
+            "y": pa.array(ys, type=pa.float32()),
+            "kind": pa.array(kinds).dictionary_encode(),
+        }
+    )
+
+    sink = pa.BufferOutputStream()
+    with ipc.new_stream(sink, table.schema) as writer:
+        writer.write_table(table)
+    arrow_buf = sink.getvalue().to_pybytes()
+    body = gzip.compress(arrow_buf, compresslevel=6)
+
+    handler.send_response(200)
+    handler.send_header("Content-Type", "application/vnd.apache.arrow.stream")
+    handler.send_header("Content-Encoding", "gzip")
+    handler.send_header("Content-Length", str(len(body)))
+    handler.send_header("Cache-Control", "max-age=60")
+    handler.end_headers()
+    handler.wfile.write(body)