feat: /map page — force-directed spec map clustered by tag similarity#5647
Open
MarkusNeusinger wants to merge 17 commits intomainfrom
Open
feat: /map page — force-directed spec map clustered by tag similarity#5647MarkusNeusinger wants to merge 17 commits intomainfrom
MarkusNeusinger wants to merge 17 commits intomainfrom
Conversation
New endpoint returns one row per spec — id, title, best-impl
preview URLs (light/dark), best-impl quality_score, spec-level tags
and impl-level tags — for the upcoming /map page that clusters
specs by tag similarity.
Best-impl selection: highest quality_score with lexicographic
library_id tiebreak for determinism. Specs without implementations
are skipped, mirroring _build_specs_list.
Route is registered before /specs/{spec_id} so the path-parameter
route doesn't capture "map". Cache key specs_map is invalidated
alongside specs_list whenever a spec changes.
Refs #5646
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New /map page renders one image-thumbnail node per spec via react-force-graph-2d, positioned by tag overlap: specs that share many tags pull together, specs that share few drift apart. Similarity is computed entirely client-side as pure functions: - flattenTags() folds nested spec.tags + impl_tags into a single category-prefixed string set (e.g. plot_type:scatter), preventing collisions like "numeric" appearing in two categories. - computeIDF() weights tags by log(N / df), so ubiquitous tags like data_type:numeric get ~zero weight and rare tags carry the signal. - weightedJaccard() + buildKNNLinks() emit each spec's top-K most similar neighbors (K=5, min-sim=0.05), producing ~1.6k edges over ~327 specs — dense enough to cluster, sparse enough to avoid hairball rendering. react-force-graph-2d's built-in d3-force engine handles the layout. Thumbnails are eager-preloaded and attached to nodes as they resolve so the canvas re-paints organically without restarting physics. Hover highlights neighbors and dims the rest. Click navigates to the spec page. A visually-hidden anchor list mirrors the canvas content for screen-reader and keyboard users. Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lazy-loaded route in router.tsx and a "map" entry in the NAV_LINKS array of NavBar.tsx so the new MapPage is reachable as a top-level section alongside specs/plots/libraries/stats/palette/mcp. Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Codecov flagged the new MapPage.tsx at 58% patch coverage (under the 80% threshold) because the smoke test only asserted top-level render output. Capture the props that get passed into the mocked ForceGraph2D wrapper and exercise them directly: - onNodeClick → navigate + analytics event - nodeCanvasObject → both branches (with/without preloaded image) - nodePointerAreaPaint → fillStyle + fillRect call - onNodeHover + linkColor — hover state propagates to the link colorer - linkWidth scales monotonically with weight Also fixed the MockResizeObserver to actually invoke its callback so the canvas mounts in tests (the size.w > 0 guard was previously keeping ForceGraph2D unmounted, hiding all of these branches from coverage). Coverage on MapPage.tsx now ~95% lines; whole-file count for the new helpers + page is 94.4% lines. Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a new “spec map” feature that exposes backend data for a tag-similarity graph and renders it as a force-directed, thumbnail-based /map page in the frontend.
Changes:
- Backend: introduce
GET /specs/mapreturning one row per spec with best-impl preview URLs, quality score, and tag bags (spec + impl), plus cache invalidation. - Frontend: add lazy-loaded
/maproute + NavBar link, and implementMapPagewith KNN links computed client-side. - Tests: add server/router tests for the new endpoint and unit/smoke tests for map helpers/page.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
api/routers/specs.py |
Adds _build_specs_map and GET /specs/map with caching |
api/schemas.py |
Defines SpecMapItem response model |
api/cache.py |
Clears specs_map cache when spec cache cleared |
tests/unit/api/test_routers.py |
Adds endpoint tests for /specs/map |
app/src/pages/MapPage.tsx |
New /map page with force graph rendering + a11y fallback list |
app/src/pages/MapPage.helpers.ts |
Pure helpers for tag flattening, IDF, similarity, KNN links, image preload |
app/src/pages/MapPage.test.tsx |
Page smoke tests with ForceGraph stub |
app/src/pages/MapPage.helpers.test.ts |
Unit tests for helper math/selection |
app/src/router.tsx |
Registers /map as lazy route |
app/src/components/NavBar.tsx |
Adds “map” link |
app/package.json / app/yarn.lock |
Adds react-force-graph-2d (+ deps) |
Comment on lines
+136
to
+143
| // 4. neighbor lookup for hover highlight (built once per links change) | ||
| const neighbors = useMemo(() => { | ||
| const map = new Map<string, Set<string>>(); | ||
| for (const l of graphData.links) { | ||
| if (!map.has(l.source)) map.set(l.source, new Set()); | ||
| if (!map.has(l.target)) map.set(l.target, new Set()); | ||
| map.get(l.source)!.add(l.target); | ||
| map.get(l.target)!.add(l.source); |
Comment on lines
+252
to
+257
| linkColor={(l: MapLink) => { | ||
| const involved = hoverId && (l.source === hoverId || l.target === hoverId); | ||
| if (involved) return colors.primary; | ||
| if (hoverId) return isDark ? 'rgba(255,255,255,0.04)' : 'rgba(0,0,0,0.04)'; | ||
| return isDark ? 'rgba(255,255,255,0.10)' : 'rgba(0,0,0,0.12)'; | ||
| }} |
Comment on lines
+167
to
+179
| return new Promise<void>(resolve => { | ||
| // document.createElement is preferred over `new Image()` here only because | ||
| // some lint configs don't surface browser globals on plain .ts files. | ||
| const img = document.createElement('img'); | ||
| img.crossOrigin = 'anonymous'; | ||
| img.onload = () => { | ||
| out.set(id, img); | ||
| onLoad?.(id, img); | ||
| resolve(); | ||
| }; | ||
| img.onerror = () => resolve(); | ||
| img.src = thumbUrl; | ||
| }); |
Comment on lines
+72
to
+81
| class SpecMapItem(BaseModel): | ||
| """One row per spec for the /map page: best-impl preview + full tag bag for client-side similarity clustering.""" | ||
|
|
||
| id: str | ||
| title: str | ||
| preview_url_light: str | None = None | ||
| preview_url_dark: str | None = None | ||
| quality_score: float | None = None | ||
| tags: dict[str, Any] | None = None | ||
| impl_tags: dict[str, Any] | None = None |
| for spec in specs: | ||
| if not spec.impls: | ||
| continue | ||
| best = max(spec.impls, key=lambda i: ((i.quality_score or 0.0), i.library_id)) |
Comment on lines
+76
to
+84
| }, | ||
| { | ||
| id: 'scatter-color-mapped', | ||
| title: 'Scatter with Color Mapping', | ||
| preview_url_light: 'https://example.com/scatter-color-light.png', | ||
| preview_url_dark: 'https://example.com/scatter-color-dark.png', | ||
| quality_score: 88, | ||
| tags: { plot_type: ['scatter'], data_type: ['numeric'], features: ['color-mapped'] }, | ||
| impl_tags: { dependencies: ['scipy'] }, |
Comment on lines
+246
to
+251
| nodePointerAreaPaint={(node, color, ctx) => { | ||
| const n = node as WithCoords; | ||
| if (n.x == null || n.y == null) return; | ||
| ctx.fillStyle = color; | ||
| ctx.fillRect(n.x - NODE_SIZE / 2, n.y - NODE_SIZE / 2, NODE_SIZE, NODE_SIZE); | ||
| }} |
Setting img.crossOrigin='anonymous' in preloadImages forces a CORS
preflight, which fails because the anyplot-images GCS bucket has no
CORS headers configured. All 312 thumbnails were getting blocked
("No 'Access-Control-Allow-Origin' header is present").
We only drawImage() these onto the canvas — never call getImageData
or toDataURL — so a tainted canvas is fine. Drop the attribute and
the browser fetches the images as a regular cross-origin GET that
GCS happily serves.
Refs #5646
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Map nodes render at ~22 px but were loading the full-size GCS PNGs
(~40–100 KB each) — total ~15 MB for 312 specs.
The responsive-image pipeline already bakes _400.webp variants
(~6 KB each), so swap selectMapThumbUrl to derive `{base}_400.webp`
from the chosen theme URL. Total payload drops to ~2 MB. Falls back
to the original URL if it isn't `.png`.
Refs #5646
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “spec map” discovery surface to anyplot: a backend endpoint that returns map-ready spec rows (best-impl preview + tags), and a new frontend /map page that renders a force-directed thumbnail graph clustered by tag similarity.
Changes:
- Backend: add
GET /specs/map(cached) returning one row per spec with best-rated implementation preview URLs, quality score, and spec + impl tag bags. - Frontend: add lazy-loaded
/maproute and NavBar entry; implementMapPageusingreact-force-graph-2dwith client-side IDF-weighted Jaccard + KNN edge construction. - Tests/deps: add unit tests for the endpoint and helper/math/page behavior; add
force-graph+react-force-graph-2ddependencies.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
api/routers/specs.py |
Adds /specs/map route and _build_specs_map() builder with caching. |
api/schemas.py |
Introduces SpecMapItem response model for the map payload. |
api/cache.py |
Extends spec cache invalidation to include specs_map. |
tests/unit/api/test_routers.py |
Adds unit tests covering /specs/map behavior and edge cases. |
app/src/pages/MapPage.tsx |
New map page UI: fetches map payload, builds graph, renders canvas + a11y fallback list. |
app/src/pages/MapPage.helpers.ts |
Pure helpers for tag flattening, IDF, weighted Jaccard, KNN links, and thumbnail URL selection/preload. |
app/src/pages/MapPage.helpers.test.ts |
Unit tests for the map helper functions. |
app/src/pages/MapPage.test.tsx |
Smoke/behavior tests for MapPage rendering, callbacks, and error states. |
app/src/router.tsx |
Registers lazy-loaded /map route. |
app/src/components/NavBar.tsx |
Adds a top-level “map” navigation link. |
app/package.json |
Adds force-graph and react-force-graph-2d dependencies. |
app/yarn.lock |
Locks new dependency tree for force-graph packages. |
| // Hairline border around a thumbnail node, theme-aware. | ||
| function strokeFor(isDark: boolean, isHover: boolean): string { | ||
| if (isHover) return colors.primary; | ||
| return isDark ? 'rgba(240,239,232,0.18)' : 'rgba(26,26,23,0.18)'; |
Comment on lines
+236
to
+240
| ctx.drawImage(n.img, x, y, baseSize, baseSize); | ||
| } else { | ||
| ctx.fillStyle = isDark ? '#242420' : '#FFFDF6'; | ||
| ctx.fillRect(x, y, baseSize, baseSize); | ||
| } |
Comment on lines
+184
to
+186
| // headers, and adding crossOrigin='anonymous' triggers a preflight that | ||
| // fails. We only ever drawImage() these onto the canvas (the canvas | ||
| // becomes "tainted", which is fine — we never read it back). |
| * `_400.webp` variant produced by the responsive-image pipeline. Map nodes | ||
| * render at ~22 px, so even 400 px is overkill — but 400 is the smallest | ||
| * pipeline-baked variant. Going from full-size (~40–100 KB) to _400.webp | ||
| * (~6 KB) cuts the 312-thumbnail payload from ~15 MB to ~2 MB. |
_400.webp pixelates very quickly when the user zooms into the force-graph canvas — the 22-graph-unit nodes blow up to hundreds of on-screen pixels and the 400px source can't keep up. Bump the variant to _800.webp: stays crisp under typical zoom-in, total payload still only ~5 MB for 312 specs (vs ~15 MB for full-size originals). Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the static _800.webp variant with a tier-aware loader: - Initial paint loads _400.webp (~6 KB, ~1.9 MB total) — fast. - nodeCanvasObject receives globalScale, computes the on-screen device-pixel size of each visible node, and picks the smallest pipeline tier (400/800/1200) that comfortably covers it. - If that tier isn't loaded yet, ensureNodeTier() kicks off a fetch and force-graph repaints when it lands (no jank, no physics restart). - pickBestLoadedTier falls back to a smaller tier during the upgrade so nodes never go blank. force-graph only invokes nodeCanvasObject for visible nodes, so zooming in fetches hi-res only for what the user is actually looking at — no fan-out to off-screen specs. New helpers (all unit-tested): RESOLUTION_TIERS, buildVariantUrl, pickTier, pickBestLoadedTier, ensureNodeTier. Smoke test fixtures updated for the new MapNode shape (imgs Map + pendingTiers Set). Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plots are typically rendered with figsize=(16, 9), but the canvas draw + hitbox code was forcing them into a square 22×22 slot, which squashed everything horizontally. Read the intrinsic aspect ratio from any loaded thumbnail variant (naturalWidth/naturalHeight) and fit the draw rect / hit area into a NODE_SIZE box without distortion — longer side stays at NODE_SIZE so nodes share a consistent layout scale. Falls back to a square when nothing is loaded yet (matches the placeholder). New helpers: nodeAspectRatio, fitToBox (both pure, unit-tested). Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “spec map” discovery surface: a backend endpoint that returns one row per spec (best implementation preview + tags), and a new frontend /map page that builds a client-side force-directed graph clustered by tag similarity.
Changes:
- Backend: introduce
GET /specs/mapreturningSpecMapItem(best-impl preview URLs + quality + spec/impl tags) and invalidate its cache on spec updates. - Frontend: add
/maproute + NavBar entry, plusMapPagewithreact-force-graph-2dand helper utilities for IDF-weighted similarity + KNN links. - Tests: add API router tests for
/specs/mapand unit/smoke tests for MapPage + helpers.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
api/schemas.py |
Adds SpecMapItem response schema for the map payload. |
api/routers/specs.py |
Implements _build_specs_map and GET /specs/map with caching. |
api/cache.py |
Ensures spec cache clears also invalidate specs_map. |
tests/unit/api/test_routers.py |
Adds unit tests covering /specs/map behaviors (no DB, best-impl pick, skips empty, empty DB). |
app/package.json |
Adds force-graph and react-force-graph-2d deps. |
app/yarn.lock |
Locks new force-graph dependency tree. |
app/src/router.tsx |
Registers lazy-loaded /map route. |
app/src/components/NavBar.tsx |
Adds top-level “map” link. |
app/src/pages/MapPage.tsx |
New map page rendering the force-directed thumbnail graph + hover/click behavior + a11y fallback list. |
app/src/pages/MapPage.helpers.ts |
Helper utilities for tag flattening, IDF, weighted Jaccard, KNN edges, and thumbnail tier loading. |
app/src/pages/MapPage.helpers.test.ts |
Unit tests for helper utilities. |
app/src/pages/MapPage.test.tsx |
Page-level smoke tests + callback behavior tests using a mocked ForceGraph component. |
| // Hairline border around a thumbnail node, theme-aware. | ||
| function strokeFor(isDark: boolean, isHover: boolean): string { | ||
| if (isHover) return colors.primary; | ||
| return isDark ? 'rgba(240,239,232,0.18)' : 'rgba(26,26,23,0.18)'; |
Comment on lines
+264
to
+269
| ctx.fillRect(x, y, w, h); | ||
| } | ||
| ctx.lineWidth = isHover ? 2 : 1; | ||
| ctx.strokeStyle = strokeFor(isDark, !!isHover); | ||
| ctx.strokeRect(x, y, w, h); | ||
| ctx.restore(); |
Comment on lines
+286
to
+291
| onNodeHover={(n: MapNode | null) => setHoverId(n?.id ?? null)} | ||
| cooldownTicks={COOLDOWN_TICKS} | ||
| onEngineStop={() => fgRef.current?.zoomToFit?.(400, 40)} | ||
| /> | ||
| )} | ||
|
|
Comment on lines
+107
to
+120
|
|
||
| // 3. derive graph data from specs/theme (pure — no setState in effect) | ||
| const graphData = useMemo<{ nodes: MapNode[]; links: MapLink[] }>(() => { | ||
| if (!specs) return { nodes: [], links: [] }; | ||
| const idf = computeIDF(specs); | ||
| const nodes: MapNode[] = specs.map(s => ({ | ||
| id: s.id, | ||
| title: s.title, | ||
| tags: flattenTags(s), | ||
| thumbUrl: selectMapThumbUrl(s, isDark), | ||
| imgs: new Map(), | ||
| pendingTiers: new Set(), | ||
| })); | ||
| const links = buildKNNLinks(specs, idf, KNN_K, KNN_MIN_SIM); |
Comment on lines
+2
to
+7
| * Pure helpers for the /map page: tag flattening, IDF weighting, | ||
| * weighted Jaccard similarity, and sparse KNN edge construction. | ||
| * | ||
| * Kept side-effect-free so the math is exhaustively unit-testable | ||
| * in MapPage.helpers.test.ts. The page component imports these and | ||
| * feeds the result into react-force-graph-2d. |
Two big UX upgrades to the /map page: 1. Cluster gravity by plot_type. Each major plot_type gets its own anchor on a ring around origin, and nodes get pulled toward their type's anchor with a gentle d3 forceX/forceY pair. The catalog has ~100 unique primary plot_types, most of them singletons; types with <3 specs bucket into a shared "other" cluster so the ring doesn't degenerate into a hundred crowded points. Result: visible "scatter neighborhood", "bar neighborhood", "line neighborhood" etc., with KNN edges still shaping the within-cluster topology. 2. Big hover preview. The previously tiny hover scale (1.6×) made it hard to see what each plot actually was. Now the hovered node gets a much larger overlay (9× node size) painted via onRenderFramePost so it always sits on top regardless of node-paint order. The preview lazy-upgrades to the highest resolution tier (1200) for crisp rendering, with a brand-green halo and border to lift it off the canvas. Plus tuning passes that fell out of MCP-driven testing: - forceManyBody strength: -160 (vs default ~-30) — more global spread - forceCollide on bounding-box radius — prevents thumbnail overlap - linkDistance/linkStrength scale with weighted-Jaccard similarity: highly related specs pull tight, weakly related drift apart - cooldownTicks 400 (vs 200) — cleaner final positions - Header overlay alignment fixed: sx `left` is NOT spacing-aware (unlike `px`/`mx`), so use raw px values matching RootLayout's container padding so "312 specs ·" lines up with the anyplot logo New helpers (all unit-tested): primaryPlotType, computeClusterAnchors (with minCount bucketing), clusterBucket. Ambient declarations for d3-force-3d added since no @types package exists. Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User feedback while iterating in the browser: - Thumbnails were too small to recognize the actual plot. Bumped NODE_SIZE 22 → 44 and HOVER_PREVIEW_SIZE down from 9× to 6× since the base node is now twice as big anyway. - Edges visually dominated the canvas. Halved both linkColor opacity (~10% → ~4%) and linkWidth multiplier, and made non-hover state even fainter when something IS hovered so the involved-edges highlight stands out. - Cluster effect was invisible. Cranked up the ring layout so each major plot_type forms a clearly distinguishable galaxy: CLUSTER_RADIUS 660 → 600 absolute graph units (decoupled from NODE_SIZE so future node-size changes don't blow up the layout), CLUSTER_STRENGTH 0.18 → 0.6. - Each bucket now paints with its own Okabe-Ito border color (cluster color = stable index into the brand palette) so the groups are obvious at a glance regardless of how the simulation positioned them. - Reduced REPULSION (-160 → -40) and capped LINK_STRENGTH (1 → 0.15) so the cluster gravity actually dominates instead of being drowned out by global repulsion + per-link pull. - Bumped zoomToFit padding to 80 px so the hover preview can bleed past a nearby canvas edge without clipping. Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “spec discovery” surface: a /map page that visualizes all plot specs as thumbnail nodes in a force-directed graph, clustered by tag similarity, backed by a new API endpoint that returns one row per spec with best-implementation preview URLs and tags.
Changes:
- Backend: introduce
GET /api/specs/map(served as/specs/mapunderAPI_URL) returningSpecMapItementries and integrate it with existing caching/invalidation. - Frontend: add a lazy-loaded
/maproute rendering areact-force-graph-2dcanvas with client-side IDF-weighted Jaccard + sparse KNN links, plus tests. - Navigation/deps: add a top-level NavBar link and new force-graph dependencies + minimal TS ambient types.
Reviewed changes
Copilot reviewed 11 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/api/test_routers.py | Adds unit tests covering /specs/map behavior (503 without DB, list shape, best-impl selection, skipping impl-less specs, empty DB). |
| api/schemas.py | Adds SpecMapItem response schema for the map endpoint. |
| api/routers/specs.py | Implements _build_specs_map and new GET /specs/map route with cache integration. |
| api/cache.py | Ensures spec cache invalidation also clears specs_map. |
| app/package.json | Adds force-graph and react-force-graph-2d dependencies. |
| app/yarn.lock | Locks new dependency tree for force-graph packages. |
| app/src/types/d3-force-3d.d.ts | Adds minimal ambient declarations for d3-force-3d used by the map page. |
| app/src/router.tsx | Registers /map as a lazy-loaded route. |
| app/src/components/NavBar.tsx | Adds a top-level “map” navigation link. |
| app/src/pages/MapPage.tsx | New page: fetches /specs/map, builds nodes/links, renders force graph, preloads thumbnails, hover/click interactions, and a11y fallback list. |
| app/src/pages/MapPage.test.tsx | Adds smoke/callback tests for the MapPage and ForceGraph integration (mocked). |
| app/src/pages/MapPage.helpers.ts | Adds pure helpers for tag flattening, IDF weighting, weighted Jaccard, KNN edge building, and thumbnail tier selection/loading. |
| app/src/pages/MapPage.helpers.test.ts | Adds unit tests for helper math + URL/tier selection utilities. |
Comment on lines
+72
to
+81
| class SpecMapItem(BaseModel): | ||
| """One row per spec for the /map page: best-impl preview + full tag bag for client-side similarity clustering.""" | ||
|
|
||
| id: str | ||
| title: str | ||
| preview_url_light: str | None = None | ||
| preview_url_dark: str | None = None | ||
| quality_score: float | None = None | ||
| tags: dict[str, Any] | None = None | ||
| impl_tags: dict[str, Any] | None = None |
Comment on lines
+326
to
+330
| } | ||
| ctx.lineWidth = isHover ? 2 : 1.5; | ||
| ctx.strokeStyle = strokeFor(isDark, !!isHover, clusterColor(n.primaryType, allBuckets)); | ||
| ctx.strokeRect(x, y, w, h); | ||
| ctx.restore(); |
Comment on lines
+437
to
+442
| <Box component="ul" sx={visuallyHiddenSx}> | ||
| {(specs ?? []).map(s => ( | ||
| <li key={s.id}> | ||
| <a href={specPath(s.id)}>{s.title}</a> | ||
| </li> | ||
| ))} |
Two changes from user feedback:
1. Weighted-Jaccard now applies a per-category multiplier on top of
the IDF weight: plot_type counts 3×, features 1.5×, dependencies
0.8×, techniques 0.7×, styling 0.5×. The rest stay at 1×. Sharing
the same plot_type now pulls two specs together harder than
sharing a common technique or visual style — which matches
intuition for a discovery interface ("show me other scatters"
beats "show me other things with alpha-blending").
2. Removed the cluster-gravity ring entirely. Top-7 plot_types still
get distinct Okabe-Ito border colors (line, scatter, bar, …) so
the catalog's biggest categories are visually obvious; everything
else paints with a neutral hairline border. The layout shape now
emerges purely from KNN edges + collision + many-body repulsion.
Plus tuning passes that fell out of MCP-driven testing:
- NODE_SIZE 22 → 60 — thumbnails are big enough to read at the zoom
level zoomToFit picks for 312 nodes. HOVER_PREVIEW_SIZE pulled
back from 9× to 5× since the base node is now much larger.
- MIN_ZOOM = 0.5 floor enforced after zoomToFit completes — without
it, a few far-flung outliers force zoomToFit to shrink the dense
central cluster down to illegible pixels.
- REPULSION -160 → -50, LINK_DISTANCE_MAX 6×NODE → 3.5×NODE, link
strength cap 1 → 0.4 — collision already enforces minimum spacing,
so strong repulsion was just blowing the graph wide enough that
zoomToFit had to zoom out further than thumbnails could survive.
- Subtler default link colors (~5% opacity) + thinner default
linkWidth, so the thumbnails dominate visually. Hover links light
up bright green for contrast.
- Header overlay alignment fixed: sx `left` is NOT spacing-aware
(unlike `px`/`mx`), so use raw px values matching RootLayout's
container padding.
New helpers (all unit-tested): primaryPlotType, topPlotTypes,
CATEGORY_WEIGHT. Removed the now-dead computeClusterAnchors and
clusterBucket. MapNode shape: dropped clusterX/clusterY/primaryType,
added colorBucket.
Refs #5646
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Legend overlay top-right of the canvas: one row per top-N plot type showing its cluster color + spec count (line 62 · scatter 35 · bar 21 · heatmap 19 · histogram 11 · map 8 · area 7). Hovering a row sets hoverType state — all nodes whose colorBucket matches stay full opacity, all others dim to 0.18 (same dimming logic as single-node hover). Intra-cluster links also stay visible while everything else fades, so the cluster's spatial shape pops out at a glance. Default link opacity bumped back up (10%/13% from the over-aggressive ~4% earlier) and minimum linkWidth raised to 0.4. Subtle still, but visible without hovering. Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new discovery surface (“/map”) that visualizes all plot specs as a force-directed graph clustered by tag similarity, backed by a new API endpoint that returns one row per spec with best-implementation previews and tag bags.
Changes:
- Backend: add
GET /specs/mapendpoint +SpecMapItemschema and cache invalidation support. - Frontend: add new lazy-loaded
/maproute with areact-force-graph-2dcanvas page and pure helper utilities for similarity + thumbnail tiering. - Tests: add unit tests for the new backend endpoint and the new frontend helpers/page.
Reviewed changes
Copilot reviewed 11 out of 13 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
api/routers/specs.py |
Adds _build_specs_map and GET /specs/map endpoint with caching. |
api/schemas.py |
Introduces SpecMapItem response schema. |
api/cache.py |
Invalidates specs_map cache when a spec cache is cleared. |
tests/unit/api/test_routers.py |
Adds unit tests covering /specs/map endpoint behavior. |
app/src/router.tsx |
Registers /map as a lazy-loaded route. |
app/src/components/NavBar.tsx |
Adds a top-level “map” nav link. |
app/src/pages/MapPage.tsx |
Implements the force-directed map page (rendering, hover/click behavior, thumbnail loading). |
app/src/pages/MapPage.helpers.ts |
Adds pure helpers for tag flattening, IDF weighting, weighted Jaccard, KNN edges, and thumbnail tier selection. |
app/src/pages/MapPage.helpers.test.ts |
Adds unit tests for helper math and URL/tiering utilities. |
app/src/pages/MapPage.test.tsx |
Adds smoke/callback tests for the Map page behavior in jsdom. |
app/src/types/d3-force-3d.d.ts |
Adds minimal ambient TS declarations for d3-force-3d. |
app/package.json |
Adds force-graph and react-force-graph-2d dependencies. |
app/yarn.lock |
Locks new dependency tree for force-graph packages. |
Comment on lines
+398
to
+404
| linkColor={(l: MapLink) => { | ||
| const involved = hoverId && (l.source === hoverId || l.target === hoverId); | ||
| if (involved) return colors.primary; | ||
| if (hoverType) { | ||
| const sId = typeof l.source === 'string' ? l.source : (l.source as { id?: string })?.id; | ||
| const tId = typeof l.target === 'string' ? l.target : (l.target as { id?: string })?.id; | ||
| const sBucket = graphData.nodes.find(n => n.id === sId)?.colorBucket; |
Comment on lines
+413
to
+416
| linkWidth={(l: MapLink) => { | ||
| const involved = hoverId && (l.source === hoverId || l.target === hoverId); | ||
| if (involved) return Math.max(1, (l.weight ?? 0.3) * 2.5); | ||
| return Math.max(0.4, (l.weight ?? 0.3) * 1.5); |
Comment on lines
+398
to
+412
| linkColor={(l: MapLink) => { | ||
| const involved = hoverId && (l.source === hoverId || l.target === hoverId); | ||
| if (involved) return colors.primary; | ||
| if (hoverType) { | ||
| const sId = typeof l.source === 'string' ? l.source : (l.source as { id?: string })?.id; | ||
| const tId = typeof l.target === 'string' ? l.target : (l.target as { id?: string })?.id; | ||
| const sBucket = graphData.nodes.find(n => n.id === sId)?.colorBucket; | ||
| const tBucket = graphData.nodes.find(n => n.id === tId)?.colorBucket; | ||
| const intra = sBucket === hoverType && tBucket === hoverType; | ||
| if (intra) return isDark ? 'rgba(255,255,255,0.18)' : 'rgba(0,0,0,0.22)'; | ||
| return isDark ? 'rgba(255,255,255,0.012)' : 'rgba(0,0,0,0.015)'; | ||
| } | ||
| if (hoverId) return isDark ? 'rgba(255,255,255,0.02)' : 'rgba(0,0,0,0.025)'; | ||
| return isDark ? 'rgba(255,255,255,0.10)' : 'rgba(0,0,0,0.13)'; | ||
| }} |
Comment on lines
+470
to
+475
| onRenderFramePost={(ctx) => { | ||
| if (!hoverId) return; | ||
| const n = graphData.nodes.find(x => x.id === hoverId) as | ||
| | (MapNode & { x?: number; y?: number }) | ||
| | undefined; | ||
| if (!n || n.x == null || n.y == null) return; |
| for spec in specs: | ||
| if not spec.impls: | ||
| continue | ||
| best = max(spec.impls, key=lambda i: ((i.quality_score or 0.0), i.library_id)) |
Comment on lines
+241
to
+243
| const colorOther = linkColor({ source: 'line-basic', target: 'scatter-color-mapped', weight: 0.5 }); | ||
| expect(colorInvolved).toMatch(/^#/); // brand color (hex) | ||
| expect(colorInvolved).not.toBe(colorOther); |
The 9 known tag categories (plot_type, features, data_type, domain, dependencies, techniques, patterns, dataprep, styling) now each have a 0–3 slider in a collapsible "weights" panel bottom-left of the canvas. Drag any slider and the KNN edges + force simulation rebuild in ~30 ms — instant visual feedback for "what if plot_type pulls harder than techniques?". API changes (helpers.ts): - Renamed CATEGORY_WEIGHT → DEFAULT_CATEGORY_WEIGHT (now exported). - weightedJaccard and buildKNNLinks accept an optional `weights` argument that overrides the defaults entirely. - Added TAG_CATEGORIES const + TagCategory type. Default weights revised after thinking about discovery semantics: plot_type 3.0 strongest signal features 1.5 techniques 1.0 patterns 1.0 dataprep 1.0 dependencies 0.8 domain 0.7 contextual, not structural data_type 0.6 too generic to discriminate styling 0.4 visual fluff UI lives bottom-left so it doesn't fight the legend (top-right) or the spec/edge counter (top-left). Toggle "weights ▸/▾" to open. Each row: category label · MUI Slider · current value. A reset button restores the defaults. Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User feedback after iterating with the weights panel: "wenn plot_type auf 3 und der rest auf 0 kommt eher so eine karte raus wie ich mir das vorgestellt hatte." Match that intuition by default: - DEFAULT_CATEGORY_WEIGHT.plot_type stays 3.0; everything else now 0. Cleanest "scatter-galaxy / bar-galaxy / line-galaxy" map out of the box. Users who want richer cross-type clustering slide up the secondary categories themselves. - Slider max bumped 3 → 5 to give more headroom for the user's experimentation (can now boost any category to 5× before relying on the IDF rarity scaling). Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new visual discovery surface (/map) that clusters plot specifications by tag similarity using a force-directed graph, backed by a new API endpoint that returns one row per spec with the best implementation’s preview URLs and tag data.
Changes:
- Backend: add
GET /specs/mapreturningSpecMapItemrows (best-impl preview + spec/impl tag bags) and invalidate it viaclear_spec_cache. - Frontend: add
/mappage built onreact-force-graph-2d, including similarity/KNN helpers with unit tests and page smoke tests. - App wiring: register the route and add a top-level NavBar link; add new frontend dependencies and minimal ambient TS types for
d3-force-3d.
Reviewed changes
Copilot reviewed 11 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
api/routers/specs.py |
Adds cached /specs/map endpoint and map payload builder. |
api/schemas.py |
Introduces SpecMapItem response model. |
api/cache.py |
Ensures spec cache invalidation also clears the map payload. |
tests/unit/api/test_routers.py |
Adds unit tests covering /specs/map behavior (503/no DB, best-impl selection, skip no-impl specs, empty DB). |
app/src/pages/MapPage.tsx |
Implements the new interactive force-graph map page UI and behavior. |
app/src/pages/MapPage.helpers.ts |
Pure helpers for tag flattening, IDF weighting, weighted Jaccard, KNN links, and thumbnail tiering. |
app/src/pages/MapPage.helpers.test.ts |
Unit tests for the helper math/utility functions. |
app/src/pages/MapPage.test.tsx |
Smoke tests for page rendering and key callbacks. |
app/src/router.tsx |
Adds lazy-loaded /map route. |
app/src/components/NavBar.tsx |
Adds a top-level “map” navigation link. |
app/src/types/d3-force-3d.d.ts |
Adds minimal ambient typings for d3-force-3d. |
app/package.json |
Adds react-force-graph-2d (and related) dependencies. |
app/yarn.lock |
Locks new dependency graph for the force-graph packages. |
Comment on lines
+360
to
+376
| <Box key={cat} sx={{ display: 'flex', alignItems: 'center', gap: 1.5 }}> | ||
| <Box component="span" sx={{ minWidth: 100, fontFamily: typography.mono, fontSize: fontSize.xs }}> | ||
| {cat} | ||
| </Box> | ||
| <Slider | ||
| value={weights[cat]} | ||
| onChange={(_, v) => setWeights(w => ({ ...w, [cat]: v as number }))} | ||
| min={0} | ||
| max={5} | ||
| step={0.1} | ||
| size="small" | ||
| sx={{ | ||
| flex: 1, | ||
| color: colors.primary, | ||
| '& .MuiSlider-rail': { opacity: 0.25 }, | ||
| }} | ||
| /> |
Comment on lines
+497
to
+505
| if (hoverType) { | ||
| const sId = typeof l.source === 'string' ? l.source : (l.source as { id?: string })?.id; | ||
| const tId = typeof l.target === 'string' ? l.target : (l.target as { id?: string })?.id; | ||
| const sBucket = graphData.nodes.find(n => n.id === sId)?.colorBucket; | ||
| const tBucket = graphData.nodes.find(n => n.id === tId)?.colorBucket; | ||
| const intra = sBucket === hoverType && tBucket === hoverType; | ||
| if (intra) return isDark ? 'rgba(255,255,255,0.18)' : 'rgba(0,0,0,0.22)'; | ||
| return isDark ? 'rgba(255,255,255,0.012)' : 'rgba(0,0,0,0.015)'; | ||
| } |
Comment on lines
+255
to
+271
| /** | ||
| * Return the highest-resolution tier that's already loaded and at least as | ||
| * big as `desired`. Falls back to a smaller tier if nothing larger is loaded | ||
| * yet (better than blank during the lazy upgrade). | ||
| */ | ||
| export function pickBestLoadedTier( | ||
| imgs: Map<ResolutionTier, HTMLImageElement>, | ||
| desired: ResolutionTier | ||
| ): HTMLImageElement | null { | ||
| for (const t of RESOLUTION_TIERS) { | ||
| if (t >= desired && imgs.has(t)) return imgs.get(t)!; | ||
| } | ||
| for (let i = RESOLUTION_TIERS.length - 1; i >= 0; i--) { | ||
| const t = RESOLUTION_TIERS[i]; | ||
| if (imgs.has(t)) return imgs.get(t)!; | ||
| } | ||
| return null; |
Comment on lines
+13
to
+22
| /** Backend response shape from GET /api/specs/map. Mirrors api/schemas.py::SpecMapItem. */ | ||
| export interface SpecMapItem { | ||
| id: string; | ||
| title: string; | ||
| preview_url_light: string | null; | ||
| preview_url_dark: string | null; | ||
| quality_score: number | null; | ||
| tags: Record<string, string[]> | null; | ||
| impl_tags: Record<string, string[]> | null; | ||
| } |
User feedback: tags like selenium (in 98% of specs), numeric (92%), html-export, basic etc. are pure noise — sharing them between two specs carries no real semantic signal but their cumulative weight across many shared common tags creates spurious cross-cluster bridges that collapse the graph into a blob whenever a secondary category is enabled. Two changes that work together: 1. computeIDF gains a `maxDfRatio` parameter (default 0.67). Tags present in more than that fraction of the corpus get IDF=0 regardless of their natural log(N/df) value. log-IDF assigns selenium ~0.016 and basic ~0.7, both small but nonzero — the sum across many shared common tags adds up. Hard-zeroing them kills the noise entirely without hurting genuinely-informative common-but-not-ubiquitous tags. 2. KNN_MIN_SIM 0.05 → 0.15. With cleaner per-tag weights, weak cross-cluster bridges (sim 0.05-0.12) that survive the IDF cutoff still get filtered at the link selector. At plot_type=3/others=0 same-type sim is ~1.0 so no real signal is lost. Net result: enabling secondary category weights now enriches the clustering instead of collapsing it. features=1 produces 924 edges with clusters still visibly distinct (heatmap, line, scatter, bar neighborhoods all readable). Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Until now the colored borders and the legend were hardcoded to the top-7 plot_types. With the weight sliders now letting the user dial up any category, the same buckets stayed colored even when "styling" became the dominant similarity signal — useful colors but mismatched to the actual clustering driver. Compute an `activeCategory` from the current weights (max wins; ties go to plot_type because it's first in TAG_CATEGORIES and we use strictly-greater compare). The legend now shows the active category name as a caption + the top-7 values from THAT category with their counts, and node border colors track the same buckets. Falls back to plot_type when all weights are zero. Refactored the helpers to be category-generic: - primaryCategoryValue(spec, category) — replaces primaryPlotType, with primaryPlotType kept as a thin wrapper for compatibility. - categoryValueCounts(specs, category) — replaces plotTypeCounts. - topCategoryValues(specs, category, n) — replaces topPlotTypes. - SPEC_LEVEL_CATEGORIES const distinguishes spec-level (plot_type, features, data_type, domain) from impl-level (the rest) so the helper looks up the right tag dict on the spec. Refs #5646 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “spec map” discovery surface by exposing a backend /specs/map payload (one row per spec with best-implementation preview + tags) and a new frontend /map page that clusters specs in a force-directed graph using tag-similarity helpers.
Changes:
- Backend: add
GET /specs/mapendpoint +SpecMapItemschema and cache invalidation. - Frontend: add
/maproute + navbar link and a newMapPageusingreact-force-graph-2d. - Frontend: add pure similarity/thumbnail helper utilities with unit tests and page smoke tests; add new deps.
Reviewed changes
Copilot reviewed 11 out of 13 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
api/routers/specs.py |
Adds /specs/map route and _build_specs_map() builder with caching. |
api/schemas.py |
Introduces SpecMapItem response schema used by the new endpoint. |
api/cache.py |
Invalidates specs_map cache when a spec-related cache clear happens. |
tests/unit/api/test_routers.py |
Adds unit tests covering /specs/map behavior (503, empty, best-impl selection, skipping impl-less). |
app/src/router.tsx |
Registers /map as a lazy-loaded route. |
app/src/components/NavBar.tsx |
Adds a top-level “map” nav link. |
app/src/pages/MapPage.tsx |
New page implementing the force-directed thumbnail graph UI + controls + a11y fallback list. |
app/src/pages/MapPage.helpers.ts |
Pure helpers for tag flattening, IDF weighting, weighted Jaccard, KNN links, and responsive thumbnail tiering. |
app/src/pages/MapPage.helpers.test.ts |
Unit tests for the helper functions. |
app/src/pages/MapPage.test.tsx |
Smoke/tests for MapPage rendering and key callback behaviors via a mocked ForceGraph2D. |
app/src/types/d3-force-3d.d.ts |
Adds minimal ambient typings for d3-force-3d to satisfy TS builds. |
app/package.json |
Adds force-graph and react-force-graph-2d dependencies. |
app/yarn.lock |
Locks transitive deps introduced by the new graph libraries. |
Comment on lines
+208
to
+224
| useEffect(() => { | ||
| if (graphData.nodes.length === 0) return; | ||
| const nodeById = new Map(graphData.nodes.map(n => [n.id, n])); | ||
| let cancelled = false; | ||
| preloadImages( | ||
| graphData.nodes.map(n => ({ id: n.id, thumbUrl: n.thumbUrl })), | ||
| (id, tier, img) => { | ||
| if (cancelled) return; | ||
| const n = nodeById.get(id); | ||
| if (n) n.imgs.set(tier, img); | ||
| fgRef.current?.refresh?.(); | ||
| } | ||
| ); | ||
| return () => { | ||
| cancelled = true; | ||
| }; | ||
| }, [graphData]); |
Comment on lines
+518
to
+532
| linkColor={(l: MapLink) => { | ||
| const involved = hoverId && (l.source === hoverId || l.target === hoverId); | ||
| if (involved) return colors.primary; | ||
| if (hoverType) { | ||
| const sId = typeof l.source === 'string' ? l.source : (l.source as { id?: string })?.id; | ||
| const tId = typeof l.target === 'string' ? l.target : (l.target as { id?: string })?.id; | ||
| const sBucket = graphData.nodes.find(n => n.id === sId)?.colorBucket; | ||
| const tBucket = graphData.nodes.find(n => n.id === tId)?.colorBucket; | ||
| const intra = sBucket === hoverType && tBucket === hoverType; | ||
| if (intra) return isDark ? 'rgba(255,255,255,0.18)' : 'rgba(0,0,0,0.22)'; | ||
| return isDark ? 'rgba(255,255,255,0.012)' : 'rgba(0,0,0,0.015)'; | ||
| } | ||
| if (hoverId) return isDark ? 'rgba(255,255,255,0.02)' : 'rgba(0,0,0,0.025)'; | ||
| return isDark ? 'rgba(255,255,255,0.10)' : 'rgba(0,0,0,0.13)'; | ||
| }} |
Comment on lines
+356
to
+368
| <Box | ||
| component="button" | ||
| onClick={() => setWeightsOpen(o => !o)} | ||
| sx={{ | ||
| all: 'unset', | ||
| cursor: 'pointer', | ||
| fontFamily: typography.mono, | ||
| fontSize: fontSize.xs, | ||
| color: weightsOpen ? 'var(--ink)' : 'var(--ink-soft)', | ||
| '&:hover': { color: colors.primary }, | ||
| userSelect: 'none', | ||
| }} | ||
| > |
| fontFamily: typography.mono, | ||
| fontSize: fontSize.xs, | ||
| color: 'var(--ink-soft)', | ||
| '&:hover': { color: colors.primary }, |
Comment on lines
+187
to
+200
| const idf = computeIDF(specs); | ||
| const topTypes = topCategoryValues(specs, activeCategory, CLUSTER_COLORS.length); | ||
| const typeCounts = categoryValueCounts(specs, activeCategory); | ||
| const nodes: MapNode[] = specs.map(s => { | ||
| const v = primaryCategoryValue(s, activeCategory); | ||
| return { | ||
| id: s.id, | ||
| title: s.title, | ||
| tags: flattenTags(s), | ||
| colorBucket: topTypes.includes(v) ? v : null, | ||
| thumbUrl: selectMapThumbUrl(s, isDark), | ||
| imgs: new Map(), | ||
| pendingTiers: new Set(), | ||
| }; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GET /api/specs/mapendpoint — one row per spec with the highest-rated implementation's preview URLs, quality score, and full tag bag (spec + impl)./mappage powered byreact-force-graph-2d— image-thumbnail nodes positioned by client-side weighted-Jaccard similarity over tags (IDF-weighted, sparse KNN edges).NavBaras a top-level link and registered as a lazy-loaded route.Closes #5646.
Approach in one paragraph
Tag similarity, IDF weighting, and KNN edge construction all happen client-side as pure helpers in
app/src/pages/MapPage.helpers.ts(19 unit tests). 327 specs × pairwise weighted-Jaccard runs in ~30 ms in the browser, so there's no need to precompute positions on the server. The wrapper's built-in d3-force physics handles the layout from a sparse link list (K=5, min-sim 0.05 → ~1.6k edges). Best-impl thumbnails are eager-preloaded and attached to nodes as they resolve, so the canvas re-paints organically without restarting physics. Hover highlights neighbors, click navigates to the spec page, and a visually-hidden anchor list mirrors the canvas for screen readers + keyboard users.Test plan
uv run ruff check api/ tests/— cleanuv run ruff format --check api/ tests/— cleanuv run pytest tests/unit/api/ tests/unit/core/— 739 passed (5 new SpecsRouter tests for the endpoint)cd app && yarn lint— no new errors in MapPage files (pre-existing baseline drift in unrelated files unchanged)cd app && yarn tsc --noEmit— cleancd app && yarn test— 421 passed (19 new helper tests + 3 new page smoke tests)cd app && yarn build— succeeds;/mapchunk is 193 KB raw / 63 KB gz, lazy-loaded only when visiting/mapNotes / known follow-ups
MapPage.tsxcancelsRootLayout's container padding so the canvas goes full-bleed. TODO comment left in place; cleaner long-term answer is a router-level layout switch analog to the existingmastheadSticksflag.react-hooks/set-state-in-effectandno-undeferrors in unrelated files were already present before this branch. None added; one previously-introduced one inMapPage.tsxwas refactored away during review.🤖 Generated with Claude Code