Skip to content

Commit 2d08128

Browse files
ttuffttuff
authored
Merge pull request #207 from CU-ESIIL/codex/add-deprecated-section-for-old-files
docs: quarantine legacy debug notes and update deprecation inventory
2 parents 47ce004 + 7dcf470 commit 2d08128

5 files changed

Lines changed: 137 additions & 108 deletions

File tree

‎docs/legacy/internal_notes/streaming_renderer.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Streaming-First Rendering & Huge Data
2+
3+
The cube viewer is designed for massive NEON, Sentinel-2, PRISM, and gridMET cubes. It iterates over time slices and never materializes the full cube in memory. **`v.plot()` and `CubePlot` are streaming-first by design.**
4+
5+
## Why streaming?
6+
7+
- Continental NDVI archives exceed laptop memory
8+
- Long PRISM or gridMET histories are easier to browse slice by slice
9+
- Progress feedback keeps notebooks responsive during big pulls
10+
11+
## How it works
12+
13+
- Works against dask-backed xarray DataArrays **and** streaming `VirtualCube` sources.
14+
- Iterates over time frames: `for t in range(0, nt, thin_time_factor)`.
15+
- Extracts only 2D slices per iteration (frame-sized `.values` are OK; full-cube `.values` are not).
16+
- Combines min/max ranges after all slices for consistent color scales without materializing the cube.
17+
- Shows a progress indicator (`progress_style="bar"` or `"pulse"`).
18+
19+
Vase volumes reuse the same pattern: `build_vase_mask` walks time slices using coordinates only, so face overlays from
20+
`geom_vase_outline` keep the streaming pipeline intact. See [Vase Volumes & Arbitrary 3-D Subsets](../../vase_volumes.md) for details.
21+
22+
## Faceting with streaming
23+
24+
Facets subset the cube one panel at a time while sharing scales:
25+
26+
```python
27+
(CubePlot(ndvi)
28+
.facet_grid(row="scenario", col="model")
29+
.geom_cube()
30+
.scale_fill_continuous(center=0)
31+
)
32+
```
33+
34+
Each facet panel pulls only its own slices, so you can fan out scenarios without memory blowups.
35+
36+
## Performance best practices
37+
38+
- Keep inputs chunked; avoid `ndvi.compute()` unless absolutely necessary
39+
- Lower `thin_time_factor` to preview long series quickly
40+
- Use `out_html` to write intermediate panels and reuse them in reports
41+
- Prefer `.facet_wrap` when the facet variable has many categories so you can control `ncol`
42+
43+
## Streaming progress in Jupyter
44+
45+
When `show_progress=True`, the renderer streams slices and displays a live progress bar. It is safe to interrupt and restart without corrupting the cube.
46+
47+
## Do / Don't for streaming safety
48+
49+
- ✅ Use xarray methods like `.isel`, `.chunk`, and `.compute()` inside targeted helper functions where you control memory.
50+
- ✅ Pass `thin_time_factor` or other downsampling parameters when you want lightweight previews.
51+
- ❌ Don't call `.values` or `np.asarray()` on the **entire** cube within plotting or verb code; rely on streamed frames instead.
52+
- ❌ Don't coerce dask arrays to NumPy arrays unless the path is explicitly documented for small cubes.

‎docs/legacy/internal_notes/viewer_debug_notes.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Viewer debug notes
2+
3+
## 1. Call graph
4+
- `v.plot()` builds a `PlotOptions` object and wraps an inner `_plot` that materializes virtual cubes, infers dims, and returns a configured `CubePlot` with geom/axes/theme set. The function returns either the `Verb` or the resulting `CubePlot`. 【F:src/cubedynamics/verbs/plot.py†L17-L108】
5+
- `CubePlot.to_html()` prepares stats/annotations and calls `_render_viewer`, which constructs the viewer HTML via `cube_from_dataarray` with styling and legend metadata. `_repr_html_` simply delegates to `to_html()`. 【F:src/cubedynamics/plotting/cube_plot.py†L582-L804】【F:src/cubedynamics/plotting/cube_plot.py†L836-L839】
6+
- `_render_viewer` hands off to `cube_from_dataarray` in `cube_viewer.py`, which renders faces/interior slices to base64 PNGs and assembles the WebGL viewer template. This template is the only HTML/JS builder on the Python side. 【F:src/cubedynamics/plotting/cube_plot.py†L629-L684】【F:src/cubedynamics/plotting/cube_viewer.py†L1-L120】
7+
8+
## 2. Python types
9+
Running `python tools/debug_viewer_pipeline.py` prints:
10+
```
11+
CubePlot type: <class 'cubedynamics.plotting.cube_plot.CubePlot'>
12+
v.plot return type: <class 'cubedynamics.plotting.cube_plot.CubePlot'>
13+
```
14+
So `v.plot()` still returns a `CubePlot` object. 【27a0d3†L1-L2】
15+
16+
## 3. Logging
17+
- Added logging hooks: `v.plot()` now logs the cube name/dims when invoked; `CubePlot._repr_html_` logs when the HTML repr is requested. 【F:src/cubedynamics/verbs/plot.py†L81-L84】【F:src/cubedynamics/plotting/cube_plot.py†L836-L839】
18+
- Enable by configuring `logging.basicConfig(level=logging.INFO)` in a notebook/kernel; watch stdout/stderr for calls when rendering.
19+
20+
## 4. Performance harness
21+
- `notebooks/cube_viewer_perf.ipynb` builds full/half/quarter-resolution NDVI cubes and renders them with `v.plot(debug=True)`.
22+
Record a Performance trace in Chrome and confirm `[CubeViewer debug] draw start/end` logs only appear while dragging or zooming.
23+
24+
## 5. HTML/JS template
25+
- The viewer HTML is entirely generated in `cube_from_dataarray` and `_render_cube_html`. It currently uses a custom WebGL wireframe cube (`canvas.getContext("webgl")` plus manual shaders) rather than the previous Lexcube integration—no references to Lexcube remain. 【F:src/cubedynamics/plotting/cube_viewer.py†L90-L195】【F:src/cubedynamics/plotting/cube_viewer.py†L374-L520】
26+
- The HTML builds a cube wrapper with `<canvas id="cube-canvas-{fig_id}">` and overlays axis labels/legend, but the JS only draws a wireframe cube; face textures are never applied to the canvas. PNG faces are still generated, but `_render_cube_html` only uses them as CSS backgrounds for `div` planes, which are missing in the current WebGL path. 【F:src/cubedynamics/plotting/cube_viewer.py†L30-L88】【F:src/cubedynamics/plotting/cube_viewer.py†L471-L520】
27+
- No alternative templates are present; the current template is the single path invoked by `CubePlot`.
28+
29+
## 6. JS console
30+
- Added guard logging around the viewer script; browser consoles should now show `[CubeViewer] script starting` and report `[CubeViewer] top-level error` if the script throws early. 【F:src/cubedynamics/plotting/cube_viewer.py†L369-L520】
31+
- To capture errors: open DevTools → Console after running `pipe(ndvi) | v.plot()`; watch for those messages alongside any WebGL errors.
32+
33+
## 7. Double render?
34+
- Searches show no `IPython.display.display` calls in plotting/verbs; rendering relies solely on the `CubePlot` return value and its `_repr_html_`. No evidence of double display. 【dbd202†L1-L1】
35+
36+
## 8. Eager loads
37+
- Face PNGs and interior planes call `.values` on slices; this is necessary for encoding small 2D images. Bulk operations include downsampling via `coarsen(...).mean()` for interior planes. Potentially heavy if the cube is large and `thin_time_factor` is small. 【F:src/cubedynamics/plotting/cube_viewer.py†L638-L718】【F:src/cubedynamics/plotting/cube_viewer.py†L828-L868】
38+
- Coordinate metadata lookups use `coord.values` but are lightweight. 【F:src/cubedynamics/plotting/cube_plot.py†L446-L474】
39+
40+
## 9. Hypotheses
41+
- **H1:** Viewer shows blank center because the JS path now draws only a green wireframe cube; PNG face textures (data slices) are not being mapped anywhere in the WebGL canvas. The DOM also lacks the Lexcube/CSS 3D elements that previously displayed face images. 【F:src/cubedynamics/plotting/cube_viewer.py†L374-L520】【F:src/cubedynamics/plotting/cube_viewer.py†L30-L88】
42+
- **H2:** Any WebGL init failure would now surface via `[CubeViewer] top-level error`; if errors appear, the script may be failing before draw (e.g., shader issues), leaving a blank canvas. 【F:src/cubedynamics/plotting/cube_viewer.py†L369-L520】
43+
- **H3:** Slowness likely comes from full-face `.values` extraction and color mapping for each face plus interior downsampling; large cubes will still materialize multiple slices eagerly before rendering. 【F:src/cubedynamics/plotting/cube_viewer.py†L638-L718】【F:src/cubedynamics/plotting/cube_viewer.py†L828-L868】
44+
45+
## 10. Interaction regressions (2025-03)
46+
- **What changed?** Drag setup was refactored to share pointer/mouse/touch start logic and attach move/end listeners to `window` so rotation keeps flowing even if the pointer leaves the drag surface. Pointer capture is attempted on the drag overlay but gracefully skipped when unsupported. Drag sessions now track the active pointer/touch identifier and clear any stale listeners before beginning a new drag to prevent cross-pointer interference. 【F:src/cubedynamics/plotting/cube_viewer.py†L353-L472】
47+
- **How to debug:**
48+
- Open DevTools and watch for `[CubeViewer] drag start/move/end` console logs when interacting with the cube. Absence of logs suggests the event listeners are not attaching (e.g., scripts blocked) or the drag surface is not present.
49+
- Verify the transparent drag surface exists with `document.getElementById("cube-drag-surface-<fig_id>")`; rotation depends on this element being on top of the cube.
50+
- Pointer capture failures are expected on some touch devices; the viewer falls back to window-level listeners. If moves stop mid-drag, confirm the move/end listeners are on `window` via `getEventListeners(window)` (in Chromium-based devtools) or by adding `window.addEventListener` breakpoints.
51+
- If drag motion stutters on multi-touch devices, inspect `activePointerId`/`activeTouchId` in the embedded script to ensure the move handler is gating events to the current pointer ID; stale listeners are cleared at drag start, so seeing multiple active IDs usually means the drag surface never received `pointerup/touchend`.
52+
- Zoom uses the wheel handler on the drag surface; if scroll-to-zoom stops working, inspect whether the `wheel` listener is blocked by the notebook or page-level scroll container.
53+
54+
## 11. Rotation/zoom expectations (2025-05)
55+
- Rotation is applied around the cube’s center via `applyCubeRotation()`; if you see skewing or off-center rotation, inspect the inline handler that updates `rotationX`/`rotationY` during drag gestures before the transform is applied. 【F:src/cubedynamics/plotting/cube_viewer.py†L590-L647】
56+
- Zoom should bring the cube closer (larger on screen). In DevTools, watch the logged `zoom` value in the `wheel` handler; if the cube shrinks when you zoom in, confirm the exponential zoom factor is clamped between `zoomMin` and `zoomMax`. 【F:src/cubedynamics/plotting/cube_viewer.py†L723-L730】
57+
58+
## 12. Interactivity hooks (2025-05)
59+
- The root viewer element now carries deterministic IDs (`cube-figure-<id>`) plus `data-debug`/`data-fig-id` attributes so the inline script can always locate the DOM node, even when Jupyter wraps outputs. The debug flag enables `[CubeViewer debug]` console logs for pointer/mouse/touch/wheel events. 【F:src/cubedynamics/plotting/cube_viewer.py†L200-L236】【F:src/cubedynamics/plotting/cube_viewer.py†L529-L596】
60+
- PointerEvents, mouse, and touch listeners are always attached to the drag surface; wheel zoom uses a non-passive handler to prevent default scrolling. Event logs emit pointer/touch identifiers to help debug Safari or embedded-notebook quirks. 【F:src/cubedynamics/plotting/cube_viewer.py†L597-L705】
61+
- `_write_demo_html()` emits a standalone `cube_demo.html` with color blocks so developers can validate drag/zoom outside of notebooks before shipping changes. 【F:src/cubedynamics/plotting/cube_viewer.py†L1032-L1072】

‎docs/project/deprecation_inventory.md‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,11 +33,22 @@ No D-class code confidently identified; uncertain items kept as legacy aliases.
3333
| `docs/vase_volumes.md` | A | Canonical vase guide referenced by legacy stub. | Keep; ensure language matches glossary. |
3434
| `docs/vase-volumes.md` | C | Legacy path kept for backward compatibility; now stub pointing to canonical page. | Keep stub; leave full content in `docs/legacy/`. |
3535
| `docs/legacy/vase-volumes.md` | C | Archived original vase volume write-up. | Keep in legacy folder; omit from nav. |
36-
| `docs/viewer_debug_notes.md`, `docs/streaming_renderer.md` | C | Developer notes not in nav; older terminology. | Move to `docs/legacy/` or annotate as internal references. |
36+
| `docs/viewer_debug_notes.md`, `docs/streaming_renderer.md` | C | Quarantined as deprecated stubs that point at archived notes under `docs/legacy/internal_notes/`. | Keep stub paths for backwards links; do not expand as active docs. |
37+
| `docs/legacy/internal_notes/*` | C | Archived engineering/debug notes retained for traceability only. | Keep out of nav; treat as historical reference. |
3738
| `docs/examples/*`, `docs/recipes/*` | B | Supplemental material referenced sporadically. | Keep; audit for vocabulary alignment. |
3839

3940
No D-class docs identified yet; treat ambiguous pages as legacy instead of deleting.
4041

42+
## Old-stuff quarantine sweep (2026-03-27)
43+
44+
Completed in this pass:
45+
- Moved standalone debug notes from top-level docs paths into `docs/legacy/internal_notes/`.
46+
- Replaced the original top-level files with lightweight deprecated stubs so old inbound links still resolve.
47+
48+
Next quarantine candidates (non-breaking, docs-only):
49+
- Root-level historical pages not in nav (`docs/pipe_syntax.md`, `docs/pipe_verbs.md`, `docs/cubeplot_grammar.md`) can follow the same stub + archive pattern.
50+
- Older conceptual snapshots (`docs/climate_cubes.md`, `docs/concepts/climate_cubes.md`) can be merged into canonical concepts pages and retained as legacy aliases.
51+
4152
## Tests and examples
4253

4354
| Path | Class | Evidence | Proposed action |

‎docs/streaming_renderer.md‎

Lines changed: 6 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,9 @@
1-
# Streaming-First Rendering & Huge Data
1+
# Streaming renderer (deprecated path)
22

3-
The cube viewer is designed for massive NEON, Sentinel-2, PRISM, and gridMET cubes. It iterates over time slices and never materializes the full cube in memory. **`v.plot()` and `CubePlot` are streaming-first by design.**
3+
This page has been quarantined as legacy/internal notes.
44

5-
## Why streaming?
5+
- Archived copy: [`docs/legacy/internal_notes/streaming_renderer.md`](legacy/internal_notes/streaming_renderer.md)
6+
- Canonical streaming guidance: [VirtualCubes](concepts/virtual_cubes.md)
7+
- Current visualization docs: [Visualization overview](viz/index.md)
68

7-
- Continental NDVI archives exceed laptop memory
8-
- Long PRISM or gridMET histories are easier to browse slice by slice
9-
- Progress feedback keeps notebooks responsive during big pulls
10-
11-
## How it works
12-
13-
- Works against dask-backed xarray DataArrays **and** streaming `VirtualCube` sources.
14-
- Iterates over time frames: `for t in range(0, nt, thin_time_factor)`.
15-
- Extracts only 2D slices per iteration (frame-sized `.values` are OK; full-cube `.values` are not).
16-
- Combines min/max ranges after all slices for consistent color scales without materializing the cube.
17-
- Shows a progress indicator (`progress_style="bar"` or `"pulse"`).
18-
19-
Vase volumes reuse the same pattern: `build_vase_mask` walks time slices using coordinates only, so face overlays from
20-
`geom_vase_outline` keep the streaming pipeline intact. See [Vase Volumes & Arbitrary 3-D Subsets](vase_volumes.md) for details.
21-
22-
## Faceting with streaming
23-
24-
Facets subset the cube one panel at a time while sharing scales:
25-
26-
```python
27-
(CubePlot(ndvi)
28-
.facet_grid(row="scenario", col="model")
29-
.geom_cube()
30-
.scale_fill_continuous(center=0)
31-
)
32-
```
33-
34-
Each facet panel pulls only its own slices, so you can fan out scenarios without memory blowups.
35-
36-
## Performance best practices
37-
38-
- Keep inputs chunked; avoid `ndvi.compute()` unless absolutely necessary
39-
- Lower `thin_time_factor` to preview long series quickly
40-
- Use `out_html` to write intermediate panels and reuse them in reports
41-
- Prefer `.facet_wrap` when the facet variable has many categories so you can control `ncol`
42-
43-
## Streaming progress in Jupyter
44-
45-
When `show_progress=True`, the renderer streams slices and displays a live progress bar. It is safe to interrupt and restart without corrupting the cube.
46-
47-
## Do / Don't for streaming safety
48-
49-
- ✅ Use xarray methods like `.isel`, `.chunk`, and `.compute()` inside targeted helper functions where you control memory.
50-
- ✅ Pass `thin_time_factor` or other downsampling parameters when you want lightweight previews.
51-
- ❌ Don't call `.values` or `np.asarray()` on the **entire** cube within plotting or verb code; rely on streamed frames instead.
52-
- ❌ Don't coerce dask arrays to NumPy arrays unless the path is explicitly documented for small cubes.
9+
This deprecated path is retained to avoid breaking old links.

0 commit comments

Comments
 (0)