fix(rendering): Introduce SequentialRenderingEngine to resolve canvas size limitation, performance degradation on high res monitors and enhance multi-monitor support

[IbrahimCSAE]

Overview

This PR introduces a new SequentialRenderingEngine to fundamentally solve the issue of viewport cropping, misalignment, and terrible performance on high-resolution displays or in layouts with many viewports.

The Problem: Offscreen Canvas Overflow & Performance Degradation Due to Ofscreen Canvas Growth

Cornerstone3D has historically used a single, large offscreen canvas for all viewports. This canvas grows horizontally as more viewports are added. This strategy is efficient until the total width of all viewports exceeds the browser's maximum canvas size (e.g., 16,384px in Chrome).

When this limit is breached, the offscreen canvas is silently cropped. Since the engine copies pixel data from fixed locations on the offscreen canvas to the on-screen viewports, this cropping results in severe visual artifacts, including misaligned and completely blank viewports. This issue is easily reproducible on high-DPI monitors or by manually increasing window.devicePixelRatio. The peformance also heavly degrades as the limit is approached.

Visual Example of the Bug:

Diagram of Current RenderingEngine:

The Journey to a Solution

Before arriving at the final solution, several alternative approaches were extensively researched and implemented. Each had significant drawbacks that made them unsuitable:

1. Multiple Offscreen Canvases: This approach attempted to distribute viewports across a few offscreen canvases.
   • Limitation: It introduced a massive performance hit and was constrained by the requirement that all VolumeViewports must share the same offscreen canvas. Only StackViewports saw a partial benefit.
2. Advanced Canvas Packing: We explored more complex packing algorithms (e.g., horizontal and vertical) to use the offscreen space more efficiently.
   • Limitation: This still ran into fundamental OS and browser memory limits. The theoretical maximum canvas size (e.g., 16k x 16k) is often not practically achievable, not even (8k x 8k) in some instances.
3. Multiple Rendering Engines: This involved spinning up a new rendering engine for each viewport.
   • Limitation: This was a dead end. Browsers limit the number of active WebGL contexts to ~16 per tab, creating a hard cap of 16 viewports. It also required all volumes to share a single engine, negating the benefit for volume rendering.

The Solution: SequentialRenderingEngine

This PR introduces a new, robust rendering strategy: the SequentialRenderingEngine.

Instead of a single, massive offscreen canvas that grows with each new viewport, this engine uses a fundamentally different approach. It renders each viewport individually to a viewport sized offscreen canvas, copies the result to the corresponding on-screen canvas, and then proceeds to the next viewport, reusing the same offscreen canvas.

This elegant solution completely bypasses the previous pain points:

• ✅ No More Aggregate Canvas Limits: The browser's maximum canvas size now applies to a single viewport, not the combined width of all viewports.
• ✅ Single WebGL Context: The engine operates with just one WebGL context, so we are no longer limited to 16 viewports.
• ✅ Memory: It avoids allocating a huge, memory-intensive offscreen canvas.
• ✅ OS & Browser Friendly: It steers clear of underlying OS memory and allocation limits that more aggressive strategies would hit.
• ✅ Enhanced Stability: Reduces the risk of WebGL context loss associated with creating and managing huge canvas surfaces.
• ✅ Multi-Monitor Support: Running OHIF or Cornerstone across a stretched tab on multiple high-resolution monitors was practically impossible before, due to canvas size limits and severe performance degradation. With the SequentialRenderingEngine, this becomes not only feasible but smooth and performant, making the use of OHIF / Cornerstone more realistic in actual diagnostic radiology environments where 2-4 monitors are common.

Visual Example of the Fix:

Diagram of Sequential RenderingEngine:

Implementation Details

This PR includes the following key changes:

• RenderingEngine Refactor: The original RenderingEngine returns an instance of either StandardRenderingEngine or SequentialRenderingEngine, based on a property that can be configured in the init function, the property is renderingEngineMode, which is a string that can be either standard or next
• BaseRenderingEngine: A new abstract base class has been created to define a common rendering engine interface.
• StandardRenderingEngine: The original RenderingEngine has been moved to the class StandardRenderingEngine which extends BaseRenderingEngine
• New SequentialRenderingEngine: The new engine is implemented as a separate class, also extending BaseRenderingEngine.
• Viewport Compatibility: The worldToCanvas and canvasToWorld methods on both StackViewport and VolumeViewport have been updated to support either rendering engine.
• New Examples: Two new examples have been added to demonstrate rendering a 6x6 grid of viewports—one using the standard engine (which will fail) and one using the new sequential engine (which will succeed). This provides a clear test of the solution.

Timeline

• April 8th: Issue reported on some Barco monitors, eg here, and 4k, 6k displays, eg here.
• April 9th: Try a multiple offscreen canvas approach here
• April 19th: Performance is horrible with multi offscreen, try a per viewport RE approach here
• May-June: Experience issues with per viewport RE, initilizing and destroying REs leads to inconsistent state and is not reliable, cornerstone3D architecture heavly relies on a single RE, many tools needed to be moved externally outside of cornerstone, such as ReferenceLines, Crosshairs, synchronizers, and core OHIF services needed to be modified heavily. Volumes need to share the same RE. Can’t have more than 16 REs, so assign 3 stacks to each RE.
• July 4th: Try canvas packing here
• July 7th: Not satisfied with canvas packing results, performance degradation still occurs, canvas cropping still occurs, hitting different canvas size limits on different OS/browsers
• July 9th: Implement sequential rendering engine, all issues resolved

Usage

The new Sequential Rendering Engine is now the default in Cornerstone3D. If you wish to continue using the old rendering engine, you can override this behavior by explicitly setting the renderingEngineMode to 'standard'.

To use the old rendering engine:

import { init } from '@cornerstonejs/core';

init({
  rendering: {
    renderingEngineMode: 'standard',
  },
});

Demos and Previews

OHIF

• OHIF before: https://coruscating-youtiao-10ee6d.netlify.app/viewer?StudyInstanceUIDs=1.3.12.2.1107.5.1.4.60175.30000008042114404745300000010&hangingProtocolId=@ohif/mnGrid

• Video: https://streamable.com/41h2xy, You can see me experience a context loss in the video, when I try to go back from one up to the grid, due to how huge the offscreen is.

• OHIF after: https://gorgeous-sable-7bbec5.netlify.app/viewer?StudyInstanceUIDs=1.3.12.2.1107.5.1.4.60175.30000008042114404745300000010&hangingProtocolId=@ohif/mnGrid

• Video: https://streamable.com/lyofjq
• Video with FPS while making each viewport 4120 x 1100 for fun, still very performant: https://streamable.com/s15zyj

Cornerstone

• CS before: https://ephemeral-raindrop-5f4044.netlify.app/live-examples/renderingengine.html

• CS After: https://ephemeral-raindrop-5f4044.netlify.app/live-examples/renderingenginesequential.html

Reference of previous attempts and useful resources

PRs

• Draft - feat(renderingEngine): multi-offscreen canvases #2159
• Wip - fix(rendering): allow users to define x num of offscreen canvas to support viewports that exceed 16384 in width #1989
• feat(canvas): canvas packing #2174
• wip-feat(renderingEngine): per viewport rendering engine flag OHIF/Viewers#4976
• Draft - feat(renderingEngine): multi rendering engine OHIF/Viewers#5180
• wip (test): per viewport OHIF/Viewers#5187

Packing

• https://www.david-colson.com/2020/03/10/exploring-rect-packing.html
• https://github.com/mapbox/potpack
• https://mapbox.github.io/potpack/

Canvas size limits per browser

• https://jhildenbiddle.github.io/canvas-size/#/?id=test-results

WebGL context limits

This is different per OS and browser

• Chrome source code (may be outdated, but its still 16): https://chromium.googlesource.com/chromium/src/+/2efb8d651d7baf2313253a01df19cbfeabd690b6/third_party/WebKit/Source/modules/webgl/WebGLRenderingContextBase.cpp#106
• Discussions: https://bugzilla.mozilla.org/show_bug.cgi?id=1421481

Thanks

Thanks to @wayfarer3130 @sedghi for their help and insights that helped me brainstorm a solution. I wouldn't have been able to resolve this without them.

[sedghi]

Great job! I'll review it, but I'm going to request changes to prevent accidental merging. This should really be a separate mode in RE, set during cscore.init.

[wayfarer3130]

Can we get the adapter/tool changes reverted, unless specifically related?

[sedghi]

I went through all 158 examples, and they're working well. It looks like only the orientation marker is broken, which I assume you're already addressing. This is a really great pull request—one of the best we've had in recent years!

[salimkanoun]

Wow that's seems really amazing, congratulations !

[wayfarer3130]

I made this one configurable with rows and columns, going to 12x22. There is a significant pause about 1/3 of the way through - not that I'm concerned, just noticing it.

[wayfarer3130]

I added a rows and columns parameter, and then tested with rows=16 and columns=32
There is a really long lag about 1/3 of the way through, then the remaining windows appear immediately after the lag. I'd suggest squashing this (I can do this), but would be nice to see if there is anything immediately obvious.

[sedghi]

As discussed in slack, we can improve this later. I'm gonna merge