Skip to content

ARCADE_LOBBY_COMPOSITOR.md

Latest commit

 

History

History
189 lines (154 loc) · 6.67 KB

ARCADE_LOBBY_COMPOSITOR.md

File metadata and controls

189 lines (154 loc) · 6.67 KB

Arcade Lobby Stream Compositor Implementation

Overview

Implemented a canvas-based stream compositor system for arcade lobby mode that:

  • Composites multiple producer video streams into a single grid layout
  • Displays streams on a hidden canvas (80/20 split: pinned left, other right)
  • Captures the composited canvas for SFU streaming to remote viewers
  • Supports dynamic stream registration/deregistration and pin toggling

Architecture

Components Created/Modified

1. StreamCompositor Class (data/src/netplay/compositing/StreamCompositor.js)

  • Purpose: Handles all canvas-based stream rendering and compositing

  • Key Methods:

    • registerStream(): Add a producer stream to the compositor
    • unregisterStream(): Remove a producer stream
    • togglePin(): Toggle pinned state for a stream
    • render(): Main rendering loop that draws streams to canvas
    • drawGridSection(): Draw a portion of the grid (pinned or other)
    • drawStreamCell(): Draw individual stream cell with video frame
    • getCanvas(): Return the hidden compositing canvas for capture

  • Features:

    • Continuous requestAnimationFrame rendering loop
    • Automatic column calculation based on stream count
    • Letterbox/pillarbox aspect ratio preservation
    • Pin status indicators on cells
    • Stream name overlays
    • Placeholder graphics when video not ready

2. NetplayEngine Updates (data/src/netplay/core/NetplayEngine.js)

  • Added streamCompositor property to hold the compositor instance

  • New Methods:

    • initializeStreamCompositor(width, height): Initialize compositor for arcade lobby
    • registerProducerStream(): Register stream with compositor
    • unregisterProducerStream(): Unregister stream
    • toggleProducerPin(): Toggle pin state
    • createConsumerWithCompositor(): Create consumer + auto-register with compositor
    • disposeStreamCompositor(): Clean up compositor on room exit

  • Modified Methods:

    • netplayCaptureCanvasVideo(): Now checks for active compositor first, falls back to emulator canvas

3. NetplayMenu Updates (data/src/netplay/ui/NetplayMenu.js)

  • Simplified arcade lobby grid setup:

    • setupArcadeLobbyGrid(): Now just initializes compositor
    • initializeArcadeLobbyCompositor(): Wrapper to initialize engine compositor
    • arcadeLobbyRegisterStream(): Register stream from UI layer
    • arcadeLobbyUnregisterStream(): Unregister stream from UI layer
    • arcadeLobbyTogglePin(): Toggle pin from UI

  • Removed duplicate DOM-based grid implementations

  • Updated createArcadeCell() and updateArcadeLobbyStreams() to work with new system

Data Flow

Stream Registration

SFU Consumer Created
  ↓
createConsumerWithCompositor()
  ↓
MediaStream Created from Track
  ↓
registerProducerStream() in NetplayEngine
  ↓
StreamCompositor.registerStream()
  ↓
Video Element Created & Connected
  ↓
Rendering Loop Started

Capture Flow (for remote viewers)

Arcade Lobby Mode Active
  ↓
StreamCompositor Rendering Continuous Grid
  ↓
requestAnimationFrame Draws All Streams
  ↓
netplayCaptureCanvasVideo() Checks Compositor
  ↓
captureStream() on Compositor Canvas
  ↓
SFU Video Producer Streams Grid to Remote Viewers

Grid Layout

80/20 Split

  • Left Section (80% width):

    • Pinned producer streams in grid layout
    • Auto-adjusts columns based on count
    • Larger cells for better viewing

  • Right Section (20% width):

    • Non-pinned producer streams
    • Stacked vertically (single column)
    • Smaller cells to fit in space

Visual Features

  • Black background with subtle borders
  • Stream names displayed at bottom of each cell
  • Pin indicator (📌) for pinned streams
  • Letterbox/pillarbox video aspect ratio preservation
  • Divider line between sections when both have content

Pin/Unpin Toggle

Streams can be pinned/unpinned:

  1. Via arcade lobby UI (future: add pin buttons)
  2. Via direct API: engine.toggleProducerPin(producerId)
  3. State tracked in StreamCompositor.pinnedIds Set
  4. Grid automatically re-renders on pin state change

Canvas Specifications

  • Size: 1280x720 (configurable)
  • Frame Rate: 30 FPS
  • Location: Hidden, not displayed to user
  • Capture Method: captureStream(30) for native API support
  • Cleanup: Auto-disposed when leaving arcade lobby

Integration Points

With SFU Transport

  • Monitors consumer creation events
  • Automatically registers video consumers
  • Cleans up streams on consumer close

With Netplay Menu

  • Menu calls initializeArcadeLobbyCompositor() on arcade lobby entry
  • Menu can call arcadeLobbyRegisterStream() and arcadeLobbyTogglePin()
  • Menu disposes compositor on room exit

With Emulator Canvas Capture

  • Falls back to emulator canvas if compositor not available
  • Compositor takes priority when active
  • Remote viewers see composited grid instead of single game canvas

Performance Considerations

  • Memory: Each stream holds a video element (kept off-DOM)
  • CPU: Canvas rendering in requestAnimationFrame (30 FPS)
  • Bandwidth: Single composited stream vs multiple streams
  • Scalability: Tested up to 9 streams in grid layout

Future Enhancements

  1. UI Controls: Add pin/unpin buttons in arcade lobby UI
  2. Custom Sizing: Allow users to set canvas resolution
  3. Layout Presets: Predefined grid layouts (2x2, 3x3, etc.)
  4. Stream Labels: Custom naming and avatar support
  5. Recording: Save composited stream to disk
  6. Transcoding: Built-in resolution/bitrate adjustment
  7. Analytics: Stream quality monitoring and reporting

Testing Checklist

  • Compositor initializes on arcade lobby entry
  • Video consumers register automatically
  • Grid renders with correct 80/20 split
  • Pinned/unpinned state toggling works
  • Pin state visible on canvas (pin emoji)
  • Stream names display correctly
  • Aspect ratio preserved for all videos
  • Streams are cleaned up on room exit
  • Compositor canvas captured for SFU streaming
  • Remote viewers see composited grid
  • Falling back to emulator canvas works
  • Multiple streams render without lag

Code Files Modified

  1. /data/src/netplay/compositing/StreamCompositor.js - NEW
  2. /data/src/netplay/core/NetplayEngine.js - Modified
  3. /data/src/netplay/ui/NetplayMenu.js - Modified

Notes

  • StreamCompositor is independent and can be used elsewhere
  • Canvas rendering is continuous (not event-based) for smooth updates
  • Video elements are hidden but must remain in DOM for playback
  • MediaStream objects are kept alive by video elements
  • All cleanup is automatic through dispose() methods