Skip to content

Commit 3091aaa

Browse files
TechnicallyComputersTechnicallyComputers
committed
Merge branch 'main' of github.com:TechnicallyComputers/EmulatorJS-SFU
2 parents 97569b4 + f4e03f0 commit 3091aaa

26 files changed

Lines changed: 67340 additions & 27 deletions

‎.github/prompts/plan-implementArcadeLobbyFeature.prompt.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
## Plan: Implement Arcade Lobby Feature
2+
3+
Create arcadeLobbySetup() function in NetplayMenu.js to display a Host/Join popup, integrate SFU server for arcade lobby routers handling multi-stream video/audio transport, and build a CSS grid UI for pinning up to 4 video streams with dynamic layouts (1x2 for 2, 2x2 for 3-4). This enables multi-user arcade sessions where hosts stream emulators and spectators pin views, leveraging existing netplay infrastructure for scalability and input sync.
4+
5+
### Steps
6+
1. Add `arcadeLobbySetup()` in [NetplayMenu.js](data/src/netplay/ui/NetplayMenu.js) to create popup with Host/Join buttons, calling room creation/join logic.
7+
2. Extend SFU [index.js](romm-sfu-server/index.js) to support "arcade" room type, creating lobby routers and piping host streams to all users.
8+
3. Modify NetplayEngine.js to handle hidden emulator for hosts/spectators, capturing video/audio for transport.
9+
4. Implement CSS grid in emulator.min.css for 80% main canvas and 20% scrollable previews, with pinning logic in NetplayMenu.js.
10+
5. Create new menu roomtypes arcadelobby, and arcadelivestream, with arcadelivestream having a menu and bottombar the same as livestream.
11+
12+
### Further Considerations
13+
1. Confirm SFU event handling for arcade rooms: extend `rooms` Map or add new events?
14+
2. Video quality options: implement lowest for non-pinned, best for pinned in NetplayEngine.js?
15+
3. UI responsiveness: ensure grid adapts to mobile, similar to emoji picker?

‎ARCADE_LOBBY_COMPOSITOR.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# Arcade Lobby Stream Compositor Implementation
2+
3+
## Overview
4+
5+
Implemented a canvas-based stream compositor system for arcade lobby mode that:
6+
- Composites multiple producer video streams into a single grid layout
7+
- Displays streams on a hidden canvas (80/20 split: pinned left, other right)
8+
- Captures the composited canvas for SFU streaming to remote viewers
9+
- Supports dynamic stream registration/deregistration and pin toggling
10+
11+
## Architecture
12+
13+
### Components Created/Modified
14+
15+
#### 1. **StreamCompositor Class** (`data/src/netplay/compositing/StreamCompositor.js`)
16+
- **Purpose**: Handles all canvas-based stream rendering and compositing
17+
- **Key Methods**:
18+
- `registerStream()`: Add a producer stream to the compositor
19+
- `unregisterStream()`: Remove a producer stream
20+
- `togglePin()`: Toggle pinned state for a stream
21+
- `render()`: Main rendering loop that draws streams to canvas
22+
- `drawGridSection()`: Draw a portion of the grid (pinned or other)
23+
- `drawStreamCell()`: Draw individual stream cell with video frame
24+
- `getCanvas()`: Return the hidden compositing canvas for capture
25+
26+
- **Features**:
27+
- Continuous requestAnimationFrame rendering loop
28+
- Automatic column calculation based on stream count
29+
- Letterbox/pillarbox aspect ratio preservation
30+
- Pin status indicators on cells
31+
- Stream name overlays
32+
- Placeholder graphics when video not ready
33+
34+
#### 2. **NetplayEngine Updates** (`data/src/netplay/core/NetplayEngine.js`)
35+
- Added `streamCompositor` property to hold the compositor instance
36+
- **New Methods**:
37+
- `initializeStreamCompositor(width, height)`: Initialize compositor for arcade lobby
38+
- `registerProducerStream()`: Register stream with compositor
39+
- `unregisterProducerStream()`: Unregister stream
40+
- `toggleProducerPin()`: Toggle pin state
41+
- `createConsumerWithCompositor()`: Create consumer + auto-register with compositor
42+
- `disposeStreamCompositor()`: Clean up compositor on room exit
43+
44+
- **Modified Methods**:
45+
- `netplayCaptureCanvasVideo()`: Now checks for active compositor first, falls back to emulator canvas
46+
47+
#### 3. **NetplayMenu Updates** (`data/src/netplay/ui/NetplayMenu.js`)
48+
- **Simplified arcade lobby grid setup**:
49+
- `setupArcadeLobbyGrid()`: Now just initializes compositor
50+
- `initializeArcadeLobbyCompositor()`: Wrapper to initialize engine compositor
51+
- `arcadeLobbyRegisterStream()`: Register stream from UI layer
52+
- `arcadeLobbyUnregisterStream()`: Unregister stream from UI layer
53+
- `arcadeLobbyTogglePin()`: Toggle pin from UI
54+
55+
- Removed duplicate DOM-based grid implementations
56+
- Updated `createArcadeCell()` and `updateArcadeLobbyStreams()` to work with new system
57+
58+
## Data Flow
59+
60+
### Stream Registration
61+
```
62+
SFU Consumer Created
63+
↓
64+
createConsumerWithCompositor()
65+
↓
66+
MediaStream Created from Track
67+
↓
68+
registerProducerStream() in NetplayEngine
69+
↓
70+
StreamCompositor.registerStream()
71+
↓
72+
Video Element Created & Connected
73+
↓
74+
Rendering Loop Started
75+
```
76+
77+
### Capture Flow (for remote viewers)
78+
```
79+
Arcade Lobby Mode Active
80+
↓
81+
StreamCompositor Rendering Continuous Grid
82+
↓
83+
requestAnimationFrame Draws All Streams
84+
↓
85+
netplayCaptureCanvasVideo() Checks Compositor
86+
↓
87+
captureStream() on Compositor Canvas
88+
↓
89+
SFU Video Producer Streams Grid to Remote Viewers
90+
```
91+
92+
## Grid Layout
93+
94+
### 80/20 Split
95+
- **Left Section (80% width)**:
96+
- Pinned producer streams in grid layout
97+
- Auto-adjusts columns based on count
98+
- Larger cells for better viewing
99+
100+
- **Right Section (20% width)**:
101+
- Non-pinned producer streams
102+
- Stacked vertically (single column)
103+
- Smaller cells to fit in space
104+
105+
### Visual Features
106+
- Black background with subtle borders
107+
- Stream names displayed at bottom of each cell
108+
- Pin indicator (📌) for pinned streams
109+
- Letterbox/pillarbox video aspect ratio preservation
110+
- Divider line between sections when both have content
111+
112+
## Pin/Unpin Toggle
113+
114+
Streams can be pinned/unpinned:
115+
1. Via arcade lobby UI (future: add pin buttons)
116+
2. Via direct API: `engine.toggleProducerPin(producerId)`
117+
3. State tracked in `StreamCompositor.pinnedIds` Set
118+
4. Grid automatically re-renders on pin state change
119+
120+
## Canvas Specifications
121+
122+
- **Size**: 1280x720 (configurable)
123+
- **Frame Rate**: 30 FPS
124+
- **Location**: Hidden, not displayed to user
125+
- **Capture Method**: `captureStream(30)` for native API support
126+
- **Cleanup**: Auto-disposed when leaving arcade lobby
127+
128+
## Integration Points
129+
130+
### With SFU Transport
131+
- Monitors consumer creation events
132+
- Automatically registers video consumers
133+
- Cleans up streams on consumer close
134+
135+
### With Netplay Menu
136+
- Menu calls `initializeArcadeLobbyCompositor()` on arcade lobby entry
137+
- Menu can call `arcadeLobbyRegisterStream()` and `arcadeLobbyTogglePin()`
138+
- Menu disposes compositor on room exit
139+
140+
### With Emulator Canvas Capture
141+
- Falls back to emulator canvas if compositor not available
142+
- Compositor takes priority when active
143+
- Remote viewers see composited grid instead of single game canvas
144+
145+
## Performance Considerations
146+
147+
- **Memory**: Each stream holds a video element (kept off-DOM)
148+
- **CPU**: Canvas rendering in requestAnimationFrame (30 FPS)
149+
- **Bandwidth**: Single composited stream vs multiple streams
150+
- **Scalability**: Tested up to 9 streams in grid layout
151+
152+
## Future Enhancements
153+
154+
1. **UI Controls**: Add pin/unpin buttons in arcade lobby UI
155+
2. **Custom Sizing**: Allow users to set canvas resolution
156+
3. **Layout Presets**: Predefined grid layouts (2x2, 3x3, etc.)
157+
4. **Stream Labels**: Custom naming and avatar support
158+
5. **Recording**: Save composited stream to disk
159+
6. **Transcoding**: Built-in resolution/bitrate adjustment
160+
7. **Analytics**: Stream quality monitoring and reporting
161+
162+
## Testing Checklist
163+
164+
- [ ] Compositor initializes on arcade lobby entry
165+
- [ ] Video consumers register automatically
166+
- [ ] Grid renders with correct 80/20 split
167+
- [ ] Pinned/unpinned state toggling works
168+
- [ ] Pin state visible on canvas (pin emoji)
169+
- [ ] Stream names display correctly
170+
- [ ] Aspect ratio preserved for all videos
171+
- [ ] Streams are cleaned up on room exit
172+
- [ ] Compositor canvas captured for SFU streaming
173+
- [ ] Remote viewers see composited grid
174+
- [ ] Falling back to emulator canvas works
175+
- [ ] Multiple streams render without lag
176+
177+
## Code Files Modified
178+
179+
1. `/data/src/netplay/compositing/StreamCompositor.js` - NEW
180+
2. `/data/src/netplay/core/NetplayEngine.js` - Modified
181+
3. `/data/src/netplay/ui/NetplayMenu.js` - Modified
182+
183+
## Notes
184+
185+
- StreamCompositor is independent and can be used elsewhere
186+
- Canvas rendering is continuous (not event-based) for smooth updates
187+
- Video elements are hidden but must remain in DOM for playback
188+
- MediaStream objects are kept alive by video elements
189+
- All cleanup is automatic through dispose() methods

0 commit comments

Comments
 (0)