FCE-3493: Add ability to use custom camera / custom source as a video track#61
Open
MiloszFilimowski wants to merge 12 commits into
Open
FCE-3493: Add ability to use custom camera / custom source as a video track#61MiloszFilimowski wants to merge 12 commits into
MiloszFilimowski wants to merge 12 commits into
Conversation
Adds the public TypeScript surface for a custom video track primitive that lets
apps feed their own GPU/CPU-rendered frames into a WebRTC video track:
- createCustomVideoTrack({width,height,poolSize}) and pushCustomVideoFrame(frame)
signatures with thorough JSDoc; stub bodies (filled in a follow-up).
- Types: CustomVideoTrackInit, CustomVideoBuffer (surfaceHandle as bigint),
CustomVideoTrack, CustomVideoFrameFence ({handle, signaledValue} bigints,
documented as platform GPU-sync primitives — MTLSharedEvent / sync-fd — not
tied to any GPU library), CustomVideoFramePush.
- Barrel exports in src/index.ts (mirrors AudioExtraction).
- common/cpp/fishjam-video/FJVideoPushJSI.h: JSI core header (install + a
registered platform deliver callback) modelled on FJAudioSinkJSI.h.
- docs/custom-video-track.md: concepts + per-frame flow + abstraction-level guide.
New-Architecture only (per-frame push is a JSI binding). Handles are bigint
end-to-end in the public API; the bridge's string transport stays internal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fills the createCustomVideoTrack/pushCustomVideoFrame bodies and adds the JSI
install lifecycle, mirroring the in-tree AudioExtraction.ts:
- ensureInstalled(): memoized install of the JSI binding via
WebRTCModule.installCustomVideoJSI(), 10s timeout, E_NO_JSI/timeout mapped to a
clear 'Custom video tracks require the New Architecture.' error.
- createCustomVideoTrack(init): ensure install, call the native bridge method,
wrap {streamId, track} into a MediaStream, and convert each bridge string
surfaceHandle to a bigint (public type) — the string stays internal.
- pushCustomVideoFrame(frame): guard + call the installed JSI global
global.__fishjamWebrtcPushCustomVideoFrame(frame).
- Named internal bridge types (BridgeCustomVideoTrack / BridgeCustomVideoBuffer)
describe the wire shape, distinct from the public bigint/MediaStream types.
References native methods landing in the platform phases; tsc + eslint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements common/cpp/fishjam-video/FJVideoPushJSI.cpp (modelled on
FJAudioSinkJSI.cpp): install() hops to the JS thread via the CallInvoker and sets
a global __fishjamWebrtcPushCustomVideoFrame(frame) HostFunction. The handler
runs synchronously (the JS thread is the caller — no thread-hop), unmarshals the
frame object (trackId/bufferIndex/timestampNs/rotation as values, the optional
fence {handle, signaledValue} as bigint->uint64), and forwards to a
platform-registered deliver callback. Absent fence => 0/0 (immediate delivery).
weak_ptr capture + expired guard; idempotent reinstall on JS reload. Aligns the
header comment to the canonical global name.
Compiles when first built into a platform (Phase 4 iOS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
iOS native side of the custom video track, verified streaming on a physical
iPhone (color-cycle -> remote peer):
- CustomVideoCaptureController.{h,m}: owns a fixed pool of IOSurface-backed,
Metal-compatible BGRA CVPixelBuffers; exposes each surface handle to JS via
bufferDescriptors; per frame waits on the MTLSharedEvent fence (handle/value as
uint64, fence==0 => immediate) then wraps the buffer in an RTCVideoFrame and
delivers to the RTCVideoSource. Drain bookkeeping for clean teardown.
- WebRTCModule+CustomVideo.mm: installCustomVideoJSI builds FJVideoPush from
self.callInvoker (associated-object box; E_NO_JSI on old arch) and setDeliver
routes (trackId,bufferIndex,timestampNs,rotation,fence) to the controller via a
weak-self trackId->controller lookup. Mirrors WebRTCModule+AudioSink.mm.
- WebRTCModule+RTCMediaStream.m: createCustomVideoTrack takes a {width,height,
poolSize} dictionary and resolves {streamId, track, buffers}.
- podspec: add common/cpp/fishjam-video to HEADER_SEARCH_PATHS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Android native side of the custom video track, verified streaming on a physical
device (color-cycle -> remote peer):
- Delivery core: AHardwareBufferPool.java + ahardware_buffer_pool.cpp allocate
renderable+sampleable AHBs; custom_video_gl.cpp imports each AHB once as a
GL_TEXTURE_EXTERNAL_OES (eglGetNativeClientBufferANDROID -> eglCreateImageKHR ->
glEGLImageTargetTexture2DOES) and waits the GPU sync-fd fence (eglWaitSyncKHR);
CustomVideoFrameDelivery.java drives it on one SurfaceTextureHelper GL thread
share-linked to WebRTC's root EGL context, then ships a TextureBufferImpl(OES)
VideoFrame via onFrameCaptured -- the same seam the Camera2 capturer uses.
- JSI install: FJVideoPushInstaller.{h,cpp,java} (fbjni HybridClass mirroring
FJAudioSinkInstaller) builds FJVideoPush from the CallInvoker and routes the
deliver callback (fence handle as a long sync-fd) into CustomVideoFrameDelivery.
- WebRTCModule.java/GetUserMediaImpl.java: createCustomVideoTrack ({w,h,poolSize}
dict) + installCustomVideoJSI gated by CallInvokerHolderImpl (E_NO_JSI on old
arch); SDK_INT < O reject before touching the AHB lib.
- minSdk kept at safeExtGet(...,24) with __builtin_available(26) guards in the AHB
cpp, so the package stays minSdk-24 and only the custom-track path needs 26.
- CMakeLists builds the new webrtc-custom-video-track lib (AHB/GL + shared
FJVideoPushJSI.cpp + the installer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- custom-video-track.md: add a Platform notes section (surface pixel format BGRA8 iOS / RGBA8 Android, New-Architecture requirement, Android API-26 floor, fence handles as platform GPU primitives) and link the worked example. - example.md: a complete, annotated color-cycle example end to end -- create the track, import the surface pool into WebGPU, run a fenced 30fps render loop, and publish via useCustomSource -- plus a 'things that bite' section (match the surface format, keep the fence alive, round-robin, same-runtime worklet import, device-only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses an independent GPU/media code review. No public API change.
Critical:
- Android: the GPU fence was waited server-side on the delivery context, which
does not gate WebRTC's hardware encoder (it samples on its own EGL context) ->
tearing under load. Switched to a client (CPU) wait on the delivery thread
(eglClientWaitSyncKHR), so the render is provably complete before any encoder
context samples it. Pixels never touch the CPU -- zero-copy preserved.
- Android: teardown freed the OES texture/EGLImage/AHB before the encoder was
quiesced (no-op TextureBufferImpl release callback) -> UAF. Reordered the
custom-video dispose: stop accepting + drain -> dispose VideoSource/VideoTrack
-> free GL imports + AHB pool.
iOS:
- Unified _accepting + _inFlightCount under one lock (was a TOCTOU race -> UAF).
- Bounded the stopCapture drain (2s) + _tornDown latch so a never-signaled fence
can't deadlock teardown; completions after teardown skip delivery.
- CFRetain/CFRelease the borrowed MTLSharedEvent for the armed-listener lifetime.
- Deliver the no-fence path off the JS thread; deliver outside the lock (retain
the CVPixelBuffer under the lock) so a WebRTC back-pressure stall can't block JS.
- Replaced the per-frame cross-thread localTracks read with a lock-guarded
weak-value trackId->controller registry.
Shared / Android level:
- JSI push validates every field and drops a malformed frame instead of throwing
on the hot path; clamps rotation to {0,90,180,270}.
- Android installCustomVideoJSI gated on API 26 before the native AHB lib loads
(keeps the package minSdk 24 safe); re-installs cleanly on JS reload.
- Comment/threading-affinity notes; the webgpu Android sync-fd is dup'd by the
exporter, so EGL owning+closing it is correct.
Builds green on both platforms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new “custom video track” capability to the React Native WebRTC package, allowing apps to render frames themselves (GPU/CPU) into a native surface pool and push those frames into a WebRTC video track via a New Architecture JSI fast-path.
Changes:
- Exposes new JS APIs (
createCustomVideoTrack,pushCustomVideoFrame) and types from the package entrypoint. - Implements native support on iOS (IOSurface-backed CVPixelBuffer pool + Metal fence handling + JSI binding install).
- Implements native support on Android (AHardwareBuffer pool + EGLImage/OES import + sync-fd fence wait + JSI installer) and adds end-to-end documentation/examples.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/index.ts | Re-exports the new custom video track APIs/types from the public entrypoint. |
| src/createCustomVideoTrack.ts | New JS wrapper: installs JSI binding, validates init, creates stream + surface pool descriptors, pushes frames via global. |
| ios/RCTWebRTC/WebRTCModule+RTCMediaStream.m | Adds createCustomVideoTrack bridge method and custom-controller teardown in track disposal. |
| ios/RCTWebRTC/WebRTCModule+CustomVideo.mm | Adds iOS JSI install method and thread-safe registry for per-track custom capture controllers. |
| ios/RCTWebRTC/WebRTCModule.m | Ensures custom capture controllers are stopped and resources released on module dealloc. |
| ios/RCTWebRTC/CustomVideoCaptureController.m | New iOS capture controller implementing IOSurface pool allocation, fence-waited delivery, and teardown. |
| ios/RCTWebRTC/CustomVideoCaptureController.h | Declares the iOS custom capture controller + WebRTCModule registry category API. |
| FishjamReactNativeWebrtc.podspec | Adds header search path for the new shared fishjam-video C++ code. |
| docs/example.md | Adds a full end-to-end example using react-native-webgpu to render + fence + publish a custom track. |
| docs/custom-video-track.md | Adds conceptual documentation for the custom video track flow and platform specifics. |
| common/cpp/fishjam-video/FJVideoPushJSI.h | Introduces shared C++ JSI core interface for installing/pushing custom video frames. |
| common/cpp/fishjam-video/FJVideoPushJSI.cpp | Implements JSI global install and per-frame parsing/dispatch to native deliver callback. |
| android/src/main/java/com/oney/WebRTCModule/WebRTCModule.java | Adds bridge methods and JSI installer wiring for custom video track push. |
| android/src/main/java/com/oney/WebRTCModule/GetUserMediaImpl.java | Implements Android createCustomVideoTrack + per-frame routing and custom teardown ordering. |
| android/src/main/java/com/oney/WebRTCModule/FJVideoPushInstaller.java | New Java wrapper for the hybrid JSI installer + per-frame routing back to Java. |
| android/src/main/java/com/oney/WebRTCModule/CustomVideoFrameDelivery.java | New delivery engine: imports AHB→EGLImage→OES texture, waits fence, delivers VideoFrame on GL thread. |
| android/src/main/java/com/oney/WebRTCModule/CustomVideoCaptureController.java | New capturer-less controller owning the AHB pool and coordinating frame delivery + teardown phases. |
| android/src/main/java/com/oney/WebRTCModule/AHardwareBufferPool.java | Java native front for allocating/releasing AHBs (API 26+). |
| android/src/main/cpp/FJVideoPushInstaller.h | New fbjni HybridClass header backing the Java push installer. |
| android/src/main/cpp/FJVideoPushInstaller.cpp | Implements hybrid install + per-frame forwarding from shared C++ JSI core to Java. |
| android/src/main/cpp/custom_video_gl.cpp | Native GL/EGL import and sync-fd fence waiting helpers for AHB-backed delivery. |
| android/src/main/cpp/CMakeLists.txt | Adds a new shared library target (webrtc-custom-video-track) and links required deps. |
| android/src/main/cpp/ahardware_buffer_pool.cpp | Native AHB allocation/release implementation with API-26 guards. |
| android/build.gradle | Uses safeExtGet for minSdk and documents API-26 load-safety assumptions for the new native lib. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+125
to
+151
| NSMutableArray *buffers = [NSMutableArray arrayWithCapacity:poolSize]; | ||
| for (NSInteger index = 0; index < poolSize; index++) { | ||
| CVPixelBufferRef buffer = NULL; | ||
| CVReturn bufferStatus = | ||
| CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, _pixelBufferPool, &buffer); | ||
| if (bufferStatus != kCVReturnSuccess || buffer == NULL) { | ||
| if (outError) { | ||
| *outError = [NSError | ||
| errorWithDomain:@"react-native-webrtc" | ||
| code:bufferStatus | ||
| userInfo:@{ | ||
| NSLocalizedDescriptionKey : [NSString | ||
| stringWithFormat:@"CVPixelBufferPoolCreatePixelBuffer failed at index %ld: %d", | ||
| (long)index, | ||
| bufferStatus] | ||
| }]; | ||
| } | ||
| // Release whatever we already allocated before bailing out. | ||
| for (NSValue *boxed in buffers) { | ||
| CVPixelBufferRelease((CVPixelBufferRef)boxed.pointerValue); | ||
| } | ||
| return NO; | ||
| } | ||
| // The NSArray keeps the +1 from CVPixelBufferPoolCreatePixelBuffer; we | ||
| // balance it in dealloc / stopCapture. | ||
| [buffers addObject:[NSValue valueWithPointer:buffer]]; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.