How to access video frames for WebRTC integration?

[noho501]

Hi,

I’m using RootEncoder and I’d like to forward video frames to WebRTC.
Is there any existing API or recommended approach to do this?

Thanks.

[pedroSG94]

If you want get video and video from the library and send it to a webrtc library you can do a class like RtmpStream replacing RtmpClient with your webrtc library
https://github.com/pedroSG94/RootEncoder/blob/master/library/src/main/java/com/pedro/library/rtmp/RtmpStream.kt
You have the buffer inside getVideoDataImp and getAudioDataImp methods

[noho501]

Thank you. I’ll do it

[noho501]

Hi @pedroSG94
I tried it, and the Surface attaches successfully. I can confirm that swapBuffer() is being called continuously. However, the onFrame() callback in my WebRTC capturer is never triggered. WebRTC only outputs a black screen, as if the frames are being silently dropped.

Could you please help me understand what I might be doing wrong?

class ExternalSurfaceCapturer(
    private val onSurfaceReady: (Surface) -> Unit,
    private val onSurfaceDestroyed: () -> Unit
) : VideoCapturer, VideoSink {

    private var capturerObserver: CapturerObserver? = null
    private var surfaceTextureHelper: SurfaceTextureHelper? = null
    private var surface: Surface? = null
    private var frameCount = 0
    private var isCapturing = false

    override fun initialize(
        surfaceTextureHelper: SurfaceTextureHelper,
        context: Context,
        capturerObserver: CapturerObserver
    ) {
        this.surfaceTextureHelper = surfaceTextureHelper
        this.capturerObserver = capturerObserver
    }

    override fun startCapture(width: Int, height: Int, framerate: Int) {
        if (isCapturing) return
        isCapturing = true

        capturerObserver?.onCapturerStarted(true)
        surfaceTextureHelper?.startListening(this)
        val surfaceTexture = surfaceTextureHelper?.surfaceTexture
        if (surfaceTexture != null) {
            surfaceTexture.setDefaultBufferSize(width, height)
            surface = Surface(surfaceTexture)
            onSurfaceReady(surface!!)
        }
    }

    override fun stopCapture() {
        if (!isCapturing) return
        isCapturing = false
        capturerObserver?.onCapturerStopped()
        surfaceTextureHelper?.stopListening()
        onSurfaceDestroyed()
        surface?.release()
        surface = null
    }

    override fun changeCaptureFormat(width: Int, height: Int, framerate: Int) {
        surfaceTextureHelper?.surfaceTexture?.setDefaultBufferSize(width, height)
    }

    override fun dispose() { stopCapture() }
    override fun isScreencast() = false

    override fun onFrame(frame: VideoFrame) {
        frameCount++
        frame.buffer.retain()
        val currentRealTimestampNs = System.nanoTime()
        val validFrame = VideoFrame(frame.buffer, frame.rotation, currentRealTimestampNs)
        capturerObserver?.onFrameCaptured(validFrame)
        validFrame.release()
    }
}

WebRtcSessionManager & LiveViewModel

private val videoCapturer: VideoCapturer by lazy {
    ExternalSurfaceCapturer(
        onSurfaceReady = { surface -> _webrtcSurfaceFlow.value = surface },
        onSurfaceDestroyed = { _webrtcSurfaceFlow.value = null }
    )
}

private val videoSource by lazy {
    peerConnectionFactory.makeVideoSource(videoCapturer.isScreencast).apply {
        videoCapturer.initialize(surfaceTextureHelper, context, this.capturerObserver)
    }
}

override fun startLocalVideoCapture(width: Int, height: Int, fps: Int) {
    videoCapturer.startCapture(width, height, fps)
}

LiveActivity

lifecycleScope.launch {
    repeatOnLifecycle(Lifecycle.State.STARTED) {
        viewModel.webrtcSurfaceFlow.collect { surface ->
            try {
                currentWebRtcSurface = surface
                if (surface != null) {
                    if (genericStream.isOnPreview && isSurfaceActive) {
                        val config = MultiPreviewConfig(width, height)
                        genericStream.getGlInterface().addMultiPreviewSurface(surface, config)
                    }
                } else {
                    genericStream.getGlInterface().removeAllMultiPreviewSurfaces()
                }
            } catch (e: Exception) {
                Log.e("WebRTC_Bridge", "Error add WebRTC Surface: ${e.message}")
            }
        }
    }
}

Modified GlStreamInterface.kt

// Thread safety fix for adding surface
fun addMultiPreviewSurface(surface: Surface, config: MultiPreviewConfig) {
    executor?.submit {
        if (surfaceManager.isReady) {
            multiPreviewSurfaceManagers.remove(surface)?.surfaceManager?.release()
            val newSurfaceManager = SurfaceManager()
            newSurfaceManager.eglSetup(surface, this@GlStreamInterface.surfaceManager)
            val finalConfig = MultiPreviewConfig(w, h, config.horizontalFlip, config.verticalFlip, config.aspectRatioMode, config.isPortrait, config.viewPort)
            multiPreviewSurfaceManagers[surface] = PreviewSurfaceInfo(newSurfaceManager, finalConfig)
        }
    }
}

// Timestamp fix in the draw() method

private fun draw(forced: Boolean) {
    if (multiPreviewSurfaceManagers.isNotEmpty()) {
        val previewSnapshot = multiPreviewSurfaceManagers.values.toList()
        previewSnapshot.forEach { info ->
            if (info.surfaceManager.isReady && info.surfaceManager.makeCurrent()) {
                mainRender.drawScreenPreview(/*...*/)
                
                // We tried forcing the timestamp here
                info.surfaceManager.setPresentationTime(mainRender.getSurfaceTexture().timestamp)
                
                info.surfaceManager.swapBuffer()
            }
        }
    }
}

[pedroSG94]

It is really difficult yo say only with the code.
For now, I recommend you use startPreview only to make sure that the problem is not related with addMultiPreviewSurface and discard it.
Also, make sure that startPreview is working as expected using a SurfaceView to check it at the same time that you use startPreview with the webrtc surface.
After that check that the swap is called inside GlStreamInterface to make sure that you are rendering the surface.

I don't know what is supposed to the onFrame in your webrtc library but if it is similar to onFrameAvailable of a SurfaceTexture and you never receive the callback. Try add this callback (onFrameAvailable) to your webrtc SurfaceTexture manually to discard problems inside the library and know if the surfacetexture is correctly renderer.