buffer dropped when decode h264 25fps video

[haohaozaici]

[REQUIRED] Searched documentation and issues

According to #2159, I checked the ffmpeg, VP9 and av1 extension related code.

[REQUIRED] Question

I am trying to implement a ffmpeg video renderer extension. When decoding h264 25fps video, I found that a large number of buffers are dropped, but it does not happen when decoding h264 60fps video.

In the decode method, one frame is indeed decoded every 40ms, but I don't know why a lot of buffers are dropped and the video does not look smooth.

I tried to modify the code for dropping the buffer in SimpleDecoderVideoRenderer. Even if the buffer is not dropped, the video(25fps) still does not look smooth. What is causing the buffer to be dropped?

thank in advance!

A full bug report captured from the device

bugreport-gemini-QQ1B.191205.011-2020-01-13-01-02-30.zip

[haohaozaici]

I use VideoDecoderOutputBuffer and VideoDecoderGLSurfaceView to display the decoded yuv data on the surface.

Here is the status of DecoderCounters when decoding

timeUs=16000000,
DecoderCounters{
    decoderInitCount=1,
    decoderReleaseCount=0,
    inputBufferCount=404,
    skippedInputBufferCount=0,
    renderedOutputBufferCount=248,
    skippedOutputBufferCount=5,
    droppedBufferCount=147,
    maxConsecutiveDroppedBufferCount=3,
    droppedToKeyframeCount=0
}

[ojw28]

This question appears to be about a custom extension, rather than about the ExoPlayer library. Hence falls outside the scope of the issue tracker.

The issue is also not reproducible because you haven't provided any information on where we might obtain your custom extension. Marking as needs more info for now, because it's not possible to provide guidance without being able to reproduce the problem.

[haohaozaici]

Thanks for reply ~

I push my code to https://github.com/haohaozaici/ExoPlayer/tree/release-v2/extensions/ffmpeg-haohao. You can run the debug apk directly, or you can download the project and run it. Maybe you need my compiled libffmpeg.so file and video file.

demo-withExtensions-debug.apk
libffmpeg.so(arm64-v8a)
720p_h264.mp4

Briefly explain what I did.

1. create FfmpegVideoRenderer same as Libgav1VideoRenderer
2. create FfmpegVideoDecoder like Gav1Decoder but decode method like FfmpegDecoder, then provide getExtraData() method for h264 type.
3. Modify the ffmpegDecode jni method from the ffmpeg_jni.cc file, copy frame data to VideoDecoderOutputBuffer.
4. create FFmpegRenderersFactory to build custom VideoRenderers.

For a quick try, only VIDEO_OUTPUT_MODE_YUV mode is currently supported.

I spent a lot of time checking the code, but I don't know where the problem is, hope this will be helpful for others like me who want to build custom extensions.

[ojw28]

This is cool! Is it something you'd consider contributing back to the core project?

The problem with your current implementation is that it assumes each input buffer will produce a corresponding output buffer with the same timestamp. The assumption is here, where you initialize the output buffer with the input buffer's timestamp.

I think this assumption is true specifically for AV1 and VP9, but for some other video codecs it's normal for the input buffers to be out-of-order, and you can see this if you log the timestamps of the buffers being queued to the codec. The decoder is responsible for re-ordering them, and provides the output buffers back in the correct order. For every buffer you give to the decoder, you might get zero, one or multiple buffers back, and the output order is not equal to the input order.

To correctly handle these cases, I think you'll need to get the output buffer timestamps back from FFmpeg along with the decoded data, rather than passing them directly from input buffer to output buffer in the Java layer. You should probably also split ffmpegDecode into separate native methods for feeding input and reading output, to account for the possibility of multiple output buffers becoming available when you feed a single input buffer.

[haohaozaici]

Thank you for your reply and patience for viewing my code ~

It's a great honor to be able to contribute back to the core project, but before that, I hope my code runs correctly for a period of time without obvious errors or efficiency problems. I'm not an experienced developer, so it may take a long time. Anyway, I will continue to share my code and what I have done.

Back to the question, I observed that the input buffers are indeed out of order, because I don't understand the detailed behavior of the SimpleDecoder#decoder() method, so I can't find the reason for the error. Next, I will modify my code according to your suggestions, and carefully read the AV1 and VP9 extensions related codes.

The Chinese New Year is coming soon, and my progress may be suspended for half a month.

Once again, thank you very much~

[andrewlewis]

@haohaozaici, please feel free to send a pull request even if the code has remaining issues or needs further testing. We'd be grateful for the contribution and can fix up any issues on our side when we merge the pull request. Thanks!