Bug description
On BlueOS 1.5.0-beta.36, mavlink-camera-manager appears to lose or delay H.264 frames/keyframes in the onboard video pipeline.
The issue causes visible multi-second glitches/gaps in the video stream. It is visible in the BlueOS stream thumbnails, Cockpit, Foxglove/MCAP playback, and onboard .mcap recordings, which suggests the problem originates before topside transport/display.
The same vehicle, camera setup, and topside setup work correctly after reverting to BlueOS 1.4.3. The issue occurs both during dives and in dry bench tests. Changing stream resolution or frame rate did not resolve it.
In my test setup, I used two H.264 camera streams: the standard BlueROV camera and an ExploreHD USB camera. Observed frame rates were lower than configured:
- ExploreHD USB: configured 30 FPS, observed ~18.5 FPS
- H.264 USB stream: configured 15 FPS, observed ~11.6 FPS
The root cause is unclear. It may be related to CPU load or SD-card I/O, but I have not confirmed that.
Steps to reproduce
- Use a BlueOS system running
1.5.0-beta.36 with an H.264 camera stream.
- Start the camera stream through
mavlink-camera-manager.
- Observe the stream in BlueOS stream thumbnails, in Cockpit on topside computer or recover it from onboard
.mcap recorder file, if vehicle was armed during tests.
- Note visible frame gaps/glitches or delayed recovery after missing keyframes/IDR frames.
- Revert the same hardware/configuration to BlueOS
1.4.3 and repeat the test.
- The stream works correctly on
1.4.3 with the same setup.
Primary pain point(s)
No response
Additional context
Only related issue I found is: #3841
BlueOS PRs show related recent MCM/streaming changes: 1.5.0-beta.36 includes “Update mavlink camera manager”, and 1.5.0-beta.34 updated MCM to t3.23.2, MCAP extractor, recorder, and Zenoh-related logging.
During robot testing, BlueOS 1.5.0-beta.36 showed a much higher 15-minute load average than 1.4.3.
A Raspberry Pi without vehicle hardware stayed around ~0.40 on both versions. On the BlueROV2 with two cameras, 1.4.3 was around ~1.40, while 1.5.0-beta.36 exceeded 4.
Several MCAP-related processes were active, including mcap doctor on multi-GB files. However, stopping mcap doctor and mcap recover did not noticeably improve the camera stream issue.
Prerequisites
Bug description
On BlueOS
1.5.0-beta.36,mavlink-camera-managerappears to lose or delay H.264 frames/keyframes in the onboard video pipeline.The issue causes visible multi-second glitches/gaps in the video stream. It is visible in the BlueOS stream thumbnails, Cockpit, Foxglove/MCAP playback, and onboard
.mcaprecordings, which suggests the problem originates before topside transport/display.The same vehicle, camera setup, and topside setup work correctly after reverting to BlueOS
1.4.3. The issue occurs both during dives and in dry bench tests. Changing stream resolution or frame rate did not resolve it.In my test setup, I used two H.264 camera streams: the standard BlueROV camera and an ExploreHD USB camera. Observed frame rates were lower than configured:
The root cause is unclear. It may be related to CPU load or SD-card I/O, but I have not confirmed that.
Steps to reproduce
1.5.0-beta.36with an H.264 camera stream.mavlink-camera-manager..mcaprecorder file, if vehicle was armed during tests.1.4.3and repeat the test.1.4.3with the same setup.Primary pain point(s)
No response
Additional context
Only related issue I found is: #3841
BlueOS PRs show related recent MCM/streaming changes:
1.5.0-beta.36includes “Update mavlink camera manager”, and1.5.0-beta.34updated MCM to t3.23.2, MCAP extractor, recorder, and Zenoh-related logging.During robot testing, BlueOS
1.5.0-beta.36showed a much higher 15-minute load average than1.4.3.A Raspberry Pi without vehicle hardware stayed around
~0.40on both versions. On the BlueROV2 with two cameras,1.4.3was around~1.40, while1.5.0-beta.36exceeded4.Several MCAP-related processes were active, including
mcap doctoron multi-GB files. However, stoppingmcap doctorandmcap recoverdid not noticeably improve the camera stream issue.Prerequisites