Fix critical crash bugs in AAC encoder plugin

[paterkleomenis]

Fix critical crash bugs in AAC encoder plugin

Summary

This PR fixes critical bugs that were causing DaVinci Resolve to crash when loading the AAC audio encoder plugin.

Problem

The plugin was crashing DaVinci Resolve on load due to several memory safety issues and unhandled error conditions in the FFmpeg integration code.

Issues Fixed

1. Ring Buffer Overflow Bug (Critical)

• Problem: m_ringBufferFill was incrementing beyond m_frameSize bounds, causing memory corruption
• Location: audio_encoder.cpp - AddPCMToRingBuffer() function
• Fix: Moved the increment inside the bounds check to only increment when data is actually written

Before:

for (size_t i = 0; i < samples; ++i) {
    for (size_t ch = 0; ch < m_channels; ++ch) {
        if (m_ringBufferFill < m_frameSize)
            m_pcmRingBuffer[ch][m_ringBufferFill] = planarPCM[ch][i];
    }
    m_ringBufferFill++;  // ALWAYS incremented, even when buffer was full!
}

After:

for (size_t i = 0; i < samples; ++i) {
    if (m_ringBufferFill < m_frameSize) {
        for (size_t ch = 0; ch < m_channels; ++ch) {
            m_pcmRingBuffer[ch][m_ringBufferFill] = planarPCM[ch][i];
        }
        m_ringBufferFill++;  // Only increment when we actually write data
    }
}

2. Missing NULL Checks for FFmpeg Allocations

• Problem: FFmpeg allocation functions can return NULL on failure, but code didn't check before using pointers
• Location: audio_encoder.cpp - DoInit() and DoProcess() functions
• Fix: Added NULL checks for:
  • avcodec_alloc_context3() - codec context allocation
  • av_frame_alloc() - frame allocation
  • av_packet_alloc() - packet allocation
  • av_channel_layout_copy() - channel layout copy
  • av_frame_get_buffer() - frame buffer allocation

All failures now properly clean up allocated resources and return errFail with error logging.

3. Initialization Validation

• Problem: No validation that encoder was properly initialized before processing audio
• Location: audio_encoder.cpp - DoProcess() function
• Fix: Added validation checks for:
  • FFmpeg context existence
  • Valid channel and frame size (non-zero)
  • Properly initialized ring buffer

4. Memory Safety in Ring Buffer Helpers

• Problem: No bounds checking in ring buffer helper functions
• Location: audio_encoder.cpp - GetFrameFromRingBuffer() and AddPCMToRingBuffer()
• Fix: Added NULL pointer checks and bounds validation

5. Duplicate Initialization Code

• Problem: Lines setting frame_size and pts were duplicated in DoInit()
• Fix: Removed the duplicate lines (lines 200-201)

6. Build System Compatibility

• Problem: Makefile was configured for clang++ with clang-specific flags, but g++ was more commonly available
• Fix:
  • Changed compiler from clang++ to g++ in .mk.defs
  • Removed clang-specific -stdlib=libstdc++ flag from Makefile
  • Removed unnecessary C++ include path specifications

Changes Made

Modified Files:

• audio_encoder.cpp - Fixed ring buffer logic, added NULL checks and validation
• .mk.defs - Changed compiler from clang++ to g++
• Makefile - Removed clang-specific flags

New Files:

• CRASH_FIXES.md - Comprehensive documentation of all fixes

Testing

✅ Build Test: Plugin compiles successfully without errors
✅ Load Test: Plugin loads in DaVinci Resolve without crashing
✅ Export Test: Successfully exports video with AAC audio encoding
✅ Sample Rate Test: Tested with 44100 Hz and 48000 Hz
✅ Bit Depth Test: Tested with 16-bit and 24-bit audio

Compatibility

• ✅ Maintains backward compatibility with the plugin API
• ✅ No breaking changes to existing functionality
• ✅ Improved error logging for easier debugging
• ✅ Works with existing FFmpeg library versions

Additional Notes

All memory management has been improved with proper cleanup on failure paths. Error messages have been enhanced to help diagnose issues if they occur. The plugin should now be stable for production use.

Related Issues

Fixes the crash issue reported where DaVinci Resolve would close immediately when the plugin was added.

[paterkleomenis]

this should work