fix: rotate/flip segfault from AV_CEIL_RSHIFT on unsigned dimensions

[SteveGilvarry]

What

Image::Rotate and Image::Flip (rewritten in the PR #4788 series) compute chroma plane dimensions with AV_CEIL_RSHIFT(width, log2_chroma_w). The macro's runtime form is -((-(a)) >> (b)) — a branchless ceiling that relies on arithmetic right shift of a negative value, valid only for signed operands (FFmpeg always passes int). Image::width/height are unsigned int, so the negation wraps, the shift is logical, and the result is 2^31 + ceil(a/2^b) instead of ceil(a/2^b).

Captured in gdb on a live 1280x720 ROTATE_270 monitor:

rotate_plane (src=..., src_linesize=640,
              src_w=2147484288,   // 2^31 + 640
              src_h=2147484008,   // 2^31 + 360
              dst=..., dst_linesize=384, bpp=1, angle=270)

The chroma loops then index ~2 billion rows out of bounds → immediate SIGSEGV.

Effect

zmc's decoder thread segfaults on the first decoded frame of any monitor with a rotated/flipped orientation and a planar pixel format. A rotated monitor with Decoding: Always crash-loops; Ondemand crashes as soon as anything views it.

Fix

Replace the macro at both sites (Rotate, Flip) with an explicit unsigned-safe ceiling shift helper, with a comment documenting why AV_CEIL_RSHIFT must not be used on unsigned operands.

Tests

New tests/zm_image.cpp: Rotate 90/180/270 and Flip on YUV420P with per-plane marker-pixel mapping checks, plus odd dimensions (101x57) exercising the chroma ceiling.

• Before the fix: the Rotate cases segfault (exit 139) — verified by stashing the fix.
• After: all pass; full suite green via ctest.
• Live verification: 1280x720 ROTATE_270 monitor with Decoding: Always — unpatched zmc crashes within seconds of decode start, patched zmc ran a clean 90s session under gdb (continuous decode + rotate).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Fixes a crash in Image::Rotate / Image::Flip for planar pixel formats by replacing AV_CEIL_RSHIFT usage on unsigned dimensions (which can produce enormous chroma plane sizes due to unsigned wrap + logical right shift), and adds regression tests to validate per-plane mapping and odd-dimension chroma sizing.

Changes:

• Replace AV_CEIL_RSHIFT(width/height, log2_chroma_*) with an unsigned-safe ceil_rshift() helper in Image::Rotate and Image::Flip.
• Add Catch2 regression tests covering Rotate 90/180/270 and Flip on YUV420P, including odd luma dimensions to exercise chroma ceiling behavior.
• Register the new test file in the tests CMake source list.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/zm_image.cpp	Introduces ceil_rshift() and uses it for planar chroma plane dimension calculation in Rotate/Flip to prevent OOB/segfault.
tests/zm_image.cpp	Adds regression tests validating Rotate/Flip correctness for YUV420P, including odd dimension chroma ceiling cases.
tests/CMakeLists.txt	Adds zm_image.cpp to the test build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[connortechnology]

I have claude working on similar issues. This is going to be a mess. Thing is: None of these things can be negative. That has to be caught much earlier.

[connortechnology]

THe thing is... standard idiom is to just be signed even though a signed value should never happen.. I might disagree, but who am I. I defer to ffmpeg.

[connortechnology]

Also We have a real crash here as a result of this code. Yet I feel that we are in slop land. We are going to have to engage our brains to help the AI out.

[SteveGilvarry]

OK definitely was not working for my monitors on dev master before this, can you reproduce that? And worked after it, so I went with sounds good. I will engage mind to check what it did.