Fix plMipmap JPEG decompression using the wrong channel for JPEG alpha

[dgelessus]

JPEG alpha images store the alpha data in the red channel, but libHSPlasma's decompression code incorrectly extracted the blue channel instead, leading to corrupted alpha channels for mipmaps with JPEG alpha.

In particular, this caused @Hoikas's script in H-uru/moul-assets#287 to corrupt the alpha channels of all Relto page symbols, making them nearly invisible in-game. (They would be completely invisible if it weren't for JPEG artifacts making the alpha data in the red channel "bleed" into the blue channel.) Thanks to @TheScarFr for reporting this.

Once this fix is merged, @Hoikas's process in H-uru/moul-assets#287 should be rerun based on the previous, non-corrupted version of the PRP. I think the process itself is safe as long as libHSPlasma is working correctly.

I'm not aware of any other textures being affected so far, but because this bug has apparently existed since 2010 (!!!) March 2024 probably, it might have caused unnoticed corruption elsewhere as well :(

---

Explanation for the code change: The uncompressed buffers in question are stored in BGRA order in memory, but the channel extraction/swapping code accesses them as an array of 32-bit ints. From this perspective, on little-endian systems, the buffer "looks like" ARGB. So to use the red channel as alpha, we need to shift each 32-bit int 8 bits to the left and mask out the low 24 bits.

[dgelessus]

To clarify: the corruption only occurred when the JPEG alpha data was decompressed. As far as I can tell, there was never any corruption when compressing alpha data to JPEG (unless, of course, the alpha data came from decompressing an alpha JPEG) or when exporting the alpha JPEG as-is in compressed form.

[dpogue]

JPEG images are used relatively infrequently in Uru data (mostly for GUI elements like books and journals), and even fewer images have alpha channels, so hopefully any breakage resulting from this would be minimal 🤞🏼

[dgelessus]

Good news: after some further testing, it seems that JPEG alpha decompression was not fully broken since 2010. For example, the last release of PlasmaShop (2023.03.21) does not seem to suffer from this bug.

It looks like #127 indirectly broke the alpha channel swapping code, because it changed plJPEG::DecompressJPEG to return the decompressed data in BGRA order instead of RGBA as before. With RGBA order (little-endian: ABGR), the previous alpha channel swapping code correctly extracted the red channel.

#127 was opened in 2018, but only merged in March 2024, so it seems this bug was only present for about a year and a half.

[Deledrius]

That explains some questions I had about the JPEG Alpha codepath. Every time I've messed with that code I've been concerned about breaking it because a lot of it felt unusual for reasons beyond my awareness.

As an aside: at this point I feel that relying on code and data that quietly stores alpha in a color channel is a trap, and we have better formats for whatever this was used for. It might be worth converting the game data that needs alpha using this format, although we'd still need to support it here for reading and writing old formats. Better documentation in the code to prevent mistakes in the future would be appreciated.

[dgelessus]

There are a lot of undocumented assumptions in the mipmap IO and conversion code, yeah. When I looked into the bug yesterday, I "livetweeted" my understanding of the code in the OpenUru Discord (thanks to @Hoikas for the initial pointers on how the memory layout is supposed to look!). I want to make another PR to document this properly in the code, but until then, here are the relevant bits of the chat, for reference.

Partial chat log

[Hoikas] The uncompressed image data should be BGRA. The jpeg cache is RGB where R is red in JCache and A in JAlpha. DXT compressed levels are RGBA
[dgelessus] when you say BGRA, is that the "bytewise" order, or dependent on host endianness (i. e. actually ARGB)?
[Hoikas] That should be host independent
[dgelessus]
okay, for all compression types except DXT, plMipmap::fImageData seems to be in byte-wise BGRA order (so ARGB when viewed as little-endian)
(I'm assuming a little-endian system here, because that's what matters practically. For big-endian systems, I think some of the code behaves inconsistently.)
If the compression type is JPEG, the RGB part and alpha part are stored separately, and each may be either JPEG-compressed or RLE-compressed
If a part is JPEG-compressed, it gets read unmodified (in JPEG format) into fJPEGCache (RGB) or fJAlphaCache (alpha) respectively, and then decompressed into fImageData in the usual format.
If a part is RLE-compressed, the corresponding fJPEGCache/fJAlphaCache is empty (length 0) and the RLE data is decompressed directly into fImageData.
The RLE compression (in the mipmap data on disk) is always based on 32-bit units, even though not all bits are used. The RGB data is actually stored as BGRx and the alpha data is actually stored as xxAx (where x = unused channel).
Based on my current understanding, the RLE alpha reading and writing does use the R channel (of BGRA) for the alpha channel, which seems to be the expected behavior
That seems consistent with Cyan's RLE reading code, best I can see
So for some reason, these JPEG-compressed textures claim to have 9 mipmap levels even though they only have 1. This causes libHSPlasma/PrpShop to crash when trying to access any mipmap level beyond the first one. We should handle that better in libHSPlasma, but I don't think it has anything to do with the alpha channel misreading.
Found it. When reading JPEG-compressed alpha, libHSPlasma used the blue channel as the alpha data, not the red channel as it's supposed to.

[Deledrius]

Thanks for sharing that info here. :)