Garbled textual metadata

[rastaman111]

Hello
My file contains LAME3.93 encoding
How can I get metadata through your library, without hieroglyphs

I get the following data: "Ñòî ÷àñîâ" and "Þðèé Ëîçà"

[sbooth]

Could you share a file that exhibits the problem?

[rastaman111]

Юрий Лоза-Сто часов.mp3.zip

[sbooth]

Thank you for the test case. Was the ID3 tag in the file generated using LAME also?

The problem seems to be one of text encoding. While ID3v1 tags use the 8859-1 charset (although sometimes the machine's local encoding is used, such as Windows-1251 which appears to be the correct encoding for this particular ID3v1 tag), ID3v2 uses UTF-8. It seems the ID3v2 tag in this file is not encoded using UTF-8 but rather a different character set, most likely Windows-1251, the same as the ID3v1 tag.

Take the "word" Þðèé (should be Юрий) from the TPE frame. The octets in UTF-8 and Windows-1251 have the following hex values:

UTF-8	Hex value	Windows-1251
Þ	0xDE	Ю
ð	0xF0	р
è	0xE8	и
é	0xE9	й

The octet values interpreted using UTF-8 give Þðèé while using Windows-1251 give Юрий. So it seems that the text in both the ID3v1 and ID3v2 tags in this file is incorrectly encoded.

[rastaman111]

It's strange when Apple's native player easily recognizes text, just like Google Translate

I'll try to search for similar files and let you know the result

[sbooth]

That is interesting. I will take a closer look at the file's tag to make sure it is being handled correctly. I've heard of charset detection for ID3v1 tags but for ID3v2 I don't think there should be any guessing involved.

[rastaman111]

Apple Music says it's version 3

I also ran it through several libraries and they all say that it is version 3

[sbooth]

It is an ID3v2.3 tag. The TPE1 frame for example contains the following bytes:

Field	Hex Bytes	Meaning
Frame ID	54 50 45 31	TPE1
Size	00 00 00 0A	10
Flags	00 00
Text Encoding	00	ISO 8859-1
Information	DE F0 E8 E9 20 CB EE E7 E0

It's possible that Music runs text reported as ISO 8859-1 through a character detection library. Based on the ID3v2 tag itself, TagLib (the metadata library used by SFBAudioEngine) is interpreting the data correctly.

It shouldn't be terribly hard to wrap uchardet to add the option for character set detection for ID3v1 or ID3v2 tags using ISO 8559-1 but I haven't investigated what it would entail.

[rastaman111]

I have the following question.
How can I understand from the text what encoding it has and show the user the appropriate text?

[sbooth]

Algorithms for character set detection are something I know little to nothing about. Perhaps an educated guess is made based on a frequency analysis of octets in the input?

For the file that you shared it should be possible to feed the C strings from the metadata to uchardet or a similar library and see what it comes back with, and then use iconv to convert to UTF-8.

[rastaman111]

This problem is not only with this file, I found a large number of such files, in the native application Files and Music the data is displayed as expected, but using the standard API does not lead to the desired result :(

[CharlesWiltgen]

@rastaman111: This problem is not only with this file, I found a large number of such files, in the native application Files and Music the data is displayed as expected…

Yes, because Apple has implemented a workaround for files with broken metadata. SFBAudioEngine does The Right Thing, so @sbooth needs to decide if they want to support files doing The Wrong Thing.

@sbooth: For the file that you shared it should be possible to feed the C strings from the metadata to uchardet or a similar library and see what it comes back with, and then use iconv to convert to UTF-8.

From what I've seen, this is the standard heuristic in libraries that opt into charset detection: run the detector when (a) the text encoding byte is 0x00 (ISO 8859-1) and (b) the payload contains any byte ≥ 0x80. The same would apply to v2.4 frames declaring 0x00.

bool needsCharsetDetection(const uint8_t *payload, size_t len) {
    for (size_t i = 0; i < len; i++) {
        if (payload[i] >= 0x80) return true;
    }
    return false;
}

Notes:

1. The detector needs the raw frame bytes, before TagLib decodes them as Latin-1 → UTF-8. Otherwise uchardet sees the UTF-8 of the mojibake, not the Cyrillic byte distribution it needs.
2. Legitimate Latin-1 (e.g. "Café") also has bytes ≥ 0x80, so the check above is just the trigger. The actual decision still has to come from a real detector (uchardet, ICU ucsdet_*).

[rastaman111]

@CharlesWiltgen Hi! I recently implemented text decoding and ran into the problem that converting to bytes is a bad idea, since different languages ​​sometimes have the same symbols, but in some places it's part of the language, and in others it's mojibake.

[CharlesWiltgen]

Yes, character encoding/decoding is strange magic, so I let libraries by people much smarter than me do any conversion work. The trigger above just lets correctly-encoded tags through, and otherwise uchardet and iconv do the heavy lifting (as I understand it). Poking at this a bit more, it looks like iOS/macOS ships with iconv equivalents (libiconv, Foundation and CoreFoundation string APIs), and apparently NSString.stringEncoding​(for:encodingOptions:​convertedString:​usedLossyConversion:) will handle common cases of the "return detected encoding plus the converted string".