Should wcswidth return Narrow (1) for Basic Emojis followed by VS-15, U+FE0E?

[Jackenmen]

Here's an example where it causes misalignment for me in Kitty terminal:

The code:

import wcwidth
default = "\N{WHITE HEAVY CHECK MARK}"
text = "\N{WHITE HEAVY CHECK MARK}\N{VARIATION SELECTOR-15}"
emoji = "\N{WHITE HEAVY CHECK MARK}\N{VARIATION SELECTOR-16}"
variations = [default, text, emoji]
for v in variations:
    width = wcwidth.wcswidth(v)
    print(v, width)
    print(" " * width, "This should align with above?")

[jquast]

Yes @Jackenmen, actually I do agree with Kovid Goyal's specification,

U+FE0E - Variation Selector 15

When the previous cell has width two and the last code point in the previous cell is one of the Basic_Emoji code points from the Wide emoji rule above that is not followed by U+FE0F then the width of the previous cell is decreased to one.

I independently came to the same conclusion, my remarks from analysis last year

And I think VS-15 serves to fulfill the need to display normally Wide emojis in a matching size and bitonal color, of only these select few early emojis, as if to show it in its historical or simplified form. I believe VS-15 exists for a mirrored property, that turning a Wide emoji into text display makes it Narrow (VS15), just as making text into emoji presentation makes it wide(VS16).

This analysis was done in response to @j4james' criticism on HN,

And at one time the East Asian Width spec specifically mentioned VS-16 making narrow characters wide (but said nothing about VS-15 making wide characters narrow).

I asked @j4james to review, responding,

Aside from the practicality, the fact that TR-11 was so explicit about the VS-16 making glyphs wider, while having no mention of VS-15 making glyphs narrow, is (I think) what convinced most terminal devs that VS-15 should only affect the visual appearance, and not the width.

It's not like this topic hasn't been discussed before. It's just that almost everyone has already agreed on the correct behavior. That's why you're seeing the results you're seeing in your tests.

And so the matter was closed and not published.

[jquast]

I renamed your title to be more clear, please check, that:

>>> wcwidth.wcswidth("\N{WHITE HEAVY CHECK MARK}\N{VARIATION SELECTOR-15}")
2

wcswidth() measures this as a "wide character" (2 cells), which matches a majority of terminals, but, in kitty, this is displayed as "narrow" (one cell).

I do agree the correct interpretation of the specification should return 1 instead of 2, but python wcwidth as it is currently published returns 2.

Happy to have any further feedback.

[Jackenmen]

Yes, returned cell width for "\N{WHITE HEAVY CHECK MARK}\N{VARIATION SELECTOR-15}" is 2, not 1.
I didn't realise that Kitty is alone in this interpretation, I thought it would be the more common one.

I don't think there's really anything better to do in wcwidth then and this should just be closed.
Thanks for a detailed response with all the references!

[jquast]

Your welcome!

Yes, Kovid and I may be wrong about the interpretation of the unicode.org standard.. it's not really our fault, the for-pay membership unicode technical committee makes little to no effort to specify any concern for fixed-width/monospace rules, so they leave a lot up for interpretation.

In these cases, this library aims to be most compatible with a majority of "progressive" or "actively developed" terminals. kitty is certainly one of the best, ranking number 2 in ucs-detect results which includes matching the specification of python wcwidth, (after ghostty, number one, which also appears to agree to display these characters Narrow by my testing).

But, they are also alone in this regard, only 2 out of 35 terminals tested match this behavior right now. note that my testing reads "python wcwidth.wcswidth() measures width 1, while [...] measures width 2.", but this statement is false, I need to update this documentation to reflect this specific "VS-15" automatic testing does not match current released python wcwidth behavior, it's tricky to describe because of this fence sitting.

I have updated ucs-detect report to reference this issue,

The interpretation of VS-15 (U+FE0E) narrowing Wide Emoji is contested. While this test expects VS-15 to make Wide Emoji Narrow, most terminal emulators do not implement this behavior, and python wcwidth.wcswidth() does not currently return a narrow width for these sequences. Only 2 of 35 terminals tested match this expectation. This score is excluded from the final ranking. See jquast/wcwidth#211.

So I will leave this issue open, anyway. I can certainly be persuaded to match kitty and ghostty, as these terminal authors are very progressive and thoughtful about their implementation compared to the other terminal authors.

[jquast]

An issue where @j4james persuaded @christianparpart against implementing VS-15/U+FE0E in Contour terminal, contour-terminal/contour#404 (comment)

[j4james]

If you think an emoji followed by VS-15 should have a width of 1, here's a little test for you to run.

import sys
check = '\u2705'
check_vs15 = check+'\uFE0E'

sys.stdout.write('<--------20-------->\n')

sys.stdout.write('\033[4h')    # enable insert mode 
sys.stdout.write('*'*10)       # write 10 asterisks
sys.stdout.write('\r')         # return to start of line
sys.stdout.write(check*5)      # insert 5 check marks
sys.stdout.write('\n')         # move to new line
sys.stdout.write('*'*10)       # write 10 asterisks
sys.stdout.write('\r')         # return to start of line
sys.stdout.write(check_vs15*5) # insert 5 check marks with VS-15
sys.stdout.write('\n')         # move to new line
sys.stdout.write('\033[4l')    # back to replace mode

sys.stdout.write('<-----15------>\n')

In the code above, we're writing out 10 asterisks, then inserting 5 emojis at the start of the same line using IRM. For the first attempt, the emojis have no VS-15 suffix, and in the second attempt, each emoji is followed by a VS-15 suffix.

If VS-15 makes the emoji narrow, then the first line should span 20 columns, and the second line should only span 15 columns.

Here's what it looks like in Ghostty:

And here's what it looks like in Kitty:

Make of that what you will.

[jquast]

We mismatch on ghostty, versions 1.3.0-main+4b7a55a50 (Feb 26), and 1.3.2-main+4903e2821 (today) appear as:

Which includes key VS-15 related PR's, ghostty-org/ghostty#9678 and ghostty-org/ghostty#9679 (Nov 24)

As for IRM behavior, you are correct that you predicted this misbehaving insert mode with a wide-to-narrow VS-15, wcwidth so far does not interpret this sequence (and I don't think it should)

[j4james]

We mismatch on ghostty

Yeah, I just came here to post an updated screenshot. My original test was on an older version. But the fundamental problem still remains. Neither of those terminals are consistently handling VS-15 characters as 1 cell wide, and any application assuming that is going to break.

[j4james]

Btw, here's another scenario for you. Imagine you're writing an editor, and you've determined that the terminal screen width is w, if wcswidth reports a string's width to be w, is it safe to assume you can fit that string on one line without wrapping?

Test case:

import struct, fcntl, termios
winsz = fcntl.ioctl(sys.stdout.fileno(), termios.TIOCGWINSZ, struct.pack('HHHH', 0, 0, 0, 0))
h,w,_,_ = struct.unpack('HHHH', winsz)

check = '\u2705'
check_vs15 = check+'\uFE0E'

print('*' * w)
print(check_vs15 * w)

With a terminal width of 40, I can fit 40 asterisks on one line, so if a VS-15 check mark is also 1 cell wide, I should also be able to fit 40 check marks on one line, right?

Here's Ghostty:

And here's Kitty:

Do you honestly still think that these are the only terminals that have got VS-15 right, and everyone else has got it wrong?