Correct VS15 handling for emojis (#9)

Author: clipperhouse

README.md

@@ -5,6 +5,7 @@ A high-performance Go package for measuring the monospace display width of strin
 [![Documentation](https://pkg.go.dev/badge/github.com/clipperhouse/displaywidth.svg)](https://pkg.go.dev/github.com/clipperhouse/displaywidth)
 [![Test](https://github.com/clipperhouse/displaywidth/actions/workflows/gotest.yml/badge.svg)](https://github.com/clipperhouse/displaywidth/actions/workflows/gotest.yml)
 [![Fuzz](https://github.com/clipperhouse/displaywidth/actions/workflows/gofuzz.yml/badge.svg)](https://github.com/clipperhouse/displaywidth/actions/workflows/gofuzz.yml)
+
 ## Install
 ```bash
 go get github.com/clipperhouse/displaywidth
@@ -34,26 +35,40 @@ func main() {
 
 For most purposes, you should use the `String` or `Bytes` methods.
 
+
 ### Options
 
-You can specify East Asian Width settings. If unspecified, the default is `EastAsianWidth: false`.
+You can specify East Asian Width settings. When false (default),
+[East Asian Ambiguous characters](https://www.unicode.org/reports/tr11/#Ambiguous)
+are treated as width 1. When true, East Asian Ambiguous characters are treated
+as width 2.
 
 ```go
-options := displaywidth.Options{
+myOptions := displaywidth.Options{
     EastAsianWidth: true,
 }
 
-width := options.String("Hello, 世界!")
+width := myOptions.String("Hello, 世界!")
 fmt.Println(width)
 ```
 
-## Details
+## Technical details
 
 This package implements the Unicode East Asian Width standard
 ([UAX #11](https://www.unicode.org/reports/tr11/)), and handles
 [version selectors](https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)),
 and [regional indicator pairs](https://en.wikipedia.org/wiki/Regional_indicator_symbol)
-(flags). We cover much of [Unicode TR51](https://unicode.org/reports/tr51/).
+(flags). We implement [Unicode TR51](https://unicode.org/reports/tr51/).
+
+`clipperhouse/displaywidth`, `mattn/go-runewidth`, and `rivo/uniseg` will
+give the same outputs for most real-world text. See extensive details in the
+[compatibility analysis](comparison/COMPATIBILITY_ANALYSIS.md).
+
+If you wish to investigate the core logic, see the `lookupProperties` and `width`
+functions in [width.go](width.go#L135). The essential trie generation logic is in
+`buildPropertyBitmap` in [unicode.go](internal/gen/unicode.go#L317).
+
+I (@clipperhouse) am keeping an eye on [emerging standards and test suites](https://www.jeffquast.com/post/state-of-terminal-emulation-2025/).
 
 ## Prior Art
 
@@ -106,12 +121,3 @@ BenchmarkRune_ASCII/mattn/go-runewidth-8                    266.4 ns/op	    480.
 BenchmarkRune_Emoji/clipperhouse/displaywidth-8            1384 ns/op	    523.02 MB/s	      0 B/op     0 allocs/op
 BenchmarkRune_Emoji/mattn/go-runewidth-8                   2273 ns/op	    318.45 MB/s	      0 B/op     0 allocs/op
 ```
-
-## Compatibility
-
-`clipperhouse/displaywidth`, `mattn/go-runewidth`, and `rivo/uniseg` should give the
-same outputs for real-world text. See [comparison/README.md](comparison/README.md).
-
-If you wish to investigate the core logic, see the `lookupProperties` and `width`
-functions in [width.go](width.go#L112). The core of the trie generation logic is in
-`BuildPropertyBitmap` in [unicode.go](internal/gen/unicode.go#L309).

comparison/COMPATIBILITY_ANALYSIS.md

@@ -1,71 +1,97 @@
 # Compatibility Analysis: displaywidth, go-runewidth, and uniseg
 
-> **Generated by:** Cursor IDE using Claude Sonnet 4.5, with a few edits by me (@clipperhouse)
-
-## Summary
+> Generated by Cursor IDE using Claude Sonnet 4.5, and edited by @clipperhouse
 
 This document summarizes the compatibility findings between three Go libraries for Unicode string width calculation:
 
 - [clipperhouse/displaywidth](https://github.com/clipperhouse/displaywidth) (this package)
 - [mattn/go-runewidth](https://github.com/mattn/go-runewidth)
 - [rivo/uniseg](https://github.com/rivo/uniseg)
 
-### 1. Regional Indicator Pairs (Flags)
+## Basic Unicode Categories
+
+Most Unicode categories show good compatibility.
+
+| Category | displaywidth | go-runewidth | uniseg |
+|----------|--------------|--------------|---------|
+| ASCII | ✅ Compatible | ✅ Compatible | ✅ Compatible |
+| Latin Extended | ✅ Compatible | ✅ Compatible | ✅ Compatible |
+| CJK (Chinese/Japanese/Korean) | ✅ Compatible | ✅ Compatible | ✅ Compatible |
+| Arabic | ✅ Compatible | ✅ Compatible | ✅ Compatible |
+| Combining Marks | ✅ Compatible | ✅ Compatible | ✅ Compatible |
+| Zero-Width Characters | ✅ Compatible | ✅ Compatible | ✅ Compatible |
+
+## Emojis
+
+Regular emojis (😀, 🚀, 🎉, etc.) behave identically:
+
+| Library | Regular Emoji Width |
+|---------|---------------------|
+| **displaywidth** | Always 2 |
+| **go-runewidth** | Always 2 |
+| **uniseg** | Always 2 |
+
+### Regional Indicator Pairs (Flags)
 
 Regional indicator pairs (flags like 🇺🇸) are composed of two Regional Indicator symbols.
 
 | Library | Behavior |
 |--------|----------|
-| **displaywidth** | Width 2 per flag (always, per Unicode TR51) |
-| **go-runewidth** | Width 1 per flag (always) |
-| **uniseg** | Width 2 per flag (always) |
+| **displaywidth** | Width 2 per flag |
+| **go-runewidth** | Width 1 per flag |
+| **uniseg** | Width 2 per flag |
 
 **Example:** `🇺🇸🇯🇵🇬🇧` (3 flags)
 - displaywidth: 6 columns (2+2+2)
 - go-runewidth: 3 columns (1+1+1)
 - uniseg: 6 columns (2+2+2)
 
-### 2. Variation Selectors
+## Variation Selectors
 
-**Variation selectors (VS15, VS16) affect emoji presentation:**
+VS15 and VS16 from [Unicode TR51](https://unicode.org/reports/tr51/#Emoji_Variation_Sequences)
 
-| Library | Behavior |
-|---------|----------|
-| **displaywidth** | Treats variation selectors as part of emoji (width 2) |
-| **go-runewidth** | Treats variation selectors as separate characters (width 1 each) |
-| **uniseg** | Treats variation selectors as part of emoji (width 2) |
+| Library | VS16 (U+FE0F) | VS15 (U+FE0E) |
+|---------|---------------|---------------|
+| **displaywidth** | Forces emoji presentation (width 2) | No effect, preserves base width |
+| **go-runewidth** | Treated as separate character (width 1) | Treated as separate character (width 1) |
+| **uniseg** | Treated as part of emoji (width 2) | Forces width 1 |
 
 **Example:** `☺️⌛︎❤️` (3 emoji with variation selectors)
-- displaywidth: 5 columns
+- displaywidth: 6 columns
 - go-runewidth: 4 columns
 - uniseg: 5 columns
 
-### 3. Keycap Sequences
+I would appear to me (@clipperhouse) that the handling of VS15 is not widely
+agreed upon. Some libraries and standards (such as wcwidth) interpret it as
+"always narrow to width 1". Others (such as this library) interpret it as
+"no effect on width, use the base character width".
+
+Here is [a conversation on GitHub](https://github.com/contour-terminal/contour/discussions/1178#discussioncomment-6778716) and an [explanation from Grok](https://grok.com/share/bGVnYWN5LWNvcHk%3D_274f540c-c9a6-47c7-9d4f-47697ed20032).
 
-**Keycap sequences (digit/symbol + VS16 + combining enclosing keycap):**
+## Keycap Sequences
 
 Keycap sequences like 1️⃣ are formed by: base character + variation selector (U+FE0F) + combining enclosing keycap (U+20E3).
 
 | Library | Behavior | Width per Keycap |
 |---------|----------|------------------|
-| **displaywidth** | Treats as single wide emoji | 2 columns |
-| **go-runewidth** | Treats base character only | 1 column |
-| **uniseg** | Treats as single narrow character | 1 column |
+| **displaywidth** | Treats as emoji | 2 columns |
+| **go-runewidth** | Treats base character | 1 column |
+| **uniseg** | Treats as base character | 1 column |
 
 **Example:** `1️⃣#️⃣` (2 keycap sequences)
 - displaywidth: 4 columns (2 per keycap)
 - go-runewidth: 2 columns (1 per keycap)
 - uniseg: 2 columns (1 per keycap)
 
-### 4. East Asian Ambiguous Width
+## East Asian Ambiguous Width
 
-**East Asian Ambiguous characters** (★, °, ±, etc.) can be rendered as either narrow (1 column) or wide (2 columns) depending on configuration.
+[East Asian Ambiguous characters](https://www.unicode.org/reports/tr11/#Ambiguous) (★, °, ±, etc.) can be rendered as either narrow (1 column) or wide (2 columns) depending on configuration.
 
 | Library | Default | With EastAsianWidth=true |
 |---------|---------|--------------------------|
 | **displaywidth** | Width 1 | Width 2 |
 | **go-runewidth** | Width 1 | Width 2 |
-| **uniseg** | Width 1 | Width 2 (usually) |
+| **uniseg** | Width 1 | Width 2 |
 
 **Example:** `★°±` (3 ambiguous characters)
 - displaywidth default: 3 columns
@@ -75,29 +101,6 @@ Keycap sequences like 1️⃣ are formed by: base character + variation selector
 - uniseg default: 3 columns
 - uniseg with EastAsianAmbiguousWidth=2: 5 columns (usually)
 
-### 5. Regular Emojis
-
-**Regular emojis (😀, 🚀, 🎉, etc.) behave identically:**
-
-| Library | Regular Emoji Width |
-|---------|---------------------|
-| **displaywidth** | Always 2 |
-| **go-runewidth** | Always 2 |
-| **uniseg** | Always 2 |
-
-### 6. Basic Unicode Categories
-
-Most Unicode categories show good compatibility.
-
-| Category | displaywidth | go-runewidth | uniseg |
-|----------|--------------|--------------|---------|
-| ASCII | ✅ Compatible | ✅ Compatible | ✅ Compatible |
-| Latin Extended | ✅ Compatible | ✅ Compatible | ✅ Compatible |
-| CJK (Chinese/Japanese/Korean) | ✅ Compatible | ✅ Compatible | ✅ Compatible |
-| Arabic | ✅ Compatible | ✅ Compatible | ✅ Compatible |
-| Combining Marks | ✅ Compatible | ✅ Compatible | ✅ Compatible |
-| Zero-Width Characters | ✅ Compatible | ✅ Compatible | ✅ Compatible |
-
 ## Detailed Test Results
 
 > Run `go test -v` in the `comparison/` directory to see comprehensive behavior comparisons between libraries.
@@ -122,3 +125,7 @@ Most Unicode categories show good compatibility.
 - displaywidth: 6 columns (2 per emoji)
 - go-runewidth: 6 columns (2 per emoji)
 - uniseg: 6 columns (2 per emoji)
+
+## Further Reading
+
+- [State of Terminal Emulation 2025](https://www.jeffquast.com/post/state-of-terminal-emulation-2025/)

comparison/behavior_test.go

@@ -73,7 +73,7 @@ func TestLibraryBehaviorComparison(t *testing.T) {
 				"displaywidth_default":   14, // 2 per emoji (properly handles Unicode 16.0)
 				"displaywidth_options{}": 14,
 				"go-runewidth_default":   7, // go-runewidth may not fully support Unicode 16.0 yet (treats as width 1)
-				"uniseg_default":         7,  // uniseg may not fully support Unicode 16.0 yet (treats as width 1)
+				"uniseg_default":         7, // uniseg may not fully support Unicode 16.0 yet (treats as width 1)
 			},
 		},
 
@@ -92,14 +92,15 @@ func TestLibraryBehaviorComparison(t *testing.T) {
 		},
 
 		// Variation selectors
+		// ☺️ (U+263A + VS16) = width 2, ⌛︎ (U+231B + VS15) = width 2 (VS15 is no-op), ❤️ (U+2764 + VS16) = width 2
 		{
 			name:  "Variation selectors",
 			input: "☺️⌛︎❤️",
 			expected: map[string]int{
-				"displaywidth_default":   5,
-				"displaywidth_options{}": 5,
+				"displaywidth_default":   6, // 2 + 2 + 2 (VS15 is no-op per Unicode TR51)
+				"displaywidth_options{}": 6,
 				"go-runewidth_default":   4,
-				"uniseg_default":         5,
+				"uniseg_default":         5, // uniseg still treats VS15 as width 1
 			},
 		},
 

internal/gen/trie.go

@@ -29,7 +29,7 @@ func GenerateTrie(data *UnicodeData) (*triegen.Trie, error) {
 			continue
 		}
 
-		props := BuildPropertyBitmap(r, data)
+		props := buildPropertyBitmap(r, data)
 
 		// Only insert characters with non-default properties
 		if props != 0 {

internal/gen/unicode.go

@@ -36,23 +36,22 @@ type PropertyDefinition struct {
 // The order matters - it defines the bit positions (via iota).
 var PropertyDefinitions = []PropertyDefinition{
 	{"Zero_Width", "Always 0 width, includes combining marks, control characters, non-printable, etc"},
-	{"Always_Wide", "Always 2 wide"},
+	{"East_Asian_Wide", "Always 2 wide (East Asian Wide F/W)"},
 	{"East_Asian_Ambiguous", "Width depends on EastAsianWidth option"},
-	{"Always_Narrow", "VARIATION SELECTOR-15 (U+FE0E) requests text presentation (width 1); not in the trie, see [width]"},
+	{"Emoji", "Extended_Pictographic + Emoji_Presentation"},
 }
 
 // these constants are used to build the property bitmap, internally.
 // the external properties are above. Keep them in the same order!
 const (
 	// ZWSP, ZWJ, ZWNJ, etc.
 	zero_Width property = iota + 1
-	// F, W
-	always_Wide
-	// A
+	// F, W (East Asian Wide)
+	east_Asian_Wide
+	// A (East Asian Ambiguous)
 	east_Asian_Ambiguous
-	// VS15 requests text presentation (width 1)
-	// not used in the trie, but noted here for reference
-	// always_Narrow
+	// Extended_Pictographic + Emoji_Presentation but not East Asian Wide
+	emoji
 )
 
 // ParseUnicodeData downloads and parses all required Unicode data files
@@ -314,8 +313,7 @@ func extractRunesFromRangeTable(table *unicode.RangeTable, target map[rune]bool)
 	}
 }
 
-// BuildPropertyBitmap creates a properties bitmap for a given rune
-func BuildPropertyBitmap(r rune, data *UnicodeData) property {
+func buildPropertyBitmap(r rune, data *UnicodeData) property {
 	if data.CombiningMarks[r] {
 		return zero_Width
 	}
@@ -326,23 +324,26 @@ func BuildPropertyBitmap(r rune, data *UnicodeData) property {
 		return zero_Width
 	}
 
-	// East Asian Width
-	// Only store properties that affect width calculation
+	// As a practical matter, we probably don't need separate properties for
+	// Emoji and East Asian Wide, as I believe they lead to the same
+	// result. I made this distinction for VS15 handling. However,
+	// eventually I came to the conclusion that VS15 is a no-op for width
+	// calculation. Keeping the distinction for now.
+
+	if data.ExtendedPictographic[r] && data.EmojiPresentation[r] {
+		return emoji
+	}
+
 	if eaw, exists := data.EastAsianWidth[r]; exists {
 		switch eaw {
 		case "F", "W":
-			return always_Wide
+			return east_Asian_Wide
 		case "A":
 			return east_Asian_Ambiguous
 			// H (Halfwidth), Na (Narrow), and N (Neutral) are not stored
 			// as they all result in width 1 (default behavior)
 		}
 	}
 
-	// Emoji properties
-	if data.ExtendedPictographic[r] && data.EmojiPresentation[r] {
-		return always_Wide
-	}
-
 	return 0
 }