unicode (proof of concept)

[koal44]

branch: https://github.com/koal44/xterm.js/tree/poc/unicode-hell

run the demo, enable the compat-addon in the width tab, and try your usual unicode test cases in bash/zsh/fish/pwsh. it renders modern unicode grapheme clusters over a legacy wcwidth-style model while remaining in sync with the backend app and supporting atomic cursor movement and deletion.

[koal44]

Measurement Viability

This branch shows that full Unicode support is possible, but that doesn't mean it's currently viable.

Right now the demo depends on preset tables that were measured offline. In practice xterm.js would need to measure during a live session -- not the full Unicode space, but some runtime probing would still be required, which raises real questions: how to avoid corrupting user input? whether to fork a shadow process? how to detect app switches? And what about apps that cannot be measured at all?

jerch pointed out there is no negotiated Unicode protocol. Measuring was effectively a way to re-derive one, but after looking at what it would take to integrate live measurement cleanly, I think the problem may be intractable unless there is a cleaner way to approach live measurement?

Salvageable Pieces

Even if live measurement is not practical, there are pieces here that still seem useful.

1. Compat table vs Unicode provider
   Instead of versioned Unicode providers, the compat table is data-driven. What matters most is alignment with backend apps, not championing a particular version. Within an environment (POSIX vs Windows), app width models tend to converge, so choose a canonical model per OS (bash on POSIX, pwsh on Windows) and derive the effective wcwidth flavor once.

2. Modern Unicode visual overlay
   In the WebglRenderer I added a visual index on top of the app index. This demonstrates that modern grapheme rendering can be layered on top of legacy column models. It adds complexity and may affect perf, but rendering itself is not fundamentally blocked.

3. Buffer ownership cleanup
   BufferSet → Buffer → BufferLine → CellData.
   The CellCompatAddon swapped BufferLine at runtime, which required tightening encapsulation and ownership boundaries. If future architectural changes are planned, like integrating Ghostty, this refactoring may be aligned with that direction.

4. Socket reporter
   This seems generally useful as a debugging and instrumentation tool.

5. Mov/Del width measurements
   These were added after tyriar mentioned prompt corruption concerns ("working emoji + broken shell prompt"). It enforces atomic editing, but adds complexity and storage overhead. Unlike app width, mov/del behavior does not converge across apps, so without reliable live measurement, this part is not practical.

Breaking Out of Unicode Hell

It is a curious situation. Apps expect terminals to mirror their implicit column layout without communicating what that layout actually is. Terminals gravitate toward a common denominator, and coordination deadlock ensues where neither side can update without breaking the implied contract.

A Unicode protocol could help, but it raises a deeper question: why are two coordinated systems doing the same width computation at all? This seems like a problem of misattributed responsibility where the conventional wisdom that terminals should remain forever dumb has led to a broken system. Width is fundamentally a rendering concern. My conclusion is that visual representation should be owned by the terminal, not the app.

One possible migration path would build on Kovid Goyal's text sizing protocol (RFC: kovidgoyal/kitty#8226). With a negotiated capability flag, apps would send OSC 66 with w=1 for everything. The terminal handles Unicode rendering. The app stops computing width. Over time, if widely adopted, width=1 becomes the baseline assumption and the protocol could be deprecated.

[koal44]

hey, did anyone get a chance to play around with this? i'm curious whether it worked for others or whether this feels worth pursuing.