Encoding issue

[eiriktsarpalis]

Originally posted by @qinyuanpei in PederHP/mcpdotnet#77.

The JSON-RPC messages displayed in the terminal have an encoding issue, as shown in the attached image.

OS: Windows 11 Pro 24H2

---

[willibrandon]

I apologize if this is unrelated, but I noticed here that Chinese characters may display as question marks in some contexts.
https://github.com/willibrandon/NetContextServer/blob/65324e660ab3d96307d3f6bf09c5b0f09215d544/tests/NetContextServer/ThinkOperationTests.cs#L194

[willibrandon]

I've investigated this issue and found that it's due to the stdio transport layer using default system encoding (Windows-1252 on Windows systems) instead of explicitly using UTF-8, which causes Unicode character corruption specifically for Chinese characters while preserving emoji.

In my fork, I've implemented a fix by ensuring consistent UTF-8 encoding throughout the transport layer:

// Applied to both client and server transports
var utf8Encoding = new UTF8Encoding(false); // No BOM
_customStdin = new StreamWriter(_process.StandardInput.BaseStream, utf8Encoding) { AutoFlush = true };
_customStdout = new StreamReader(_process.StandardOutput.BaseStream, utf8Encoding);

I've written comprehensive tests that verify both Chinese characters ("上下文伺服器") and emoji (🔍🚀👍) are correctly preserved in transport messages. All tests are now passing, confirming this approach resolves the issue regardless of system locale or default encoding settings.

I'd be happy to submit a PR with this fix if you're interested.

[stephentoub]

Thanks, I actually already have a fix along those lines. I was just waiting to add tests. But let me put up my fix; if you could pr your tests to validate it, that'd be great.

[stephentoub]

Actually, @willibrandon, why don't you just put up your whole fix + tests. It'll be easier to manage as a unit. Thanks!

[willibrandon]

@stephentoub - Will do. I'll prepare the PR now and will submit it here shortly.

[willibrandon]

@stephentoub I've submitted a PR that addresses this encoding issue (#73). As this is my first contribution to this codebase, I'm admittedly a bit nervous about it. I've done my best to follow the project's patterns and conventions, but would really appreciate your thorough review and critical feedback. I'm particularly interested in your thoughts on the approach to handling the UTF-8 encoding and the test implementation. Thanks for the encouragement to submit the fix!