Skip to content

Commit 4dc0808

Browse files
VortexUKclaude
andcommitted
fix: non-ASCII Discord names render as boxes in test-connection status
User reported their test-connection label showed "Connected as □□□□□" where their actual Discord name has non-ASCII characters. Three layered fixes, all in the same v0.1.9 release (commit goes on top of f3df6a1, no tag yet). PRIMARY (the user-visible fix): SettingsPanel labels that display user-supplied strings now render via GDI / Uniscribe instead of GDI+ / Graphics.DrawString. GDI does proper Unicode font fallback — when a glyph isn't in the declared font (Segoe UI), Windows falls back through Microsoft YaHei (CJK), Segoe UI Symbol, Segoe UI Emoji, etc. GDI+ has no fallback chain and renders missing glyphs as tofu boxes (□). Applied to four labels that show user data: _testStatusLabel, _currentCharLabel, _uploadStatusLabel, _captureLabel — via a small EnableUnicodeFontFallback() helper that sets UseCompatibleTextRendering = false. Application.SetCompatibleTextRenderingDefault is the process-wide switch but ACT owns that — we can only set this per-label. DEFENSIVE — UTF-8 decoding: HttpContent.ReadAsStringAsync on .NET Framework 4.8 has a long-standing charset-fallback issue when the server omits charset=utf-8 in Content-Type. FastAPI's JSONResponse intentionally omits it (RFC 8259 mandates UTF-8 for application/json), which hits the fallback. Replaced with ReadAsByteArrayAsync + explicit Encoding.UTF8.GetString in a new ReadBodyUtf8Async helper. Same result on every framework version. dotnet/runtime#28658 DEFENSIVE — JSON escape handling: ExtractJsonString's escape branch took the NEXT char literally rather than decoding it. \uXXXX therefore became the literal text "uXXXX", \n became literal "n", etc. FastAPI's ensure_ascii=False means we don't hit this in production today, but any other JSON producer (or a server config change) would silently bite. Now handles every escape from RFC 8259 § 7: \" \ \/ \b \f \n \r \t \uXXXX Explicit hex-digit validation on \uXXXX — caught in test that NumberStyles.HexNumber tolerates trailing whitespace, so "\u4f6 bar" would otherwise have silently parsed as U+04F6 (Cyrillic Komi Gha) instead of returning null for the malformed input. 6 new tests in UploadClientTests pinning every escape and the real-world raw-UTF-8 happy path. Total 131/131. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent f3df6a1 commit 4dc0808

3 files changed

Lines changed: 173 additions & 7 deletions

File tree

src/Core/UploadClient.cs

Lines changed: 82 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ public async Task<Result> TestConnectionAsync(string serverUrl, string apiToken)
116116
{
117117
using (var resp = await _http.SendAsync(req).ConfigureAwait(false))
118118
{
119-
var body = await resp.Content.ReadAsStringAsync().ConfigureAwait(false);
119+
var body = await ReadBodyUtf8Async(resp.Content).ConfigureAwait(false);
120120
if (!resp.IsSuccessStatusCode)
121121
{
122122
return new Result
@@ -212,7 +212,7 @@ public async Task<Result> UploadEncounterAsync(string serverUrl, string apiToken
212212
{
213213
using (var resp = await _http.SendAsync(req).ConfigureAwait(false))
214214
{
215-
var body = await resp.Content.ReadAsStringAsync().ConfigureAwait(false);
215+
var body = await ReadBodyUtf8Async(resp.Content).ConfigureAwait(false);
216216
if (!resp.IsSuccessStatusCode)
217217
{
218218
// Pull a friendly error out of FastAPI's {"detail": "..."} shape if present.
@@ -258,6 +258,13 @@ public async Task<Result> UploadEncounterAsync(string serverUrl, string apiToken
258258
// -------------------------------------------------------------------
259259
// Tiny JSON string-field extractor — avoids pulling in a full JSON
260260
// parser just for one field. NOT a general-purpose JSON parser.
261+
//
262+
// Handles all the escape sequences defined in RFC 8259 § 7:
263+
// \" \\ \/ \b \f \n \r \t \uXXXX
264+
// The \uXXXX path is the one that matters for non-ASCII Discord
265+
// names if a server is ever configured with ensure_ascii=True —
266+
// without it, "discord_name": "你好" came out as the
267+
// literal text "u4f60u597d" in the test-connection label.
261268
// -------------------------------------------------------------------
262269

263270
internal static string? ExtractJsonString(string json, string fieldName)
@@ -276,18 +283,86 @@ public async Task<Result> UploadEncounterAsync(string serverUrl, string apiToken
276283
for (int i = start; i < json.Length; i++)
277284
{
278285
char ch = json[i];
279-
if (ch == '\\' && i + 1 < json.Length)
286+
if (ch == '"') return sb.ToString();
287+
if (ch != '\\')
280288
{
281-
sb.Append(json[i + 1]);
282-
i++;
289+
sb.Append(ch);
283290
continue;
284291
}
285-
if (ch == '"') return sb.ToString();
286-
sb.Append(ch);
292+
// Escape sequence — need at least one more char.
293+
if (i + 1 >= json.Length) return null;
294+
char esc = json[i + 1];
295+
switch (esc)
296+
{
297+
case '"': sb.Append('"'); i++; break;
298+
case '\\': sb.Append('\\'); i++; break;
299+
case '/': sb.Append('/'); i++; break;
300+
case 'b': sb.Append('\b'); i++; break;
301+
case 'f': sb.Append('\f'); i++; break;
302+
case 'n': sb.Append('\n'); i++; break;
303+
case 'r': sb.Append('\r'); i++; break;
304+
case 't': sb.Append('\t'); i++; break;
305+
case 'u':
306+
// \uXXXX — 4 hex digits → 1 char. Surrogate
307+
// pairs (e.g. emoji) come as two \uXXXX in
308+
// sequence; we just emit each char, and
309+
// System.String stores them correctly as a
310+
// UTF-16 surrogate pair.
311+
//
312+
// Validate each char is a real hex digit before
313+
// TryParse — NumberStyles.HexNumber tolerates
314+
// leading/trailing whitespace which would make
315+
// "\u4f6 bar" silently parse as U+04F6.
316+
if (i + 5 >= json.Length) return null;
317+
for (int h = i + 2; h < i + 6; h++)
318+
{
319+
char hc = json[h];
320+
if (!((hc >= '0' && hc <= '9')
321+
|| (hc >= 'a' && hc <= 'f')
322+
|| (hc >= 'A' && hc <= 'F')))
323+
{
324+
return null;
325+
}
326+
}
327+
var hex = json.Substring(i + 2, 4);
328+
if (!ushort.TryParse(hex, System.Globalization.NumberStyles.HexNumber,
329+
System.Globalization.CultureInfo.InvariantCulture, out var code))
330+
{
331+
return null;
332+
}
333+
sb.Append((char)code);
334+
i += 5;
335+
break;
336+
default:
337+
// Unknown escape — preserve the backslash so the
338+
// user at least sees something they can report.
339+
sb.Append('\\');
340+
sb.Append(esc);
341+
i++;
342+
break;
343+
}
287344
}
288345
return null;
289346
}
290347

348+
// -------------------------------------------------------------------
349+
// .NET Framework's HttpContent.ReadAsStringAsync has a long-standing
350+
// charset-fallback issue when Content-Type lacks a charset directive.
351+
// FastAPI's JSONResponse intentionally omits the charset (the body
352+
// is always UTF-8 per RFC 8259), which hits the fallback path on
353+
// 4.8 and can mis-decode non-ASCII names. Read bytes + decode
354+
// explicitly as UTF-8 — same result on every framework version.
355+
// https://github.com/dotnet/runtime/issues/28658
356+
// -------------------------------------------------------------------
357+
358+
internal static async Task<string> ReadBodyUtf8Async(HttpContent content)
359+
{
360+
if (content == null) return "";
361+
var bytes = await content.ReadAsByteArrayAsync().ConfigureAwait(false);
362+
if (bytes == null || bytes.Length == 0) return "";
363+
return Encoding.UTF8.GetString(bytes);
364+
}
365+
291366
internal static string Truncate(string s, int max)
292367
{
293368
if (string.IsNullOrEmpty(s)) return "";

src/SettingsPanel.cs

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,7 @@ public SettingsPanel(PluginConfig config, Action<PluginConfig> onSave, UploadCli
230230
Text = "",
231231
MaximumSize = new Size(InputWidth - 240, 0),
232232
};
233+
EnableUnicodeFontFallback(_testStatusLabel);
233234
cfgButtons.Controls.Add(_testStatusLabel);
234235
cfgCard.Controls.Add(cfgButtons);
235236

@@ -246,6 +247,7 @@ public SettingsPanel(PluginConfig config, Action<PluginConfig> onSave, UploadCli
246247
Margin = new Padding(0, 0, 0, 10),
247248
Text = "",
248249
};
250+
EnableUnicodeFontFallback(_currentCharLabel);
249251
logCard.Controls.Add(_currentCharLabel);
250252

251253
logCard.Controls.Add(MakeFieldLabel("Don't upload as"));
@@ -280,6 +282,7 @@ public SettingsPanel(PluginConfig config, Action<PluginConfig> onSave, UploadCli
280282
BackColor = T.Card,
281283
Margin = new Padding(0, 0, 0, 10),
282284
};
285+
EnableUnicodeFontFallback(_captureLabel);
283286
capCard.Controls.Add(_captureLabel);
284287

285288
var capButtons = new FlowLayoutPanel
@@ -306,6 +309,7 @@ public SettingsPanel(PluginConfig config, Action<PluginConfig> onSave, UploadCli
306309
BackColor = T.Card,
307310
Text = "",
308311
};
312+
EnableUnicodeFontFallback(_uploadStatusLabel);
309313
capCard.Controls.Add(_uploadStatusLabel);
310314

311315
Controls.Add(stack);
@@ -575,6 +579,25 @@ private void UpdateCurrentCharLabel()
575579
}
576580
}
577581

582+
/// <summary>
583+
/// Force the label to render via GDI (TextRenderer / Uniscribe)
584+
/// instead of GDI+ (Graphics.DrawString). GDI does proper Unicode
585+
/// font fallback through Uniscribe — when a glyph isn't in the
586+
/// declared font (Segoe UI), Windows automatically substitutes
587+
/// from Microsoft YaHei (CJK), Segoe UI Symbol, Segoe UI Emoji,
588+
/// etc. GDI+ doesn't fall back and renders missing glyphs as
589+
/// tofu boxes (□). Apply to any label that displays user-supplied
590+
/// strings (Discord names, encounter titles, character names).
591+
///
592+
/// Application.SetCompatibleTextRenderingDefault is the
593+
/// process-wide knob but ACT owns that — we can only set this
594+
/// per-label.
595+
/// </summary>
596+
private static void EnableUnicodeFontFallback(Label label)
597+
{
598+
label.UseCompatibleTextRendering = false;
599+
}
600+
578601
// ──────────────────────────────────────────────────────────────────
579602
// Builders — themed widget factories
580603
// ──────────────────────────────────────────────────────────────────

tests/EQ2Lexicon.ACTPlugin.Tests/UploadClientTests.cs

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,74 @@ public void ExtractJsonString_HandlesEscapedQuote()
122122
Assert.Equal("He said \"hi\" loudly", got);
123123
}
124124

125+
// ── JSON escape sequences (RFC 8259 § 7) ───────────────────────────
126+
// FastAPI's JSONResponse emits raw UTF-8 (ensure_ascii=False) so
127+
// \uXXXX escapes don't show up in practice today, but any other
128+
// JSON producer (or a config change) would emit them — and the
129+
// pre-v0.1.9 extractor turned "你" into the literal text
130+
// "u4f60", which is how the non-ASCII Discord name rendered as
131+
// boxes in the test-connection label. Pin every escape RFC 8259
132+
// defines so a regression here can't slip through silently.
133+
134+
[Fact]
135+
public void ExtractJsonString_HandlesAllStandardEscapes()
136+
{
137+
// \\ \/ \b \f \n \r \t — order matches the RFC table.
138+
var body = "{\"detail\":\"a\\\\b\\/c\\bd\\fe\\nf\\rg\\th\"}";
139+
Assert.Equal("a\\b/c\bd\fe\nf\rg\th", UploadClient.ExtractJsonString(body, "detail"));
140+
}
141+
142+
[Fact]
143+
public void ExtractJsonString_HandlesUnicodeEscapes()
144+
{
145+
// 你好 = 你好 (CJK "hello"). Pre-fix this came out as
146+
// the literal text "u4f60u597d" and rendered as boxes in the
147+
// label. Mojibake-shaped bug we'd have shipped to every
148+
// non-ASCII Discord username if we hadn't caught it.
149+
var body = "{\"discord_name\":\"\\u4f60\\u597d\"}";
150+
Assert.Equal("你好", UploadClient.ExtractJsonString(body, "discord_name"));
151+
}
152+
153+
[Fact]
154+
public void ExtractJsonString_HandlesEmojiSurrogatePair()
155+
{
156+
// Emoji outside the BMP are emitted as a UTF-16 surrogate
157+
// pair of \uXXXX escapes. 😀 (U+1F600) = D83D + DE00.
158+
// System.String stores the two chars as a valid surrogate
159+
// pair so concatenation just works.
160+
var body = "{\"discord_name\":\"\\uD83D\\uDE00\"}";
161+
Assert.Equal("😀", UploadClient.ExtractJsonString(body, "discord_name"));
162+
}
163+
164+
[Fact]
165+
public void ExtractJsonString_HandlesMixedAsciiAndEscapes()
166+
{
167+
// The real-world shape: a name that's mostly ASCII but
168+
// contains a single accented char that the server escaped.
169+
var body = "{\"discord_name\":\"caf\\u00e9\"}";
170+
Assert.Equal("café", UploadClient.ExtractJsonString(body, "discord_name"));
171+
}
172+
173+
[Fact]
174+
public void ExtractJsonString_MalformedUnicodeEscapeReturnsNull()
175+
{
176+
// \u followed by anything that isn't 4 hex digits is a
177+
// protocol error — bail rather than guess.
178+
var body = "{\"detail\":\"foo \\u4f6 bar\"}"; // only 3 hex digits
179+
Assert.Null(UploadClient.ExtractJsonString(body, "detail"));
180+
}
181+
182+
[Fact]
183+
public void ExtractJsonString_HandlesRawUtf8FromFastApi()
184+
{
185+
// The default FastAPI path: bytes-on-the-wire are raw UTF-8,
186+
// ReadBodyUtf8Async produces a string with the actual
187+
// characters (not escapes). Extractor passes them through
188+
// unchanged — this is the happy path that the field today.
189+
var body = "{\"discord_name\":\"你好\"}";
190+
Assert.Equal("你好", UploadClient.ExtractJsonString(body, "discord_name"));
191+
}
192+
125193
[Fact]
126194
public void ExtractJsonString_MissingFieldReturnsNull()
127195
{

0 commit comments

Comments
 (0)