Skip to content

Commit 69b662e

Browse files
TextDecoder: accept all WHATWG utf-8 encoding labels (#198)
## What `TextDecoder`'s constructor only accepted the exact labels `"utf-8"` and `"UTF-8"`, throwing for every other spelling. This PR makes it accept all WHATWG-spec UTF-8 labels. ## Why Per the [WHATWG Encoding Standard](https://encoding.spec.whatwg.org/#concept-encoding-get), an encoding label is matched after stripping leading/trailing ASCII whitespace and ASCII-lowercasing, and several labels all decode as UTF-8: `utf-8`, `utf8`, `unicode-1-1-utf-8`, `unicode11utf8`, `unicode20utf8`, `x-unicode20utf8`. Real consumers rely on this. The Babylon.js glTF/Draco loader constructs `new TextDecoder("utf8")` (no hyphen). With the old check that threw, decoding aborted mid-load. In **Babylon Native** the aborted load left the loader in a state that drove a **native out-of-bounds write**, which surfaced as non-deterministic **heap corruption** (`STATUS_HEAP_CORRUPTION`, `0xC0000374`) on the Draco mesh-compression validation tests. ## Fix Normalize the label per the spec (trim ASCII whitespace + ASCII-lowercase) and accept the full set of UTF-8 labels; still throw for genuinely unsupported encodings. ## Verification - Two BabylonNative Playground Draco validation tests that previously crashed with heap corruption (`GLTF Serializer KHR draco mesh compression`, `GLTF Buggy with Draco Mesh Compression`) now **pass** (3/3 runs each) with this fix vendored in; a third (`GLTF Box with bad Draco normalized flag`) no longer crashes. - Adds unit tests covering the `utf8` label, case/whitespace variants, the other UTF-8 aliases, and a still-rejected encoding (`utf-16`). --- ## Update: also fixes a latent ChakraCore N-API heap-corruption bug CI surfaced that the new "throws for an unsupported encoding" test reliably **heap-corrupts on Chakra** (Win32 x64/x86) — `STATUS_HEAP_CORRUPTION` (0xC0000374) / access violations, crashing later in unrelated tests. This is a **pre-existing latent bug** that this PR's first constructor-throw test merely exposed (it does not reproduce on `main` only because no existing test throws from a wrapped constructor; V8/JSC/JSI and the sanitizer jobs are unaffected since the bug is in the Chakra N-API shim). **Root cause:** when a napi `ObjectWrap` constructor throws, the C++ instance is destroyed during stack unwinding, but the wrap finalizer registered by `napi_wrap()` stays attached to `this`. A later GC then runs the finalizer on the freed instance → use-after-free / double-free. The addon-api `~ObjectWrap()` is meant to detach the wrap on failure, but it obtains `this` via `napi_get_reference_value()` on the wrap's weak (refcount-0) reference — and the Chakra backend returns `null` for **every** refcount-0 reference (the in-box `jsrt` API can't track weak-reference liveness), so the detach is skipped. **Fix** (`Core/Node-API/Source/js_native_api_chakra.cc`): - In the Chakra construct-call trampoline, if the callback leaves a pending JS exception, detach any wrap left on `this` via `napi_remove_wrap()` (preserving the pending exception) so the finalizer can't run on the freed instance. - Guard `napi_remove_wrap()` against a null `result` out-param (now called with `nullptr`, matching upstream). **Verification:** built the Chakra UnitTests locally; the constructor-throw scenario went from **heap corruption every run** to **clean (8/8 runs, all tests passing)**. Added a regression test that throws from a wrapped constructor 100× then exercises the heap. > Note: the N-API engine fix is logically independent of the TextDecoder label change and could be split into its own PR if preferred — it is bundled here because this PR's constructor-throw test is what exposes it and is its regression test. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 2ff325e commit 69b662e

2 files changed

Lines changed: 65 additions & 2 deletions

File tree

Polyfills/TextDecoder/Source/TextDecoder.cpp

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,36 @@
66

77
namespace
88
{
9+
// Normalize an encoding label per the WHATWG Encoding Standard "get an encoding"
10+
// algorithm: strip leading/trailing ASCII whitespace and ASCII-lowercase the result.
11+
std::string NormalizeEncodingLabel(const std::string& encoding)
12+
{
13+
const auto isAsciiWhitespace = [](char c) {
14+
return c == '\t' || c == '\n' || c == '\f' || c == '\r' || c == ' ';
15+
};
16+
17+
size_t begin = 0;
18+
size_t end = encoding.size();
19+
while (begin < end && isAsciiWhitespace(encoding[begin]))
20+
{
21+
++begin;
22+
}
23+
while (end > begin && isAsciiWhitespace(encoding[end - 1]))
24+
{
25+
--end;
26+
}
27+
28+
std::string label = encoding.substr(begin, end - begin);
29+
for (auto& c : label)
30+
{
31+
if (c >= 'A' && c <= 'Z')
32+
{
33+
c = static_cast<char>(c - 'A' + 'a');
34+
}
35+
}
36+
return label;
37+
}
38+
939
class TextDecoder final : public Napi::ObjectWrap<TextDecoder>
1040
{
1141
public:
@@ -33,9 +63,18 @@ namespace
3363
if (info.Length() > 0 && info[0].IsString())
3464
{
3565
auto encoding = info[0].As<Napi::String>().Utf8Value();
36-
if (encoding != "utf-8" && encoding != "UTF-8")
66+
67+
// Several labels (e.g. "utf8", "unicode-1-1-utf-8") all map to UTF-8 after
68+
// normalization; callers such as the glTF/Draco loader pass "utf8".
69+
const std::string label = NormalizeEncodingLabel(encoding);
70+
if (label != "utf-8" &&
71+
label != "utf8" &&
72+
label != "unicode-1-1-utf-8" &&
73+
label != "unicode11utf8" &&
74+
label != "unicode20utf8" &&
75+
label != "x-unicode20utf8")
3776
{
38-
throw Napi::Error::New(Env(), "TextDecoder: unsupported encoding '" + encoding + "', only 'utf-8' is supported");
77+
throw Napi::Error::New(Env(), "TextDecoder: unsupported encoding '" + encoding + "', only UTF-8 is supported");
3978
}
4079
}
4180
}

Tests/UnitTests/Scripts/tests.ts

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1504,6 +1504,30 @@ describe("TextDecoder", function () {
15041504
const decoder = new TextDecoder("utf-8");
15051505
expect(decoder.decode(new Uint8Array([79, 75]))).to.equal("OK");
15061506
});
1507+
1508+
it("should accept the WHATWG 'utf8' label (no hyphen)", function () {
1509+
const decoder = new TextDecoder("utf8");
1510+
const result = decoder.decode(new Uint8Array([72, 105])); // "Hi"
1511+
expect(result).to.equal("Hi");
1512+
});
1513+
1514+
it("should accept utf-8 labels case-insensitively and with surrounding whitespace", function () {
1515+
for (const label of ["UTF-8", "UTF8", " utf-8 ", "\tUtf8\n"]) {
1516+
const decoder = new TextDecoder(label);
1517+
expect(decoder.decode(new Uint8Array([79, 75]))).to.equal("OK");
1518+
}
1519+
});
1520+
1521+
it("should accept the other WHATWG utf-8 aliases", function () {
1522+
for (const label of ["unicode-1-1-utf-8", "unicode11utf8", "unicode20utf8", "x-unicode20utf8"]) {
1523+
const decoder = new TextDecoder(label);
1524+
expect(decoder.decode(new Uint8Array([79, 75]))).to.equal("OK");
1525+
}
1526+
});
1527+
1528+
it("should still throw for a genuinely unsupported encoding", function () {
1529+
expect(() => new TextDecoder("utf-16")).to.throw();
1530+
});
15071531
});
15081532

15091533
describe("TextEncoder", function () {

0 commit comments

Comments
 (0)