fix(js-exec): honor encoding arg in Buffer.from / toString / write / byteLength

[Hazzng]

Problem

The Buffer shim inside the QuickJS-backed js-exec runtime ignored the encoding argument on every encoding-aware method. All calls silently fell through to UTF-8, so base64, hex, latin1, ascii, and utf16le were broken. Encode + decode tests masked this because both sides were no-ops.

Failing cases (before fix)

Run inside js-exec (QuickJS), compared against real Node:

Expression	Before (broken)	After (Node-correct)
Buffer.from([0xff,0xfe,0x00]).toString('base64')	""	"//4A"
Buffer.from([0xff,0xfe,0x00]).toString('hex')	""	"fffe00"
Buffer.from('//4A', 'base64')	<Buffer 2f 2f 34 41> (raw utf8 of the string)	<Buffer ff fe 00>
Buffer.from('68656c6c6f', 'hex').toString()	"68656c6c6f"	"hello"
Buffer.byteLength('aGVsbG8=', 'base64')	8 (utf8 length of input string)	5
Buffer.from('hello', 'latin1').toString('latin1')	worked only because both sides no-op'd	matches Node, also works for non-ASCII (0x80–0xff)

Root cause: every entry point built the byte array via _utf8Encode(str) regardless of the encoding argument; toString symmetrically decoded as UTF-8.

Solution

Added pure-JS encode/decode helpers inside BUFFER_MODULE_SOURCE (the IIFE that runs in QuickJS where TextEncoder/atob/btoa are unavailable), and wired them into the four entry points:

Method	What changed
Buffer.from(str, encoding)	dispatches to the right encoder
buf.toString(encoding)	dispatches to the right decoder
Buffer.byteLength(str, encoding)	returns decoded byte count, not UTF-8 length
buf.write(str, offset, length, encoding)	full Node overload signature

Encodings supported: utf8, base64, base64url, hex, latin1/binary, ascii, utf16le/ucs2. Node's lenient parsing is matched (base64 ignores whitespace + missing padding; hex stops at first invalid char). Unknown encodings throw TypeError.

After fix — verified against node -e

$ node ./dist/cli/just-bash.js -c 'js-exec <<<"
  const b = Buffer.from([0xff,0xfe,0x00]);
  console.log(b.toString(\"base64\"));      // //4A
  console.log(b.toString(\"hex\"));         // fffe00
  console.log(Buffer.from(\"//4A\",\"base64\")); // <Buffer ff fe 00>
  console.log(Buffer.from(\"68656c6c6f\",\"hex\").toString()); // hello
  console.log(Buffer.byteLength(\"aGVsbG8=\",\"base64\"));     // 5
"' --root .
//4A
fffe00
<Buffer ff fe 00>
hello
5

Byte-for-byte match with node -e running the same snippet.

Tests

New file js-exec.buffer-encoding.test.ts — 29 cases:

• Encode-only (assert exact output constant, not round-trip)
• Decode-only
• Binary bytes (0xff, 0xfe, 0x00) — these were the broken path
• Alias names (binary, ucs2, utf-8)
• Forgiving inputs (missing base64 padding, odd-length hex, invalid hex char)
• Unknown encoding throws

Existing js-exec.node-compat.test.ts (120 tests) stays green — no regressions.

Verification

pnpm exec vitest run --config vitest.wasm.config.ts src/commands/js-exec/js-exec.buffer-encoding.test.ts
# 29/29 passed

pnpm exec vitest run --config vitest.wasm.config.ts src/commands/js-exec/js-exec.node-compat.test.ts
# 120/120 passed

pnpm typecheck && pnpm lint:fix && pnpm knip
# all clean

Manual spot-checks against node -e confirmed byte-for-byte match for aGVsbG8=, hello, //4A, 68656c6c6f, ff00ab, and byteLength = 5.

[vercel]

@Hazzng is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[Hazzng]

@cramforce i just did some fixes for The Buffer shim inside the js-exec runtime (encoding issues). Would you mind having a look please ? Thanks

[cramforce]

From codex review:

• Findings:

• packages/just-bash/src/commands/js-exec/module-shims.ts:448 regresses Buffer.from(ArrayBuffer) sharing. This now wraps the ArrayBuffer in a Uint8Array, then new Buffer(Uint8Array) copies it.
  Node’s Buffer.from(arrayBuffer[, offset[, length]]) shares the backing store, and the previous no-offset path did too. Mutations through the original Uint8Array or through the Buffer will no
  longer reflect on the other side.
• packages/just-bash/src/commands/js-exec/module-shims.ts:545 silently accepts invalid write() lengths. Node throws RangeError for negative lengths and for lengths greater than the remaining buffer
  capacity; this code clamps or can even return a negative count. Examples: Buffer.alloc(2).write("abc", 0, 3) should throw, but this writes 2 bytes; write("abc", 0, -1) returns -1.
• packages/just-bash/src/commands/js-exec/module-shims.ts:502 uses Uint8Array.subarray directly for toString(..., start, end), so negative end is treated relative to the end of the buffer. Node
  clamps negative end to 0, so Buffer.from("abc").toString("utf8", 0, -1) should return ""; this returns "ab".

I did not run the project test suite. I only used local Node snippets to verify the expected Node behavior for these edge cases.

[Hazzng]

@cramforce are there any updates on this fix as i kinda need this. Thanks