Skip to content

Commit f0bf4f8

Browse files
brandonpaytonclaude
andcommitted
perl: run on the default/unset locale without a startup panic
musl's setlocale(LC_ALL,"") returns a positional ';'-separated composite (e.g. "C.UTF-8;C;C;C;C;C") when the categories differ, but perl-cross configured the wasm target with glibc's name=value assumption (PERL_LC_ALL_USES_NAME_VALUE_PAIRS). perl parsed musl's first field "C.UTF-8" as name=value, found no '=', and aborted at interpreter startup (exit 29) for every locale except literally LC_ALL=C -- making perl unusable with the default or any UTF-8 locale on Kandelo. build-perl.sh now patches the target config.h to perl's positional LC_ALL mode with musl's category order (CTYPE;NUMERIC;TIME;COLLATE;MONETARY;MESSAGES), the mechanism perl already ships for *BSD; get_category_index() maps each position to its internal index and a STATIC_ASSERT guards the count. Host miniperl (xconfig.h) is left matching the build machine's libc. build.toml revision 1->2 (output bytes change). Adds a Node + browser locale smoke and a porting-guide troubleshooting note. Verified on both kernel hosts: perl -e runs with an unset locale, LC_ALL=C, LC_ALL=C.UTF-8, LANG=C.UTF-8, LANG=en_US.UTF-8, and a disparate composite (all exit-29 panics before); built-in ${^UTF8LOCALE} confirms per-category parsing routes CTYPE correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent f433983 commit f0bf4f8

5 files changed

Lines changed: 370 additions & 1 deletion

File tree

docs/porting-guide.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -776,3 +776,20 @@ accepted. See [fork-instrumentation.md](fork-instrumentation.md).
776776
**Process hangs on read**: The fd might be in blocking mode waiting for data. Check that writers are properly closing their end of the pipe.
777777

778778
**Browser SharedArrayBuffer unavailable**: Ensure COOP/COEP headers are set. In production, the service worker handles this. In dev, Vite's config sets them.
779+
780+
**Locale panic / `setlocale(LC_ALL, "")` format mismatch**: Kandelo's libc is
781+
musl, whose `setlocale(LC_ALL, "")` returns a *positional*, `;`-separated
782+
composite of the per-category locales in category order
783+
(`CTYPE;NUMERIC;TIME;COLLATE;MONETARY;MESSAGES`) when the categories differ
784+
e.g. with no locale env set it returns `C.UTF-8;C;C;C;C;C`. This is POSIX-legal
785+
(the `LC_ALL` return string is unspecified and need only round-trip through
786+
`setlocale`), but it is *not* glibc's `LC_CTYPE=…;LC_NUMERIC=…` `name=value`
787+
form. A runtime that assumes the glibc formatoften because a cross-build
788+
tool defaulted to itwill mis-parse musl's output and can abort at startup.
789+
Perl 5.40 hit exactly this: perl-cross configured the target with
790+
`PERL_LC_ALL_USES_NAME_VALUE_PAIRS`, so perl parsed `C.UTF-8` as `name=value`,
791+
found no `=`, and panicked (`packages/registry/perl/build-perl.sh` now patches
792+
the target `config.h` to perl's positional mode with musl's category order; see
793+
kd-dvph). If you port another locale-aware runtime and see startup failures
794+
tied to `LC_ALL`, check whether it assumes the glibc composite format and point
795+
it at musl's positional notation instead of forcing `LC_ALL=C`.

packages/registry/perl/build-perl.sh

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -503,6 +503,72 @@ if [ ! -f config.sh ]; then
503503
Makefile
504504
fi
505505

506+
# --- Fix the default-locale startup panic (kd-dvph) ---
507+
#
508+
# musl's setlocale(LC_ALL,"") returns a POSITIONAL, ';'-separated composite of
509+
# the per-category locale names in musl category order
510+
# (CTYPE;NUMERIC;TIME;COLLATE;MONETARY;MESSAGES) whenever the categories are not
511+
# all identical -- e.g. with no locale env set it returns the default
512+
# "C.UTF-8;C;C;C;C;C" (musl gives LC_CTYPE a UTF-8 C locale, the rest plain C).
513+
# This is POSIX-legal: the LC_ALL return string is unspecified and need only
514+
# round-trip through setlocale, which musl's does.
515+
#
516+
# perl-cross cross-configures the TARGET with glibc's assumption
517+
# (d_perl_lc_all_uses_name_value_pairs=define), so perl compiles out its
518+
# positional LC_ALL parser and parses musl's first field "C.UTF-8" as a glibc
519+
# "name=value" pair. There is no '=', so perl panics during the startup locale
520+
# scan (locale.c "needs an '=' to split name=value") and aborts with exit 29.
521+
# The effect is that perl is unusable unless LC_ALL is forced to a single value
522+
# like "C" -- even LC_ALL=C.UTF-8 or LANG=... trips it.
523+
#
524+
# Fix the TARGET config header. In perl-cross, config.h is the primary
525+
# (cross-compiled perl.wasm) config -- Makefile builds target objects with
526+
# `%$o: %.c config.h` using the cross compiler -- while xconfig.h configures the
527+
# build-time miniperl that runs on the build machine (`%$O: %.c xconfig.h`,
528+
# HOSTCC). So patch config.h (target) and leave xconfig.h (host miniperl)
529+
# matching the build machine's libc.
530+
# This is the mechanism perl already uses for positional platforms (*BSD):
531+
# - undef PERL_LC_ALL_USES_NAME_VALUE_PAIRS (musl is positional, not name=value)
532+
# - define PERL_LC_ALL_SEPARATOR ";"
533+
# - define PERL_LC_ALL_CATEGORY_POSITIONS_INIT to the LC_* categories in
534+
# musl's emit order; perl's get_category_index() maps each to its internal
535+
# index and a compile-time STATIC_ASSERT checks the count == LC_ALL_INDEX_.
536+
# Idempotent + applied on every build so incremental rebuilds can't ship the
537+
# unpatched interpreter.
538+
echo "==> Patching config.h for musl positional LC_ALL notation (kd-dvph)..."
539+
python3 - "$SRC_DIR/config.h" << 'PYLOCALE'
540+
import re, sys
541+
542+
path = sys.argv[1]
543+
with open(path) as f:
544+
original = f.read()
545+
546+
def sub_macro(content, macro, replacement):
547+
# Rewrite the single '#define'/'#undef' line for `macro` (whether currently
548+
# defined or emitted as the commented "/*#define MACRO */" template form),
549+
# leaving the doc-comment header (which has no #define/#undef) untouched.
550+
pat = re.compile(r'^.*#\s*(?:define|undef)\s+' + re.escape(macro) + r'\b.*$', re.M)
551+
new, n = pat.subn(lambda _m: replacement, content)
552+
if n != 1:
553+
sys.stderr.write("kd-dvph WARNING: %s matched %d lines (expected 1)\n" % (macro, n))
554+
return new
555+
556+
content = original
557+
content = sub_macro(content, "PERL_LC_ALL_USES_NAME_VALUE_PAIRS",
558+
"#undef PERL_LC_ALL_USES_NAME_VALUE_PAIRS\t/* kd-dvph: musl uses positional LC_ALL */")
559+
content = sub_macro(content, "PERL_LC_ALL_SEPARATOR",
560+
'#define PERL_LC_ALL_SEPARATOR ";"\t/**/')
561+
content = sub_macro(content, "PERL_LC_ALL_CATEGORY_POSITIONS_INIT",
562+
"#define PERL_LC_ALL_CATEGORY_POSITIONS_INIT { LC_CTYPE, LC_NUMERIC, LC_TIME, LC_COLLATE, LC_MONETARY, LC_MESSAGES }\t/**/")
563+
564+
if content != original:
565+
with open(path, 'w') as f:
566+
f.write(content)
567+
print("Patched config.h for musl positional LC_ALL notation (kd-dvph)")
568+
else:
569+
print("config.h already patched for musl positional LC_ALL notation (kd-dvph)")
570+
PYLOCALE
571+
506572
# --- Build ---
507573
echo "==> Building Perl (this takes a while)..."
508574
make -j"$(sysctl -n hw.ncpu 2>/dev/null || nproc)" perl 2>&1 | tee "$SCRIPT_DIR/build.log" | tail -80

packages/registry/perl/build.toml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
script_path = "packages/registry/perl/build-perl.sh"
22
repo_url = "https://github.com/brandonpayton/kandelo.git"
33
commit = "8c53383229fab78f97b098c3207a655159c03041"
4-
revision = 1
4+
# rev2: build-perl.sh patches the target config.h to musl's positional LC_ALL
5+
# notation, fixing the default-locale startup panic (kd-dvph). Output bytes
6+
# change (locale.o + relink), so the archive cache must invalidate.
7+
revision = 2
58

69
[binary]
710
index_url = "https://github.com/Automattic/kandelo/releases/download/binaries-abi-v{abi}/index.toml"
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
/**
2+
* locale-browser-smoke.ts — kd-dvph Perl default-locale startup smoke in a real
3+
* browser (headless Chromium via Playwright + the browser-demos test-runner
4+
* page, which drives BrowserKernel).
5+
*
6+
* Confirms the fix is host-agnostic: the same perl.wasm that no longer panics
7+
* under the Node kernel host also starts cleanly under the browser kernel host
8+
* with the browser's default locale env. Runs `perl -e 'print "R=",2+3'` with:
9+
* - no locale env (the disparate musl composite that used to panic), and
10+
* - LANG=en_US.UTF-8 (the exact env live-setup.ts / browser-kernel-host.ts
11+
* pass in the browser UI).
12+
*
13+
* Usage (needs playwright + a chromium build available):
14+
* npx tsx packages/registry/perl/demo/locale-browser-smoke.ts
15+
*/
16+
import { readFileSync, existsSync } from "node:fs";
17+
import { resolve, dirname } from "node:path";
18+
import { spawn, type ChildProcess } from "node:child_process";
19+
import { chromium, type Browser, type Page } from "playwright";
20+
21+
const scriptDir = dirname(new URL(import.meta.url).pathname);
22+
const repoRoot = resolve(scriptDir, "../../../..");
23+
const BROWSER_DIR = resolve(repoRoot, "apps/browser-demos");
24+
const VITE_PORT = 5207;
25+
const ARITH = 'print "R=", 2 + 3, "\\n"';
26+
27+
interface Case { name: string; env: string[]; }
28+
const CASES: Case[] = [
29+
{ name: "unset-locale (no LC_ALL/LANG)", env: [] },
30+
{ name: "LANG=en_US.UTF-8 (browser default)", env: ["LANG=en_US.UTF-8"] },
31+
];
32+
33+
function startVite(): Promise<ChildProcess> {
34+
return new Promise((resolvePromise, reject) => {
35+
const proc = spawn(
36+
"npx",
37+
["vite", "--config", resolve(BROWSER_DIR, "vite.config.ts"), "--port", String(VITE_PORT)],
38+
{ cwd: BROWSER_DIR, stdio: ["ignore", "pipe", "pipe"], env: { ...process.env } },
39+
);
40+
let started = false;
41+
const timer = setTimeout(() => { if (!started) { proc.kill(); reject(new Error("Vite did not start in 60s")); } }, 60_000);
42+
proc.stdout!.on("data", (d: Buffer) => {
43+
if (!started && d.toString().includes("Local:")) {
44+
started = true; clearTimeout(timer); setTimeout(() => resolvePromise(proc), 500);
45+
}
46+
});
47+
proc.on("exit", (code) => { if (!started) { clearTimeout(timer); reject(new Error(`Vite exited ${code}`)); } });
48+
});
49+
}
50+
51+
async function runCase(page: Page, perlBytes: Buffer, c: Case) {
52+
return page.evaluate(
53+
async ({ bytes, argv, env }) => {
54+
const ab = new Uint8Array(bytes).buffer;
55+
return await (window as any).__runTest(ab, argv, 60_000, { env });
56+
},
57+
{ bytes: Array.from(perlBytes), argv: ["perl", "-e", ARITH], env: c.env },
58+
);
59+
}
60+
61+
async function main() {
62+
const perlWasm = resolve(repoRoot, "packages/registry/perl/bin/perl.wasm");
63+
if (!existsSync(perlWasm)) {
64+
console.error("perl.wasm not found. Run: bash packages/registry/perl/build-perl.sh");
65+
process.exit(1);
66+
}
67+
const perlBytes = readFileSync(perlWasm);
68+
69+
let vite: ChildProcess | undefined;
70+
let browser: Browser | undefined;
71+
const passed: string[] = [];
72+
const failed: string[] = [];
73+
try {
74+
vite = await startVite();
75+
browser = await chromium.launch();
76+
const page = await browser.newPage();
77+
await page.goto(`http://localhost:${VITE_PORT}/pages/test-runner/`);
78+
await page.waitForFunction(() => (window as any).__testRunnerReady === true, {}, { timeout: 60_000 });
79+
80+
for (const c of CASES) {
81+
let r: any;
82+
try {
83+
r = await runCase(page, perlBytes, c);
84+
} catch (err) {
85+
failed.push(`${c.name}: threw ${String(err)}`);
86+
continue;
87+
}
88+
const ok = r.exitCode === 0 && (r.stdout || "").includes("R=5");
89+
const detail = `exit=${r.exitCode} stdout=${JSON.stringify((r.stdout || "").trim())} stderr=${JSON.stringify((r.stderr || "").trim())}`;
90+
(ok ? passed : failed).push(`${c.name}: ${detail}`);
91+
}
92+
} finally {
93+
if (browser) await browser.close();
94+
if (vite) vite.kill();
95+
}
96+
97+
console.log("=== PASSED (browser) ===");
98+
for (const p of passed) console.log(" " + p);
99+
if (failed.length) { console.log("=== FAILED (browser) ==="); for (const f of failed) console.log(" " + f); }
100+
const ok = failed.length === 0 && passed.length === CASES.length;
101+
console.log(ok ? "PERL_LOCALE_BROWSER_SMOKE_PASS" : "PERL_LOCALE_BROWSER_SMOKE_FAIL");
102+
process.exit(ok ? 0 : 1);
103+
}
104+
105+
main().catch((err) => { console.error("Fatal error:", err); process.exit(1); });
Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
/**
2+
* locale-smoke.ts — kd-dvph Perl default/locale startup smoke on Kandelo.
3+
*
4+
* Regression guard for the default-locale startup panic: perl 5.40 built for
5+
* the wasm32 musl target used to abort at interpreter startup when no locale
6+
* env was set, because perl-cross defaulted the target to glibc's
7+
* "cat=value;cat=value" LC_ALL notation while Kandelo's libc (musl) returns a
8+
* POSITIONAL ";"-separated composite ("C.UTF-8;C;C;C;C;C"). perl parsed the
9+
* first field "C.UTF-8" as name=value, found no '=', and panicked (exit 29).
10+
*
11+
* Each case spawns the built perl.wasm under the Node kernel host with a
12+
* controlled guest environment and asserts a clean run (exit 0 + expected
13+
* stdout). The empty-env case is the primary regression; the disparate case
14+
* exercises the exact multi-component positional path that used to panic.
15+
*
16+
* If a Perl runtime tree (PERL5LIB with POSIX.pm/XS, e.g. kd-k7zy's
17+
* perl-runtime) is provided as argv[2], an extra case round-trips
18+
* POSIX::setlocale(LC_ALL) to verify per-category parsing; otherwise that case
19+
* is skipped (baseline `make perl` ships no modules).
20+
*
21+
* Usage:
22+
* bash build.sh && bash packages/registry/perl/build-perl.sh
23+
* npx tsx packages/registry/perl/demo/locale-smoke.ts [PERL5LIB_DIR]
24+
*/
25+
import { existsSync } from "fs";
26+
import { resolve, dirname } from "path";
27+
import { runCentralizedProgram } from "../../../../host/test/centralized-test-helper";
28+
import { NodePlatformIO } from "../../../../host/src/platform/node";
29+
30+
const scriptDir = dirname(new URL(import.meta.url).pathname);
31+
const repoRoot = resolve(scriptDir, "../../../..");
32+
33+
interface Case {
34+
name: string;
35+
env: string[];
36+
argv: string[];
37+
expect: string;
38+
optional?: boolean;
39+
}
40+
41+
// A trivial arithmetic program that loads no modules, so it works against the
42+
// baseline `make perl` build (no staged runtime). The only thing under test is
43+
// that the interpreter reaches its body instead of panicking during the
44+
// startup locale scan.
45+
const ARITH = 'print "R=", 2 + 3, "\\n"';
46+
47+
function buildCases(perl5lib?: string): Case[] {
48+
const cases: Case[] = [
49+
// PRIMARY regression: no locale env at all -> musl returns the disparate
50+
// positional composite "C.UTF-8;C;C;C;C;C"; used to panic (exit 29).
51+
{ name: "unset-locale (no LC_ALL/LANG)", env: [], argv: ["perl", "-e", ARITH], expect: "R=5" },
52+
{ name: "LC_ALL=C", env: ["LC_ALL=C"], argv: ["perl", "-e", ARITH], expect: "R=5" },
53+
{ name: "LC_ALL=C.UTF-8", env: ["LC_ALL=C.UTF-8"], argv: ["perl", "-e", ARITH], expect: "R=5" },
54+
{ name: "LANG=C.UTF-8", env: ["LANG=C.UTF-8"], argv: ["perl", "-e", ARITH], expect: "R=5" },
55+
// Browser demos default the guest env to LANG=en_US.UTF-8 (live-setup.ts,
56+
// browser-kernel-host.ts). musl maps that to a disparate composite too, so
57+
// this covers the exact env the browser host passes.
58+
{ name: "LANG=en_US.UTF-8 (browser default)", env: ["LANG=en_US.UTF-8"], argv: ["perl", "-e", ARITH], expect: "R=5" },
59+
// Explicitly disparate categories -> forces musl's multi-field positional
60+
// composite; this is the exact code path that panicked, now positionally
61+
// parsed. If the positional category map were wrong the interpreter would
62+
// mis-route or NULL-deref a category during startup.
63+
{
64+
name: "disparate (LC_CTYPE=C.UTF-8, others C)",
65+
env: ["LC_CTYPE=C.UTF-8", "LC_NUMERIC=C", "LC_TIME=C", "LC_COLLATE=C", "LC_MONETARY=C", "LC_MESSAGES=C"],
66+
argv: ["perl", "-e", ARITH],
67+
expect: "R=5",
68+
},
69+
// Category-mapping correctness (no modules needed): perl's built-in
70+
// ${^UTF8LOCALE} is true iff the *LC_CTYPE* startup locale is UTF-8. In a
71+
// disparate composite it must reflect field 0 (CTYPE) only, so these prove
72+
// the positional map routes CTYPE to the right slot rather than merely not
73+
// panicking. If the order were wrong, CTYPE would pick up a different
74+
// field's value and these would flip.
75+
{
76+
name: "category map: CTYPE=C.UTF-8 disparate -> UTF8LOCALE=1",
77+
env: ["LC_CTYPE=C.UTF-8", "LC_NUMERIC=C", "LC_TIME=C", "LC_COLLATE=C", "LC_MONETARY=C", "LC_MESSAGES=C"],
78+
argv: ["perl", "-e", 'print "U=", (${^UTF8LOCALE} ? 1 : 0), "\\n"'],
79+
expect: "U=1",
80+
},
81+
{
82+
name: "category map: LC_ALL=C -> UTF8LOCALE=0",
83+
env: ["LC_ALL=C"],
84+
argv: ["perl", "-e", 'print "U=", (${^UTF8LOCALE} ? 1 : 0), "\\n"'],
85+
expect: "U=0",
86+
},
87+
];
88+
89+
if (perl5lib) {
90+
// With a runtime tree, verify per-category parsing by round-tripping the
91+
// composite through POSIX::setlocale under the disparate env above. This
92+
// fails loudly if the positional order/count is wrong.
93+
const PROG = [
94+
"use POSIX qw(setlocale LC_ALL LC_CTYPE LC_NUMERIC);",
95+
'my $all = setlocale(LC_ALL);',
96+
'my $ctype = setlocale(LC_CTYPE);',
97+
'my $num = setlocale(LC_NUMERIC);',
98+
'print "LC_ALL=$all\\nLC_CTYPE=$ctype\\nLC_NUMERIC=$num\\n";',
99+
'print "POSIX_OK\\n" if $ctype =~ /UTF-?8/i && $num eq "C";',
100+
].join(" ");
101+
cases.push({
102+
name: "POSIX::setlocale round-trip (disparate)",
103+
env: [
104+
`PERL5LIB=${perl5lib}`,
105+
"LC_CTYPE=C.UTF-8",
106+
"LC_NUMERIC=C",
107+
"LC_TIME=C",
108+
"LC_COLLATE=C",
109+
"LC_MONETARY=C",
110+
"LC_MESSAGES=C",
111+
],
112+
argv: ["perl", "-e", PROG],
113+
expect: "POSIX_OK",
114+
optional: true,
115+
});
116+
}
117+
return cases;
118+
}
119+
120+
async function main() {
121+
const perlWasm = resolve(repoRoot, "packages/registry/perl/bin/perl.wasm");
122+
const perl5lib = process.argv[2];
123+
if (!existsSync(perlWasm)) {
124+
console.error("perl.wasm not found. Run: bash packages/registry/perl/build-perl.sh");
125+
process.exit(1);
126+
}
127+
128+
const passed: string[] = [];
129+
const failed: string[] = [];
130+
const skipped: string[] = [];
131+
132+
for (const c of buildCases(perl5lib)) {
133+
let result;
134+
try {
135+
result = await runCentralizedProgram({
136+
programPath: perlWasm,
137+
argv: c.argv,
138+
env: c.env,
139+
io: new NodePlatformIO(),
140+
timeout: 60_000,
141+
});
142+
} catch (err) {
143+
failed.push(`${c.name}: threw ${String(err)}`);
144+
continue;
145+
}
146+
const ok = result.exitCode === 0 && result.stdout.includes(c.expect);
147+
const detail = `exit=${result.exitCode} stdout=${JSON.stringify(result.stdout.trim())}`;
148+
if (ok) {
149+
passed.push(`${c.name}: ${detail}`);
150+
} else if (c.optional && result.exitCode !== 0) {
151+
// Optional cases (need runtime modules) are skipped, not failed, when the
152+
// interpreter can't load the module — the required cases still gate.
153+
skipped.push(`${c.name}: ${detail} stderr=${JSON.stringify(result.stderr.trim())} (runtime module unavailable)`);
154+
} else {
155+
failed.push(`${c.name}: ${detail} stderr=${JSON.stringify(result.stderr.trim())}`);
156+
}
157+
}
158+
159+
console.log("=== PASSED ===");
160+
for (const p of passed) console.log(" " + p);
161+
if (skipped.length) {
162+
console.log("=== SKIPPED ===");
163+
for (const s of skipped) console.log(" " + s);
164+
}
165+
if (failed.length) {
166+
console.log("=== FAILED ===");
167+
for (const f of failed) console.log(" " + f);
168+
}
169+
170+
const allRequiredPass = failed.length === 0;
171+
console.log(allRequiredPass ? "PERL_LOCALE_SMOKE_PASS" : "PERL_LOCALE_SMOKE_FAIL");
172+
process.exit(allRequiredPass ? 0 : 1);
173+
}
174+
175+
main().catch((err) => {
176+
console.error("Fatal error:", err);
177+
process.exit(1);
178+
});

0 commit comments

Comments
 (0)