Project flag fields as flat string arrays in canonical JSON

A flag-word spec entry now surfaces in canonical-doc as a flat sorted
`string[]` instead of a `{flags, flagsRaw}` wrapper. Each entry is either
a canonical slug for a named set bit or `bit<N>` for set bits the spec
table doesn't name. Canonical sort: named slugs alphabetical first, then
`bit<N>` numerically.

Why: the wrapper had a redundant key collision when the parent field
itself was named `flags` (e.g. `header.flags = {flags: [...], flagsRaw: ...}`),
the wrapper-vs-array split made named and unnamed bits diff differently,
and `flagsRaw` was hostile to hand-edits since toggling one bit required
recomputing the hex. The flat array gives uniform diffs across named and
unnamed bits, makes hand-edits one entry per bit, and preserves the
strict-disjoint invariant via per-element refines (a `bit<N>` whose
position falls inside the named-mask is rejected at both the schema
layer and the wire boundary, and `bit<N>` with N >= codec width is also
rejected).

`slugifyCodedName` rejects display strings whose slug would collide with
the `bit<N>` sentinel namespace, reserving it for the projection's
synthesized entries.

This is a wire-shape change for `*.pro.json` snapshots. Snapshots
produced by previous versions need to be re-saved through this version's
CLI before they can be loaded.

Author: burner1024

binary/INTERNALS.md

@@ -321,15 +321,15 @@ These are non-negotiable across PRO and MAP:
 5. **Linked structures.** Same-struct: array length drives count via `enforceLinkedCounts(spec, doc)` + zod refinement. Cross-struct (`fromCtx`): orchestrator owns the binding; the count flows in via the read-time ctx.
 6. **No work-time artifacts in the repo.** Exception: `tmp/` (in `.gitignore`).
 
-7. **Sorted-array projection for flag fields.** A flag-word spec entry (`{codec, flags: Table}`) surfaces in canonical-doc as a `{flags: string[], flagsRaw?: hexString}` strict-object wrapper, not as the raw int. `flags` lists the slugified-camelCase names of every set bit, alphabetically sorted; toggling one bit adds or removes one entry. `flagsRaw` is an optional hex-string reservoir for unnamed bits in the wire word, omitted when every set bit has a name. Strict-disjoint invariant: named bits never appear in `flagsRaw`, enforced at the wire boundary by `flagArrayToInt`. `compileFlagTable` slugifies the table's display strings to camelCase canonical keys (`"NoBlock"` -> `noBlock`); `intToFlagArray` / `flagArrayToInt` translate at the wire codec boundary via the `FlagArraySchema` wrapper.
+7. **Flat-array projection for flag fields.** A flag-word spec entry (`{codec, flags: Table}`) surfaces in canonical-doc as a flat sorted `string[]`, not as the raw int. Each entry is either a named slug (slugified-camelCase from the table's display string) or `bit<N>` (zero-based bit position) for set bits the table doesn't name. Canonical sort order: named slugs first alphabetically, then `bit<N>` in ascending bit position. Toggling one bit adds or removes one entry at its sorted position - same shape for named and unnamed bits, so diffs read uniformly. `compileFlagTable` slugifies display strings to camelCase canonical keys (`"NoBlock"` -> `noBlock`); `slugifyCodedName` rejects display strings whose slug would collide with the `bit<N>` sentinel namespace. `intToFlagArray` / `flagArrayToInt` translate at the wire codec boundary via the `FlagArraySchema` wrapper.
 
-    Slugified identifiers (rather than the raw display strings) are the canonical token shape because the construction API (`docs/todo.md`) surfaces flags as TS members typed against a literal-name union - identifier-shaped names get the canonical dot-trigger autocomplete with per-flag JSDoc visible inline, which a quoted-display-string union does not. Schema validation messages and JSON Schema `items.enum` autocomplete also benefit from identifier tokens (no spacing/casing ambiguities like "No LOS required" vs "No los required"). The display string remains the parsed-tree label; the slug is the toolchain token, with one translation point (label <-> slug) at the projection boundary.
+    Strict-disjoint invariant: a `bit<N>` entry whose position falls inside the named-mask is rejected at both the schema layer and the wire boundary - hand-edits must use the canonical slug for any spec-named bit. `bit<N>` with N >= codec width is also rejected at both layers, so synthesized entries cannot reference a bit past the wire word.
 
-    The wrapper-at-canonical-doc shape (`header.flags = {flags: [...], flagsRaw: "..."}`) keeps the spec -> canonical-doc orchestration one-key-per-spec-field; a flat sidecar `header.flagsRaw` would require a special case in `toZodSchema` to emit two parent keys per flag spec entry.
+    Slugified identifiers (rather than the raw display strings) are the canonical token shape because the construction API (`docs/todo.md`) surfaces flags as TS members typed against a literal-name union - identifier-shaped names get the canonical dot-trigger autocomplete with per-flag JSDoc visible inline, which a quoted-display-string union does not. Schema validation messages and JSON Schema `items.enum` autocomplete also benefit from identifier tokens (no spacing/casing ambiguities like "No LOS required" vs "No los required"). The display string remains the parsed-tree label; the slug is the toolchain token, with one translation point (label <-> slug) at the projection boundary.
 
-    This is a consistent application of rule #1 - `packedAs`+`bitRange` already exposes byte-packed sub-fields as peer scalar entries; the named-bit projection exposes bit-packed sub-fields the same way (the wire packs N independent semantic units into one int; canonical separates them).
+    This is a consistent application of rule #1 - `packedAs`+`bitRange` already exposes byte-packed sub-fields as peer scalar entries; the flat-array projection exposes bit-packed sub-fields the same way (the wire packs N independent semantic units into one int; canonical separates them into one entry per set bit).
 
-8. **Lossless reservoirs preserve unnamed bits.** Adding a name to a flag table is a non-breaking spec evolution: old snapshots load via the `flagsRaw` path, re-saving promotes the bit to its new name. `schemaVersion` does NOT bump for additive name changes; bumping is reserved for _re-interpretive_ spec changes (a previously-parsed field's meaning changes), which require explicit migration code in the snapshot codec. The byte round-trip invariant `serialize(parse(b)) === b` is the load-bearing property - every existing `*-roundtrip.test.ts` enforces it.
+8. **Lossless preservation of unnamed bits.** Adding a name to a flag table is a non-breaking spec evolution: old snapshots load via `bit<N>` entries, re-saving promotes the bit to its new slug. `schemaVersion` does NOT bump for additive name changes; bumping is reserved for _re-interpretive_ spec changes (a previously-parsed field's meaning changes), which require explicit migration code in the snapshot codec. The byte round-trip invariant `serialize(parse(b)) === b` is the load-bearing property - every existing `*-roundtrip.test.ts` enforces it.
 
 9. **Enums and PIDs stay numeric in canonical-doc by design.** Where flag fields project to sorted-array name lists (rule #7), enum and PID fields stay as raw integers - the diff-friendliness gain doesn't justify the complication. Half the enum fields drive dispatch (`objectType`, `subType`, `scriptType`, MAP `version` / `rotation` / `elevation`) and would force a conversion at every dispatch site if projected to strings; the rest produce diffs of the same line count whether named or numeric (`5 -> 0` vs `"Items" -> "Background"`), unlike flags where the diff is fundamentally lossy. The display layer's `enum` table resolves names for editor dropdowns and hover; the snapshot stays close to the wire.
 

binary/src/itm/schemas.ts

@@ -15,17 +15,17 @@ import { itmAbilitySpecAnnotated } from "./specs/ability.overrides";
 // Wire codecs use the *annotated* specs so flag fields project through
 // `intToFlagArray` / `flagArrayToInt` at the byte boundary - the
 // canonical-doc surface (which is built off the same annotated specs) sees
-// flags as sorted-array `{flags, flagsRaw?}` projections, matching what the
-// zod schema validates.
+// flags as flat sorted `string[]` projections (named slugs + `bit<N>`
+// sentinels), matching what the zod schema validates.
 export const itmHeaderSchema = toTypedBinarySchema(itmHeaderSpecAnnotated);
 export const itmAbilitySchema = toTypedBinarySchema(itmAbilitySpecAnnotated);
 export const effectSchema = toTypedBinarySchema(effectSpecAnnotated);
 
 // Re-export the data types projected from the *annotated* specs so flag
-// fields surface as `{flags: string[], flagsRaw?: string}` (the sorted-array
-// projection) rather than `number`. The bare-spec types in `./specs/header`
-// etc. remain the underlying source for the spread, but consumers (parser,
-// canonical reader/writer) should consume the annotated projection.
+// fields surface as `string[]` (the flat sorted-array projection) rather
+// than `number`. The bare-spec types in `./specs/header` etc. remain the
+// underlying source for the spread, but consumers (parser, canonical
+// reader/writer) should consume the annotated projection.
 import type { SpecData } from "../spec/types";
 
 export type ItmHeaderData = SpecData<typeof itmHeaderSpecAnnotated>;

binary/src/map/canonical-reader.ts

@@ -308,7 +308,7 @@ export function rebuildMapCanonicalDocument(parseResult: ParseResult): MapCanoni
         ...headerScalars,
         version: headerScalars.version >>> 0,
         filename: readString(headerGroup, "Filename"),
-        // `flags` is a sorted-array projection (`{flags, flagsRaw?}`) produced
+        // `flags` is a flat sorted-array projection (`string[]`) produced
         // by `walkGroup`; no signedness coercion applies.
         timestamp: headerScalars.timestamp >>> 0,
         defaultElevation: clampNumericValue(headerScalars.defaultElevation, "int32", {

binary/src/map/schemas.ts

@@ -23,12 +23,13 @@ export interface MapHeader {
     numLocalVars: number;
     scriptId: number;
     /**
-     * `flags` is the sorted-array projection produced by the wire codec
-     * (`FlagArraySchema`). Consumers that need a numeric mask use the
-     * `FlagArray` overload of `hasElevation` (see `./types.ts`); raw-int
-     * access goes through `flagArrayToInt(MapFlags, ...)` if needed.
+     * `flags` is the flat sorted-array projection produced by the wire
+     * codec (named slugs first, then `bit<N>` for unnamed set bits).
+     * Consumers that need a numeric mask use the `FlagArray` overload of
+     * `hasElevation` (see `./types.ts`); raw-int access goes through
+     * `flagArrayToInt(MapFlags, ..., 32)` if needed.
      */
-    flags: { flags: string[]; flagsRaw?: string };
+    flags: string[];
     darkness: number;
     numGlobalVars: number;
     mapId: number;

binary/src/map/types.ts

@@ -131,11 +131,11 @@ export const ObjectFlags: Record<number, string> = {
  * bitmask form is preserved as a fallback for callers that still hold a
  * raw int (e.g., low-level parsers).
  */
-export function hasElevation(flags: number | { flags: string[]; flagsRaw?: string }, elevation: number): boolean {
+export function hasElevation(flags: number | string[], elevation: number): boolean {
     if (typeof flags === "number") {
         return (flags & (0x2 << elevation)) === 0;
     }
     const key =
         elevation === 0 ? "skipElevation0Tiles" : elevation === 1 ? "skipElevation1Tiles" : "skipElevation2Tiles";
-    return !flags.flags.includes(key);
+    return !flags.includes(key);
 }

binary/src/pro/canonical-reader.ts

@@ -105,11 +105,10 @@ function readClampedFieldNumber(
 }
 
 /**
- * Read a flag-word field from the display tree and project it to the
- * sorted-array `{flags, flagsRaw?}` shape canonical-doc expects. Width is
- * hard-coded per call site to match the underlying spec codec - every PRO
- * flag word is u8 / u24 / u32 in the spec, mapped to the matching
- * `codecBitWidth` here.
+ * Read a flag-word field from the display tree and project it to the flat
+ * sorted `string[]` shape canonical-doc expects. Width is hard-coded per
+ * call site to match the underlying spec codec - every PRO flag word is
+ * u8 / u24 / u32 in the spec, mapped to the matching `codecBitWidth` here.
  */
 function readFlagArray(
     group: ParsedGroup,

binary/src/pro/index.ts

@@ -319,7 +319,7 @@ function parseCritter(data: CritterData): ParsedGroup[] {
     // flagsField. The dynamic-access subset for fieldsFromDefs is the
     // numeric-only view.
     const critterData = data as unknown as Record<string, number>;
-    const critterFlagsInt = flagArrayToInt(CritterFlags, data.critterFlags);
+    const critterFlagsInt = flagArrayToInt(CritterFlags, data.critterFlags, 32);
 
     return [
         group("Critter Properties", [

binary/src/spec/coded-projection.ts

@@ -69,9 +69,21 @@ export function slugifyCodedName(displayName: string): string {
             `slugifyCodedName: name "${displayName}" produced "${camel}" which is not a valid JS identifier`,
         );
     }
+    if (RESERVED_BIT_PATTERN.test(camel)) {
+        // `bit<N>` is the reserved sentinel for unnamed set bits in
+        // FlagArray projections (see `intToFlagArray`). Spec authors must
+        // not pick a display name whose slug collides with that namespace,
+        // since the projection cannot distinguish a spec-named `bit13`
+        // from "bit at position 13" on encode.
+        throw new Error(
+            `slugifyCodedName: name "${displayName}" produced reserved sentinel "${camel}"; flag display names must not slugify to bit<N>`,
+        );
+    }
     return camel;
 }
 
+const RESERVED_BIT_PATTERN = /^bit\d+$/;
+
 export interface FlagBitEntry {
     readonly key: string;
     readonly mask: number;
@@ -82,9 +94,9 @@ export interface FlagBitEntry {
  * Compile a flag table (`{[mask]: displayName}`) into a sorted entry list with
  * canonical keys, plus the OR'd `namedMask` covering every named bit.
  *
- * Sorted alphabetically by canonical key so the projected `flags` array
- * serialises in stable order - toggling one bit adds or removes one entry
- * at its alphabetical position, regardless of bit position.
+ * Sorted alphabetically by canonical key so the projected array serialises
+ * in stable order - toggling a named bit adds or removes one entry at its
+ * alphabetical position, regardless of bit position.
  *
  * Returns frozen entries to discourage in-place mutation.
  */
@@ -108,90 +120,109 @@ export function compileFlagTable(table: Readonly<Record<number, string>>): {
 }
 
 /**
- * Build a default array projection - empty `flags`, no `flagsRaw`. Used by
- * structural-edit transitions and as a default in test fixtures or
- * construction APIs.
+ * Sorted-array projection of a flag word. Each entry is one of:
+ *
+ * - A canonical (slugified-camelCase) key from the spec table, identifying
+ *   a named set bit (e.g. `lightThru`).
+ * - `bit<N>` where N is the zero-based bit position, identifying a set bit
+ *   the spec table doesn't name (e.g. `bit5`, `bit13`).
+ *
+ * Canonical sort order: named keys alphabetically first, then `bit<N>`
+ * entries in ascending bit-position order. Toggling one bit adds or removes
+ * exactly one entry at its sorted position - same shape for named and
+ * unnamed bits, so diffs read uniformly.
+ *
+ * Strict-disjoint invariant: a `bit<N>` entry is rejected at the wire
+ * boundary if `1 << N` falls inside the spec table's named-mask, since a
+ * hand-edit must use the canonical name for any spec-named bit. Encode
+ * also rejects N >= codecBitWidth (no synthetic bits past the wire word).
  */
-export function emptyFlagArray(_table: Readonly<Record<number, string>>): FlagArray {
-    return { flags: [] };
-}
+export type FlagArray = string[];
 
 /**
- * Sorted-array projection of a flag word - `flags` lists every set bit by its
- * canonical (slugified-camelCase) name, `flagsRaw` carries any wire bits the
- * spec table doesn't name as a hex string. Both fields are wire-shape:
- * `flags` order is alphabetical for stable diffs, `flagsRaw` is omitted in
- * the common case where every set bit has a name.
+ * Build a default flag-array projection (empty array). Used by
+ * structural-edit transitions and as a default in test fixtures or
+ * construction APIs.
  */
-export interface FlagArray {
-    flags: string[];
-    flagsRaw?: string;
+export function emptyFlagArray(_table: Readonly<Record<number, string>>): FlagArray {
+    return [];
 }
 
 /**
- * Project an integer flag word to a sorted array of slugified names. Each
- * named bit that's set contributes its canonical key; unnamed bits land in
- * `flagsRaw` as a lowercase hex string. `flagsRaw` is omitted when all set
- * bits are named.
+ * Project an integer flag word to a sorted FlagArray. Named set bits
+ * contribute their canonical key (alphabetical); unnamed set bits within
+ * the codec's bit width contribute `bit<N>` entries (numeric).
  *
- * `codecBitWidth` (8 / 16 / 24 / 32) masks `flagsRaw` to the wire width so
- * sign-extended bits a JS bit-OR might surface don't leak in.
+ * `codecBitWidth` (8 / 16 / 24 / 32) bounds the per-bit scan so sign-
+ * extended bits a JS bit-OR might surface don't leak in.
  */
 export function intToFlagArray(
     table: Readonly<Record<number, string>>,
     value: number,
     codecBitWidth: number,
 ): FlagArray {
     const { entries, namedMask } = compileFlagTable(table);
-    const flags: string[] = [];
+    const named: string[] = [];
     for (const entry of entries) {
-        if ((value & entry.mask) !== 0) flags.push(entry.key);
+        if ((value & entry.mask) !== 0) named.push(entry.key);
     }
     const codecMask = codecBitWidth >= 32 ? 0xffffffff : (1 << codecBitWidth) - 1;
     const reservoir = (value & ~namedMask & codecMask) >>> 0;
-    if (reservoir !== 0) {
-        return { flags, flagsRaw: `0x${reservoir.toString(16)}` };
+    const bits: string[] = [];
+    for (let i = 0; i < codecBitWidth; i++) {
+        if ((reservoir & (1 << i)) !== 0) bits.push(`bit${i}`);
     }
-    return { flags };
+    return [...named, ...bits];
 }
 
 /**
- * Pack a flag array back to an integer. Every name in `flags` contributes its
- * mask; `flagsRaw` (hex) ORs in. Throws on unknown names, duplicate names,
- * malformed `flagsRaw`, or a `flagsRaw` value overlapping a named bit
- * (strict-disjoint invariant - the hand-edit surface should not let the same
- * bit be specified twice).
+ * Pack a FlagArray back to an integer. Each entry contributes its bit(s):
+ * named keys via the spec table, `bit<N>` via `1 << N`. Throws on:
+ *
+ * - unknown names that match neither the table nor `bit<N>`,
+ * - duplicate entries,
+ * - `bit<N>` whose position overlaps a named-bit mask (strict-disjoint
+ *   invariant - the hand-edit surface should not let the same bit be
+ *   specified twice),
+ * - `bit<N>` with N >= codecBitWidth (no synthetic bits past the wire
+ *   word).
  */
-export function flagArrayToInt(table: Readonly<Record<number, string>>, projection: FlagArray): number {
+export function flagArrayToInt(
+    table: Readonly<Record<number, string>>,
+    projection: FlagArray,
+    codecBitWidth: number,
+): number {
     const { entries, namedMask } = compileFlagTable(table);
     const byKey = new Map(entries.map((entry) => [entry.key, entry.mask]));
     const seen = new Set<string>();
     let value = 0;
-    for (const name of projection.flags) {
+    for (const name of projection) {
         if (seen.has(name)) {
             throw new Error(`flagArrayToInt: duplicate flag name "${name}"`);
         }
         seen.add(name);
         const mask = byKey.get(name);
-        if (mask === undefined) {
+        if (mask !== undefined) {
+            value = (value | mask) >>> 0;
+            continue;
+        }
+        const bitMatch = RESERVED_BIT_PATTERN.exec(name);
+        if (!bitMatch) {
             throw new Error(`flagArrayToInt: unknown flag "${name}" (known: ${entries.map((e) => e.key).join(", ")})`);
         }
-        value = (value | mask) >>> 0;
-    }
-    if (projection.flagsRaw !== undefined) {
-        if (typeof projection.flagsRaw !== "string" || !/^0x[0-9a-f]+$/i.test(projection.flagsRaw)) {
-            throw new TypeError(
-                `flagArrayToInt: flagsRaw must be a hex string ("0x..."); got ${String(projection.flagsRaw)}`,
+        const position = Number(bitMatch[0].slice(3));
+        if (position >= codecBitWidth) {
+            throw new Error(
+                `flagArrayToInt: bit position ${position} exceeds codec width ${codecBitWidth} for "${name}"`,
             );
         }
-        const reservoir = Number.parseInt(projection.flagsRaw, 16);
-        if ((reservoir & namedMask) !== 0) {
-            const overlapHex = (reservoir & namedMask).toString(16);
+        const bitMask = (1 << position) >>> 0;
+        if ((bitMask & namedMask) !== 0) {
             throw new Error(
-                `flagArrayToInt: flagsRaw ${projection.flagsRaw} overlaps named-bit mask 0x${overlapHex}; named bits must be set via the flags array`,
+                `flagArrayToInt: bit position ${position} overlaps named-bit mask; named bits must be set by their canonical name, not "${name}"`,
             );
         }
-        value = (value | reservoir) >>> 0;
+        value = (value | bitMask) >>> 0;
     }
     return value;
 }