feat(container): honor --spaces for legible-text containers (flagless crc-probe)

--spaces was silently ignored in -C/--container mode. Now literal spaces are
kept in the .pbf.json `data` value so text (e.g. Markdown) reads naturally
inside valid single-line JSON (letters/digits/. @ ^ _ already pass through;
newlines stay ¶ glyphs). No schema flag and version stays 1: the crc32_encoded
oracle disambiguates spaces-as-data from transport noise via a keep-then-strip
probe. An ambiguity warning fires when literal spaces coexist with the space
glyph (formatting injected into a non-spaces container); the glyph is read from
the map, not hardcoded. --tabs/--crlf/-w/--preserve + -C now hard-error.

Applied across all 5 container surfaces (Lua, Node, Zig, standalone-C, FFI-C)
and the shared test/test_container guard. Also: documented the printable-ASCII
passthrough in character_map.txt (verified behavior-neutral); fixed an FFI
NULL-deref (preserve_chars is a char* there, not an array).

All 17 aarch64-darwin nix checks green.

Author: pmarreck

PLAN.md

@@ -6,6 +6,32 @@ maintained_by: agent
 
 # PLAN — issue #1: decode workflow + printable-binary-file.json
 
+## Container honors `--spaces` (legible-markdown containers) — DONE (2026-07-21 EDT)
+
+Goal (Peter): put a markdown file into a `.pbf.json` container but keep it legible;
+`--spaces` was silently ignored in container mode. Fixed across all 5 container
+surfaces (Lua, Node, Zig, standalone-C, FFI-C) + shared `test/test_container`.
+
+- [x] `--spaces` honored in container encode: literal spaces in the JSON `data`
+      value (letters/digits/`. @ ^ _` already pass through, so markdown reads
+      naturally; newlines stay `¶` glyphs → still valid single-line JSON). (2026-07-21 03:00 PM EDT)
+- [x] **Flagless** design (Peter's call): NO `spaces` schema field, `version` stays 1.
+      Decode uses a **crc-probe** — try keeping literal spaces; if `crc32_encoded`
+      mismatches, strip them as transport noise. The crc is the disambiguation oracle. (2026-07-21 03:00 PM EDT)
+- [x] Ambiguity warning: a non-spaces container (space-glyph present) with a
+      transport-injected literal space → recover by stripping + WARN that literal
+      spaces were assumed formatting because the space glyph was also present.
+      Space glyph read from the map (DRY), never hardcoded. (2026-07-21 03:00 PM EDT)
+- [x] `--tabs`/`--crlf`/`-w`/`--preserve` + `--container` = HARD ERROR (would break
+      JSON-validity + transport-resistance). (2026-07-21 03:00 PM EDT)
+- [x] Corrected `character_map.txt` header comment: documents the 66/94 printable-ASCII
+      passthrough (the legibility property), verified behavior-neutral. (2026-07-21 03:00 PM EDT)
+- [x] Guard #4 (map redefines space→space) proven UNNECESSARY: the "first
+      whitespace-delimited token" parser + 256-glyph-count check make a literal-space
+      glyph structurally unloadable (physics over policy). (2026-07-21 03:00 PM EDT)
+- [x] Fixed FFI NULL-deref: `preserve_chars` is `char *` (NULL) in FFI opts, not an
+      array — guard now NULL-checks. Caught by hermetic nix check, not exit code. (2026-07-21 03:00 PM EDT)
+
 ## Docs and benchmark parity — active (2026-07-17 EDT)
 
 - [x] Document every supported implementation—especially Rust, WebAssembly, and

README.md

@@ -207,6 +207,25 @@ paste) and restores the file under its original name. Optional metadata
 (timestamps, POSIX permissions/owner) is included when the tool can read it and
 omitted otherwise; decode never fails on a missing optional field.
 
+**Legible-text containers with `-s`/`--spaces`.** Add `--spaces` to keep literal
+spaces in the `data` value instead of encoding them to the space glyph. Since
+letters, digits and `. @ ^ _` already pass through as themselves, a text file —
+e.g. Markdown — reads naturally inside the container while staying valid,
+single-line JSON (newlines remain `¶` glyphs):
+
+```bash
+./bin/printable-binary -C --spaces notes.md > notes.md.pbf.json
+# data value reads like:  "♯ Title¶¶Some ⁎⁎bold⁎⁎ text.¶˗ a list item¶"
+```
+
+No schema flag records this — the format stays `version: 1`. Decode disambiguates
+with the `crc32_encoded` oracle: it keeps literal spaces if they check out as
+data, otherwise strips them as transport formatting (and warns if the space glyph
+was also present, i.e. the spaces were injected into a non-`--spaces` container).
+`--spaces` is the only preserve flag allowed with `-C`; `--tabs`/`--crlf`/`-w`/
+`--preserve` are rejected, since raw tab/CR/LF would break both JSON validity and
+the whitespace-stripping transport-resistance.
+
 ### Hexlike Mode (`-X`)
 
 `-X`/`--hexlike` is a hybrid view: printable ASCII passes through untouched while

bin/printable-binary

@@ -21,8 +21,13 @@ local function crc32_hex(data)
   return bit.tohex(bit.bxor(crc, 0xFFFFFFFF))
 end
 
--- Strip transport whitespace (space/tab/CR/LF) from an encoded payload.
-local function canonical_payload(s)
+-- Strip transport whitespace from an encoded payload. Normally all of space/tab/
+-- CR/LF are transport noise. When keep_spaces is set (container --spaces), literal
+-- spaces are DATA, so only tab/CR/LF are stripped as noise.
+local function canonical_payload(s, keep_spaces)
+  if keep_spaces then
+    return (s:gsub("[\t\r\n]", ""))
+  end
   return (s:gsub("[ \t\r\n]", ""))
 end
 
@@ -32,8 +37,6 @@ end
 local function json_get_string(json, key)
   return json:match('"' .. key .. '"%s*:%s*"(.-)"')
 end
-
-
 -- PrintableBinary: A utility to encode/decode binary data into human-readable UTF-8
 -- Module definition
 local PrintableBinary = {}
@@ -1028,19 +1031,45 @@ if arg then
   -- Container (.pbf.json) mode (issue #1): assemble/parse the flat-JSON envelope
   -- over the shared encode/decode + crc32 (A2). Transport-resistant.
   if container_mode then
+    -- Only --spaces is honored in container mode. --tabs/--crlf/-w/--preserve would
+    -- put raw tab/CR/LF (or arbitrary chars) into the JSON value, breaking both
+    -- single-line-JSON validity and the tab/CR/LF-stripping transport-resistance.
+    if tabs_mode or crlf_mode or preserve_chars ~= "" then
+      io.stderr:write("Error: --tabs/--crlf/-w/--preserve are not supported with "
+        .. "--container (only --spaces is honored; other whitespace stays encoded)\n")
+      os.exit(1)
+    end
     if decode_mode then
       local data_raw = json_get_string(input_data, "data")
       if not data_raw then
         io.stderr:write("Error: not a printable-binary-file container (missing 'data')\n")
         os.exit(1)
       end
-      local clean = canonical_payload(data_raw)
+      -- Flagless crc-probe: no schema flag records whether --spaces was used; the
+      -- crc32_encoded oracle disambiguates. Try keeping literal spaces (they're DATA
+      -- in a --spaces container); if the crc mismatches, strip them as transport noise.
+      local clean = canonical_payload(data_raw, true)
+      local spaces = true
       local ce = json_get_string(input_data, "crc32_encoded")
       if ce and crc32_hex(clean) ~= ce then
-        io.stderr:write("Error: container crc32_encoded mismatch (data corrupted)\n")
-        os.exit(1)
+        local stripped = canonical_payload(data_raw, false)
+        if crc32_hex(stripped) == ce then
+          -- Literal spaces were noise. If the space glyph is ALSO present, the payload
+          -- mixed real (glyph) spaces with formatting spaces -> warn we dropped them.
+          local space_glyph = encode_map[32]
+          if space_glyph and clean:find(space_glyph, 1, true) then
+            io.stderr:write("Warning: literal spaces in container data were assumed to be "
+              .. "ignorable formatting because the space glyph " .. space_glyph
+              .. " was also present; stripping them\n")
+          end
+          clean = stripped
+          spaces = false
+        else
+          io.stderr:write("Error: container crc32_encoded mismatch (data corrupted)\n")
+          os.exit(1)
+        end
       end
-      local decoded = PrintableBinary.decode(clean, {})
+      local decoded = PrintableBinary.decode(clean, { spaces = spaces })
       local co = json_get_string(input_data, "crc32")
       if co and crc32_hex(decoded) ~= co then
         io.stderr:write("Error: container crc32 mismatch (decoded data corrupted)\n")
@@ -1049,8 +1078,8 @@ if arg then
       io.write(decoded)
       os.exit(0)
     else
-      local data = PrintableBinary.encode(input_data, {})
-      local clean = canonical_payload(data)
+      local data = PrintableBinary.encode(input_data, { spaces = spaces_mode })
+      local clean = canonical_payload(data, spaces_mode)
       local crc_orig = crc32_hex(input_data)
       local crc_enc = crc32_hex(clean)
       local fname = ""

bin/printable-binary-node.js

@@ -361,6 +361,13 @@ async function main() {
     }
 
     if (containerMode) {
+      // Only --spaces is honored in container mode; --tabs/--crlf/-w/--preserve
+      // would put raw tab/CR/LF (or arbitrary chars) into the JSON value, breaking
+      // single-line-JSON validity and the tab/CR/LF-stripping transport-resistance.
+      if (tabsMode || crlfMode || preserveChars) {
+        process.stderr.write('Error: --tabs/--crlf/-w/--preserve are not supported with --container (only --spaces is honored; other whitespace stays encoded)\n');
+        process.exit(1);
+      }
       if (decodeMode) {
         const res = pb.decodeText(input.toString('utf8'));
         stats(`Decoded ${res.kind} container${res.filename && res.filename !== 'decoded.bin' ? ' (' + res.filename + ')' : ''}: ${res.bytes.length} bytes`);
@@ -376,7 +383,7 @@ async function main() {
             meta.mode = '0' + (st.mode & 0o777).toString(8);
           } catch (_e) { /* metadata is best-effort */ }
         }
-        const container = pb.encodeToContainer(new Uint8Array(input), meta);
+        const container = pb.encodeToContainer(new Uint8Array(input), meta, { spaces: spacesMode });
         stats(`Encoded ${input.length} bytes -> .pbf.json container (crc32 ${container.crc32})`);
         process.stdout.write(JSON.stringify(container, null, 2) + '\n');
       }

character_map.txt

@@ -5,6 +5,13 @@
 ## is the glyph for byte N. Exactly 256 glyph lines. The glyph is the FIRST
 ## whitespace-delimited token on the line.
 ##
+## ASCII passthrough (the core point): letters (A–Z, a–z), digits (0–9), and
+## . @ ^ _ map to THEMSELVES — 66 of the 94 printable non-space bytes — so ASCII
+## text stays legible in the encoded form even when mixed with control/high bytes.
+## The other printable bytes take look-alike glyphs to stay unambiguous: 0x20
+## space -> ␣, plus the 28 punctuation ! " # $ % & ' ( ) * + , - / : ; < = > ? [ \ ] ` { | } ~
+## (see per-line trailing notes). Byte 0x23 '#' -> ♯ is discussed below.
+##
 ## Comments:
 ##   - A line beginning with `##` (two or more '#') is a full-line comment, skipped.
 ##   - A trailing `<glyph> ## comment` is ignored (only the first token is taken).

js/printable_binary.js

@@ -683,8 +683,10 @@ class PrintableBinary {
    * wrap, reflow) added — making the container as whitespace-tolerant as raw
    * printable-binary while keeping crc integrity intact for real corruption.
    */
-  _canonicalPayload(data) {
-    return String(data).replace(/[\r\n\t ]/g, '');
+  _canonicalPayload(data, keepSpaces) {
+    // keepSpaces (container --spaces): literal spaces are DATA, so strip only
+    // tab/CR/LF as transport noise; otherwise strip spaces too.
+    return String(data).replace(keepSpaces ? /[\r\n\t]/g : /[\r\n\t ]/g, '');
   }
 
   /**
@@ -699,12 +701,15 @@ class PrintableBinary {
   }
 
 
-  encodeToContainer(bytes, meta = {}) {
+  encodeToContainer(bytes, meta = {}, opts = {}) {
     if (!(bytes instanceof Uint8Array)) {
       throw new Error("encodeToContainer expects a Uint8Array");
     }
-    const data = this.encode(bytes);
-    const dataBytes = sharedTextEncoder.encode(this._canonicalPayload(data));
+    // opts.spaces: preserve literal spaces in the payload (legible ASCII). No schema
+    // flag is written -- decode's crc-probe disambiguates spaces-as-data vs noise.
+    const spaces = opts.spaces === true;
+    const data = this.encode(bytes, { spaces });
+    const dataBytes = sharedTextEncoder.encode(this._canonicalPayload(data, spaces));
     const container = {
       format: "printable-binary-file",
       version: 1,
@@ -742,14 +747,30 @@ class PrintableBinary {
     if (typeof c.data !== "string") {
       throw new Error("Container 'data' must be a string");
     }
-    const cleanData = this._canonicalPayload(c.data);
+    // Flagless crc-probe: no schema flag records whether --spaces was used; the
+    // crc32_encoded oracle disambiguates. Try keeping literal spaces (DATA in a
+    // --spaces container); if the crc mismatches, strip them as transport noise.
+    let cleanData = this._canonicalPayload(c.data, true);
+    let spaces = true;
     if (c.crc32_encoded !== undefined && c.crc32_encoded !== null) {
-      const got = this.crc32hex(sharedTextEncoder.encode(cleanData));
-      if (got !== String(c.crc32_encoded).toLowerCase()) {
-        throw new Error(`Container crc32_encoded mismatch (got ${got}, expected ${c.crc32_encoded}) — 'data' is corrupted`);
+      const want = String(c.crc32_encoded).toLowerCase();
+      if (this.crc32hex(sharedTextEncoder.encode(cleanData)) !== want) {
+        const stripped = this._canonicalPayload(c.data, false);
+        if (this.crc32hex(sharedTextEncoder.encode(stripped)) === want) {
+          // Literal spaces were noise. If the space glyph is ALSO present, the payload
+          // mixed real (glyph) spaces with formatting spaces -> warn we dropped them.
+          const spaceGlyph = this.encodeMap.get(0x20);
+          if (spaceGlyph && cleanData.includes(spaceGlyph)) {
+            console.warn(`Warning: literal spaces in container data were assumed to be ignorable formatting because the space glyph ${spaceGlyph} was also present; stripping them`);
+          }
+          cleanData = stripped;
+          spaces = false;
+        } else {
+          throw new Error(`Container crc32_encoded mismatch (got ${this.crc32hex(sharedTextEncoder.encode(cleanData))}, expected ${c.crc32_encoded}) — 'data' is corrupted`);
+        }
       }
     }
-    const bytes = this.decode(cleanData);
+    const bytes = this.decode(cleanData, { spaces });
     if (c.byte_length !== undefined && c.byte_length !== null && bytes.length !== c.byte_length) {
       throw new Error(`Container byte_length mismatch (decoded ${bytes.length}, expected ${c.byte_length})`);
     }

src/container_json.h

@@ -14,20 +14,33 @@
 #include <stdlib.h>
 #include <string.h>
 
-/* Strip transport whitespace (space/tab/CR/LF). malloc'd result (caller frees);
- * *out_len set. Clean payloads (no literal whitespace) are unchanged. */
-static char *cj_canonical(const char *data, size_t len, size_t *out_len) {
+/* Strip transport whitespace from an encoded payload. malloc'd result (caller
+ * frees); *out_len set. Normally space/tab/CR/LF are all transport noise. When
+ * keep_spaces is nonzero (container --spaces), literal spaces are DATA, so only
+ * tab/CR/LF are stripped as noise. */
+static char *cj_canonical(const char *data, size_t len, size_t *out_len, int keep_spaces) {
     char *out = (char *)malloc(len ? len : 1);
     if (!out) { *out_len = 0; return NULL; }
     size_t n = 0;
     for (size_t i = 0; i < len; i++) {
         char c = data[i];
-        if (c != ' ' && c != '\t' && c != '\r' && c != '\n') out[n++] = c;
+        int strip = (c == '\t' || c == '\r' || c == '\n') || (!keep_spaces && c == ' ');
+        if (!strip) out[n++] = c;
     }
     *out_len = n;
     return out;
 }
 
+/* Nonzero iff `needle` (nlen bytes) occurs within `hay` (hlen bytes). Used to
+ * detect the space glyph inside a payload for the ambiguity warning. */
+static int cj_contains(const char *hay, size_t hlen, const char *needle, size_t nlen) {
+    if (nlen == 0 || nlen > hlen) return 0;
+    for (size_t i = 0; i + nlen <= hlen; i++) {
+        if (memcmp(hay + i, needle, nlen) == 0) return 1;
+    }
+    return 0;
+}
+
 /* Raw string value of "key" in a flat JSON object: pointer into `json` (not
  * NUL-terminated), *out_len set; NULL if absent. Whitespace-tolerant. */
 static const char *cj_get_string(const char *json, size_t json_len, const char *key, size_t *out_len) {

src/printable_binary.c

@@ -1777,19 +1777,51 @@ int main(int argc, char *argv[]) {
     }
 
     if (opts.container_mode) {
+        /* Only --spaces is honored in container mode; --tabs/--crlf/-w/--preserve
+         * would put raw tab/CR/LF (or arbitrary chars) into the JSON value, breaking
+         * single-line-JSON validity and the tab/CR/LF-stripping transport-resistance. */
+        if (opts.tabs_mode || opts.crlf_mode || opts.preserve_chars[0] != '\0') {
+            fprintf(stderr, "Error: --tabs/--crlf/-w/--preserve are not supported with --container (only --spaces is honored; other whitespace stays encoded)\n");
+            return 1;
+        }
         if (opts.decode_mode) {
             size_t dlen;
             const char *draw = cj_get_string(input.data, input.size, "data", &dlen);
             if (!draw) { fprintf(stderr, "Error: not a printable-binary-file container (missing 'data')\n"); return 1; }
+            /* Flagless crc-probe: no schema flag records whether --spaces was used; the
+             * crc32_encoded oracle disambiguates. Try keeping literal spaces (DATA in a
+             * --spaces container); if the crc mismatches, strip them as transport noise. */
             size_t clen;
-            char *clean = cj_canonical(draw, dlen, &clen);
+            char *clean = cj_canonical(draw, dlen, &clen, 1);
+            bool spaces = true;
             size_t celen;
             const char *ce = cj_get_string(input.data, input.size, "crc32_encoded", &celen);
             if (ce) {
                 char hx[9]; snprintf(hx, 9, "%08x", (unsigned int)pb_crc32(clean, clen));
-                if (celen != 8 || memcmp(hx, ce, 8) != 0) { free(clean); fprintf(stderr, "Error: container crc32_encoded mismatch (data corrupted)\n"); return 1; }
+                if (celen != 8 || memcmp(hx, ce, 8) != 0) {
+                    size_t slen;
+                    char *stripped = cj_canonical(draw, dlen, &slen, 0);
+                    char hs[9]; snprintf(hs, 9, "%08x", (unsigned int)pb_crc32(stripped, slen));
+                    if (celen == 8 && memcmp(hs, ce, 8) == 0) {
+                        /* Literal spaces were noise. If the space glyph is ALSO present, the
+                         * payload mixed real (glyph) spaces with formatting spaces -> warn. */
+                        const char *sg = (const char *)encode_table[' '].bytes;
+                        size_t sglen = encode_table[' '].length;
+                        if (cj_contains(clean, clen, sg, sglen)) {
+                            fprintf(stderr, "Warning: literal spaces in container data were assumed to be ignorable formatting because the space glyph %.*s was also present; stripping them\n", (int)sglen, sg);
+                        }
+                        free(clean);
+                        clean = stripped;
+                        clen = slen;
+                        spaces = false;
+                    } else {
+                        free(stripped); free(clean);
+                        fprintf(stderr, "Error: container crc32_encoded mismatch (data corrupted)\n");
+                        return 1;
+                    }
+                }
             }
-            buffer_t dec = decode_data((uint8_t *)clean, clen, false);
+            buffer_t dec = decode_data((uint8_t *)clean, clen, spaces);
             free(clean);
             size_t colen;
             const char *co = cj_get_string(input.data, input.size, "crc32", &colen);
@@ -1801,13 +1833,12 @@ int main(int argc, char *argv[]) {
             return 0;
         } else {
             options_t enc_opts = opts;
-            enc_opts.spaces_mode = false;
             enc_opts.tabs_mode = false;
             enc_opts.crlf_mode = false;
             enc_opts.preserve_chars[0] = '\0';
             buffer_t enc = encode_data((uint8_t *)input.data, input.size, &enc_opts);
             size_t clen;
-            char *clean = cj_canonical(enc.data, enc.size, &clen);
+            char *clean = cj_canonical(enc.data, enc.size, &clen, opts.spaces_mode ? 1 : 0);
             char crc_orig[9], crc_enc[9];
             snprintf(crc_orig, 9, "%08x", (unsigned int)pb_crc32(input.data, input.size));
             snprintf(crc_enc, 9, "%08x", (unsigned int)pb_crc32(clean, clen));