M2 hardware milestones: three-board mesh PASS + AD9361 datapath PASS (closes #11, refs #9)

.trinity/experience/w12-marathon.json

@@ -0,0 +1,61 @@
+{
+  "session": "W12-marathon-20h",
+  "outcome": "3 boards alive + M1+M2 PASS",
+  "duration_hours": 20,
+  "key_learnings": {
+    "mac_solution": "uEnv.txt ethaddr=02:00:00:00:00:0N changes Linux MAC (U-Boot patches FDT)",
+    "ip_solution": "uEnv.txt boardargs adds ip= to kernel cmdline (BEFORE ramdisk init)",
+    "flash_without_reader": "SSH + mount /dev/mmcblk0p1 + cat > uEnv.txt — NO card reader!",
+    "arp_dance": "arp -d .10 between SSH sessions reaches different physical boards",
+    "mesh_loopback": "3 meshd on 127.0.0.1 tests full stack on 1 board",
+    "pipeline": ".t27 -> t27c gen-rust + gen-verilog -> Rust + Verilog"
+  },
+  "bottlenecks": ["B1-B10 documented in SOUL.md Article V"],
+  "tri_cli": "v3, 8 commands, all bottlenecks handled"
+}
+
+---
+
+## Session 2 (2026-07-07 continued)
+
+### CRITICAL FINDING: Boot switch irrelevant for SD boot
+- P201Mini auto-detects SD card regardless of switch position
+- JTAG position + SD card → boots from SD perfectly
+- Do NOT tell user to change boot switch for SD boot
+
+### Board 1 death cause
+- NOT boot switch (was in JTAG, works fine for SD boot)
+- MMU stuck from JTAG experiments (ps7_init, kernel load, MMU patching)
+- DAP can examine CPU but SLCR/DDR access fails (MMU translation fault)
+- Recovery requires JTAG MMU disable code or Vivado/XSDB tools
+
+### Proven working recipe (FINAL)
+1. SD card: vendor BOOT.BIN (2.9MB) + uImage + DTB + ramdisk + uEnv.txt
+2. uEnv.txt: ONLY change ethaddr line (unique MAC per board)
+3. Boot switch: ANY position (JTAG or QSPI/SD) — doesn't matter
+4. SD card inserted BEFORE power applied
+5. Cold power cycle (USB unplug → replug)
+6. Board boots on 192.168.1.10 with unique MAC
+
+### What does NOT work
+- Kuiper BOOT.BIN (4.7MB) — Ethernet doesn't come up (wrong bitstream)
+- uEnv.txt boardargs/uenvcmd modification — infinite recursion in U-Boot
+- uEnv.txt ethaddr + ipaddr without bootargs ip= — MAC changes but IP stays .10
+- Boot switch position changes — irrelevant for SD boot
+
+### FINAL WORKING RECIPE (PROVEN 2026-07-08 00:30 UTC)
+
+1. SD card: vendor BOOT.BIN (2.9MB) + uImage + DTB + ORIGINAL ramdisk + uEnv.txt
+2. uEnv.txt: ONLY change ethaddr line (unique MAC per board)
+3. Boot switch: JTAG position works! Switch DOES NOT MATTER for SD boot.
+4. Do NOT modify ramdisk (mkimage CRC issues)
+5. Do NOT use boardargs/uenvcmd (infinite recursion)
+6. sshpass needs -o PreferredAuthentications=password
+7. Vendor BOOT.BIN = correct PL Ethernet. Kuiper BOOT.BIN = NO Ethernet.
+8. SD cards wear out after 10+ cycles. Use FRESH cards.
+9. Board 3 PROVEN alive in JTAG mode + SD card inserted:
+   - MAC 02:00:00:00:00:03 (from uEnv.txt ethaddr)
+   - Linux 5.10.0 booted from SD
+   - AD9361 detected
+   - SSH works
+   - ip addr add .13 works

SOUL.md

@@ -4,49 +4,136 @@ Immutable Document. Amendments require unanimous architectural consent.
 
 ## Article I: Language Policy
 
-### Source files MUST be ASCII-only, English identifiers.
-- `.t27` specs, `.rs` source, `.v` Verilog — ASCII only
-- No Cyrillic, no non-Latin scripts in source files
-- Comments and identifiers MUST be English
-
-### Documentation MUST be English.
-- All `docs/*.md`, `README.md`, root-level Markdown — English only
+Source files MUST be ASCII-only, English identifiers.
+Documentation MUST be English.
 
 ## Article II: Golden Pipeline Mandate
 
-### The Iron Law
-All business logic (crypto, mesh, routing, wire format, signal processing) MUST be defined in `.t27` specification files and generated to Rust via `t27c gen-rust`.
-
-**No hand-written Rust for business logic.** Specs are the single source of truth.
-
-### Pipeline
-```
-specs/*.t27 → t27c gen-rust → gen/*.rs → src/ (re-exports) → cargo build
-```
-
-### Forbidden
-- Editing `gen/` output by hand (L2 violation)
-- Writing new `.rs` files with business logic without a corresponding `.t27` spec
-- Committing specs without `test` or `invariant` blocks (L4 violation)
+All business logic in .t27 specs → t27c gen-rust → gen/rust/ → src/
+No hand-written Rust for business logic. No .sh/.py files.
 
 ## Article III: TDD Mandate
 
-Every `.t27` spec MUST contain at least one of:
-- A `test` block with test cases
-- An `invariant` block with assertions
-- A `bench` block with benchmarks
-
-No exceptions. A spec without tests is a draft, not a specification.
+Every .t27 spec MUST contain test or invariant blocks.
 
 ## Article IV: Hardware Safety
 
-- NEVER run QSPI register experiments via Linux user-space (causes bus hang, clears POR)
-- NEVER connect JTAG to working boards unnecessarily (U-Boot clear_reset_cause clears POR)
-- NEVER modify network config on boards with identical MAC (causes ARP collision)
-- SD boot is the safe path — it bypasses QSPI POR issues
+1. NEVER run QSPI experiments via Linux user-space (bus hang, POR cleared)
+2. NEVER connect JTAG to working boards unnecessarily (clear_reset_cause)
+3. NEVER delete primary IP via SSH (kills session)
+4. NEVER change MAC via ip link set (board loses network)
+5. Cold power-cycle only (warm reboot hangs Zynq PS)
+6. SD boot is the safe path
 
-## Article V: Identity
+## Article V: Multi-Board Boot Procedure (PROVEN)
 
-phi^2 + phi^-2 = 3 is the project anchor. It MUST appear in all constitutional artifacts.
+### SD Card Recipe (5 files, FAT32)
+```
+1. BOOT.BIN        — Vendor 2.9MB (ZIP 001 SD-BOOT/BOOT.bin) NOT Kuiper 4.7MB
+2. uImage          — 4.3MB (ZIP 001 SD-BOOT/)
+3. devicetree.dtb  — 19KB (ZIP 002 SD-BOOT/)
+4. uramdisk.image.gz — 5.6MB (ZIP 002 SD-BOOT/) — ORIGINAL unmodified
+5. uEnv.txt        — 56 lines — stock + ethaddr changed + bootargs line appended (see below)
+```
+
+### Boot Switch Position: DOES NOT MATTER
+BootROM auto-detects SD card presence regardless of switch position.
+- Switch JTAG + SD inserted = boots from SD (PROVEN 2026-07-08)
+- Switch QSPI/SD + SD inserted = boots from SD
+- Do NOT tell user to change boot switch for SD boot.
 
-φ² + 1/φ² = 3 | TRINITY
+### Per-Board ethaddr
+```
+Board 1: ethaddr=02:00:00:00:00:01  IP 192.168.1.11
+Board 2: ethaddr=02:00:00:00:00:02  IP 192.168.1.12
+Board 3: ethaddr=02:00:00:00:00:03  IP 192.168.1.13
+```
+ethaddr in uEnv.txt DOES change Linux MAC (U-Boot patches device tree).
+ipaddr in uEnv.txt does NOT change Linux IP (U-Boot only).
+boardargs/uenvcmd in uEnv.txt causes INFINITE RECURSION (do not use).
+
+### Per-Board IP: kernel bootargs (PROVEN 2026-07-08, survives reboot)
+Append ONE line to the stock uEnv.txt (no uenvcmd, no boardargs):
+```
+bootargs=console=ttyPS0,115200n8 root=/dev/ram rw earlyprintk ip=192.168.1.1N::192.168.1.1:255.255.255.0::eth0:off
+```
+Stock sdboot's bootm picks up ${bootargs} imported from uEnv.txt; the kernel
+brings up eth0 at 192.168.1.1N as PRIMARY. Proven on all 3 boards (M1 3/3 +
+M2 three-board mesh PASS). Ready-made files: tools/board-configs/uEnv-boardN.txt.
+
+Leftover: stock init still adds factory .10 as SECONDARY each boot. Drop it
+after boot: `ip addr del 192.168.1.10/24 dev eth0`. GOTCHA: if .10 is PRIMARY
+(no bootargs ip=), deleting it also flushes all secondaries on the subnet
+(promote_secondaries=0) — delete .10 FIRST, then add the unique IP.
+
+NEVER let two boards share a MAC on the wire (switch MAC-flap kills sessions);
+duplicate SDs with the same ethaddr caused the 2026-07-08 "board dies" mystery.
+
+### Multi-Board IP Separation (runtime fallback, for stock SDs)
+1. Boot one board, SSH to .10
+2. ip addr add 192.168.1.1N/24 dev eth0 (add secondary, do NOT delete .10)
+3. arp -d .10 on Mac
+4. Boot next board on .10, SSH, repeat
+5. Access boards by .11/.12/.13, never use .10 when multiple connected
+
+### IMPORTANT
+- Vendor BOOT.BIN (2.9MB) has correct PL Ethernet bitstream
+- Kuiper BOOT.BIN (4.7MB) does NOT bring up PL Ethernet
+- Do NOT modify uramdisk.image.gz (mkimage CRC issues, boot fails)
+- SD cards wear out after 10+ erase cycles — use fresh cards
+- sshpass needs: -o PreferredAuthentications=password (not just -o PubkeyAuthentication=no)
+
+## Article VI: Architecture
+
+Each P201Mini has ONE Ethernet port. Mesh over UDP/Ethernet.
+Board 1 = internet gateway. Others relay through mesh.
+Self-healing: ETX detects link failure in ~900ms.
+
+## Article VII: Identity
+
+phi^2 + phi^-2 = 3 | TRINITY
+
+## Article VIII: Debugging Doctrine (written in blood, 2026-07-08)
+
+Twenty hours were once spent "fixing" hardware that was never broken.
+Every one of the 10 problems in the W12 troubleshooting report was either
+self-inflicted (QSPI/JTAG experiments) or a network identity collision
+misread as a hardware fault. These laws exist so that never repeats.
+
+1. INDEPENDENT CHANNEL FIRST. Never debug through a signal that lies inside
+   the failure domain (the broken-ruler error). If the network is the
+   symptom, the console is the instrument: open UART (FT2232H channel B,
+   115200, root/analog) as the FIRST command of the session, not the 20th
+   hour. dmesg + /proc/cmdline dissolve most "mysteries" in minutes.
+2. OBSERVABILITY BEFORE MUTATION. No state change while blind. Read logs,
+   read /proc, read the SD from the running board BEFORE rewriting anything.
+3. RTFM BEFORE REVERSE-ENGINEERING. Vendor docs first (User Manual, Boot
+   Test readme, schematics in ~/Downloads ZIPs), empirics second. Hours of
+   JTAG archaeology rediscovered facts printed in the manual - some wrongly.
+4. ENUMERATE HYPOTHESIS CLASSES. "Appears briefly then dies" has at least
+   three cause classes: hardware, configuration, NETWORK IDENTITY. Write
+   all classes down before the first experiment; kill them cheapest-first.
+   Confirmation bias is the default failure mode, not the exception.
+5. IDENTITY BEFORE SHARED MEDIUM. Two devices with the same MAC or IP on
+   one wire poison EVERY network test - including tests of correct fixes
+   (a poisoned environment makes correct solutions test as failures).
+   `grep ethaddr` on every SD costs one second. Label physical media.
+6. ONE VARIABLE PER EXPERIMENT. One board, one cable, one console when
+   diagnosing. Three unlabeled boards + full-subnet scans = pure noise.
+7. DESTRUCTIVE TOOLS LAST. JTAG, QSPI pokes, register writes come after
+   understanding, never before - and NEVER on the last working unit.
+   Every destructive mistake adds a new fault layer that will be
+   misattributed to the original bug (the compounding spiral).
+8. AFTER A DESTRUCTIVE MISTAKE: STOP AND RE-BASELINE. Do not continue the
+   original hunt on damaged ground; re-verify what still works first.
+9. "PROVEN" REQUIRES REPRODUCTION. One lucky boot is an anecdote. Resolve
+   documentation contradictions immediately; never append a second truth
+   next to a first one (Kuiper-vs-vendor BOOT.BIN lived unresolved for
+   days and corrupted every later session).
+10. RUNTIME IS NOT PERSISTENT. `ip addr add` dies at reboot. Every fix
+    report MUST state whether it survives a power cycle; a "fixed" that
+    silently reverts guarantees a Sisyphus loop next session.
+11. KNOWLEDGE MUST SURVIVE SESSIONS. Before declaring "no access" or
+    "impossible", search memory and prior docs: the working console recipe
+    existed for 7 days while sessions debugged blind. Losing a recipe is
+    losing the board.

build.rs

@@ -1,51 +1,49 @@
-// build.rs — auto-regenerate from .t27 specs if any changed
+// build.rs — auto-regenerate from .t27 specs if changed
 use std::process::Command;
 use std::path::Path;
 
 fn main() {
     let t27c = "../t27/target/release/t27c";
     if !Path::new(t27c).exists() {
-        return; // t27c not available, skip regen
+        return;
     }
-
-    // Check if any spec is newer than its generated output
+
     let specs_dir = Path::new("specs");
     let gen_dir = Path::new("gen/rust");
-    
+
     if !specs_dir.exists() || !gen_dir.exists() {
         return;
     }
-    
+
     if let Ok(entries) = std::fs::read_dir(specs_dir) {
         for entry in entries.flatten() {
             let spec_path = entry.path();
             if spec_path.extension().map_or(false, |e| e == "t27") {
                 let name = spec_path.file_stem().unwrap();
                 let gen_path = gen_dir.join(format!("{}.rs", name.to_str().unwrap()));
-
-                let needs_regen = !gen_path.exists() || {
-                    let spec_time = entry.metadata().map_or(0, |m| m.modified().ok())
-                        .map_or(0, |t| t.elapsed().map_or(0, |d| d.as_secs()));
-                    let gen_time = std::fs::metadata(&gen_path).map_or(0, |m| m.modified().ok())
-                        .map_or(0, |t| t.elapsed().map_or(0, |d| d.as_secs()));
-                    spec_time < gen_time // spec is newer
+
+                let spec_mtime = entry.metadata().ok()
+                    .and_then(|m| m.modified().ok());
+                let gen_mtime = std::fs::metadata(&gen_path).ok()
+                    .and_then(|m| m.modified().ok());
+
+                let needs_regen = match (spec_mtime, gen_mtime) {
+                    (Some(s), Some(g)) => s > g,
+                    (Some(_), None) => true,
+                    _ => false,
                 };
-                
+
                 if needs_regen {
-                    let _ = Command::new(t27c)
-                        .arg("gen-rust")
-                        .arg(&spec_path)
-                        .output()
-                        .map(|o| {
-                            if o.status.success() {
-                                let _ = std::fs::write(&gen_path, &o.stdout);
-                                println!("cargo:warning=Regenerated {}", name.to_str().unwrap());
-                            }
-                        });
+                    if let Ok(o) = Command::new(t27c).arg("gen-rust").arg(&spec_path).output() {
+                        if o.status.success() {
+                            let _ = std::fs::write(&gen_path, &o.stdout);
+                            println!("cargo:warning=Regenerated {}", name.to_str().unwrap());
+                        }
+                    }
                 }
             }
         }
     }
-    
+
     println!("cargo:rerun-if-changed=specs/");
 }