test(ios): add tart ephemeral e2e reproduction pipeline

Host-orchestrated Tart VMs with detached iteration, session-scoped artifacts,
virtiofs completion polling, and optional SCP harvest (--no-sync-artifacts).

Author: mikehardy

scripts/tart/.gitignore

@@ -0,0 +1,4 @@
+# Local iOS e2e reproduction run outputs (see okf-bundle/testing/local-ios-e2e-reproduction.md)
+artifacts/
+manifests/golden-baked.json
+manifests/warmed-main.json

scripts/tart/README.md

@@ -0,0 +1,216 @@
+# Local iOS E2E reproduction (Tart)
+
+Self-contained Tart VM pipeline for reproducing **Testing E2E iOS** locally. Full design: [okf-bundle/testing/local-ios-e2e-reproduction.md](../../okf-bundle/testing/local-ios-e2e-reproduction.md).
+
+**Isolation:** everything lives under `scripts/tart/` plus OKF documentation. This system does not modify `.github/`, CI workflows, or package scripts.
+
+## Prerequisites (host)
+
+- Apple Silicon Mac with [Tart](https://tart.run/) installed
+- `sshpass` (`brew install sshpass`)
+- **≥ 80 GiB free** on the volume holding `~/.tart` (symlink to external disk if needed)
+- Git worktree for the branch under test (not the primary clone)
+
+## Quick start
+
+```bash
+# 1. Bake golden toolchain (once, after disk cleanup)
+./scripts/tart/bake-golden.sh
+
+# 2. Prewarm caches on origin/main (once per main bump)
+./scripts/tart/bake-warmed.sh /path/to/any/rnfb-clone-or-worktree
+
+# 3. Run one ephemeral iteration on your worktree
+./scripts/tart/run-ephemeral.sh --worktree /path/to/worktree --mode debug
+
+# 4. Stress loop
+./scripts/tart/run-stress.sh --worktree /path/to/worktree --count 10 --mode debug
+```
+
+## Yarn cache
+
+All tart scripts export:
+
+```bash
+YARN_CACHE_FOLDER=$HOME/.yarn-rnfb-cache
+```
+
+The repo uses `enableGlobalCache: false` in `.yarnrc.yml`; the env var keeps Berry cache outside the virtiofs worktree without changing repo config.
+
+## Scripts
+
+| Script | Where |
+|--------|-------|
+| `bake-golden.sh` | Host — seed → golden VM |
+| `bake-warmed.sh` | Host — golden → warmed VM |
+| `run-ephemeral.sh` | Host — one iteration, delete VM |
+| `run-stress.sh` | Host — N iterations |
+| `stop-ephemeral.sh` | Host — stop/delete a kept ephemeral VM |
+| `provision-toolchain.sh` | In-VM (via SSH during bake) |
+| `prewarm-caches.sh` | In-VM (via SSH during warm bake) |
+| `run-iteration.sh` | In-VM (via SSH during ephemeral run) |
+| `verify-manifest.sh` | Host |
+| `report-disk-usage.sh` | Host |
+
+CI helper equivalents are vendored under `lib/` and `vendor/homebrew-rnfb/` — not imported from `.github/`.
+
+## Completion, artifacts, and cleanup
+
+`run-ephemeral.sh` launches the iteration **detached inside the VM** (`nohup`), then polls virtiofs for `iteration-complete.json`. No long-lived SSH log stream.
+
+### How completion is detected
+
+1. Host SSHs once to start `run-iteration.sh` under `nohup` on VM disk (`~/rnfb-prewarm/...`).
+2. In-VM `run-iteration.sh` logs to `run-iteration.log` on VM disk (not over SSH).
+3. When finished (or on failure via `EXIT` trap), it writes **`iteration-complete.json`** and syncs small files to virtiofs.
+4. Host polls the worktree for `iteration-complete.json`, reads `rc`, then **SCPs bulk logs** from the VM (while it is still running, unless `--no-sync-artifacts`).
+5. Orchestrator tears down the VM on exit (or leaves it running with `--keep-vm`).
+
+**Artifact flow (two paths):**
+
+| Stage | What | Where |
+|-------|------|--------|
+| In-VM write | Full iteration output during the run | `~/rnfb-prewarm/scripts/tart/artifacts/<SESSION_ID>/<mode>-iter<N>/` on VM disk |
+| Virtiofs sync | Small completion files (`iteration-complete.json`, `detox-step.log`, `ccache-stats.txt`, …) | Same path under the mounted worktree (host-visible without SSH) |
+| Host SCP harvest (default) | Bulk files (`run-iteration.log`, `simulator.log`, `simulator.mp4`, …) | Host pulls from VM → worktree after completion marker |
+
+Use **`--no-sync-artifacts`** to skip the SCP harvest step. Essentials still land on the worktree via virtiofs; bulk logs remain on the VM until teardown (or inspect with `--keep-vm` + SSH).
+
+The VM **stays running** from `tart run` through iteration and SCP harvest. Nothing stops it until the host orchestrator calls teardown at exit (or you pass `--keep-vm` to leave it up).
+
+Live status (optional):
+
+```bash
+ssh admin@$(tart ip rnfb-e2e-<uuid>) tail -f ~/rnfb-prewarm/scripts/tart/artifacts/<SESSION_ID>/<mode>-iter<N>/run-iteration.log
+```
+
+`bake-warmed.sh` uses a similar marker pattern: in-VM `prewarm-caches.sh` writes **`prewarm-complete.json`**; the host polls for it.
+
+### Artifact layout
+
+All run outputs live under **`scripts/tart/artifacts/<SESSION_ID>/`**, where `SESSION_ID` is the UTC run start timestamp (e.g. `20260623T062250Z`). The directory is gitignored.
+
+| Path | Purpose |
+|------|---------|
+| `<SESSION_ID>/run-ephemeral.log` | Host orchestrator log (APFS, not virtiofs) |
+| `<SESSION_ID>/<mode>-iter<N>/` | Per-iteration artifacts (see below) |
+| `<SESSION_ID>/prewarm.log` | `bake-warmed.sh` prewarm log |
+
+### Artifact files (per iteration)
+
+Iteration dir: `scripts/tart/artifacts/<SESSION_ID>/<mode>-iter<N>/` (e.g. `debug-iter0`).
+
+| File | Purpose |
+|------|---------|
+| `iteration-complete.json` | Authoritative completion marker + exit code |
+| `run-iteration.log` | Full in-VM iteration log (SCP'd on harvest) |
+| `iteration-env.txt` | Lockfile SHAs, warmed-manifest comparison, build duration |
+| `ccache-stats.txt` | `ccache -s` snapshots at each phase |
+| `detox-step.log` | Detox/Jet test output |
+| `flake-summary.txt` | Post-run flake digest |
+
+Host wrapper log: `scripts/tart/artifacts/<SESSION_ID>/run-ephemeral.log` (host APFS, not virtiofs).
+
+## Operational pitfalls (learned in practice)
+
+### Do not stream iteration output over SSH or virtiofs
+
+Build output (xcbeautify note spam) can block `tee` on virtiofs and wedge the whole pipeline. Iteration stdout belongs on **VM disk**; the host waits on **`iteration-complete.json`** only.
+
+Bulk artifacts (`simulator.log`, `testing.log`, `simulator.mp4`, `run-iteration.log`) are pulled with **host-side SCP** after the completion marker appears.
+
+### Do not rsync large logs over virtiofs
+
+The iteration exit path **must not** block on copying hundreds of MB through virtiofs. Essential files only (`iteration-complete.json`, `detox-step.log`, etc.).
+
+### SSH sessions can outlive the VM
+
+`rnfb_tart_ssh` connects to `tart ip <vm>`. Problems observed on older flows:
+
+- Long-lived SSH streams blocking cleanup (mitigated by detached iteration launch).
+- **Deleted VMs leave zombie `sshpass` processes** still connected to stale `192.168.64.*` addresses.
+
+Mitigations in `lib/common.sh`:
+
+- `ServerAliveInterval=15` / `ServerAliveCountMax=4` — dead connections drop in ~60s
+- `timeout` wrapper on SSH/SCP (default **`RNFB_TART_SSH_TIMEOUT=3600`** seconds for launch/harvest)
+- **`RNFB_TART_ITERATION_TIMEOUT=7200`** — max wait for `iteration-complete.json` on virtiofs
+
+### Low CPU does not mean the pipeline finished
+
+A completed test run can still leave:
+
+- An ephemeral VM running (host script blocked on SSH)
+- Zombie `sshpass` / `run-ephemeral.sh` / `bake-warmed.sh` wrappers on the host
+- `tart run` background processes
+
+Always check process state explicitly.
+
+## Troubleshooting
+
+### Check for zombie host processes
+
+```bash
+pgrep -fl 'sshpass|run-ephemeral|bake-warmed|tart run'
+tart list
+```
+
+- Any **`sshpass` to `192.168.64.*`** with no matching **`rnfb-e2e-*`** VM in `tart list` → safe to kill.
+- Multiple overlapping `run-ephemeral.sh` from retries → kill extras; only one should own the ephemeral VM.
+
+```bash
+pkill -f 'sshpass.*192.168.64'
+pkill -f 'scripts/tart/run-ephemeral.sh'
+```
+
+### Check iteration outcome
+
+```bash
+ls scripts/tart/artifacts/*/
+cat scripts/tart/artifacts/<SESSION_ID>/<mode>-iter<N>/iteration-complete.json
+cat scripts/tart/artifacts/<SESSION_ID>/<mode>-iter<N>/ccache-stats.txt
+tail scripts/tart/artifacts/<SESSION_ID>/<mode>-iter<N>/detox-step.log
+```
+
+### Manual VM cleanup
+
+Ephemeral VMs are named `rnfb-e2e-<uuid>`. By default `run-ephemeral.sh` stops and deletes them on exit. With `--keep-vm`, the orchestrator leaves the VM running:
+
+```bash
+./scripts/tart/stop-ephemeral.sh rnfb-e2e-<uuid>           # stop only
+./scripts/tart/stop-ephemeral.sh rnfb-e2e-<uuid> --delete  # stop + delete image
+```
+
+Or manually:
+
+```bash
+tart list
+tart stop rnfb-e2e-<uuid>
+tart delete rnfb-e2e-<uuid>
+```
+
+Preserve **`rnfb-ios-e2e-golden`** and **`rnfb-ios-e2e-warmed`** (stopped is normal).
+
+### `bake-warmed` appeared finished but host hung
+
+If in-VM prewarm logged `[prewarm] complete` but the host never reached `[bake-warmed] tagging`:
+
+1. Check for stuck `sshpass` / `bake-warmed.sh`.
+2. Confirm `~/rnfb-tart-manifests/prewarm-complete.json` exists in the build VM (or fetch `warmed-main.json` manually).
+3. Tag manually: `tart clone rnfb-ios-e2e-warmed-build rnfb-ios-e2e-warmed`, then delete the `-build` VM.
+
+### When to rebake warmed
+
+Re-run `bake-warmed.sh` when `origin/main` changes **`yarn.lock`** or **`tests/ios/Podfile.lock`** (compare SHAs in `manifests/warmed-main.json`). Golden rebake only when CI toolchain pins change (`manifests/golden-expected.json`).
+
+## Environment variables (SSH / caches)
+
+| Variable | Default | Notes |
+|----------|---------|-------|
+| `RNFB_TART_SSH_TIMEOUT` | `3600` | Max seconds per SSH/SCP call (`timeout` wrapper) |
+| `RNFB_TART_ITERATION_TIMEOUT` | `7200` | Max seconds to poll for `iteration-complete.json` |
+| `RNFB_TART_SSH_USER` / `RNFB_TART_SSH_PASSWORD` | `admin` / `admin` | Cirrus Tart images |
+| `YARN_CACHE_FOLDER` | `~/.yarn-rnfb-cache` | Outside virtiofs worktree |
+| `CCACHE_DIR` | `~/.ccache` | Shared between prewarm and iteration via `~/rnfb-prewarm` path |
+| `RNFB_TART_PREWARM_DIR` | `~/rnfb-prewarm` | Iteration materializes worktree here for ccache key alignment |
+

scripts/tart/bake-golden.sh

@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+# Host: clone Cirrus seed, provision toolchain, tag rnfb-ios-e2e-golden.
+set -euo pipefail
+
+# shellcheck source=lib/common.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/common.sh"
+
+rnfb_tart_require tart
+rnfb_tart_require sshpass
+
+EXPECTED="${RNFB_TART_ROOT}/manifests/golden-expected.json"
+SEED_IMAGE="$(rnfb_tart_read_manifest_field "$EXPECTED" tart_seed_image)"
+BUILD_VM="${RNFB_TART_GOLDEN_VM}-build"
+
+MIN_FREE_GIB="${RNFB_TART_MIN_FREE_GIB:-80}"
+avail_kib="$(df -k "${HOME}" | awk 'NR==2 {print $4}')"
+avail_gib=$((avail_kib / 1024 / 1024))
+if (( avail_gib < MIN_FREE_GIB )); then
+  echo "warning: only ~${avail_gib} GiB free on ${HOME}; recommend >= ${MIN_FREE_GIB} GiB before baking" >&2
+fi
+
+echo "[bake-golden] pulling ${SEED_IMAGE} if needed..."
+tart pull "$SEED_IMAGE" || true
+
+tart stop "$BUILD_VM" 2>/dev/null || true
+tart delete "$BUILD_VM" 2>/dev/null || true
+tart stop "$RNFB_TART_GOLDEN_VM" 2>/dev/null || true
+tart delete "$RNFB_TART_GOLDEN_VM" 2>/dev/null || true
+
+echo "[bake-golden] cloning seed → ${BUILD_VM}"
+tart clone "$SEED_IMAGE" "$BUILD_VM"
+
+echo "[bake-golden] starting ${BUILD_VM} with tart-scripts mount..."
+tart run --no-graphics --dir=tart-scripts:"${RNFB_TART_ROOT}":ro "$BUILD_VM" &
+run_pid=$!
+sleep 5
+
+cleanup() {
+  kill "$run_pid" 2>/dev/null || true
+  tart stop "$BUILD_VM" 2>/dev/null || true
+}
+trap cleanup EXIT
+
+rnfb_tart_wait_ssh "$BUILD_VM" 240
+
+echo "[bake-golden] provisioning toolchain..."
+rnfb_tart_ssh "$BUILD_VM" "bash '${RNFB_TART_SCRIPTS_MOUNT}/provision-toolchain.sh'"
+
+rnfb_tart_ssh "$BUILD_VM" "cat ~/rnfb-tart-manifests/golden-manifest.json" \
+  > "${RNFB_TART_ROOT}/manifests/golden-baked.json"
+
+kill "$run_pid" 2>/dev/null || true
+wait "$run_pid" 2>/dev/null || true
+tart stop "$BUILD_VM" 2>/dev/null || true
+
+echo "[bake-golden] tagging ${RNFB_TART_GOLDEN_VM}"
+tart clone "$BUILD_VM" "$RNFB_TART_GOLDEN_VM"
+tart delete "$BUILD_VM"
+trap - EXIT
+
+echo "[bake-golden] done — local VM: ${RNFB_TART_GOLDEN_VM}"
+echo "[bake-golden] manifest: ${RNFB_TART_ROOT}/manifests/golden-baked.json"

scripts/tart/bake-warmed.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# Host: clone golden VM, prewarm caches on origin/main, tag rnfb-ios-e2e-warmed.
+set -euo pipefail
+
+# shellcheck source=lib/common.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/common.sh"
+
+rnfb_tart_require tart
+rnfb_tart_require sshpass
+
+HOST_REPO="${1:-${RNFB_HOST_REPO:-}}"
+if [[ -z "$HOST_REPO" ]]; then
+  echo "usage: $0 [path-to-host-git-repo]" >&2
+  echo "  Uses git remote.origin.url and resolves origin/main HEAD for prewarm." >&2
+  exit 1
+fi
+HOST_REPO="$(cd "$HOST_REPO" && pwd)"
+
+if ! rnfb_tart_vm_exists "${RNFB_TART_GOLDEN_VM}"; then
+  echo "error: golden VM ${RNFB_TART_GOLDEN_VM} not found — run bake-golden.sh first" >&2
+  exit 1
+fi
+
+RNFB_REPO_URL="$(git -C "$HOST_REPO" remote get-url origin)"
+RNFB_REPO_URL="$(rnfb_tart_https_repo_url "$RNFB_REPO_URL")"
+RNFB_MAIN_REF="main"
+RNFB_MAIN_SHA="$(git -C "$HOST_REPO" ls-remote origin refs/heads/main | awk '{print $1}')"
+
+BUILD_VM="${RNFB_TART_WARMED_VM}-build"
+tart stop "$BUILD_VM" 2>/dev/null || true
+tart delete "$BUILD_VM" 2>/dev/null || true
+tart stop "$RNFB_TART_WARMED_VM" 2>/dev/null || true
+tart delete "$RNFB_TART_WARMED_VM" 2>/dev/null || true
+
+echo "[bake-warmed] cloning ${RNFB_TART_GOLDEN_VM} → ${BUILD_VM}"
+tart clone "$RNFB_TART_GOLDEN_VM" "$BUILD_VM"
+
+BAKE_SESSION_ID="$(date -u +%Y%m%dT%H%M%SZ)"
+BAKE_ARTIFACT_DIR="$(rnfb_tart_artifact_session_dir "$RNFB_TART_ROOT" "$BAKE_SESSION_ID")"
+mkdir -p "$BAKE_ARTIFACT_DIR"
+
+echo "[bake-warmed] prewarm origin/main @ ${RNFB_MAIN_SHA:0:12}"
+tart run --no-graphics --dir=tart-scripts:"${RNFB_TART_ROOT}":ro "$BUILD_VM" &
+run_pid=$!
+sleep 5
+
+cleanup() {
+  kill "$run_pid" 2>/dev/null || true
+  tart stop "$BUILD_VM" 2>/dev/null || true
+}
+trap cleanup EXIT
+
+rnfb_tart_wait_ssh "$BUILD_VM" 240
+
+set +e
+rnfb_tart_ssh "$BUILD_VM" \
+  "RNFB_REPO_URL='${RNFB_REPO_URL}' RNFB_MAIN_REF='${RNFB_MAIN_REF}' \
+   bash '${RNFB_TART_SCRIPTS_MOUNT}/prewarm-caches.sh'" \
+  > "${BAKE_ARTIFACT_DIR}/prewarm.log" 2>&1 &
+prewarm_pid=$!
+
+while kill -0 "$prewarm_pid" 2>/dev/null; do
+  if rnfb_tart_ssh "$BUILD_VM" "test -f ~/rnfb-tart-manifests/prewarm-complete.json" 2>/dev/null; then
+    rnfb_tart_log "[bake-warmed] prewarm completion marker detected"
+    sleep 3
+    kill "$prewarm_pid" 2>/dev/null || true
+    wait "$prewarm_pid" 2>/dev/null || true
+    break
+  fi
+  sleep 15
+done
+
+if kill -0 "$prewarm_pid" 2>/dev/null; then
+  wait "$prewarm_pid"
+fi
+set -e
+
+rnfb_tart_ssh "$BUILD_VM" "cat ~/rnfb-tart-manifests/warmed-main.json" \
+  > "${RNFB_TART_ROOT}/manifests/warmed-main.json"
+
+kill "$run_pid" 2>/dev/null || true
+wait "$run_pid" 2>/dev/null || true
+tart stop "$BUILD_VM" 2>/dev/null || true
+
+echo "[bake-warmed] tagging ${RNFB_TART_WARMED_VM}"
+tart clone "$BUILD_VM" "$RNFB_TART_WARMED_VM"
+tart delete "$BUILD_VM"
+trap - EXIT
+
+echo "[bake-warmed] done — local VM: ${RNFB_TART_WARMED_VM}"
+echo "[bake-warmed] warmed-main.json commit: $(python3 -c "import json; print(json.load(open('${RNFB_TART_ROOT}/manifests/warmed-main.json'))['main_commit'][:12])")"

scripts/tart/launch-iteration-detached.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# In-VM: one-shot SSH target — start run-iteration.sh detached, print pid, exit.
+set -euo pipefail
+
+REMOTE_DIR="$1"
+RNFB_REPO_ROOT="$2"
+RNFB_BUILD_MODE="$3"
+RNFB_RUN_ID="$4"
+RNFB_ITERATION="$5"
+RNFB_HEADLESS_SIMULATOR="$6"
+
+# shellcheck source=lib/common.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/common.sh"
+
+mkdir -p "$REMOTE_DIR"
+export RNFB_REPO_ROOT RNFB_BUILD_MODE RNFB_RUN_ID RNFB_ITERATION RNFB_HEADLESS_SIMULATOR
+
+nohup bash "${RNFB_TART_SCRIPTS_MOUNT}/run-iteration.sh" </dev/null >/dev/null 2>&1 &
+disown -h 2>/dev/null || disown
+echo $! > "${REMOTE_DIR}/iteration.pid"