|
| 1 | +# WiFlow Browser Trainer (`wiflow_browser.html`) |
| 2 | + |
| 3 | +A **single self-contained HTML page** that does the entire camera-supervised |
| 4 | +WiFi-pose loop **in your browser, in your laptop camera's coordinate frame**, as |
| 5 | +a **4-stage gated flow** with a progress stepper (each stage unlocks the next): |
| 6 | + |
| 7 | +0. **CALIBRATE** *(ADR-151 empty-room baseline)* — you step OUT of the space; the |
| 8 | + page captures ~10 s of the quiescent CSI and computes a per-feature running |
| 9 | + **mean + std (Welford)** over the 410-d vector. Every CSI vector afterwards is |
| 10 | + expressed as **deviation from baseline** |
| 11 | + (`x_norm = (x − base_mean) / (base_std + ε)`), so a body's perturbation stands |
| 12 | + out from the static channel. Persisted to IndexedDB. *Can't capture without it.* |
| 13 | +1. **CAPTURE** — MediaPipe Pose runs on your laptop camera → 17 COCO keypoints |
| 14 | + (the *label*), paired with the **baseline-normalized** 410-d ESP32 CSI vector |
| 15 | + (the *input*). A **guided, balanced routine** cycles big on-screen prompts |
| 16 | + (stand / turn / walk / arms / crouch / sit / reach) with a countdown, and a |
| 17 | + **per-pose coverage meter** so you build a balanced dataset, not 2 000 frames |
| 18 | + of standing. |
| 19 | +2. **TRAIN** — a TensorFlow.js MLP learns `CSI → pose` in-browser. Honest |
| 20 | + held-out PCK@0.10 / PCK@0.05 / MPJPE, plus a **mean-pose baseline** the model |
| 21 | + must beat (the project's whole ethos — no baseline-beating signal, it says so). |
| 22 | + *Can't train with <200 samples.* |
| 23 | +3. **INFER** — the trained model drives a skeleton **from WiFi CSI only** |
| 24 | + (baseline-normalized → standardized → model), drawn over the **same** camera |
| 25 | + frame it trained in — so the inferred skeleton **aligns** with the camera |
| 26 | + image. That alignment is the entire point of doing this in-browser instead of |
| 27 | + with a separate Python camera. *Can't infer without a model.* |
| 28 | + |
| 29 | +## Why in-browser |
| 30 | + |
| 31 | +The Python pipeline (`wiflow_capture.py` → `wiflow_train.py` → `wiflow_infer.py`) |
| 32 | +proved the signal is real (held-out PCK@0.10 ≈ 59.5% vs a 50% mean-pose baseline |
| 33 | += +9.4 pp). But it trained in a *different* camera's frame, so the inferred |
| 34 | +skeleton never lined up with the laptop camera. Doing capture + train + infer all |
| 35 | +in the browser with the **same** camera makes the training frame and the |
| 36 | +inference frame identical → the skeleton aligns. |
| 37 | + |
| 38 | +## Compute backends (WebGPU / WASM / WebGL) |
| 39 | + |
| 40 | +Training and inference run on TensorFlow.js. The page selects the backend at |
| 41 | +startup, preferring the fastest available: |
| 42 | + |
| 43 | +- **WebGPU** (Chrome / Edge, secure context — `localhost` qualifies) — GPU compute. |
| 44 | +- **WASM-SIMD** fallback (`tfjs-backend-wasm`, SIMD enabled, `.wasm` from the CDN). |
| 45 | +- **WebGL** last-resort fallback (ships inside tfjs core). |
| 46 | + |
| 47 | +The **active backend is shown as a badge in the header** (`compute: WebGPU` / |
| 48 | +`WASM-SIMD` / `WebGL`) so it's honest about what's actually running. The model |
| 49 | +code is backend-agnostic — tf.js abstracts the device. |
| 50 | + |
| 51 | +## Honesty (baked in) |
| 52 | + |
| 53 | +- The **CAPTURE** skeleton (blue) is the camera = ground truth, labeled as such. |
| 54 | +- The **INFER** skeleton (green) is **CSI-only**, labeled, and **coarse** — the |
| 55 | + real measured held-out PCK is shown, not a marketing number. |
| 56 | +- The **mean-pose baseline** is always computed and shown in TRAIN; the verdict |
| 57 | + states plainly whether the model **beats** it (real signal) or **does not** |
| 58 | + (no usable signal). This guards against the project's retracted 92.9% that |
| 59 | + failed exactly this check. |
| 60 | +- Status banner is strict and mutually exclusive: |
| 61 | + **LIVE** (real `source: "esp32"`) / **SIMULATED — not real** (any other source) |
| 62 | + / **NO-CSI-SERVER**. The page never invents frames. |
| 63 | + |
| 64 | +## How to run |
| 65 | + |
| 66 | +### 1. Start the real sensing-server (provides the CSI WebSocket on :8765) |
| 67 | + |
| 68 | +```bash |
| 69 | +cd v2 |
| 70 | +cargo build -p wifi-densepose-sensing-server |
| 71 | +./target/debug/sensing-server.exe --ws-port 8765 --udp-port 5005 |
| 72 | +``` |
| 73 | + |
| 74 | +A real ESP32-S3 must be provisioned and streaming for `source` to read `esp32` |
| 75 | +(see `CLAUDE.local.md` for the firmware build/provision steps). The page expects |
| 76 | +the verified live endpoint **`ws://localhost:8765/ws/sensing`** with |
| 77 | +`source:"esp32"`, nodes `[9, 13]`, `features.*`, `node_features[].features.*`, |
| 78 | +and `signal_field.values` (400 floats). |
| 79 | + |
| 80 | +### 2. Serve this page over localhost (camera + WebGPU need a localhost/secure origin) |
| 81 | + |
| 82 | +Any static localhost server works. For example: |
| 83 | + |
| 84 | +```bash |
| 85 | +python -m http.server 8099 |
| 86 | +# then open: http://localhost:8099/examples/through-wall/wiflow_browser.html |
| 87 | +``` |
| 88 | + |
| 89 | +(8099 is just the static file server — 8765 is a separate process, the CSI |
| 90 | +WebSocket.) Allow camera access when the browser prompts. |
| 91 | + |
| 92 | +Point at a CSI server on another host with `?ws=`: |
| 93 | + |
| 94 | +``` |
| 95 | +http://localhost:8099/examples/through-wall/wiflow_browser.html?ws=ws://192.168.1.20:8765/ws/sensing |
| 96 | +``` |
| 97 | + |
| 98 | +### 3. Use it |
| 99 | + |
| 100 | +1. **CAPTURE** tab → *enable laptop camera* → *start recording*. Follow the guided |
| 101 | + routine (stand / turn / walk / arms / crouch / sit). A pair is stored only when |
| 102 | + a confident pose AND a fresh live `esp32` CSI frame coexist. Aim for a few |
| 103 | + thousand samples. Samples persist in IndexedDB across refreshes. |
| 104 | +2. **TRAIN** tab → *train model*. Watch the live loss curve, held-out PCK, and the |
| 105 | + baseline verdict. The model saves to IndexedDB. |
| 106 | +3. **INFER** tab → the green skeleton is now driven by WiFi CSI only, aligned over |
| 107 | + your camera. Toggle *hide camera* to see the CSI-only skeleton on black. |
| 108 | + |
| 109 | +## The 410-d CSI vector (matches the Python pipeline exactly) |
| 110 | + |
| 111 | +``` |
| 112 | +[ mean_rssi, variance, motion_band_power, breathing_band_power ] # 4 (features.*) |
| 113 | ++ for node 9 then node 13: [ mean_rssi, variance, motion_band_power ] # 6 (node_features[].features.*) |
| 114 | ++ signal_field.values, padded / truncated to 400 # 400 |
| 115 | += 410-d |
| 116 | +``` |
| 117 | + |
| 118 | +Verified against a real live frame: the in-browser `csiVector()` produces the |
| 119 | +identical 410 vector as `wiflow_capture.py`'s `csi_vector()` (node 9 first, then |
| 120 | +node 13; field zero-padded). |
| 121 | + |
| 122 | +## Libraries (CDN only, no bundler) |
| 123 | + |
| 124 | +| Library | CDN | |
| 125 | +|---|---| |
| 126 | +| TensorFlow.js core | `@tensorflow/tfjs@4.22.0/dist/tf.min.js` | |
| 127 | +| TF.js WebGPU backend | `@tensorflow/tfjs-backend-webgpu@4.22.0/dist/tf-backend-webgpu.min.js` | |
| 128 | +| TF.js WASM backend | `@tensorflow/tfjs-backend-wasm@4.22.0/dist/tf-backend-wasm.min.js` | |
| 129 | +| MediaPipe Pose 0.5 (legacy solutions) | `@mediapipe/pose@0.5/pose.js` | |
| 130 | + |
| 131 | +## Scope / honesty caveats |
| 132 | + |
| 133 | +Same person, same room, same session. **Not** validated cross-day, cross-room, or |
| 134 | +through-wall. The inferred pose is coarse (PCK@0.05 is typically weak). If the |
| 135 | +model does not beat the mean-pose baseline, the page says so — that is a feature. |
0 commit comments