docs(gb10): memory reclaim recipe — fuser /dev/nvidia-uvm + SIGTERM frees orphan GPU mem without reboot; drop_caches is cache-only

Captures the incident: kill -9 of a batch=4 m04 mid-CUDA-alloc left ~100GB
device-mapped (AnonPages tiny, ps/pgrep/nvidia-smi blind). fuser -v /dev/nvidia-uvm
finds the orphan; SIGTERM releases it. drop_caches freed only ~0.5G (cache-only).
Plus the 70GB benchmark memory cap.

Author: apstenku123

docs/GB10-MEMORY-RECLAIM.md

@@ -0,0 +1,58 @@
+# gb10 — reclaiming "stuck" memory WITHOUT a reboot
+
+On the GB10 (Grace‑Blackwell, unified LPDDR5X), GPU allocations are carved out of
+system RAM. When a CUDA process is `kill -9`'d **mid‑allocation**, the runtime
+never releases the device memory, so `free` keeps showing it as **used** even
+though no process appears to own it. This looks like a 100 GB leak and a near‑frozen
+box (`avail` collapses to single‑digit GB).
+
+## Why `ps`/`pgrep`/`nvidia-smi` don't find it
+
+- The leaked memory is **device‑mapped**, not anonymous: `grep AnonPages /proc/meminfo`
+  stays tiny (e.g. 495 MB) while `free` shows ~100 GB used. `ps --sort=-rss` shows
+  nothing large (top process ~0.2 GB).
+- `nvidia-smi --query-compute-apps` / FB memory report **N/A** on GB10 (unified mem).
+- A pattern `pkill -f m04_train_step` can **miss** the holder: a `timeout 1800 python …`
+  wrapper or a re‑exec'd child has a different cmdline, so the orphan survives.
+
+## The reclaim recipe (no reboot, no `--gpu-reset`)
+
+1. Find the real holder of CUDA managed memory — it lives on **`/dev/nvidia-uvm`**,
+   not `/dev/nvidia0`:
+   ```sh
+   ssh gb10 'fuser -v /dev/nvidia-uvm /dev/nvidia-uvm-tools'
+   ```
+   This lists PIDs that `ps`/`pgrep`‑by‑name do not. The anomalous `python` entry
+   (next to legit Xorg/gnome/firefox display holders) is the orphan.
+2. Kill it **cleanly** so the CUDA runtime releases the allocation:
+   ```sh
+   ssh gb10 'kill -TERM <pid>'      # SIGTERM first — lets CUDA free device mem
+   # only escalate to kill -9 if TERM fails; a -9 mid-alloc is what leaks in the first place
+   ```
+   Memory returns to `free` within a few seconds once the true holder dies.
+3. (Optional, cache pressure only) drop the page cache — reclaims **buff/cache**,
+   NOT process/GPU memory:
+   ```sh
+   ssh gb10 "sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'"
+   ```
+   In the leak case above this freed ~0.5 GB (cache was only ~1.3 GB) — it is the
+   right tool for cache pressure, but it does **not** fix a GPU‑orphan leak. Use the
+   `fuser` step for that.
+
+## Verify
+
+```sh
+ssh gb10 'free -g | awk "/^Mem:/{print \"used=\"\$3\"G avail=\"\$7\"G\"}"'
+# healthy gb10 idle baseline: used≈5-6G, avail≈115-119G of 121G
+```
+
+## Prevention
+
+- Never `kill -9` a CUDA process unless `kill -TERM` failed. SIGTERM lets MLX/torch
+  tear down the context and release device memory; SIGKILL mid‑alloc leaks it.
+- After any kill, confirm with `fuser -v /dev/nvidia-uvm` that no orphan remains,
+  then re‑check `free`.
+- Memory discipline for benchmarks on this box: ramp slowly (start bs=1 seq=512),
+  sample `free -g` live, and abort if used grows past **70 GB** — `bs=4 × seq=4096`
+  (= 16,384 tok/step, 32× the bs=1/seq=512 footprint) does not fit this config in
+  MLX‑eager and must not be forced.