Skip to content

Commit 5438cce

Browse files
committed
docs(gb10): clarify two reclaim cases — fuser+kill (process holds GPU) vs drop_caches (page-cache accumulation)
1 parent ab024f1 commit 5438cce

1 file changed

Lines changed: 16 additions & 5 deletions

File tree

docs/GB10-MEMORY-RECLAIM.md

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,14 +30,25 @@ box (`avail` collapses to single‑digit GB).
3030
# only escalate to kill -9 if TERM fails; a -9 mid-alloc is what leaks in the first place
3131
```
3232
Memory returns to `free` within a few seconds once the true holder dies.
33-
3. (Optional, cache pressure only) drop the page cache — reclaims **buff/cache**,
34-
NOT process/GPU memory:
33+
3. Drop the page cache — reclaims **buff/cache** (file-backed pages), NOT memory
34+
held by a live process:
3535
```sh
3636
ssh gb10 "sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'"
3737
```
38-
In the leak case above this freed ~0.5 GB (cache was only ~1.3 GB) — it is the
39-
right tool for cache pressure, but it does **not** fix a GPU‑orphan leak. Use the
40-
`fuser` step for that.
38+
39+
## Two distinct cases — pick the right tool
40+
41+
The `free` "used" can stay high for **two different reasons**; the fix differs:
42+
43+
| symptom | cause | fix |
44+
|---|---|---|
45+
| `fuser -v /dev/nvidia*` shows an **anomalous python/m04** PID (next to Xorg/gnome/firefox); `nvidia_uvm` refcount > 0 | a live/orphan CUDA process still **holds** GPU memory (device-mapped; `ps`/`nvidia-smi` may not show it) | `kill -TERM <pid>` the holder (step 2). Memory returns when it dies. |
46+
| `fuser` shows **no** compute holder (even under `sudo`); `nvidia_uvm` refcount **0**; `AnonPages` tiny; but `used` is tens of GB | **page-cache accumulation** — e.g. a fresh-process-per-cell sweep where each process re-reads weights/parquet/tilelang artifacts from disk | `sync; echo 3 > /proc/sys/vm/drop_caches` (step 3). Frees it instantly. |
47+
48+
Observed both in one session: a `kill -9`'d batch=4 m04 left an orphan **holding** ~100 GB
49+
(fixed by `fuser`+kill); a 54-cell fresh-process regen left ~60 GB of **page cache** with
50+
no holder (fixed by `drop_caches`, which dropped 65 G → 5 G). Check `fuser` first to tell
51+
which case you're in.
4152

4253
## Verify
4354

0 commit comments

Comments
 (0)