You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/GB10-MEMORY-RECLAIM.md
+16-5Lines changed: 16 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,14 +30,25 @@ box (`avail` collapses to single‑digit GB).
30
30
# only escalate to kill -9 if TERM fails; a -9 mid-alloc is what leaks in the first place
31
31
```
32
32
Memory returns to `free` within a few seconds once the true holder dies.
33
-
3.(Optional, cache pressure only) drop the page cache — reclaims **buff/cache**,
34
-
NOT process/GPU memory:
33
+
3.Drop the page cache — reclaims **buff/cache** (file-backed pages), NOT memory
34
+
held by a live process:
35
35
```sh
36
36
ssh gb10 "sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'"
37
37
```
38
-
In the leak case above this freed ~0.5 GB (cache was only ~1.3 GB) — it is the
39
-
right tool for cache pressure, but it does **not** fix a GPU‑orphan leak. Use the
40
-
`fuser` step for that.
38
+
39
+
## Two distinct cases — pick the right tool
40
+
41
+
The `free` "used" can stay high for **two different reasons**; the fix differs:
42
+
43
+
| symptom | cause | fix |
44
+
|---|---|---|
45
+
|`fuser -v /dev/nvidia*` shows an **anomalous python/m04** PID (next to Xorg/gnome/firefox); `nvidia_uvm` refcount > 0 | a live/orphan CUDA process still **holds** GPU memory (device-mapped; `ps`/`nvidia-smi` may not show it) |`kill -TERM <pid>` the holder (step 2). Memory returns when it dies. |
46
+
|`fuser` shows **no** compute holder (even under `sudo`); `nvidia_uvm` refcount **0**; `AnonPages` tiny; but `used` is tens of GB |**page-cache accumulation** — e.g. a fresh-process-per-cell sweep where each process re-reads weights/parquet/tilelang artifacts from disk |`sync; echo 3 > /proc/sys/vm/drop_caches` (step 3). Frees it instantly. |
47
+
48
+
Observed both in one session: a `kill -9`'d batch=4 m04 left an orphan **holding**~100 GB
49
+
(fixed by `fuser`+kill); a 54-cell fresh-process regen left ~60 GB of **page cache** with
50
+
no holder (fixed by `drop_caches`, which dropped 65 G → 5 G). Check `fuser` first to tell
0 commit comments