Skip to content

Commit fa31632

Browse files
Add: Monitor RAM usage for local pipelines with WSL
Closes #216
1 parent 18f8ff2 commit fa31632

1 file changed

Lines changed: 40 additions & 0 deletions

File tree

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
id: 4f69163546
3+
question: How to monitor RAM usage when running data pipelines locally?
4+
sort_order: 58
5+
---
6+
7+
Monitoring RAM usage when running data pipelines locally is important to prevent out-of-memory failures. The following steps show how to monitor memory on Windows with WSL (Ubuntu by default):
8+
9+
1) Install Windows Subsystem for Linux with Ubuntu as the default distribution:
10+
11+
```
12+
wsl --install
13+
```
14+
15+
2) To see static memory usage (RAM) consumed up to the current time (within WSL):
16+
17+
```
18+
free -h
19+
```
20+
21+
3) To see memory usage in real time while a process is running (refreshes at your chosen interval):
22+
23+
```
24+
watch -n 1 free -h
25+
```
26+
27+
4) If you are using Windows, allocate resources to WSL by configuring the .wslconfig file in your Windows home directory (e.g., C:\\Users\\<your_user>\\.wslconfig). A simple setup:
28+
29+
```
30+
[wsl2]
31+
memory=6GB # Memory to be allocated
32+
processors=4 # Processors to be allocated
33+
swap=10GB # Disk memory used to avoid crashes (slower but safer)
34+
```
35+
36+
5) Restart WSL to apply changes:
37+
38+
```
39+
wsl --shutdown
40+
```

0 commit comments

Comments
 (0)