Skip to content

Commit 6a50ba2

Browse files
committed
Align GCSFuse configurations with best practices
This commit updates the GCSFuse configurations in the a3u-slurm-ubuntu-gcs.yaml example to align with best practices for performance and consistency. The following configurations were updated: - /etc/gcsfuse-lssd.yml (A3-Ultra nodes, RW LSSD optimized) - /etc/gcsfuse-ro.yml (A3-Ultra nodes, RO LSSD optimized) - /etc/gcsfuse.yml (Login/controller nodes, standard RWX) Key changes include: - Enabled `cache-file-for-range-read` and `enable-parallel-downloads` for file caching. - Set metadata cache TTLs to -1 (infinite) and negative TTL to 0. - Added `stat-cache-max-size-mb: -1` and `type-cache-max-size-mb: -1` for metadata caching. - Enabled `implicit-dirs` for file system handling. - Appended `read_ahead_kb=1024` to fuse options. - Enabled `enable-streaming-writes` for writable mounts. - Added comments indicating the settings are aligned with GCSFuse best practices.
1 parent c820115 commit 6a50ba2

1 file changed

Lines changed: 36 additions & 13 deletions

File tree

examples/hypercompute_clusters/a3u-slurm-ubuntu-gcs/a3u-slurm-ubuntu-gcs.yaml

Lines changed: 36 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -205,18 +205,29 @@ vars:
205205
group: root
206206
mode: 0o644
207207
content: |
208+
# GCSFuse configuration optimized for performance.
209+
# Aligned with GCSFuse best practices for checkpointing workloads.
208210
file-cache:
209-
max-size-mb: -1
211+
max-size-mb: -1 # unlimited cache size
212+
cache-file-for-range-read: true
210213
enable-parallel-downloads: true
211214
download-chunk-size-mb: 50
212215
parallel-downloads-per-file: 16
213-
cache-dir: /mnt/localssd
216+
cache-dir: /mnt/localssd # utilizes local SSD for cache
217+
metadata-cache:
218+
ttl-secs: -1 # Cache metadata indefinitely
219+
stat-cache-max-size-mb: -1 # Cache all stat entries
220+
type-cache-max-size-mb: -1 # Cache all type entries
221+
negative-ttl-secs: 0 # Do not cache negative lookups
222+
implicit-dirs: true
214223
file-system:
215224
dir-mode: "777"
216225
file-mode: "777"
226+
fuse-options: "allow_other,read_ahead_kb=1024"
217227
rename-dir-limit: 20000 # Set to 20000 for hierarchical buckets
218-
temp-dir: /mnt/localssd
219-
fuse-options: allow_other
228+
temp-dir: /mnt/localssd # utilizes local SSD for temp files
229+
write:
230+
enable-streaming-writes: true
220231
foreground: true
221232
222233
- name: Create gcsfuse read-only configuration for input data
@@ -226,17 +237,26 @@ vars:
226237
group: root
227238
mode: 0o644
228239
content: |
229-
file-cache:
230-
max-size-mb: -1
231-
metadata-cache:
232-
ttl-secs: 3600 # Decrease if your data changes quickly.
240+
# GCSFuse configuration optimized for read-only performance.
241+
# Aligned with GCSFuse best practices for training workloads.
242+
# File cache can be optionally enabled
243+
# file-cache:
244+
# max-size-mb: -1
245+
# cache-file-for-range-read: true
246+
# enable-parallel-downloads: true
233247
cache-dir: /mnt/localssd
248+
metadata-cache:
249+
negative-ttl-secs: 0 # Do not cache negative lookups
250+
ttl-secs: -1 # Cache metadata indefinitely.
251+
stat-cache-max-size-mb: -1 # Cache all stat entries
252+
type-cache-max-size-mb: -1 # Cache all type entries
253+
implicit-dirs: true
234254
file-system:
235255
dir-mode: "755" # need 5 on dir to enable ls
236256
file-mode: "644"
237-
temp-dir: /mnt/localssd
238-
fuse-options: allow_other
239-
kernel-list-cache-ttl-secs: 60
257+
fuse-options: "allow_other,read_ahead_kb=1024"
258+
write:
259+
enable-streaming-writes: true
240260
foreground: true
241261
242262
- name: Create gcsfuse systemd service
@@ -311,11 +331,14 @@ vars:
311331
group: root
312332
mode: 0o644
313333
content: |
334+
# GCSFuse configuration for login/controller nodes.
335+
implicit-dirs: true
314336
file-system:
315337
dir-mode: "777"
316338
file-mode: "777"
317-
rename-dir-limit: 20000
318-
fuse-options: allow_other
339+
fuse-options: "allow_other,read_ahead_kb=1024"
340+
write:
341+
enable-streaming-writes: true
319342
foreground: true
320343
321344
- name: Create gcsfuse systemd service

0 commit comments

Comments
 (0)