Summary
In run.py (confirmed on main @ 224943f), LOCAL_RANK defaults to 1:
RANK = int(os.environ.get('RANK', 0))
WORLD_SIZE = int(os.environ.get('WORLD_SIZE', 1))
LOCAL_WORLD_SIZE = int(os.environ.get("LOCAL_WORLD_SIZE", 1))
LOCAL_RANK = int(os.environ.get("LOCAL_RANK", 1)) # <-- should default to 0
|
LOCAL_RANK = int(os.environ.get("LOCAL_RANK", 1)) |
Every other distributed variable defaults to its identity value (RANK=0, WORLD_SIZE=1, LOCAL_WORLD_SIZE=1). LOCAL_RANK defaulting to 1 is inconsistent and incorrect — a local rank starts at 0.
Impact
LOCAL_RANK is used to compute the per-process GPU slice:
DEVICE_START_IDX = GPU_PER_PROC * LOCAL_RANK
If a multi-process run sets LOCAL_WORLD_SIZE but does not export LOCAL_RANK for rank 0 (e.g. a custom launcher rather than torchrun), process 0 computes DEVICE_START_IDX = GPU_PER_PROC * 1 and binds to the wrong GPU slice; the highest-rank process can index past the GPU list. On a single-GPU machine there is no device index >= 1.
Fix
LOCAL_RANK = int(os.environ.get("LOCAL_RANK", 0))Happy to open a one-line PR.
Summary
In
run.py(confirmed onmain@224943f),LOCAL_RANKdefaults to1:VLMEvalKit/run.py
Line 36 in 224943f
Every other distributed variable defaults to its identity value (
RANK=0,WORLD_SIZE=1,LOCAL_WORLD_SIZE=1).LOCAL_RANKdefaulting to1is inconsistent and incorrect — a local rank starts at0.Impact
LOCAL_RANKis used to compute the per-process GPU slice:If a multi-process run sets
LOCAL_WORLD_SIZEbut does not exportLOCAL_RANKfor rank 0 (e.g. a custom launcher rather thantorchrun), process 0 computesDEVICE_START_IDX = GPU_PER_PROC * 1and binds to the wrong GPU slice; the highest-rank process can index past the GPU list. On a single-GPU machine there is no device index >= 1.Fix
Happy to open a one-line PR.