Skip to content

Commit bfeb3e4

Browse files
committed
[Diag] Revert app_launcher torch-defer (isaac-sim#5633) to reintroduce atfork bug
Surgical hunk-level revert of commit a5eb9ad ("Fixes OmniHub startup in Docker tests", isaac-sim#5633) applied only to source/isaaclab/isaaclab/app/app_launcher.py. Removes the defer-torch mechanism so that `import torch` (and transitively `import numpy`) happens in AppLauncher.__init__ BEFORE SimulationApp's fork() through libomni.platforminfo. If the resolved numpy is 2.3.5, its bundled OpenBLAS pthread_atfork handler will SIGSEGV the canary jobs. Unlike the prior whole-file revert, this preserves PR isaac-sim#5449's `--deterministic` CLI flag and RTX-determinism logic, which landed after isaac-sim#5633 and was wiped as collateral damage in the previous attempt. Companion to the relocated diagnostic conftest in the next commit: source/isaaclab/test/conftest.py prints the resolved numpy + OpenBLAS hash so we can confirm which numpy actually landed. Refs PR isaac-sim#5656 (numpy!=2.3.5 exclusion fix being validated).
1 parent 4f7dd10 commit bfeb3e4

1 file changed

Lines changed: 6 additions & 15 deletions

File tree

source/isaaclab/isaaclab/app/app_launcher.py

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -242,7 +242,6 @@ def __init__(self, launcher_args: argparse.Namespace | dict | None = None, **kwa
242242
# Exposed to train scripts
243243
self.device_id: int # device ID for GPU simulation (defaults to 0)
244244
self.device: str # resolved device string (e.g. "cuda:0" or "cpu")
245-
self._deferred_cuda_device_id: int | None = None
246245
self.local_rank: int # local rank of GPUs in the current node
247246
self.global_rank: int # global rank for multi-node training
248247

@@ -251,7 +250,6 @@ def __init__(self, launcher_args: argparse.Namespace | dict | None = None, **kwa
251250

252251
# Create SimulationApp, passing the resolved self._config to it for initialization
253252
self._create_app()
254-
self._set_deferred_cuda_device()
255253
# Load IsaacSim extensions
256254
self._load_extensions()
257255

@@ -1007,26 +1005,19 @@ def _resolve_device_settings(self, launcher_args: dict):
10071005
launcher_args["physics_gpu"] = self.device_id
10081006
launcher_args["active_gpu"] = self.device_id
10091007

1010-
# Defer importing torch until after SimulationApp starts. Importing
1011-
# torch can import NumPy/OpenBLAS, whose at-fork handlers can crash
1012-
# Kit's platform-info fork during startup.
1008+
# Set the current CUDA device early so that physics backends (e.g. Newton/Warp)
1009+
# that allocate on the "current" device during initialization get the correct GPU.
1010+
# Without this, all ranks may default to cuda:0 for early allocations.
10131011
if "cuda" in device:
1014-
self._deferred_cuda_device_id = self.device_id
1012+
import torch
1013+
1014+
torch.cuda.set_device(self.device_id)
10151015

10161016
# Store the resolved device string for downstream consumers (e.g. sim_launcher)
10171017
self.device = device
10181018

10191019
logger.info("Using device: %s", device)
10201020

1021-
def _set_deferred_cuda_device(self) -> None:
1022-
"""Set the current torch CUDA device after Kit startup."""
1023-
if self._deferred_cuda_device_id is None:
1024-
return
1025-
1026-
import torch
1027-
1028-
torch.cuda.set_device(self._deferred_cuda_device_id)
1029-
10301021
def _resolve_experience_file(self, launcher_args: dict):
10311022
"""Resolve experience file related settings."""
10321023
# Check if input keywords contain an 'experience' file setting

0 commit comments

Comments
 (0)