Cleanups: pytetwild aarch64 + Dockerfile.arm-ci

hujc7 · hujc7 · commit b3d7f7c48163 · 2026-05-21T07:39:53.000Z
- source/isaaclab/setup.py: gate pytetwild==0.2.3 with
  `platform_machine != 'aarch64'`. No aarch64 wheel on PyPI; source build
  fails because the transitive `geogram` dep hardcodes `-m64` in its
  CMakeLists. The single call site at sim/schemas/schemas.py already
  lazy-imports it with a clear "install manually" message, so aarch64
  users keep everything except automatic volume-deformable tetrahedralization.

- docker/Dockerfile.arm-ci: new lightweight Dockerfile that layers
  cmake / build-essential / git + EULA env vars + a bash entrypoint onto
  the multi-arch Isaac Sim base image. Replaces the previous inline
  `docker run + apt-get install + docker commit` chain in arm-ci.yaml.

- docker/Dockerfile.base: reverted the libgmp / libmpfr / libeigen3 /
  libcgal / libboost additions from the arm64-conditional apt block —
  those were only needed by pytetwild's fTetWild build, which we no
  longer install on aarch64.

- .github/workflows/arm-ci.yaml: build via docker/Dockerfile.arm-ci
  instead of the inline apt-install-and-commit pattern. Test steps no
  longer need to re-specify --entrypoint or EULA env vars on every
  docker run.
diff --git a/.github/workflows/arm-ci.yaml b/.github/workflows/arm-ci.yaml
@@ -141,93 +141,63 @@ jobs:
             echo "${NGC_API_KEY}" | docker login nvcr.io --username '$oauthtoken' --password-stdin
           fi
 
-      - name: Pull arm64 Isaac Sim image
+      - name: Build arm-ci docker image
         shell: bash
         run: |
           set -euo pipefail
-          # Multi-arch manifest at this tag has both linux/arm64 and linux/amd64.
-          # Docker on an aarch64 host auto-resolves to the arm64 variant.
-          docker pull --platform linux/arm64 \
-            "${{ needs.config.outputs.isaacsim_image_name }}:${{ needs.config.outputs.isaacsim_image_tag }}"
-
-      - name: Install system build deps inside the container
+          # Build the lightweight arm-ci image from docker/Dockerfile.arm-ci.
+          # That Dockerfile layers cmake/build-essential/git plus EULA env vars
+          # and a bash entrypoint onto the multi-arch Isaac Sim base image
+          # (auto-resolves to the arm64 manifest on aarch64 hosts).
+          docker build \
+            --platform linux/arm64 \
+            --build-arg "ISAACSIM_BASE_IMAGE_ARG=${{ needs.config.outputs.isaacsim_image_name }}" \
+            --build-arg "ISAACSIM_VERSION_ARG=${{ needs.config.outputs.isaacsim_image_tag }}" \
+            -f docker/Dockerfile.arm-ci \
+            -t isaac-lab-arm-ci:${{ github.sha }} \
+            docker
+
+      - name: Install isaaclab via ./isaaclab.sh -i + run all tier 1/2 tests
         shell: bash
+        timeout-minutes: 90
         run: |
           set -euo pipefail
-          # pytetwild's fTetWild source build needs libgmp / libmpfr / libeigen3 /
-          # libcgal / libboost; isaaclab's editable install pulls pytetwild as a
-          # hard dep (added in PR isaac-sim/IsaacLab#5710 on 2026-05-20).
-          # We install into a fresh container layer per run so the apt cost (~1-2
-          # min) shows up only when this workflow runs, not on the Sim image.
-          # Persist by committing into a per-run image tagged isaac-lab-arm-ci.
-          # Build a tagged image with deps baked in so subsequent test runs are fast.
-          # Override the Sim image's entrypoint (Kit launcher) with bash so we can
-          # run plain shell commands. EULA env vars set per build.yaml conventions.
-          docker run --name arm-deps-prep --user root \
-            --entrypoint bash \
-            -e OMNI_KIT_ACCEPT_EULA=yes -e ACCEPT_EULA=Y -e ISAAC_SIM_HEADLESS=1 \
-            -e PRIVACY_CONSENT=Y \
-            "${{ needs.config.outputs.isaacsim_image_name }}:${{ needs.config.outputs.isaacsim_image_tag }}" \
-            -c "
-              set -euo pipefail
-              apt-get update
-              apt-get install -y --no-install-recommends \
-                libgmp-dev libmpfr-dev libeigen3-dev libcgal-dev libboost-all-dev \
-                cmake build-essential git
-              rm -rf /var/lib/apt/lists/*
-            "
-          docker commit arm-deps-prep isaac-lab-arm-ci:${{ github.sha }}
-          docker rm arm-deps-prep
-
-      - name: Editable install of isaaclab + isaaclab_tasks in a uv venv
-        shell: bash
-        timeout-minutes: 25
-        run: |
-          set -euo pipefail
-          # All Tier 2 jobs need both packages installed once. We do the install
-          # inside a uv venv mounted under /workspace/isaaclab so subsequent
-          # docker run invocations see the same env_isaaclab_uv directory.
+          # Single docker run because uv-managed Python lives in
+          # $HOME/.cache/uv/python inside the container and is discarded on
+          # container exit, leaving env_isaaclab_uv/bin/python as a dangling
+          # symlink in subsequent containers.
+          #
+          # Install via ./isaaclab.sh -i (the canonical user-facing install
+          # entry point) instead of hand-rolled uv pip install lines. This
+          # picks up _ensure_cuda_torch (re-installs cu130 torch on aarch64
+          # after isaacsim downgrades it to a CPU wheel), nlopt arm prep,
+          # pin-pink dependency probe, etc. — same install path real users
+          # hit on `./isaaclab.sh -i`, so CI failures here are real user bugs.
+          mkdir -p reports
           docker run --rm --user root \
-            --entrypoint bash \
-            -e OMNI_KIT_ACCEPT_EULA=yes -e ACCEPT_EULA=Y -e ISAAC_SIM_HEADLESS=1 \
-            -e PRIVACY_CONSENT=Y \
             -v "${{ github.workspace }}":/workspace/isaaclab \
             -w /workspace/isaaclab \
             --gpus all \
-            isaac-lab-arm-ci:${{ github.sha }} \
-            -c "
+            isaac-lab-arm-ci:${{ github.sha }} "
               set -e
               if ! command -v uv >/dev/null 2>&1; then
                 curl -LsSf https://astral.sh/uv/install.sh | sh
                 export PATH=\$HOME/.local/bin:\$PATH
               fi
+
               uv venv --python 3.12 env_isaaclab_uv
               source env_isaaclab_uv/bin/activate
-              uv pip install -e source/isaaclab
-              uv pip install -e source/isaaclab_assets
-              uv pip install -e source/isaaclab_tasks
               uv pip install pytest pytest-timeout
-              python -c 'import isaaclab, isaaclab_assets, isaaclab_tasks; print(\"editable imports ok\")'
-            "
 
-      - name: Tier 1 — general-arm smoke (torch + scipy)
-        shell: bash
-        continue-on-error: true
-        timeout-minutes: 10
-        run: |
-          set -e
-          mkdir -p reports
-          docker run --rm --user root \
-            --entrypoint bash \
-            -e OMNI_KIT_ACCEPT_EULA=yes -e ACCEPT_EULA=Y -e ISAAC_SIM_HEADLESS=1 \
-            -e PRIVACY_CONSENT=Y \
-            -v "${{ github.workspace }}":/workspace/isaaclab \
-            -w /workspace/isaaclab \
-            --gpus all \
-            isaac-lab-arm-ci:${{ github.sha }} \
-            -c "
-              source env_isaaclab_uv/bin/activate
-              python -m pytest \
+              # Use ./isaaclab.sh -i (mirrors user install path).
+              # -i none installs core submodules + isaacsim + restores cu130 torch on aarch64.
+              ./isaaclab.sh -i none
+              python -c 'import isaaclab, isaaclab_assets, isaaclab_physx, isaaclab_tasks; print(\"editable imports ok\")'
+
+              set +e  # individual tier failures do not abort the script
+
+              echo '::group::Tier 1 — general-arm smoke (torch + scipy)'
+              ./isaaclab.sh -p -m pytest \
                 source/isaaclab/test/deps \
                 --ignore=tools/conftest.py \
                 -m arm_ci \
@@ -236,88 +206,20 @@ jobs:
                 --timeout-method=signal \
                 -v \
                 --junitxml=reports/general-arm.xml
-            "
-
-      - name: Tier 1 — kit-launch-arm (boot Kit headless)
-        shell: bash
-        continue-on-error: true
-        timeout-minutes: 10
-        run: |
-          set -e
-          docker run --rm --user root \
-            --entrypoint bash \
-            -e OMNI_KIT_ACCEPT_EULA=yes -e ACCEPT_EULA=Y -e ISAAC_SIM_HEADLESS=1 \
-            -e PRIVACY_CONSENT=Y \
-            -v "${{ github.workspace }}":/workspace/isaaclab \
-            -w /workspace/isaaclab \
-            --gpus all \
-            isaac-lab-arm-ci:${{ github.sha }} \
-            -c "
-              source env_isaaclab_uv/bin/activate
-              uv pip install --extra-index-url https://pypi.nvidia.com 'isaacsim[all]'
-              timeout 120 python - <<'EOF'
-              import sys
-              from isaaclab.app import AppLauncher
-              sim = AppLauncher(headless=True).app
-              assert sim is not None, 'AppLauncher did not return a SimulationApp'
-              sim.close()
-              sys.exit(0)
-              EOF
-            "
-
-      - name: Tier 2 — kitless-arm (Warp + OvRTX rendering on aarch64)
-        shell: bash
-        continue-on-error: true
-        timeout-minutes: 30
-        run: |
-          set -e
-          docker run --rm --user root \
-            --entrypoint bash \
-            -e OMNI_KIT_ACCEPT_EULA=yes -e ACCEPT_EULA=Y -e ISAAC_SIM_HEADLESS=1 \
-            -e PRIVACY_CONSENT=Y \
-            -v "${{ github.workspace }}":/workspace/isaaclab \
-            -w /workspace/isaaclab \
-            --gpus all \
-            isaac-lab-arm-ci:${{ github.sha }} \
-            -c "
-              source env_isaaclab_uv/bin/activate
-              python -m pytest \
-                source/isaaclab_tasks/test \
-                --ignore=tools/conftest.py \
-                -m arm_ci \
-                --continue-on-collection-errors \
-                --timeout=300 \
-                --timeout-method=signal \
-                -v \
-                --junitxml=reports/kitless-arm.xml
-            "
-
-      - name: Tier 2 — determinism-arm (controllers / math)
-        shell: bash
-        continue-on-error: true
-        timeout-minutes: 20
-        run: |
-          set -e
-          docker run --rm --user root \
-            --entrypoint bash \
-            -e OMNI_KIT_ACCEPT_EULA=yes -e ACCEPT_EULA=Y -e ISAAC_SIM_HEADLESS=1 \
-            -e PRIVACY_CONSENT=Y \
-            -v "${{ github.workspace }}":/workspace/isaaclab \
-            -w /workspace/isaaclab \
-            --gpus all \
-            isaac-lab-arm-ci:${{ github.sha }} \
-            -c "
-              source env_isaaclab_uv/bin/activate
-              python -m pytest \
-                source/isaaclab/test \
-                --ignore=tools/conftest.py \
-                --ignore=source/isaaclab/test/deps \
-                -m arm_ci \
-                --continue-on-collection-errors \
-                --timeout=180 \
-                --timeout-method=signal \
-                -v \
-                --junitxml=reports/determinism-arm.xml
+              echo '::endgroup::'
+
+              echo '::group::Tier 1 — kit-launch-arm (boot Kit headless)'
+              timeout 120 ./isaaclab.sh -p -c \"from isaaclab.app import AppLauncher; sim = AppLauncher(headless=True).app; assert sim is not None, 'AppLauncher did not return a SimulationApp'; sim.close()\"
+              echo '::endgroup::'
+
+              # Tier 2 uses tools/conftest.py's subprocess-per-file orchestrator
+              # (CI_MARKER=arm_ci) so aarch64 Kit re-init across multiple
+              # AppLauncher-at-module-level test files doesn't SIGSEGV — each
+              # test file gets its own Python process.
+              echo '::group::Tier 2 — arm_ci marker discovery (subprocess-per-file)'
+              CI_MARKER=arm_ci python -m pytest tools -v --junitxml=reports/tier2-arm.xml
+              echo '::endgroup::'
+              true
             "
 
       - name: Upload test reports
diff --git a/docker/Dockerfile.arm-ci b/docker/Dockerfile.arm-ci
@@ -0,0 +1,48 @@
+# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+# Lightweight base image for ARM/Spark CI (arm-ci.yaml).
+#
+# Layers on top of the multi-arch nvcr.io/nvidian/isaac-sim image (arm64
+# manifest on aarch64 hosts) and adds:
+#   - cmake / build-essential / git: source-build infra for aarch64 Python
+#     packages that have no prebuilt aarch64 wheel (pin-pink, daqp, etc.).
+#   - EULA env vars + bash entrypoint so the workflow can run plain shell
+#     commands without re-specifying flags per `docker run`.
+#
+# This is intentionally NOT Dockerfile.base — that one builds a full
+# isaac-lab-ci image with isaaclab pre-installed. arm-ci.yaml instead
+# mounts source/ at test time and does a uv editable install, so we only
+# need the apt layer here.
+
+ARG ISAACSIM_BASE_IMAGE_ARG=nvcr.io/nvidian/isaac-sim
+ARG ISAACSIM_VERSION_ARG=latest-develop
+
+FROM ${ISAACSIM_BASE_IMAGE_ARG}:${ISAACSIM_VERSION_ARG}
+
+USER root
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        cmake build-essential git \
+        # imgui-bundle has no aarch64 wheel and source-builds need GL/X11 headers.
+        # swig is needed for the nlopt aarch64 source build.
+        libgl1-mesa-dev libopengl-dev libglx-dev \
+        libx11-dev libxcursor-dev libxi-dev libxinerama-dev libxrandr-dev \
+        swig && \
+    apt-get -y autoremove && apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+ENV ACCEPT_EULA=Y \
+    OMNI_KIT_ACCEPT_EULA=yes \
+    ISAAC_SIM_HEADLESS=1 \
+    PRIVACY_CONSENT=Y \
+    # aarch64 scipy/numpy have a known OpenMP thread-safety issue that
+    # requires libgomp.so.1 to be LD_PRELOAD-ed before scipy imports. Without
+    # it, pytest collection trips an InitError. The scipy error message
+    # spells out this exact fix.
+    LD_PRELOAD=/lib/aarch64-linux-gnu/libgomp.so.1
+
+ENTRYPOINT ["/bin/bash", "-c"]
diff --git a/docker/Dockerfile.base b/docker/Dockerfile.base
@@ -52,14 +52,11 @@ RUN apt-get update && \
 # arm64-only build deps:
 # - imgui-bundle has no prebuilt arm64 wheel; needs GL/X11 dev headers.
 # - swig is required for the nlopt source build (see arm64 nlopt step below).
-# - libgmp-dev / libmpfr-dev / libeigen3-dev / libcgal-dev / libboost-all-dev:
-#   needed by pytetwild's fTetWild source build (no arm64 wheel on PyPI).
 RUN if [ "$(dpkg --print-architecture)" = "arm64" ]; then \
         apt-get update && \
         apt-get install -y --no-install-recommends \
         libgl1-mesa-dev libopengl-dev libglx-dev \
         libx11-dev libxcursor-dev libxi-dev libxinerama-dev libxrandr-dev \
-        libgmp-dev libmpfr-dev libeigen3-dev libcgal-dev libboost-all-dev \
         swig; \
     fi
 
diff --git a/source/isaaclab/changelog.d/jichuanh-apppath-exp-fallback.rst b/source/isaaclab/changelog.d/jichuanh-apppath-exp-fallback.rst
@@ -0,0 +1,11 @@
+Fixed
+^^^^^
+
+* Added a defensive fallback in :class:`isaaclab.app.AppLauncher` so it derives
+  ``EXP_PATH`` from the installed ``isaacsim`` package when the env var is not
+  set. ``isaacsim.bootstrap_kernel`` normally sets ``EXP_PATH`` on first import,
+  but the early-return path in its bootstrap (triggered under some pip install
+  layouts on aarch64) skips the env-var setup. Previously this caused
+  ``KeyError: 'EXP_PATH'`` deep inside ``_resolve_experience_file``; now
+  AppLauncher resolves the path from ``isaacsim.__file__`` and stores it back
+  into the environment so subsequent code can rely on it.
diff --git a/source/isaaclab/changelog.d/jichuanh-conftest-ci-marker.skip b/source/isaaclab/changelog.d/jichuanh-conftest-ci-marker.skip
@@ -0,0 +1 @@
+Skip changelog: CI-infrastructure only. Generalizes tools/conftest.py to read a CI_MARKER env var (defaulting to ISAACSIM_CI_SHORT=true → "isaacsim_ci" for back-compat). Lets cross-platform CI workflows reuse the same subprocess-per-test orchestrator with their own markers (arm_ci for ARM/Spark, windows_ci for Windows) instead of forking conftest.py per platform.
diff --git a/source/isaaclab/changelog.d/jichuanh-pytetwild-aarch64.rst b/source/isaaclab/changelog.d/jichuanh-pytetwild-aarch64.rst
@@ -0,0 +1,8 @@
+Fixed
+^^^^^
+
+* Excluded ``pytetwild`` install on aarch64 platforms. The package has no aarch64 wheel on PyPI and its
+  source build fails (the ``geogram`` CMake dep hardcodes ``-m64``). The single call site in
+  :mod:`isaaclab.sim.schemas` already raises a clear "install pytetwild manually or provide a
+  pre-tetrahedralized UsdGeom.TetMesh" message when the lazy import fails, so aarch64 users keep
+  everything except automatic volume-deformable tetrahedralization.
diff --git a/source/isaaclab/isaaclab/app/app_launcher.py b/source/isaaclab/isaaclab/app/app_launcher.py
@@ -1036,8 +1036,22 @@ def _resolve_experience_file(self, launcher_args: dict):
             launcher_args.get("deterministic", AppLauncher._APPLAUNCHER_CFG_INFO["deterministic"][1])
         )
 
-        # If nothing is provided resolve the experience file based on the headless flag
-        kit_app_exp_path = os.environ["EXP_PATH"]
+        # If nothing is provided resolve the experience file based on the headless flag.
+        # EXP_PATH is normally set by ``isaacsim.bootstrap_kernel()`` on first import.
+        # If it is not set (e.g. on aarch64 where the bootstrap early-return triggered
+        # under certain install layouts), derive it from the installed isaacsim package.
+        kit_app_exp_path = os.environ.get("EXP_PATH")
+        if not kit_app_exp_path:
+            try:
+                import isaacsim as _isaacsim_for_paths
+            except ImportError as e:
+                raise RuntimeError(
+                    "EXP_PATH is not set and the 'isaacsim' package is not importable."
+                    " Install Isaac Sim (`pip install isaacsim` or the binary distribution)"
+                    " before launching AppLauncher."
+                ) from e
+            kit_app_exp_path = os.path.join(os.path.dirname(_isaacsim_for_paths.__file__), "apps")
+            os.environ["EXP_PATH"] = kit_app_exp_path
         isaaclab_app_exp_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), *[".."] * 4, "apps")
         # For Isaac Sim 4.5 compatibility, we use the 4.5 app files in a different folder
         # if launcher_args.get("use_isaacsim_45", False):
diff --git a/source/isaaclab/setup.py b/source/isaaclab/setup.py
@@ -30,8 +30,12 @@
     # procedural-generation
     "trimesh",
     "pyglet>=2.1.6,<3",
-    # tetrahedralization for deformable bodies (pinned: >=0.3 unconditionally imports pyvista at package import time)
-    "pytetwild==0.2.3",
+    # tetrahedralization for deformable bodies (pinned: >=0.3 unconditionally imports pyvista at package import time).
+    # Skip on aarch64: pytetwild has no aarch64 wheel on PyPI and its source build fails because the geogram CMake
+    # dep hardcodes -m64 (x86_64-only). The single call site (sim/schemas/schemas.py) already raises a clear
+    # "install pytetwild manually" error if the lazy import fails, so aarch64 users keep everything except
+    # automatic volume-deformable tetrahedralization.
+    "pytetwild==0.2.3 ; platform_machine != 'aarch64'",
     # image processing
     "transformers==4.57.6",
     "einops",  # needed for transformers, doesn't always auto-install
diff --git a/tools/conftest.py b/tools/conftest.py

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+Skip changelog: CI-infrastructure only. Generalizes tools/conftest.py to read a CI_MARKER env var (defaulting to ISAACSIM_CI_SHORT=true → "isaacsim_ci" for back-compat). Lets cross-platform CI workflows reuse the same subprocess-per-test orchestrator with their own markers (arm_ci for ARM/Spark, windows_ci for Windows) instead of forking conftest.py per platform.`