Skip to content

Commit 4acba70

Browse files
pbarejkopbarejko
authored
Merge branch 'develop' into fix/fabric-prepare-for-reuse
Signed-off-by: Piotr Barejko <pbarejko@nvidia.com>
2 parents 9e3dc50 + 261d077 commit 4acba70

23 files changed

Lines changed: 693 additions & 184 deletions

File tree

‎.github/workflows/wheel.yml‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,28 @@ jobs:
4646
python-version: "3.12"
4747
architecture: x64
4848

49+
# Compose Docker-image-style metadata for the artifact. The artifact
50+
# name is what QA sees in `gh run download`, so we make it scannable:
51+
# isaaclab-<VERSION>-build<RUN_NUMBER>-<SHA7>. The wheel inside follows
52+
# PEP 440 (VERSION+buildN.SHA7) since pip requires that format.
53+
- name: Compute wheel metadata
54+
id: meta
55+
run: |
56+
set -euo pipefail
57+
version=$(cat VERSION)
58+
sha_slug="${GITHUB_SHA:0:7}"
59+
echo "artifact_name=isaaclab-${version}-build${{ github.run_number }}-${sha_slug}" >> "$GITHUB_OUTPUT"
60+
4961
- name: Build wheel
62+
env:
63+
WHEEL_BUILD_NUMBER: ${{ github.run_number }}
64+
WHEEL_SHA: ${{ github.sha }}
5065
run: bash tools/wheel_builder/build.sh
5166

5267
- name: Upload wheel artifact
5368
uses: actions/upload-artifact@v7
5469
with:
55-
name: ${{ github.event_name == 'pull_request' && format('isaaclab-wheel-pr-{0}-{1}', github.event.pull_request.number, github.sha) || format('isaaclab-wheel-{0}-{1}', github.ref_name, github.sha) }}
70+
name: ${{ steps.meta.outputs.artifact_name }}
5671
path: tools/wheel_builder/build/dist/isaaclab-*.whl
5772
if-no-files-found: error
5873
retention-days: 30

‎README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,11 @@ This branch is a development branch for Isaac Sim 6.0, which is currently only a
1818
For installation, please refer to the Isaac Sim GitHub repo to build the latest Isaac Sim branch, and follow the binary installation method in the
1919
Isaac Lab documentation for Isaac Lab installation.
2020

21+
> [!WARNING]
22+
> A recent breaking change on the Isaac Lab `develop` branch is not compatible with the `develop` branch of Isaac Sim on GitHub.
23+
> To run Isaac Lab with Isaac Sim's GitHub `develop` branch, use Isaac Lab commit [`f0234a82e432e2a0b0f0a26ca3c5b59e527ddaaa`](https://github.com/isaac-sim/IsaacLab/commit/f0234a82e432e2a0b0f0a26ca3c5b59e527ddaaa) or an earlier commit.
24+
> Alternatively, use the Isaac Lab [`v3.0.0-beta`](https://github.com/isaac-sim/IsaacLab/tree/v3.0.0-beta) tag.
25+
2126
Note that this branch is currently under active development and may experience breaking changes or error messages.
2227
Performance issues and regressions may also be observed in some use cases.
2328

‎docs/source/_static/how-to/howto_profile_nsys_example.png‎

89.9 KB
Loading

‎docs/source/how-to/index.rst‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,3 +207,14 @@ This guide explains 2 features that can speed up stage initialization, **fabric
207207
:maxdepth: 1
208208

209209
optimize_stage_creation
210+
211+
212+
Profiling Isaac Lab with Nsight Systems
213+
---------------------------------------
214+
215+
This guide explains how to profile Isaac Lab tasks with NVIDIA Nsight Systems for runtime performance analysis.
216+
217+
.. toctree::
218+
:maxdepth: 1
219+
220+
profile_with_nsys

‎docs/source/how-to/profile_with_nsys.rst‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
Profiling Isaac Lab with Nsight Systems
2+
=======================================
3+
4+
.. currentmodule:: isaaclab
5+
6+
Isaac Lab supports CPU and GPU profiling via **NVIDIA Nsight Systems (nsys)** for runtime performance analysis. This can help identify GPU/CPU bottlenecks and determine the best configuration for your environments and tasks. Profiling adds modest runtime overhead and produces large output files, so it's best suited to targeted investigations rather than long unattended training runs.
7+
8+
Common Use Cases
9+
----------------
10+
11+
- **My training iteration is slow** - capture 3-5 iterations to see whether time is mostly spent on physics, rendering, env reset, observation, etc.
12+
- **Env init takes 40 seconds** - profile a single launch to see the import times for each module, kernel compilation, etc.
13+
- **Which physics/renderer backend should I use?** - profile your task with different backend combinations to find the best fit.
14+
- **Are my GPU kernels efficient?** - dive into the CUDA rows in nsys to identify when the GPU is idle (optimization opportunity).
15+
- **Did my code changes make things slower?** - A/B profile comparison before and after your change.
16+
17+
18+
Quick Start
19+
-----------
20+
21+
This section walks you through everything needed to capture your first nsys profile.
22+
23+
Prerequisites
24+
~~~~~~~~~~~~~
25+
26+
- `Nsight Systems <https://developer.nvidia.com/nsight-systems>`_ - install for your platform.
27+
- ``nvtx`` Python package for source code instrumentation, installed into your Isaac Lab environment:
28+
29+
.. code-block:: bash
30+
31+
./isaaclab.sh -p -m pip install nvtx
32+
33+
34+
Running a Profile
35+
~~~~~~~~~~~~~~~~~
36+
37+
The following command shows how to capture a profile for the ``Isaac-Cartpole-v0`` task via the ``rsl_rl`` training framework with 3 iterations:
38+
39+
.. code-block:: bash
40+
41+
nsys profile \
42+
-t nvtx,cuda \
43+
--python-functions-trace=scripts/benchmarks/nsys_trace.json \
44+
-o my_profile \
45+
./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py \
46+
--task=Isaac-Cartpole-v0 \
47+
--headless \
48+
--max_iterations=3
49+
50+
Flags:
51+
52+
- ``-t nvtx,cuda`` - capture NVTX ranges (CPU swim-lanes) and CUDA activity (GPU row).
53+
- ``--python-functions-trace=...`` - the function annotations file; ships with Isaac Lab.
54+
- ``-o my_profile`` - output path; nsys appends ``.nsys-rep``.
55+
56+
Reading the Resulting Profile
57+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
58+
59+
Launch the Nsight Systems UI, go to **File > Open**, and select your ``.nsys-rep`` file that you generated in the previous step. Expand the **Threads** section and the domain swim-lanes will appear as separate rows. Clicking on a function will highlight related functions in other lanes. Expand the **CUDA HW** row to cross-reference with GPU kernels fired during that CPU range.
60+
61+
.. image:: ../_static/how-to/howto_profile_nsys_example.png
62+
:alt: Domain swim-lanes in nsys-ui
63+
64+
65+
The Trace JSON
66+
--------------
67+
68+
The trace definition file lives at ``scripts/benchmarks/nsys_trace.json`` and defines metadata for the Python functions of interest in Isaac Lab. nsys monkey-patches these functions to automatically emit NVTX ranges at runtime.
69+
70+
Schema:
71+
72+
.. code-block:: json
73+
74+
[
75+
{
76+
"domain": "MyDomain",
77+
"color": "0x9C27B0",
78+
"module": "isaaclab.envs.manager_based_env",
79+
"functions": [
80+
"ManagerBasedEnv.step",
81+
{"function": "ManagerBasedEnv.reset", "color": "0xAB47BC"}
82+
]
83+
}
84+
]
85+
86+
87+
After editing the JSON, run the sync test to confirm every entry resolves:
88+
89+
.. code-block:: bash
90+
91+
./isaaclab.sh -p -m pytest scripts/benchmarks/test/test_nsys_trace.py
92+
93+
94+
Troubleshooting
95+
---------------
96+
97+
- **An expected domain doesn't appear in the timeline:**
98+
99+
- Confirm ``nvtx`` is installed in your Isaac Lab environment (``./isaaclab.sh -p -m pip show nvtx``).
100+
- Make sure the function is actually called during the profiled run.
101+
- Verify the JSON entry by running the sync test (above).
102+
103+
- **Sync test fails with an** ``AttributeError`` - a function listed in the JSON no longer exists at the given path.
104+
105+
- Fix the path to point at the renamed function, or
106+
- Remove the entry if the function was deleted.
107+
108+
109+
See Also
110+
--------
111+
112+
- :doc:`simulation_performance` - broader simulation performance tuning tips.
113+
- `Nsight Systems User Guide <https://docs.nvidia.com/nsight-systems/UserGuide/index.html>`_ - official ``nsys`` documentation.
114+
- `NVTX Python package <https://nvtx.readthedocs.io/>`_ - the package nsys uses to emit NVTX ranges from Python.

‎docs/source/setup/installation/index.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,13 @@ installation methods.
4242

4343
.. caution::
4444

45+
**Compatibility warning for Isaac Sim GitHub develop:** A recent breaking change on the Isaac Lab
46+
``develop`` branch is not compatible with the ``develop`` branch of Isaac Sim on GitHub. To run
47+
Isaac Lab with Isaac Sim's GitHub ``develop`` branch, use Isaac Lab commit
48+
`f0234a82e432e2a0b0f0a26ca3c5b59e527ddaaa <https://github.com/isaac-sim/IsaacLab/commit/f0234a82e432e2a0b0f0a26ca3c5b59e527ddaaa>`__
49+
or an earlier commit. Alternatively, use the Isaac Lab
50+
`v3.0.0-beta <https://github.com/isaac-sim/IsaacLab/tree/v3.0.0-beta>`__ tag.
51+
4552
We have dropped support for Isaac Sim versions 5.1.0 and below. We recommend using the latest
4653
Isaac Sim 6.0.0 release to benefit from the latest features and improvements.
4754

‎docs/source/setup/installation/source_installation.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,15 @@ or want to test Isaac Lab with the nightly version of Isaac Sim.
2323
The following instructions are adapted from the `Isaac Sim documentation <https://github.com/isaac-sim/IsaacSim?tab=readme-ov-file#quick-start>`_
2424
for the convenience of users.
2525

26+
.. warning::
27+
28+
**Compatibility warning for Isaac Sim GitHub develop:** A recent breaking change on the Isaac Lab
29+
``develop`` branch is not compatible with the ``develop`` branch of Isaac Sim on GitHub. To run
30+
Isaac Lab with Isaac Sim's GitHub ``develop`` branch, use Isaac Lab commit
31+
`f0234a82e432e2a0b0f0a26ca3c5b59e527ddaaa <https://github.com/isaac-sim/IsaacLab/commit/f0234a82e432e2a0b0f0a26ca3c5b59e527ddaaa>`__
32+
or an earlier commit. Alternatively, use the Isaac Lab
33+
`v3.0.0-beta <https://github.com/isaac-sim/IsaacLab/tree/v3.0.0-beta>`__ tag.
34+
2635
.. attention::
2736

2837
Building Isaac Sim from source requires Ubuntu 22.04 LTS or higher.

‎scripts/benchmarks/nsys_trace.json‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
[
2+
{
3+
"_comment": "=== PYTHON IMPORTS (tracks module loading) ===",
4+
"domain": "Python-Imports",
5+
"color": "0x9E9E9E",
6+
"module": "importlib",
7+
"functions": ["import_module"]
8+
},
9+
{
10+
"domain": "Python-Imports",
11+
"color": "0x9E9E9E",
12+
"module": "importlib._bootstrap",
13+
"functions": ["_find_and_load", "_load_unlocked"]
14+
},
15+
{
16+
"_comment": "=== WARP (verified working) ===",
17+
"domain": "Warp",
18+
"color": "0xF44336",
19+
"module": "warp",
20+
"functions": ["launch", "synchronize", "copy", "zeros", "empty"]
21+
},
22+
{
23+
"_comment": "=== PYTORCH (verified working) ===",
24+
"domain": "PyTorch",
25+
"color": "0x3F51B5",
26+
"module": "torch.autograd",
27+
"functions": ["backward"]
28+
},
29+
{
30+
"_comment": "=== USD (verified working) ===",
31+
"domain": "USD",
32+
"color": "0x795548",
33+
"module": "pxr.UsdGeom",
34+
"functions": ["Xform", "Mesh", "Sphere", "Cube"]
35+
},
36+
{
37+
"_comment": "=== ISAACLAB ENVIRONMENTS ===",
38+
"domain": "IsaacLab-Env",
39+
"color": "0x9C27B0",
40+
"module": "isaaclab.envs.manager_based_rl_env",
41+
"functions": [
42+
{"function": "ManagerBasedRLEnv.__init__", "color": "0x9C27B0"},
43+
{"function": "ManagerBasedRLEnv.step", "color": "0xAB47BC"}
44+
]
45+
},
46+
{
47+
"domain": "IsaacLab-Env",
48+
"color": "0x9C27B0",
49+
"module": "isaaclab.envs.manager_based_env",
50+
"functions": [
51+
{"function": "ManagerBasedEnv.__init__", "color": "0x9C27B0"},
52+
{"function": "ManagerBasedEnv.step", "color": "0xAB47BC"},
53+
{"function": "ManagerBasedEnv.reset", "color": "0xBA68C8"},
54+
{"function": "ManagerBasedEnv._reset_idx", "color": "0xCE93D8"}
55+
]
56+
},
57+
{
58+
"_comment": "=== ISAACLAB SIMULATION ===",
59+
"domain": "IsaacLab-Sim",
60+
"color": "0x4CAF50",
61+
"module": "isaaclab.sim.simulation_context",
62+
"functions": [
63+
{"function": "SimulationContext.__init__", "color": "0x4CAF50"},
64+
{"function": "SimulationContext.reset", "color": "0x66BB6A"},
65+
{"function": "SimulationContext.step", "color": "0x81C784"}
66+
]
67+
},
68+
{
69+
"_comment": "=== ISAACLAB SCENE ===",
70+
"domain": "IsaacLab-Scene",
71+
"color": "0x2196F3",
72+
"module": "isaaclab.scene.interactive_scene",
73+
"functions": [
74+
{"function": "InteractiveScene.__init__", "color": "0x2196F3"},
75+
{"function": "InteractiveScene.reset", "color": "0x42A5F5"},
76+
{"function": "InteractiveScene.write_data_to_sim", "color": "0x64B5F6"},
77+
{"function": "InteractiveScene.update", "color": "0x90CAF9"}
78+
]
79+
},
80+
{
81+
"_comment": "=== ISAACLAB MANAGERS ===",
82+
"domain": "IsaacLab-Managers",
83+
"color": "0xFF9800",
84+
"module": "isaaclab.managers.observation_manager",
85+
"functions": [
86+
{"function": "ObservationManager.__init__", "color": "0xFF9800"},
87+
{"function": "ObservationManager.reset", "color": "0xFFA726"},
88+
{"function": "ObservationManager.compute", "color": "0xFFB74D"}
89+
]
90+
},
91+
{
92+
"domain": "IsaacLab-Managers",
93+
"color": "0xFF9800",
94+
"module": "isaaclab.managers.action_manager",
95+
"functions": [
96+
{"function": "ActionManager.__init__", "color": "0xFF9800"},
97+
{"function": "ActionManager.reset", "color": "0xFFA726"},
98+
{"function": "ActionManager.process_action", "color": "0xFFB74D"},
99+
{"function": "ActionManager.apply_action", "color": "0xFFCC80"}
100+
]
101+
},
102+
{
103+
"domain": "IsaacLab-Managers",
104+
"color": "0xFF9800",
105+
"module": "isaaclab.managers.reward_manager",
106+
"functions": [
107+
{"function": "RewardManager.__init__", "color": "0xFF9800"},
108+
{"function": "RewardManager.reset", "color": "0xFFA726"},
109+
{"function": "RewardManager.compute", "color": "0xFFB74D"}
110+
]
111+
},
112+
{
113+
"_comment": "=== ISAACLAB ASSETS ===",
114+
"domain": "IsaacLab-Assets",
115+
"color": "0x607D8B",
116+
"module": "isaaclab.assets.articulation.articulation",
117+
"functions": [
118+
{"function": "Articulation.__init__", "color": "0x607D8B"},
119+
{"function": "Articulation.reset", "color": "0x78909C"},
120+
{"function": "Articulation.write_data_to_sim", "color": "0x90A4AE"},
121+
{"function": "Articulation.update", "color": "0xB0BEC5"}
122+
]
123+
},
124+
{
125+
"_comment": "=== ISAACLAB SENSORS ===",
126+
"domain": "IsaacLab-Sensors",
127+
"color": "0x00BCD4",
128+
"module": "isaaclab.sensors.camera.camera",
129+
"functions": [
130+
{"function": "Camera.__init__", "color": "0x00BCD4"},
131+
{"function": "Camera.reset", "color": "0x26C6DA"},
132+
{"function": "Camera.update", "color": "0x4DD0E1"}
133+
]
134+
},
135+
{
136+
"_comment": "=== RSL-RL ===",
137+
"domain": "RSL-RL",
138+
"color": "0x673AB7",
139+
"module": "rsl_rl.runners.on_policy_runner",
140+
"functions": [
141+
{"function": "OnPolicyRunner.__init__", "color": "0x673AB7"},
142+
{"function": "OnPolicyRunner.learn", "color": "0x7E57C2"}
143+
]
144+
},
145+
{
146+
"domain": "RSL-RL",
147+
"color": "0x673AB7",
148+
"module": "rsl_rl.algorithms.ppo",
149+
"functions": [
150+
{"function": "PPO.__init__", "color": "0x673AB7"},
151+
{"function": "PPO.act", "color": "0x7E57C2"},
152+
{"function": "PPO.update", "color": "0x9575CD"}
153+
]
154+
},
155+
{
156+
"_comment": "=== NEWTON WARP RENDERER ===",
157+
"domain": "NewtonWarpRenderer",
158+
"color": "0xE91E63",
159+
"module": "isaaclab_newton.renderers.newton_warp_renderer",
160+
"functions": [
161+
{"function": "NewtonWarpRenderer.update_transforms", "color": "0x2196F3"},
162+
{"function": "NewtonWarpRenderer.render", "color": "0x4CAF50"},
163+
{"function": "NewtonWarpRenderer.read_output", "color": "0xFF9800"}
164+
]
165+
}
166+
]

0 commit comments

Comments
 (0)