[Bug Report] Hand Tracking does not work with beta release

[Penchekrak]

Describe the bug

Hand tracking for Apple Vision Pro VR teleoperation does not work in beta release. Video is being streamed and application does send hand poses to the server. However problems arise as soon as one enables XR in Isaac Lab kit visualisation.

Steps to reproduce

• local installation -> not containerised
• uv pip installation of isaac sim
• ./isaaclab.sh --install installation of isaac lab 3.0.0-beta2
• sample teleop client built from source with full capalbilities

after running

./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py     --task Isaac-PickPlace-GR1T2-WaistEnabled-Abs-v0     --visualizer kit    --xr  --cloudxr_env avp

and clicking Start XR
following logs conclude the experiment with no visible red markers on user hand joints and zero movement of robot hands/arms

[INFO]: XR enabled changed to: True
[INFO]: Required extensions: ['XR_EXT_hand_tracking', 'XR_KHR_convert_timespec_time', 'XR_MNDX_xdev_space', 'XR_NVX1_action_context', 'XR_NV_opaque_data_channel']
[INFO]: Required extensions: ['XR_EXT_hand_tracking', 'XR_KHR_convert_timespec_time', 'XR_MNDX_xdev_space', 'XR_NVX1_action_context', 'XR_NV_opaque_data_channel']
LOG in xrCreateInstance: Instance created
	createInfo->applicationInfo.applicationName: IsaacLab
	createInfo->applicationInfo.applicationVersion: 1
	createInfo->applicationInfo.engineName: Omniverse Kit XR
	createInfo->applicationInfo.engineVersion: 110001
	createInfo->applicationInfo.apiVersion: 1.1.53
	appinfo.detected.engine.name: Omniverse
	appinfo.detected.engine.version: 0.0.0
	quirks.disable_vulkan_format_depth: false
	quirks.disable_vulkan_format_depth_stencil: false
	quirks.disable_quad_views: true
	quirks.no_validation_error_in_create_ref_space: true
	quirks.skip_end_session: false
	quirks.parallel_views: false
	quirks.no_texture_source_alpha: false

LOG in xrGetSystem: System has quad views but it is not allowed to expose them.
LOG in xrGetSystem: Selected devices
	Head: 'Head Device'
	Eyes: '<none>'
	Left: '<none>'
	Right: '<none>'
	Gamepad: '<none>'
	Hand-Tracking Left (unobstructed): 'Head Device'
	Hand-Tracking Right (unobstructed): 'Head Device'
	Hand-Tracking Left (conforming): '<none>'
	Hand-Tracking Right (conforming): '<none>'
XR_ERROR_FORM_FACTOR_UNSUPPORTED: xrGetSystem(getInfo->formFactor) no matching system (given: 2, first: 1)
LOG in xrCreateVulkanInstanceKHR: Creation of VkInstance:
	result: VK_SUCCESS
	vulkanInstance: 0x76c8e5c27060
	extensions:
		VK_KHR_external_fence_capabilities
		VK_KHR_external_memory_capabilities
		VK_KHR_external_semaphore_capabilities
		VK_KHR_get_physical_device_properties2
		VK_EXT_debug_utils
LOG in xrGetVulkanGraphicsDeviceKHR: Creation of VkDevice:
	result: VK_SUCCESS
	vulkanDevice: 0x76c8f8608890
	vulkanInstance: 0x76c8e5c27060
	external_fence_fd: true
	external_semaphore_fd: true
	timelineSemaphore: true
	extensions:
		VK_KHR_dedicated_allocation
		VK_KHR_external_fence
		VK_KHR_external_memory
		VK_KHR_external_semaphore
		VK_KHR_get_memory_requirements2
		VK_KHR_external_memory_fd
		VK_EXT_depth_range_unrestricted
		VK_KHR_external_semaphore_fd
		VK_KHR_external_fence_fd
		VK_KHR_image_format_list
		VK_KHR_timeline_semaphore
2026-06-18T01:49:29Z [2,338,964ms] [Warning] [omni.usd] Coding Error: in AddXformOp at line 172 of /builds/omniverse/usd-ci/conan/src/0.25.11.kit.2/pxr/usd/usdGeom/xformable.cpp -- The xformOp 'xformOp:transform' already exists in xformOpOrder [[xformOp:transform]].

[INFO]: Acquired OpenXR handles from Kit XR bridge
DeviceIOSession: Creating session with 3 trackers
name: Head Device
serial: N/A
name: Left Head Device Hand
serial: Head Device (0)
name: Right Head Device Hand
serial: Head Device (1)
name: DebugController Left
serial: N/A
name: DebugController Right
serial: N/A
name: Quest Touch Plus Left
serial: N/A
name: Quest Touch Plus Right
serial: N/A
name: PS Sense Left
serial: N/A
name: PS Sense Right
serial: N/A
name: Pico 4 Ultra Left
serial: N/A
name: Pico 4 Ultra Right
serial: N/A
name: Generic Controller Left
serial: N/A
name: Generic Controller Right
serial: N/A
name: Simple Controller Left
serial: N/A
name: Simple Controller Right
serial: N/A
name: EXT Hand Interaction Left
serial: N/A
name: EXT Hand Interaction Right
serial: N/A
name: Derived EXT Hand Interaction Left
serial: N/A
name: Derived EXT Hand Interaction Right
serial: N/A
name: Xbox Controller
serial: N/A
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 45) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 45) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 46) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 46) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 47) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 47) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 48) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 48) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 49) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 49) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 50) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 50) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 51) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 51) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 52) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 52) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 53) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 53) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 54) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 54) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 55) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 55) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 56) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 56) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 57) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 57) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 58) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 58) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 59) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 59) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 60) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 60) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 61) XDev does not support hand tracking
XR_ERROR_FEATURE_UNSUPPORTED: xrCreateHandTrackerEXT(createInfo->next->id == 61) XDev does not support hand tracking
ERROR [ipc_client_system_devices_create_hand_tracker] ipc_call_hand_tracker_create failed: XRT_ERROR_FEATURE_NOT_SUPPORTED [/builds/cloudxr/cloudxr-openxr-runtime/deps/monado/src/xrt/ipc/client/ipc_client_system_devices.c:105]
XR_ERROR_FEATURE_UNSUPPORTED in xrCreateHandTrackerEXT: Did not find any hand-tracking inputs that matched the given hand.
ERROR [ipc_client_system_devices_create_hand_tracker] ipc_call_hand_tracker_create failed: XRT_ERROR_FEATURE_NOT_SUPPORTED [/builds/cloudxr/cloudxr-openxr-runtime/deps/monado/src/xrt/ipc/client/ipc_client_system_devices.c:105]
XR_ERROR_FEATURE_UNSUPPORTED in xrCreateHandTrackerEXT: Did not find any hand-tracking inputs that matched the given hand.
HandTracker initialized (left candidates: 3, right candidates: 3)
ControllerTracker initialized (left + right) with action context
[INFO]: IsaacTeleop session started: IsaacLabTeleop
[INFO]: XR enabled changed to: False

System Info

Describe the characteristic of your environment:

• Commit: [e.g. 28a37ce]
• Isaac Sim Version: 6.0.0.1
• OS: Ubuntu 24.04.4 LTS (noble)
• GPU: RTX 5090
• CUDA: 13.0
• GPU Driver: 580.159.03

Additional context

I'm very sad that happened, let me know if i can contribute to fixing this error and where to look for bugs. I will try with stable release, but the setup overhead of previous release was enough to consider trying this beta one first.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo. -> I'm not sure, probably have something to do with CloudXR

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

• Teleop runs successfully with current release

[Penchekrak]

I have noticed these issues, and maybe they are indeed related but all of them have hand/finger tracking working
#4314
#4352

[Penchekrak]

After updating the AVP client app (https://github.com/isaac-sim/isaac-xr-teleop-sample-client-apple) to utilise CloudXR 6.1.0 the hand tracking started working as expected (almost, we are still missing red spheres as joint markers, however that may be an settings issue).

[RandomOakForest]

Thanks for posting this and posting the workaround.