Skip to content

Commit f777efe

Browse files
jonyMarinoMarinaGlocMarinaGloc
authored
Gpu lidar 360 scan (#160)
* Fix nng build with UE5 toolchain and replace deprecated pkg_resources Pass UE5 sysroot and find-root-path-mode args to nng's ExternalProject_Add to match how assimp is already configured, fixing undefined symbol errors (arc4random, strlcpy, __isoc23_strtoull) when linking against UE5's packaged toolchain. Replace deprecated pkg_resources with importlib.resources in the Python client for Python 3.12+ compatibility. * Add GPU lidar example and fix UE 5.7 render pass - Example script lidar_basic_gpu.py plus scene + robot configs that exercise the UGPULidar path via lidar-type: gpu_cylindrical. - Drop redundant VS/PS SetParameters in the UE 5.7 branch of the lidar intensity scene view extension. They dirtied scratch params before SetShaderParameters could commit, tripping the 5.7 !HasParameters() ensure. The View UB is still bound through the shader parameter struct. - Throttled [GPULidar DEBUG] logs across Simulate, TickComponent, and PrePostProcessPass_RT for pipeline diagnosis. * chmod +x blocks_genprojfiles_vscode.sh * Fix GPU lidar point cloud generation and visualization Root cause was a stride mismatch on the readback path: UE5's FVector4 is TVector4<double> (32 bytes), but the compute shader writes 16 bytes per point, so the memcpy silently shifted data across rows. Switching the receiving buffer to FVector4f restores alignment. Other fixes layered on top: - Rebuild Cam1ProjData from the live scene-capture transform every tick (init-time capture happens before the component is parented, collapsing the inverse view-projection to near-identity). - Compute and bind a forward ProjectionMatrix from a fresh FSceneViewProjectionData with axis-swap rotation, transposed for HLSL column-major mul. - Restructure the compute kernel to unconditionally write all four components per thread (sentinel xyz=-1 for misses, w=intensity). - Remove the VS/PS->SetParameters(GetScratchShaderParameters()) calls in the UE 5.7 path that dirtied scratch and tripped the !HasParameters() ensure inside SetShaderParameters. - Visualization: pass color_mode=COLOR_INTENSITY in lidar_basic_gpu.py (segmentation_cloud is filled with -1, which renders black on black). Configure symmetric +/-15 deg vertical FOV to work around the upper*2 / hardcoded-symmetric VFOV bugs in GPULidar.cpp and the shader. * Bisect GPU lidar fix down to minimal set vs main Revert the speculative pieces layered on top of the FVector4f stride fix, keeping only the changes that are demonstrably needed: - GPULidar.cpp: revert per-tick Cam1ProjData rebuild (back to main). - LidarPointCloudCS.usf: revert kernel restructure (back to main's conditional indexed writes). - LidarPointCloudCS.h: restore dead SHADER_PARAMETER_UAV comment. - LidarIntensitySceneViewExtension.cpp: revert UnrealLogger include, readback-bool refactor, and the AddClearUAVFloatPass removal — but keep the SetParameters fix and the FwdProjData → ProjectionMatrix block (the latter is needed for the shader's ProjectWorldToScreen). Net delta vs main is now: FVector4f data type, UE 5.7 SetParameters fix, and forward ProjectionMatrix wiring. * Restore trailing whitespace on blank line in lidar readback block Drops an incidental whitespace-only delta vs main so the file diff shows only the intentional SetParameters and ProjectionMatrix changes. * enabled 360 GPU lidar * LiDAR type detection test * modified mid70 settings --------- Co-authored-by: MarinaGloc <95144231+MarinaGloc@users.noreply.github.com> Co-authored-by: MarinaGloc <tu-correo-de-github@example.com>
1 parent fd2d265 commit f777efe

20 files changed

Lines changed: 2581 additions & 32 deletions

‎CMakeLists.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,11 @@ ExternalProject_Add(nng-external
141141
INSTALL_COMMAND ${CMAKE_COMMAND} --build <BINARY_DIR> --config ${NNG_BUILD_TYPE} --target install
142142
TEST_COMMAND "" # disable test step
143143
CMAKE_ARGS
144+
-DCMAKE_SYSROOT=${CMAKE_SYSROOT}
145+
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=${CMAKE_FIND_ROOT_PATH_MODE_PROGRAM}
146+
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=${CMAKE_FIND_ROOT_PATH_MODE_LIBRARY}
147+
-DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=${CMAKE_FIND_ROOT_PATH_MODE_INCLUDE}
148+
-DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE=${CMAKE_FIND_ROOT_PATH_MODE_PACKAGE}
144149
-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
145150
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
146151
-DCMAKE_EXE_LINKER_FLAGS=${NNG_CMAKE_LINKER_MANIFEST_FLAG}

‎client/python/example_user_scripts/lidar_basic_gpu.py‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
"""
2+
Copyright (C) Microsoft Corporation.
3+
Copyright (C) 2025 IAMAI CONSULTING CORP
4+
MIT License.
5+
6+
Demonstrates using the GPU lidar sensor (lidar-type: gpu_cylindrical) with a
7+
cylindrical scan pattern. This is a variant of lidar_basic.py that loads a
8+
scene whose robot config sets "lidar-type": "gpu_cylindrical", which routes
9+
sensor creation to UGPULidar in UnrealSensorFactory.
10+
"""
11+
12+
import asyncio
13+
14+
from projectairsim import ProjectAirSimClient, Drone, World
15+
from projectairsim.utils import projectairsim_log
16+
from projectairsim.image_utils import ImageDisplay
17+
from projectairsim.lidar_utils import LidarDisplay
18+
19+
20+
# Async main function to wrap async drone commands
21+
async def main():
22+
# Create a Project AirSim client
23+
client = ProjectAirSimClient()
24+
25+
# Initialize an ImageDisplay object to display camera sub-windows
26+
image_display = ImageDisplay()
27+
28+
# ----------------------------------------------------------------------------------
29+
30+
# Initialize a LidarDisplay object for a point cloud visualization sub-window.
31+
# Use intensity coloring — segmentation_cloud is currently filled with -1
32+
# by GPULidar (TODO in source), which would render every point black on
33+
# the black background and make the cloud invisible.
34+
lidar_subwin = image_display.get_subwin_info(2)
35+
lidar_display = LidarDisplay(
36+
x=lidar_subwin["x"],
37+
y=lidar_subwin["y"] + 30, # add 30 y-pix for window title bar
38+
color_mode=LidarDisplay.COLOR_INTENSITY,
39+
)
40+
41+
# ----------------------------------------------------------------------------------
42+
43+
try:
44+
# Connect to simulation environment
45+
client.connect()
46+
47+
# Load the GPU-lidar scene (robot config sets "lidar-type": "gpu_cylindrical")
48+
world = World(client, "scene_lidar_drone_gpu.jsonc", delay_after_load_sec=2)
49+
50+
# Create a Drone object to interact with a drone in the loaded sim world
51+
drone = Drone(client, world, "Drone1")
52+
53+
# Subscribe to chase camera sensor
54+
# chase_cam_window = "ChaseCam"
55+
# image_display.add_chase_cam(chase_cam_window)
56+
# client.subscribe(
57+
# drone.sensors["Chase"]["scene_camera"],
58+
# lambda _, chase: image_display.receive(chase, chase_cam_window),
59+
# )
60+
61+
# Subscribe to the Drone's sensors with a callback to receive the sensor data
62+
rgb_name = "RGB-Image"
63+
image_display.add_image(rgb_name, subwin_idx=0)
64+
client.subscribe(
65+
drone.sensors["DownCamera"]["scene_camera"],
66+
lambda _, rgb: image_display.receive(rgb, rgb_name),
67+
)
68+
69+
depth_name = "Depth-Image"
70+
image_display.add_image(depth_name, subwin_idx=1)
71+
client.subscribe(
72+
drone.sensors["DownCamera"]["depth_camera"],
73+
lambda _, depth: image_display.receive(depth, depth_name),
74+
)
75+
76+
image_display.start()
77+
78+
# ------------------------------------------------------------------------------
79+
80+
lidar_msg_counter = 0
81+
82+
def on_lidar(_, lidar):
83+
nonlocal lidar_msg_counter
84+
lidar_msg_counter += 1
85+
if lidar_msg_counter % 10 == 1:
86+
pc = lidar.get("point_cloud", []) if isinstance(lidar, dict) else []
87+
n_points = len(pc) // 3
88+
if n_points:
89+
xs = pc[0::3]
90+
ys = pc[1::3]
91+
zs = pc[2::3]
92+
ranges = [(x * x + y * y + z * z) ** 0.5 for x, y, z in zip(xs, ys, zs)]
93+
fwd = sum(1 for x in xs if x > 0)
94+
projectairsim_log().info(
95+
f"[client] lidar msg #{lidar_msg_counter}: n={n_points} "
96+
f"forward(x>0)={fwd} ({100*fwd//n_points}%) "
97+
f"range[min={min(ranges):.2f} mean={sum(ranges)/n_points:.2f} max={max(ranges):.2f}] m, "
98+
f"x[{min(xs):.2f},{max(xs):.2f}] "
99+
f"y[{min(ys):.2f},{max(ys):.2f}] "
100+
f"z[{min(zs):.2f},{max(zs):.2f}] "
101+
f"sample={list(pc[:6])}"
102+
)
103+
else:
104+
projectairsim_log().info(
105+
f"[client] lidar msg #{lidar_msg_counter}: 0 points"
106+
)
107+
lidar_display.receive(lidar)
108+
109+
client.subscribe(drone.sensors["lidar1"]["lidar"], on_lidar)
110+
111+
lidar_display.start()
112+
113+
# ------------------------------------------------------------------------------
114+
115+
# Set the drone to be ready to fly
116+
drone.enable_api_control()
117+
drone.arm()
118+
119+
# Fly the drone around the scene
120+
projectairsim_log().info("Move up")
121+
move_task = await drone.move_by_velocity_async(
122+
v_north=0.0, v_east=0.0, v_down=-3.0, duration=4.0
123+
)
124+
await move_task
125+
126+
projectairsim_log().info("Move north")
127+
move_task = await drone.move_by_velocity_async(
128+
v_north=4.0, v_east=0.0, v_down=0.0, duration=12.0
129+
)
130+
await move_task
131+
132+
projectairsim_log().info("Move north-east")
133+
move_task = await drone.move_by_velocity_async(
134+
v_north=4.0, v_east=4.0, v_down=0.0, duration=8.0
135+
)
136+
await move_task
137+
138+
projectairsim_log().info("Move north")
139+
move_task = await drone.move_by_velocity_async(
140+
v_north=4.0, v_east=0.0, v_down=0.0, duration=3.0
141+
)
142+
await move_task
143+
144+
projectairsim_log().info("Move down")
145+
move_task = await drone.move_by_velocity_async(
146+
v_north=0.0, v_east=0.0, v_down=3.0, duration=4.0
147+
)
148+
await move_task
149+
150+
# Shut down the drone
151+
drone.disarm()
152+
drone.disable_api_control()
153+
154+
except Exception as err:
155+
projectairsim_log().error(f"Exception occurred: {err}", exc_info=True)
156+
157+
finally:
158+
# Always disconnect from the simulation environment to allow next connection
159+
client.disconnect()
160+
161+
image_display.stop()
162+
163+
# ------------------------------------------------------------------------------
164+
165+
lidar_display.stop()
166+
167+
# ------------------------------------------------------------------------------
168+
169+
170+
if __name__ == "__main__":
171+
asyncio.run(main()) # Runner for async main function

0 commit comments

Comments
 (0)