inconsistent intrinsic camera and projection parameters

[christian-rauch]

System Environment

• Ubuntu Version: 24.04
• ROS Version: jazzy
• Camera Model: Femto Bolt
• Firmware Version: 1.1.2
• Branch: v2-main
• Commit: v2.5.5

Description

In the function convertToCameraInfo, the projection matrix P is set with the same intrinsic parameters as the camera matrix K:

OrbbecSDK_ROS2/orbbec_camera/src/utils.cpp

Lines 22 to 60 in 7d517ea

	sensor_msgs::msg::CameraInfo convertToCameraInfo(OBCameraIntrinsic intrinsic,
	OBCameraDistortion distortion, int width) {
	(void)width;
	sensor_msgs::msg::CameraInfo info;
	info.distortion_model = getDistortionModels(distortion);
	info.width = intrinsic.width;
	info.height = intrinsic.height;
	info.d.resize(8, 0.0);
	info.d[0] = distortion.k1;
	info.d[1] = distortion.k2;
	info.d[2] = distortion.p1;
	info.d[3] = distortion.p2;
	info.d[4] = distortion.k3;
	info.d[5] = distortion.k4;
	info.d[6] = distortion.k5;
	info.d[7] = distortion.k6;
	bool all_zero = std::all_of(info.d.begin(), info.d.end(), [](double val) { return val == 0.0; });
	info.roi.do_rectify = all_zero;
	info.k.fill(0.0);
	info.k[0] = intrinsic.fx;
	info.k[2] = intrinsic.cx;
	info.k[4] = intrinsic.fy;
	info.k[5] = intrinsic.cy;
	info.k[8] = 1.0;
	info.r.fill(0.0);
	info.r[0] = 1;
	info.r[4] = 1;
	info.r[8] = 1;
	info.p.fill(0.0);
	info.p[0] = info.k[0];
	info.p[2] = info.k[2];
	info.p[5] = info.k[4];
	info.p[6] = info.k[5];
	info.p[10] = 1.0;
	return info;
	}

According to the ROS documentation for the sensor_msgs/msg/CameraInfo, the camera (K) and projection (P) matrices are for the raw/distorted/unrectified and undistorted/rectified images, respectively.

Since the distortion coefficients d in the intrinsics are not 0, hence the camera publishes the raw unrectified images, both matrices should be different.

Should the published intrinsics in the K and P matrices be used for the raw images, i.e. K contains the true values, or for the rectified images, i.e. P contains the true values?

Also, can you just fix this issue by setting the K and P matrices correctly?

[ob-yalian]

@christian-rauch Thanks for the detailed analysis and for pointing this out.

By default, our ROS 2 driver publishes image_raw, which is not rectified / undistorted. In this default mode, the published CameraInfo corresponds to the raw image stream. Therefore, having identical K and P matrices is intentional and correct, even if the distortion coefficients D are non-zero. In other words, both K and P describe the same raw camera model when no rectification is applied.

When image undistortion is enabled (e.g. via enable_color_undistortion), the behavior should be different:

• The projection matrix P should be recomputed to reflect the rectified image geometry.

You are correct that this logic is currently incomplete in our implementation — P is not recalculated accordingly. This is an issue on our side.

Thank you for bringing this to our attention. We will fix this behavior and ensure that K, D, and P are published consistently when undistortion is enabled in a future release.

[christian-rauch]

In this default mode, the published CameraInfo corresponds to the raw image stream. Therefore, having identical K and P matrices is intentional and correct, even if the distortion coefficients D are non-zero. In other words, both K and P describe the same raw camera model when no rectification is applied.

This means, that K is the camera matrix that should be used for these raw/distorted images. Rectifying these images and using P on these rectified images would be wrong. See the documentation about the k and p matrices in sensor_msgs/msg/CameraInfo:

# Intrinsic camera matrix for the raw (distorted) images.
#     [fx  0 cx]
# K = [ 0 fy cy]
#     [ 0  0  1]

# Projection/camera matrix
#     [fx'  0  cx' Tx]
# P = [ 0  fy' cy' Ty]
#     [ 0   0   1   0]
# By convention, this matrix specifies the intrinsic (camera) matrix
#  of the processed (rectified) image. That is, the left 3x3 portion
#  is the normal camera intrinsic matrix for the rectified image.

I think this distinction ($f_x \neq f_x^{'}$, ...) quite important to avoid using the wrong intrinsics. IMHO, k and p should only be identical, when the images are already rectified and the distortion coefficients in d are all 0.

When image undistortion is enabled (e.g. via enable_color_undistortion), the behavior should be different:

* The distortion coefficients D should be zeroed.

* The projection matrix P should be recomputed to reflect the rectified image geometry.

I did not know about the parameter enable_color_undistortion, as it is not exposed in the launch file.

If I start the orbbec_camera_node node in colour only mode, with enable_color_undistortion enabled:

ros2 run orbbec_camera orbbec_camera_node --ros-args -p enable_depth:=false -p enable_color:=true -p enable_color_undistortion:=true

it indeed exposes a new topic /color/image_undistorted but I am not receiving images on this topic (ros2 topic echo --no-arr /color/image_undistorted never returns). Also, /color/camera_info still contains identical intrinsics in k and p.

Btw, you can set enable_color_undistortion at runtime (i.e. the parameter is not read-only):

ros2 param set /camera/camera enable_color_undistortion True

but the node will reject this:

Parameter enable_color_undistortion can not be changed in runtime.

It would be better, if you declare this parameter read-only (set read_only in the ParameterDescriptor when declaring the parameter) in the first place, to avoid this issue.

You are correct that this logic is currently incomplete in our implementation — while D is cleared during undistortion, P is not recalculated accordingly. This is an issue on our side.

Thank you for bringing this to our attention. We will fix this behavior and ensure that K, D, and P are published consistently when undistortion is enabled in a future release.

[ob-yalian]

@christian-rauch I understand your point now.

Let me clarify our original design intent in the code. In the default mode, we only publish raw (distorted) images, so users are expected to use K together with D. In this case, P is effectively irrelevant, because no rectification is applied. For simplicity, we filled P with the same values as K. Conceptually, P only needs to be meaningful when rectified images are actually published.But we agree this should be corrected, so that K, D, and P are published consistently.

Regarding the undistortion logic (enable_color_undistortion): you are also right here. At the moment, this functionality is incomplete. Undistortion is not fully applied, yet the distortion coefficients D are incorrectly set to zero, which is wrong. My earlier message was meant to convey that P should only be set (and recomputed) when undistortion is enabled — however, the current implementation does not correctly perform this recomputation. Sorry for my previous explanation wasn't clear enough.

We will fix these problems in the next release.

[bbferka]

Dear @ob-yalian we also came across the same problem. Do you perhaps have an estimate on when the fix would land in a future release?

[ob-yalian]

@bbferka Since v2.6.3, you can enable color undistortion by setting:

enable_color_undistortion := true

After enabling it, the node will publish:

/camera/color/image_undistorted

At the same time, the P matrix in /camera/color/camera_info will be updated to the rectified projection parameters that match the undistorted image.

Note: This topic does not support standalone subscription. You still need to subscribe to:

/camera/color/image_raw

[christian-rauch]

@ob-yalian Would it be possible to use "standard" topic names for the rectified and raw images?

E.g. have a look at the RealSense or the original Azure Kinect node. The topic name image_undistorted is very uncommon. I haven't seen this pattern before. Usually, it is just image_raw for the original unrectified image and image or image_rect for the rectified or otherwise post-processed image.

[dirksavage88]

Hi, I am running into this very issue with the IR camera frames on the gemini 2L. It looks like in ob_camera_node.cpp, you use the video stream profile to calculate the projection matrix extrinsics (which is fine for undistorted images), but I do not see where camera_info.k.at(0) and camera_info.k(4) are coming from? It is not clear to the user in any case.

I would echo @christian-rauch concerns that this deviates from what ros2 packages normally do in regards to camera info and distorted/undistorted images.

In my case I am using the raw distorted left and right IR camera streams and just need K and D. I see that the parameters include an ir_info_url, which appears like it would take in a calibration yam file, but it is unclear where we can pull in individual camera info yaml files (for left and right IR streams for example). Hence I am looking at the code and scratching my head trying to figure out where my actual calibration data goes.

For context, I am using the ros2 camera calibration and a checkerboard pattern to determine D, K, and P