How to project 3D data into pixel coordinate?

Hello,

The scenario is I have a RGB-D camera in robot, and by subscribing camera/depth_registered/points (sensor_msgs/PointCloud2) get the point cloud data. What I want is to project the PointCloud2 data into pixel coordinate. My question is that is there any module / library which is hardware independent (the camera could either be Kinect or RealSense or what ever RGB-D camera) could help to do this?

Thanks

You don’t need a library for that, just use the pinhole camera model
here is a small python function:
def world_to_pixel(self, point):
x = point[0]
y = point[1]
z = point[2]
px = xself.K[0][0]/z + self.K[0][2]
py = y
self.K[1][1]/z + self.K[1][2]
return np.asarray([px, py])

K is the 3x3 camera matrix you get from camera_info
self.K[0][0] = cam_info.K[0]
self.K[0][2] = cam_info.K[2]
self.K[1][1] = cam_info.K[4]
self.K[1][2] = cam_info.K[5]
self.K[2][2] = 1

1 Like

Even simpler: all the examples you gave of “depth_registered” point clouds
are organized: point (u, v) corresponds to pixel (u, v) in the image.

If you want the points in a different RGB frame and you have the depth
image, it’s best to register the depth image into that RGB frame and then
calculate the point cloud (which will also be organized). Both can be done
using depth_image_proc.

If you have an arbitrary point cloud, you’ll have to use the pinhole camera
model as suggested above.

P.S.: The proper place to ask this kind of question is ROS answers.

Also note that when using the pinhole camera model, multiple points can
correspond to the same pixel, so you might have to do a z buffer check
(only keep the closest pixel).

@mehditlili @Martin_Guenther really appreciate your helps. thank you very much.