Celebrating the launch of ROS2 Humble, new AI stereo vision perception packages for Isaac ROS will be released.
(IsaacSIM generated image with columns left to right containing, stereo disparity, original image, BI3D, and ESS)
BI3D is a DNN for vision based obstacle prediction (https://arxiv.org/pdf/2005.07274.pdf). The DNN is enhanced to simultaneously detect freespace & obstacles, and optimized to run on DLA hardware. The DNN predicts if an obstacle is within one of 4 programmable proximity fields from a stereo camera.
ESS is a DNN for stereo camera disparity prediction (https://arxiv.org/pdf/1803.09719.pdf). The network provides vision based continuous depth perception providing predictions for areas of left / right camera occlusion.
Both BI3D and ESS are pre-trained for robotics applications using synthetic and real data intended for commercial use; there is no research limitation on these pre-trained DNN’s. The AI perception functions are designed for color input which yields the best accuracy while maintaining compatibility with monochrome stereo camera inputs. They join stereo_image_proc which provides a hardware accelerated classic CV function to computing stereo depth/disparity providing a set of three diverse independent functions for stereo camera depth perception.
(comparison of results from synthetic camera image top, and RGB stereo camera image capture with no active projection bottom; image on left, BI3D DNN prediction for 4 proximity fields with ground freespace 2nd from left, ESS DNN prediction for continuous depth 2nd from right, and classic CV stereo disparity function on the right)
These new packages for ROS2 Humble will be available at the end of June in the next Isaac ROS update.