Ohh, yeah add me in some robust VIO grounded in SE2 space for ground-robots and I’m very happy (in addition to filtering, filtering is a must).
3 Likes
Thanks for the feedback.
- Better short range. It makes it complicated for mobile robots to be blind at close range.
We have one coming out soon github issue here
Do you have something specific in mind. Would love to see options and evaluate them.
Absolutely
Hhmm. SInce it is mostly over NN from our side for now. I’m not sure about that one.
1 Like
Do you have any suggestions on the noise filtering algorithms ? I would like to take a look and see what we can do.
1 Like
Well since you’re taking suggestions I’d mention some way to set temperatures for thermal throttling if that’s possible. Bought an Oak-1 a while back and haven’t really done much with it since it always turns into a miniature star every time I run anything notable on it 
It also would’ve been great to have a way to just pre-upload the entire model and script setup to the camera once and have it then spit out the processed data in a standard webcam format. That way it can be moved between robots without having to install openvino and the rest of the requirements which is a real time sink.
I’ve found the options and flexibility for on-board processing on the Oak-D to be impressive, and we believe that these cameras can meet our needs for both computer vision and observation. However, it is currently a pain to use Oak-D’s for remote observation compared to a simple IP camera. I have found the WebRTC demo and the RTSP demo to both be very buggy.
What I would love to see are production ready instructions/programs that:
- serve H.264/H.265 directly from the Oak-D over HLS, WebRTC, or RTSP
- connect to the camera with depthai, read H.264/H.265 frames over XLink, and then repackage these frames into RTSP, HLS, or WebRTC.
Otherwise, it’s little annoying that you have to write a bunch of custom Python or C++ to do what a COTS IP camera can do out of the box.
I don’t want to muddle the conversation with specific algorithms. I think to a great level that depends on the type of error and occurrences in the sensors. I haven’t had a Luxonis sensor to play with before so I don’t have first hand experience to provide a set of options. 2-3 stages might be necessary / optimal though. Sensors infrequently have a single-algorithm golden bullet.
I would say that it would be useful to have a way to filter the point-cloud with the AI semantic segmentation outputs so this filtered point-cloud can feed the VSLAM / VIO.
Also, to provide better IMU calibration methods or scripts so the IMU is easily calibrated with the cameras.
What about more robust hardware interfaces like GMSL? GMSL is nearly essential for autonomous vehicles, but there’s not a lot of great hardware for it outside of NVIDIA. It could also be a great option for robots. I worked at a large robotics company who used custom realsense modules designed for GMSL instead of USB, because USB simply wasn’t robust enough for 24/7 reliability with multiple sensors
1 Like
There usually is an Ethernet version of OAK cams. Is GMSL better in some regard?
1 Like
Maybe it’s too niche, but I’ve worked with the FLIR blackfly Ethernet cams and they easily saturate gigabit Ethernet, so you can’t receive the full image at the full framerate. Especially for anything outdoors, you’re going to want 12 or 16 bit depth for high dynamic range. For distant object detection you may want high resolution, and for SLAM or anything like that you’re going to want high FPS. You can’t get all of these with Ethernet cameras, and GMSL solves these challenges. If you want multiple cams, you lose more bandwidth or need to get a 10G NIC/switch. Lastly, GMSL can be lower latency, since I believe Jetson hardware allows for DMA.
1 Like
GMSL is high data rate, low latency, and low jitter on data transmission – no relation to DMA.
GMSL uses the industry standard CSI interface. GMSL uses a serdes in the camera module, and a de-serdes at Tegra to increase CSI cable length from 15cm to 15m. Multiple camera’s can go over the same connection. This is common practice in AV as 15m covers distances needed to place cameras around the exterior body of passenger cars and trucks. We re-use this work in robotics.
GMSL has the added benefit that Tegra can trigger frame capture simultaneously across all cameras on GMSL (not possible on Ethernet or USB), for synchronous capture which is of high value for surround perception. Some camera implementations include IMU’s which can transmit I2C data over GMSL. GMSL has the added benefit that the camera images never hit the CPU for data movement, and can be delivered directly to hardware accelerated memory.
We use GMSL RGB cameras for our development with both stereo and fish-eye variations.
Intel launched RealSense D457 (GMSL version) of their camera.
GMSL has the drawback that while locking connectors are industrial grade, they are rated to be connected and disconnected a few times, making RJ45 connections more convenient in early development (RJ45 is not however has it’s own reliability issues in robots). If the camera needs to >15m away from the processor, ethernet can handle the distance. GMSL comes with added cost and complexity, which may not be justifiable when using a single camera on a robot.
It would be great for Luxonis to provide a GMSL option, but it’s understandable if it does not fit within your target market.
Thanks
2 Likes
As a suggestion for next-gen camera’s have you considered a RGGB CFA on the global-shutter imagers?
While many CV functions use monochrome data, DNN’s benefit from color, allowing for stereo DNN functions on the left & right images, or a DNN to process the left image with corresponding depth image for perception functions.
There is a high resolution RGGB camera in products which is rolling shutter, making the visual artifacts in high vibration environments difficult to process in practice. It it very challenging to simulate rolling shutter vibration artifacts reducing the benefits of SIM to real transfer.
Thanks
Interesting, I wasn’t aware the D457 was publicly available now. Amazon was using it for their robots, I assumed it was private IP but apparently just pre-production hardware. Isn’t DMA the same thing you referred to though with GMSL bypassing the CPU?
Perhaps one might call it DMA, as it’s direct memory access from CSI HW. Typically I would refer to DMA when the CPU needs to move data, and requests a HW block to perform the work instead of a memcpy. In this case, the CPU is uninvolved in receiving the data off CSI as it occurs entirely in HW.
Thanks.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.