ROS Resources: Documentation | Support | Discussion Forum | Service Status | Q&A answers.ros.org

OpenCV AI Kit (OAK)

Hi Everyone,

Brandon from Luxonis here, one of the many folks behind the OpenCV AI Kit!

So we wanted to announce here that our campaign for the kit is live!


TLDR: AI + depth in a small, modular, and embeddable device.

We have prototyped (proof-of-concept’ed) initial support for ROS (and have some Alpha testers using it in production actually!) on our host driver, and will be delivering full support by the time the KickStarter hardware ships.

Please feel free to ping us with any and all questions!

Thanks,
Brandon Gilles
OpenCV AI Kit Team

8 Likes

So we hit $250k so OAK-D will now include an integrated IMU for all the backers!

1 Like

Do you perhaps mean “will now include”? :stuck_out_tongue:

1 Like

Oh yes thank you! Just edited!

1 Like

Now that it includes an IMU, will the IMU be hardware synchronized with the cameras?

1 Like

Great question thanks. Yes we will be timestamping with the MIPI receipts so it will be synced to the micro-second level.

1 Like

Huge fan of the work @Luxonis-Brandon and team are doing, I backed it!

Offloading edge compute in a small, modular, and embeddable device away from host CPU will enable a lot safer, less expensive robotic solutions.

Keep up the great work.

1 Like

Thanks @SeeBQ ! (And cool to see you on here!)

2 Likes

is it possible to build robot control like of https://hello-robot.com STRETCH with OAK-D?
how does OAK-D compare to intel realsense (there is realsense d435i with imu)?
maybe frame rate speed or accuracy, compute power, simplicity of integration.

how to build a system like that in ros, like a general road map?

1 Like

Yes, I think so. Would love to see it used in that application.

WRT the D435i. So OAK-D’s premise is real-time spatial AI. So the RealSense cameras can’t do any neural inference/etc.

So here’s the lineage:
D4 chipset (Gen0): Intel D400 series (D410, D415 D435/i, D455): Depth only
Myriad 2 (Gen1): Intel T265: Depth + Tracking (i.e. great for SLAM)
Myriad X (Gen2): OpenCV AI Kit: Depth + AI + a whole slew of accelerated computer vision functions like H.265 encoding. Stereo neural inference, etc.

So to get the equivalent functionality of OAK-D you’d have to buy 3 things:

  1. 12MP camera
  2. Depth Camera (e.g. Intel D455)
  3. AI processor (e.g. NCS2).

And unlike previous depth or SLAM cameras, OAK-D isn’t for making maps of rooms or objects or for doing SLAM… it’s new use-case:
Giving real-time information on the physical location of objects (e.g. people, strawberries, fish and how big they are, seeds when seeding a farm, etc.).

It supports two ways of doing this:

Monocular Neural inference (e.g. off the color camera) fused with stereo depth. An example of that is below:
Spatial AI

Or stereo neural inference, like below:
Stereo AI

So the monocular AI + stereo depth is useful for objects, while the stereo AI can be used for objects or for features (e.g. facial landmarks, pose information, etc.)

Both modes support standard 2D models (since 3D training data is extremely limited compared to 2D data) and use the stereo cameras to produce 3D results all onboard.

Sorry that was a bit long but I hope it helps!

Thanks,
Brandon

2 Likes

Brandon & Jon will demo OAK during 2020-07-23T22:00:00Z call which is open to all ROS 2 Edge AI WG Meeting - Thursday July 23rd 15:00 PST (UTC-8). Fernando will demo ROS YOLACT, we will review results of ROS user ML survey and make a plan

0b6dd1a76adcd835d6ac0d9faa6bb2b5_original d23ce8700f65a9edd76d9cb3490cfb73_original 85575aa41244de7c44270520fececdd1_original fede55b7bb767a7336cd564a761685d3_original

1 Like

Thanks @joespeed! It was great being able to share on the talk. Oh and if anyone is interested on more strawberry picking, see below:
https://youtu.be/Okjh2OCP-o8

The student who is making this system is also going to record a version with a bunch of clutter/grass/etc. being put into the scene while it picks to show off that it is indeed ML-based on not just hue-detection against a background. :slight_smile:

2 Likes