Given Augmented Reality is increasing in popularity and available on many platforms. Additionally, AR provides 3D contextual data which would be very useful in visualizing data such as point clouds and markers. Is there any interest in converting some visualization tools such as RVIZ and MoveIt to an AR platform? With multiplatform support in ROS 2, it would likely easier than ROS 1.
Point clouds anchored to real world objects visualize from an AR device could be infinitely more useful. Augmented motion plans displayed on real world objects could help identify collisions with items not sensed or modeled. I’m sure there are many more possible use cases.
rviz porting has already started, first by @bponsler:
And then by me and some other developers, see:
And:
The latter work (that I’m involved in) is still a work in progress, but we’re close to having a usable rviz application in the ros 2 master checkout and some version of it will definitely be in the release at the end of this year.
I’m not aware of anyone working on AR stuff yet, or integration with third party libraries or hardware.
A friend did an AR rviz-like tool for his PhD about 15 years ago. He did a lot of research into effective data visualisation methods. The tool was great to use. He wasn’t the only one, either; there have been many papers and projects over the years looking at how to use AR to develop and debug robots. The concept definitely has a lot of value.
Is the ROS 2 version of rviz targeting only Ubuntu or do you plan to make it multiplatform? I’m specifically thinking if it can be built for Windows then combining with Unity Game Engine should be easier. Since Unity is the AR development environment of choice, this would be a great first step.
I also remember mention at ROSCon of UML’s work with ROS1 and Unity. That might also be a good option.
It works on Windows, macOS, and Linux, but integrating it with Unity is probably not easy because rviz uses Ogre as the 3D engine, and that’s not easy to change. I don’t know why you need Unity to do what you want. Also Unity works on Linux and macOS too, as well as so many others:
So I don’t know if there are Windows only features you need.
At least initially I was thinking about AR devices that are primarily targeted for Windows development like the Hololens or Meta2.
I found some sites that talk about porting from Ogre to Unity. It certainly doesn’t sound easy. I’ll start taking a look soon just to satisfy my curiosity.
I’m thinking about see-through camera support of point clouds and markers. Looks like there is some precedent of using Ogre3d with ARToolKit. What do you think about something like that?