Hello everyone,
Have you ever wondered how to build a fancy touch friendly HMI for ROS including 3D visualization? Well, then you might like this idea, concept, and prototype.
I’ve been working with Qt UIs for more than 10 years now and I have seen the transition from desktop focus, to mobile with Nokia to industrial and automotive HMIs. The trend is clear, the introduction of smartphones and other touch devices made a focus on user experience easier, more accessible and a necessity for modern HMIs. More and more industrial machines and robots are equipped with touch-enabled tablets rather than mouse and keyboard.
So far RViz only supports Qt Widgets based interfaces. Qt Widgets are great for normal desktop applications, but they have many drawbacks when it comes to HMIs, embedded mobile, and touch applications.
Since a few years, Qt also ships with a technology called QtQuick. QtQuick is paradigm shift compared to widgets. QtQuick uses the declarative QML programming language (think of HTML+JS with JSON syntax) to build the UIs rather than XML based descriptions. The rendering happens in OpenGL, everything is based on base types such as Rectangles. This means it is possible to create literally any design you can think of, which is impossible with widgets and hard with HTML5.
Long story short, since I work on HMIs for industrial machines and robots professionally, I was looking for ways to build UIs with QtQuick and ROS. Using ROS and QtQuick together is not very hard. It’s just a matter of using the ROS bindings for C++ or Python and building an application.
One of the greatest things about ROS when it comes to UI is RViz. However, RViz strongly depends on QtWidgets internally. Using QtWidgets in combination with QtQuick is possible, and after some research, it turns out that the actual problem is not combining QtQuick and Qt Widgets, but rather combining RViz Ogre-based OpenGL renderer with QtQuick. Long story short, the easy way of adding QtQuick to an rqt/RViz application using QtQuickWidget does not work out.
That’s why I invested some time into exploring the correct approach of integrating RViz into QtQuick or to be exact, to integrate the RViz render panel into QtQuick. Turns out that is possible, but only with some major architectural changes inside RViz. In many places inside RViz it’s assumed that Qt Widgets are used, this includes things such as changing the mouse pointer or setting properties of the render panel. It’s possible to refactor out the differences between Qt Widgets and QtQuick Items, but incompatibilities still reside at the moment.
I have created a GitHub issue and pull request
The patches work and I’m already working on an HMI for an industrial robot application. Unfortunately, I can’t show some screenshots or videos of this application at the moment.
I’m also aware that the changes probably will not make into upstream ROS1 RViz, but I want to use this discourse to spur some interest in QtQuick support for ROS2. In my opinion, QtQuick support would be a major benefactor for I-ROS applications, as Qt is becoming a bigger and bigger player in the industrial embedded HMI space.
If you wonder what QtQuick applications look like, I encourage you to take a look at the [Qt user stories](https://resources.qt.io/customer-stories-all and some) or at my CNC examples on my website.
If you are fellow HMI engineers interested in QtQuick support for RViz please let me know. Maybe we can pull together to get a ready-to-use upstream version of RViz QtQuick going.
Machine Koder