I had the need to develop a simple HMI for a previous project, and since I have NO experience at all in developing GUI applications, I resorted to using QtDesigner to create the interface, and then added ROS 2 callbacks to each button press and list information. It worked quite well for the project, but I was looking for a way to integrate the core visualisation component of RViz2 into my HMI, so that I do not have to open a separate program to “see” my robot and what it does. I know that FoxGlove Studio does a great job of visualising robot/sensor data, but in this context I was looking to have my HMI essentially “wrapped around” Rviz2.
HMI development and management is not always a core focus of robotics engineers, as it takes a predominantly different set of skills to achieve a professional looking UI, skills that not all of us possess.
I would like to know if you ever had to create an HMI for your application, how you did it, and what tradeoffs you had to make (for my application, I sacrificed the aesthetics and a major part of visualisation to have ease of development). I know that the ROS client library rcl has been bindings for JavaScript, which I know to be used in UI development, but this question is coming more from a C++/Python user, who just needs a HMI with some buttons, and a way to visualise the robot inside the application.
Sorry for the long posts. This is a highly unedited version, if the mods wish to suggest changes or to edit the question further, that would be appreciated.
P.S.: I could not release the entire HMI I developed during my project, but I released a skeletal version of the HMI which can be edited by the user, and also documented my thoughts on this question. I welcome any improvements to my answer or my code.
Since it’s relevant I’ll shamelessly plug that you can use my ROS(2) module for QML to prototype and develop UIs quickly.
QML is IMHO magnitudes better than QtWidget-based UIs since responsiveness is very easy to develop and using and choosing the right animations to guide the user’s attention properly is integral to good UX.
Also, you don’t have to compile anything and can have the definition of the button and the logic of e.g. calling a service on click in the same file.
Here’s the ROS 1 version:
and here’s the ROS 2 version:
The ROS 1 version can also be overlaid on top of the RViz 3D view using this plugin:
A port to RViz2 should be possible but is still on the roadmap.
And here’s a short video of our interface built on top of RViz in use at a robotics competition:
I still have to finish that one PR I have open for it, but the QML plugin @StefanFabian mentions above would be my go-to for “a C++/Python user”.
I still fight with layouts, but that’s my shortcoming as a non-UI-guy. Hooking up some buttons and text boxes to topics and services is really not that hard any more with the QML plugin and a simple UI is easily created.
This looks quite nice ! I don’t have a lot of experience with QML to give you more feedback on the logic.
Maybe you can add some images of the generated HMI to show users what they can do with your code?
EDIT : Oops, was a bit late, I see you already added the visualisation, and oh wow, it looks very nice ! I am going to have to check out the code over the weekend
Mostly because I’m not a fan of companies leeching off the work of open-source devs as they rarely contribute their fixes back and was intending more of a dual license model but since that would complicate the situation with PRs, I think I’ll just switch to MIT, as well, as in this instance this works better with the open-source spirit.
I am going to have to check out the code over the weekend