Just though I’d gather some feedback from the community regarding ROS2 on mobile devices, i.e. iOS and Android. With ROS1, support for these platforms was pretty limited due to the lack of a core ROS implementation, but now RCL and ROS2 has enabled creations such as ros2_java for Android and rclobj for iOS. However, to my knowledge it seems that adoption has been limited. As a summer project, I’m working on a client library for Swift, but I wanted to see how the community feels with regard to mobile device use cases. Do you think first-class mobile device support has a meaningful place in the ROS ecosystem?
A few applications that come to mind for me are introspection/diagnostic apps for research or commercial settings, i.e. a tablet HMI for interfacing with an Autoware pc in a vehicle, or planning missions on an AGV. Another use case might be visualization, either something more traditional like an RViz clone, or something novel integrating AR. Some might argue this is a gimmicky application, but I think it would be really cool to be able to see a robot’s sensors such as lidar or object detections overlaid into the real world (ARViz!). Also maybe just another option for developers in the field. Sometimes I find a laptop clunky, and I could probably have a decent portable workflow on an iPad with ssh and visualization tools.
Also, for such applications, do you think mobile client libraries add something meaningful in comparison to more web-oriented workflows like the json bridge? I can’t help but notice that ros2-web-bridge currently has more GitHub stars than the objective c and java client libraries combined.
Anyways, would greatly appreciate your feedback on how you think these kinds of applications would/would not be useful!
I like the idea, however a bit reserved on the matter. Two things that come to mind is hackers gaining access to your mobile device and not having control of the bot. The other is the amount of service currently on the cell towers… With normal users then kids playing games on their cell phones, I really think 5G cell system that is coming out (which is in some locations already) will provide some additional cell expansion. My cell (LTE/4G) drops out from time to time and always have to make a callback. My only concern would be using the cell and having to hit the emergency stop for the bot or having some control when using the phone without fear of losing signal by a dropout. If all systems work accordingly and have security, I’m In! But we need to assure ourselves we have total control using ROS with cells.
The OpenROV project uses Android for it’s Trident Controller. If i recall correctly, this was prior to ROS2, the development team had been using DDS to exchange ROV telemetry and teleop control over the two wire tether to the surface level WiFi base station that PCs or the tablet proxy through to connect to the underwater robot. I recall they had to do a bit of hacking to get the optimal QoS working for streaming encoded video for low latency feedback with the android device’s network stack, given the Android’s Linux kernel lags quite a bit behind mainline, and that some may not even have multicast enabled.
Here was some of the old promo and slides that relates to talk OpenROV gave during Connext Conference 2017. Perhaps @spiderkeys who spoke then would know more?
I’m sure there are many more ground, aerial, service, and industrial robot based use cases for native mobile interfaces in integrating with ROS2 enabled applications. In particular it’d be nice to see something general, like an open source android app, either for viewing raw ROS2 messages, publishing smart phone sensor data, or something more substantial, equivalent to rviz2, so people would have a cookie cutter demo project to fork from for getting started on there own custom ROS2 apps. Mobile development is a bit of a bear to break into, and takes a lot of effort to stay afloat given how much of a moving target the ecosystem can be. Having a community maintained app/example to spring board from could help a lot of other ROS users.
Thanks for the mention @ruffsl. Happy to answer any questions anyone might have about those projects and the various implications relating to DDS, video, wifi, and whatnot. There are many things I would do differently, but some of the approach in the presentations above (though definitely DDS-specific) could be useful as a starting point. I’ve learned quite a lot about various techniques for video and control transmission over lossy networks in the last couple of years, having spent some of it focused on developing real-time teleops solutions for self-driving cars.
On the GUI front, myself and two other former colleagues from OpenROV recently co-founded a new marine robotics company and one of the things we are working on solving is creating a cross platform ROS2 GUI application using Qt, which we hope to open source in the future. There are a number of pain points involved in making this process clean and easy to replicate, mainly due to how ROS2 software is currently packaged.
One of the things I am exploring is how to package ament_cmake projects with Conan (the C++ package manager) via a custom Colcon plugin, which would trivialize the process of pulling into a CMake project 1.) any ROS2 library or package and 2.) Qt itself. From there, Conan can be used to build the GUI application and all of its dependencies for multiple architectures with different settings, including Android. The final step is to use the conan deploy feature to install the binary, all of its resources, and its entire dependency tree to a folder, where it can be packaged appropriately (apk, debian package, flatpak, snap, etc). With this approach, I hope to greatly simplify pure C++ cross-platform GUI development with ROS2, allowing you to use a single codebase for both desktop and mobile, relying on minimal system-installed dependencies. Note that the same approach would also work with other cross-platform frameworks like SDL, if Qt isn’t your cup of tea.
If anyone has any ideas or feedback around the above, or would be interested in helping out, I’d be happy to chat.
So I would go about this more generically and try to take advantage of rosllibjs. Utilizing all of the great projects/ui libraries being developed for the browser it would make this option really powerful.
Personally I am a big fan of React and all it has (including the three.js wrapper react-three-fiber) and have already started to work on a react-ros component library to try and allow for quick and easy integration of ROS/ROS2 via websockets into a webpage. Building on top of this is another react component library for 3D stuff utlizing react-three-fiber is react-ros-three which I’m also currently working on.
Removing barriers for people to connect websites and webapps to ROS/ROS2 would be extremely powerful. I would guess many developers in the web dev world would jump at the chance at this - connecting their roombas to their websites
Thanks for the clarification … The way it sounded from [tomoyafujita] (which I may have misunderstood his comment) referring to a mobile device such as a “cell”. I’m dealing with cell communications with my drone and still testing it as time progresses. However, I would have to say that ROS will eventually (if not already) would be capable of communications using cell but foresee it being in the next generation wave using 5G cellular networks for higher speed capabilities.
I cooked up a demo with RBSManager and roslibjs + new iOS Vision APIs:
I think mobile devices, along with the perks they have (huge companies developing vision software, or really fast processors on iPads) will be an important player in the success of ROS2 and its high-performance DDS.
Do you think first-class mobile device support has a meaningful place in the ROS ecosystem?
I’ve worked with rosjava and ROS on Android (for ROS1) in the past, and here’s my 2 cents.
In general I’m aligned with the fact that robot / ROS developers are not very fond of mobile platforms as a way of developing robots. It’s definitely feasible to build a robot controlled by an Android device, but as you mention the adoption has been limited indeed.
I do see value however in the other kind of applications named above: visualization, monitoring, and remote operation are good examples where mobile platforms could add value for users.
On the other hand, making use of the sensors of an Android device is compelling as well; this is particularly true for AR-enabled devices. With the (now deprecated) Tango platform you’d be able to get pointclouds, images, and visual-inertial odometry from a single device; all that data is valuable for a robotics application. You’d be able to get more or less the same with an AR-enabled device today (except for dense depth clouds), and there are plenty of devices that support that technology nowadays.
Last but not least, Android has a few advantages in the UX side: it’s rather easy to build user-friendly interfaces, keeping apps up to date, etc. Overall, there definitely are some compelling reasons to push ROS2 integration on Android.
