Project Proposal: Mobile App for publishing sensor data, AR pose and depth

ROS Projects
, ,
1

Hi all, I’m preparing a project proposal for a team of computer science undergrads (600 hours). I was hoping some input from the ROS community could help scope the project, provide interesting directions to pursue, or recommend not to proceed with the project if it is unlikely to be fruitful!

As background, I’ve been wondering why there is currently no app in the play store for publishing my sensor data (camera, GPS, IMU), or ARCore outputs (VIO, depth data) from my android phone to ROS. It seems plain that such an app would be of great value and would be widely adopted/utilised, particularly in educational and hobbyist robotics. So my project idea was to explore developing such an app.

(A side interest for the project could be to try setting up a full ros install on the phone (in docker?), and/or to connect the phone to a simple diff drive platform via usb)

Note I’m aware there has been plenty of work/discussion regarding smart phone apps and ROS over the past 10-ish years, I acknowledge it is not a new concept, but it’s relatively new to me and I’m trying to get my head around the landscape of options in 2023.

Approaches I’m aware of: ROS APIs for mobile apps.
I’ve put together a spreadsheet capturing options I’m aware of, after a couple night’s googling. In broad terms I see three possible approaches:

  1. write a native android app in java
  2. use unity/c# to build to an android app
  3. perhaps use some framework to write apps from pure javascript (React native? I don’t really know how that would work, I have no experience in this area).

There are also distinctions regarding whether to use DDS directly or websockets and a bridge node.

My spreadsheet is here, you can see I’ve identified 10 possible options:

My first thought right now is to take the AR-RViz project as a starting point (Unity with TCP bridge to ROS), as that is a recent and impressive project. Still, I think the javascript and java approaches have their own merits, so I’m open to suggestions.

Questions for community:
Do you think there is a possible project here that is:

  • likely to deliver a valuable outcome
  • achievable by a computer science undergrad team with a total of 600 hrs to spend

If so, any comments on particular approaches which may be promising? Or on the otherhand, approaches likely to be a dead end? Any input or feedback would be greatly appreciated!

2 Likes
2

Tagging a few users who I thought could be interested, based on previous activity on ROS discourse
@atyshka ( Discussion: ROS2 on mobile devices)
@yasuch GitHub - YasuChiba/ROS2-Mobile-Android: Visualization and controlling application for Android
@NicoS (GitHub - ROS-Mobile/ROS-Mobile-Android: Visualization and controlling application for Android)
@fmrico (ROsBOBO – ROSIN)
@esteve (GitHub - ros2-java/ros2_java: Java and Android bindings for ROS2)
@kodieartner (GitHub - kodie-artner/AR-RViz: Unity Project for visualization and control of ROS systems in augmented reality)

Any comments/insights you could provide would be greatly appreciated!

3

I’ll also point out GitHub - sloretz/sensors_for_ros: An Android app that publishes sensor and camera data on ROS 2 topics, which was the subject of a ROSCon talk last year: Building ROS 2 enabled Android apps with C++.mp4 on Vimeo

2 Likes
4

Thanks, somehow I’d missed that project!

Very interesting as I hadn’t considered a pure cpp approach. I’ll have to watch the talk and update my spreadsheet of options.

@sloretz I’d be curious if you have general thoughts about the cpp-based approach. Do you think it would provide a solid foundation for such an app, which would be supportable/maintainable by the community in the longer term?
I see you have a very nice overview of some problems encountered - I’m wondering if its better to just avoid any DDS-based options in general to avoid these issues - perhaps an app using only websocket/tcp for comms will be easier to support and maintain in the long run and easier for students to get their head around in a short amount of time!

It was this comment that got me thinking down these lines:

After spending significant time on this, I created a trivial Websocket/JSON bridge (using our Java implementation) to talk to ROS2 which is much less work to support than even just installing/building ROS2.
I think that unfortunately, the current ROS2 ecosystem makes the easiest way to expose ROS2 in an existing application written in a different language than C/C++ (or a C/C++ application with a legacy build system/main) is to use a service like https://github.com/RobotWebTools/ros2-web-bridge

5

Also tagging @eshu (Foxglove Studio - Visualizing and debugging your robotics data - Foxglove)

Would it be feasible to base the app on roslibjs or the foxglove websocket protocol, where the app is written in javascript and somehow built into a native app (react native? I’m far out of my area of expertise talking about web technologies for app development!).

Do you think a websocket bridge would lead to significant latency or bandwidth limitations? Do you see such limitations in Foxglove studio?

6

Quite old by now, but as another point of reference, there is this app for ROS1:

https://wiki.ros.org/android_sensors_driver/

It supports publishing IMU and Camera data over wifi. Still works on current android phones. Unfortunately it does not appear to be on the play store anymore.

7

Yes, if I remember correctly, I had that (or a fork) running on my phone about 5 years ago, publishing IMU. It came in handy a couple times!

The idea for my proposed project is basically a revival of that, perhaps adding AR core publishing as well. I’m also wondering (out loud) if the rosjava dependency has proved too difficult for the community to maintain over the years; perhaps that’s why apps such as this come up from time to time but eventually become unsupported. And this leads me to wonder (out loud) if a bridge-based comms approach is worth looking at first.

8

I did a quick proof-of-concept for an iOS version of this a couple of years ago: GitHub - christophebedard/ipad-lidar2ros: Publish iPad Pro LiDAR data & more in ROS 2 - source code & assets only

I used ros2-web-bridge, which has now been replaced with rosbridge_suite. While it was somewhat easy to set up on the iOS side, it didn’t seem to work great. Perhaps rosbridge_suite’s performance is better.

I think doing this project and using & building native ROS 2 on Android would be an achievement in itself, but it might take a lot of those 600 hours. However, if @sloretz’s work (and fixes) can save you a bunch of work, then it might be worth it. Otherwise, it might be easier to go down the JSON+bridge route.

Overall, I think this is a nice project!

9

Nice, impressive project :slight_smile: I’ve added another row in my spreadsheet for “custom rosbridge client in swift”, I now have 14 options for app-to-ROS comms! (I guess “custom rosbridge client” could be achieved in any language that can be used to make apps).

it didn’t seem to work great .

Were you seeing issues related to latency and/or bandwidth? That would be my concern with the bridge in general. Yes, perhaps rosbridge_suite would be better, maybe some of the students’ efforts could be in benchmarking comms performance.

10

I believe rosbridge’s performance issues were one of the reasons for Foxglove to implement their own protocol. From Announcing the New Foxglove Bridge for Live ROS Data - Foxglove :

Since we’ve learned that Rosbridge can struggle with streaming high-frequency topics and large messages like images and point clouds – both from Foxglove Studio users and our own testing – we designed Foxglove bridge to tackle these performances issues and to provide a dramatically streamlined experience for Studio users debugging their robot.

1 Like
11

Hmm yes I’d seen that they’d implemented their own protocol for performance reasons. That makes me interested in the path of writing the app in javascript and using their js client. But as mentioned previously, I’m less clear on how that would work, I don’t have much experience in that area. In particular I’m unclear how to access device sensors and ARCore APIs from javascript. But I assume the project as a whole would look something like this.

12

@clalancette thanks again for sharing the link to GitHub - sloretz/sensors_for_ros: An Android app that publishes sensor and camera data on ROS 2 topics.

I’ve just built it, installed on my phone, and started receiving camera and IMU messages on my laptop! My motivation to follow through on the student project is greatly reduced :rofl:

The publish rate was a bit too slow to be usable, but that could easily be me setting something up incorrectly. I logged an issue Slow publish rate · Issue #5 · sloretz/sensors_for_ros · GitHub

13

I don’t remember much about the latency, but yeah I had issues with the bandwidth and LiDAR & camera data. I’m assuming Foxglove’s solution is much better.

That could indeed be an interesting part of your project! I don’t know if you have to do any sort of report, but if you do, you could talk about whatever you’ve done to improve performance (if you can), what else could be done, what are the main limitations, etc.

14

It’s definitely better, but for video you really need to compress using h264 (or similar) and not send individual images, which is why I developed this live-connector:
Foxglove WebRTC | Transitive Robotics

1 Like
15

for video you really need to compress using h264

Is this still true if I assume a LAN connection between the phone and the robot in my application? I’m not sure if the compression requirement is more to do with operating over limited bandwidth connections (e.g. cellular), or a workaround to fundamental bandwith limitations in the rosbridge/websocket approach.

16

You are right, this only matters for remote connections, not local ones. It saves bandwidth at the expense of CPU.

17

Not that I’d want to rule that out as an option - a phone which is usable as a remote sensor device publishing to ROS via mobile network could have interesting applications! But support for low bandwidth networks would not be a priority to start.

18

A bit of an update - the project has been picked up, so we’ll continue scoping out the options and investigating interesting avenues until the project wraps up in November. Throughout this time any suggestions or comments from the community would be very welcome!

I’ve been thinking more about the AR side of things - I assume Google and Apple have spent a decent amount of engineering time and expertise developing SLAM for AR applications - so why don’t we already have an app which makes this available for robotics applications? The ideal would be to map out say, my house using ARcore/ARKit etc using an old phone, then mount that phone to a robot for persistent/repeatable localisation. Has anyone looked into this use case? Are there any technical blockers that make it not feasible?

I do note that relocalisation is an open issue in ARCore. There are proposed workarounds using anchors, but I’m not sure how well that would work.

2 Likes
19

Slightly unfortunate update - at last minute the project was cancelled, so will not be going ahead after all (at least this year).

At least I hope the spreadsheet from my original post can help anyone else who is considering work in this area.

Also, as a quick proof of concept I put together a web_xr pose publisher, for publishing the ARCore pose from my phone. It was surprisingly easy to get running, and pose accuracy looks good enough for many applications. More info on the repo: GitHub - tim-fan/webxr_ros_pub: Experiment publishing webxr pose to ROS via rosbridge

I still think this is an interesting area for investigation, if I have more time in future I’ll look more into publishing sensor data via javascript/rosbridge, as it is so straight forward to get running. If anyone else makes progress on this, I’d be keen to hear about it!

2 Likes