I’ve spent over a hundred hours connecting robots with LTE - Ask Me Anything

General
1

Hey there,

My name is Alex Cutting and I’m the Head of Hardware over at Freedom Robotics. I have worked with many different robotics companies deploying robots remotely and I’ve seen lots of common mistakes. Feel free to ask me anything!

I have also put together a blog post to help others avoid many of the mistakes that we have seen. Hoping that this helps save some pain for anyone out there looking to connect their robot with LTE. In the blog you can expect to read things like:

  • Some tips on how to pick the right solution for your application
  • How to avoid runaway robots
  • How to not get charged a thousand dollar bill on data overages

You can check out the blog here. I’m always happy to hear what roboticist are working on nowadays so let me know what project you have active and how you’re connecting it to the internet, if at all! Share your robots and/or questions below!

25 Likes
2

We’re operating a robot with LTE. But the robot also logs a lot of data which is useful in the future, not at that time, this data is too much for the dataplan to handle.
We’re using a LTE router which can also connect to Wifi.

Question: what would be the most robust way to detect that we’ve switch to a wifi connection? Unfortunately the router does not provide us with that information.

I’m hoping there are ways of detecting this without switching routers.

2 Likes
3

Hi Tim,

Great question - this comes up a lot. First, I am going to answer the general question of swapping networks for people and then get in to how to detect your network topology to know if you are on LTE.

Swapping Networks Cleanly

If your LTE Modem supports both Wifi Bridging and LTE

Many higher end LTE modems (Peplink, Cradlepoint, etc.) allow you to set up a priority of wifi networks which it will use to route internet traffic with and then, if there aren’t any stable routes, will fall back to LTE on the SIM card. If this is the case, you just have to set this up in the settings for your modem.

If your LTE Modem just connects to Cellular

For simpler modems which just have cellular connectivity, then you will need to set up simple routing yourself inside linux. It has more steps than this and depends on your exact use case, but these are the basic steps:

  • Continuously connect using Wifi to a prioritized list of wifi networks
  • Check if there is connectivity check wifi status and also ping the outside world
  • If wifi disconnects or is connected but not able to get out, change the routing to put everything through eth0
  • If wifi comes back, swap everything to go back through wlan0

Knowing where you are connected

If your router has a programmatic interface

Some routers have a local API (Usually accessible at http:/192.168.0.1/index.html for login) and possibly with a status page you can see information. A few of them, like Peplink (With their InControl API), actually have an API where you can view the status of WAN connections, cellular or otherwise. It also lets you pull bandwidth consumption data for each device.

If your router does not have a programmatic interface

It does get a little more tricky, but there are a few hacks.

Use Traceroute

traceroute [google.com](http://google.com/) will tell you the IP address of each hop as data goes out. In my case, when I am on my wifi, it has one set of IPs and goes through known comcast.net servers. When I swap to an AT&T hotspot,

When connected to WIFI:


> traceroute [8](http://google.com/).8.8.8

1 192.168.0.1 (192.168.0.1) 3.005 ms 4.607 ms 5.202 ms

…

 6 [be-297-ar01.santaclara.ca.sfba.comcast.net](http://be-297-ar01.santaclara.ca.sfba.comcast.net/) (96.108.99.13) 51.731 ms 53.496 ms 51.366 ms

…

When connected to LTE:


> traceroute [8](http://google.com/).8.8.8

1 172.20.10.1 (172.20.10.1) 4.431 ms 8.919 ms 1.906 ms

…

6 [sffca22crs.ip.att.net](http://sffca22crs.ip.att.net/) (12.122.3.70) 39.999 ms 60.329 ms 82.778 ms

…

The challenge in here overall is that this trace route is a slow script, but what it does mean is that you can do very simple pattern matching of both your local IP address and ones down the trail and cache it so that you can then just check if your IP changed. If you don’t have a classification for that IP address or address family, then you run trace route with some simple regex based on your LTE provider to differentiate the two, cache it and then use that data.

2 Likes
4

Hi Alex,

I’m a hobbyist. I’ve recently purchased a NETGEAR Nighthawk M1 with 4 antennas to make use of the 4x4 MIMO. I’m planning to connect this to an embedded computer like an NVIDIA Jetson over ethernet to drive an RC car/drone around my suburb and map it using depth camera/lidar and perhaps perform some object detection/classification along the way. I have a handful of questions:

  • What are the minimum requirements (download/upload speeds) to live stream video of decent latency, resolution and frames. Doesn’t need to be 4k 60FPS. The maximum I’ve seen in Australia with the Nighthawk M1 is 120 mbit/s down and 70 mbit/s up on 4G.
  • What are the issues to expect if my robot is travelling long distances like 200km. For example, will there be issues in ROS every time the tower providing 4G changes? Or switching between 4G/3G networks.
  • Do you think something like this would suffice for my small project instead of a Nighthawk M1: https://auvidea.eu/product/38372/
  • Everyone suggests to use and external antenna but how do you choose one? What size/dB/connectors/etc?
  • What’s the difference between portforwarding vs. VPN. vs. websocket server vs. android, and when should you use them?

Thanks for your time! :grinning:

2 Likes
5

Hi, ed-muthiah. I’ve had experience with this and thought I’d chime in. I don’t use Freedom Robotics. Instead, I use my service: https://bthere.ai.

We’ve had good experience with 1Mbps upload speed supporting good video quality. It depends on your goals, but you can get something like 1280x720 at 15fps at decent quality at that bitrate. That’s assuming you’re using h264 or vp8, which are the standards for WebRTC.
Download speed won’t matter much for video quality coming off the robot.
My experience with 4G modems is that 10+Mbps down is common, as is 2.5Mbps up. But it does vary a lot. If you’re getting 70Mbps up then you’re in a great situation, but might want to watch your data usage.

Traveling over 200km you will undoubtedly switch towers and get variation in available bitrate. I haven’t done that, but I would expect the IP address to change, so your software would need to reconnect, at a minimum. You’d also want to handle variations in bitrate, ideally scaling down video when bitrate decreases.

Not sure about the alternate radio device. Not sure about antennae either. Hopefully Alex has good suggestions.

Port forwarding essentially allows somebody out on the internet to get to a specific port on your computer. It essentially opens a small hole in the firewall just for that port. On cell connections the firewall is often external to your modem. You can sometimes (often?) get the carrier to open up your modem to external connections. At which point you need your cell modem to act as the firewall. And that leads to having the modem do port forwarding.

With a VPN your computer (robot) makes an outbound connection to a proxy somewhere on the internet. For example, it could be OpenVPN running on a server in EC2. Then, other computers could connect to your robot by contacting the proxy. You can configure which ports on the proxy are forwarded to your robot. Of course, VPN’s also work the other way: your computer can make outbound connections through the VPN proxy, but that’s probably not relevant to your robot.

A websocket is essentially a type of socket connection from a computer to a server. It’s often used for browsers to make data connections to server. It is also often used for robots to connect to servers from a cell phone connection. This is sometimes used to connect to a service (e.g. bThere.ai, Freedom Robotics) from a robot over a cell phone connection. Firewalls like those on cell modems or cell networks almost always allow outbound websocket connections. The websockets are then bidirectional, allowing the server to send data back to the robot. A downside of websockets is that they go over TCP, which means they can experience stalls when packets are lost. This can produce latency in video and control. WebRTC data channels and UDP are a way to get outbound (and bidirectional) connections without that problem.

If you’re using a service like bThere.ai or Freedom Robotics the client on your robot will automatically handle connecting your robot to the outside world. If you’re building your own solution then a websocket from the robot out to a server is a reasonable first step. It gets complicated, though. Services like bThere.ai and Freedom Robotics exist to solve that complexity for you.

Stuart

1 Like
6

Hi Ed,

Great questions! Sounds like you are working on a really cool project!

What are the minimum requirements to live video stream?

When it comes to required connection speeds and video quality with Freedom Robotics you have a good amount of flexibility. You can set the maximum resolution and frame rate you want which will depend on your data plan and how good an image you want. From there Freedom will automatically downgrade the image resolution to maintain a connection as the network connection speed decreases. We can maintain a robust connection down to 100 Kbps with an image quality good enough for reliable tele-op. We have users with everything from <100Kbps to >1Mbps. One additional learning if you are looking to minimize data is that you don’t really need 15 or 30 hz video to pilot a robot. Many times, 10 or even 5fps can get the job done depending on your robot’s speed.

What to expect when traveling long distances (200 km)

Traveling long distances shouldn’t pose much of a problem as long as there is decent network coverage along your route. As Stuart pointed out if you are traveling long distances you will switch towers, however with the NightHawk M1 and the 4x4 MIMO your router will be able to monitor multiple bands in the same network simultaneously. As long as there is good total coverage you shouldn’t have any hard coverage gaps during switchover as your router will dynamically switch bands. ROS will handle these switches without a problem.

Will the https://auvidea.eu/product/38372/ suffice for my small project?

I am not familiar with this product (https://auvidea.eu/product/38372/) specifically, but I assure you that the Nighhawk M1 will be much easier to get operational and cause you less headaches throughout your project. Connectivity solutions like the Auvidea are best if you are doing large production runs where driving down cost is extremely important and you have lots of time to invest upfront in configuring them correctly. You will run into driver issues and also as this unit connects via USB I would not recommend it on a mobile robot, as vibration does not pair well with USB connectors (hot glue helps a bit). If you do want to pursue something similar I would recommend looking at Mikrotik as they sell router boards with mini-PCIe slots and mini-PCIe modems that can attach to those boards.

What is the best external antenna?

When it comes to external antennae the most important thing is just to have one. For most robotics applications here will be an order of magnitude difference between having one and not having one. Then maybe a 10% difference between having the best one for your application vs. a good antenna. The antennas that come with your Nighthawk M1 are likely pretty good and the important thing is to get them mounted as high as possible and ensure that they won’t get blocked during operation. If you are looking to squeeze the most out of your antenna I recommend Taoglas. They have really good antennas and sales support that will help you find the best antenna for your application.

Let me know if you have any more questions!

1 Like
7

Hi Alex, We make a box with 6x LTE-A modules + WiFi + CPU. What could we do with it?:thinking:

2 Likes
8

Hi joe,

That is a lot of LTE modules! Does it support monitoring multiple bands at the same time? If so I’m sure it gets really fast connection speeds with good reliability. In robotics, the autonomous car and trucking industry are likely a good fit as they often have very high connection speed and reliability requirements.

2 Likes
9

We got a lot of positive feedback from the community when we first published this blog. One of our good friends Nicolas Beucher, a veteran in the robotics space and lead on one of the most popular telepresence robot deployments, shared another common issue he has worked through and we wanted to make sure we passed it along!

Not Getting Network firewall permissions

When accessing commercial networks, IT firewalls can pose a serious barrier to a good connection. Nicolas shared that on some of the bigger robotics deployments he has done, the biggest challenge in connectivity is enabling wifi networks to correctly allow robot data through firewalls. Commonly IT departments won’t have the necessary ports open for the sake of security and often critical services can get blocked by a firewall. Nicolas did a lot of coordinating directly with IT departments and found that most of the time they are more than happy to help, you just have to speak their language. The principal concern with most IT departments is security which means they default to keeping all doors shut and only open them when they know precisely why they are needed and have clear direction on exactly what to open. Nicolas has had great success working with a myriad of different IT departments by developing a standard document to give to them that clearly communicates his exact needs. In his experience IT managers don’t read lots of text so he kept the format simple with the bottom line up front, bullet points and a clear layout.

Our Recommendation:

Be like Nicolas! Get all your ducks in a row before talking to IT and make their job as easy as possible for them, use minimal text and highlight requirements in a bulleted format. Here is a list of the information that Nicolas would communicate to IT departments in his documentation:

  • Router / Firewall configuration requirements and port forwarding
  • Local Area Network Internet Protocol addressing requirements
  • Inbound Traffic (Network Address Translation and Port Forwarding)
  • List TCP ports and usage
  • Outbound Traffic
  • List client, TCP ports and usage
  • Needed signal strength
  • Signal to Noise ratio requirements
  • Bandwidth needed
  • Latency requirements
1 Like
10

Thanks for posting this. We’ve gone through quite a few revisions of hardware and mobile carriers too over here at Greenzie.

Currently using Digimodem and Microhard, and I believe we are on AT&T’s network. We’re in Atlanta, GA though, and they have the “curiosity” 5G network they are pushing us to use, but it is only in certain areas.

3 Likes
11

Hi Charles,

So glad you found it useful. Your robots look really cool!

5G definitely has a lot of potential for robotics and it is exciting to see the early networks coming to life.

12

Hello Alex and everyone!
This is a terrific and timely discussion for us as we’ve been mulling over the addition of LTE services to our marine robots systems for a year and were getting ready to purchase some hardware to test just before Covid 19. Alex’s blog post, makes some excellent points - things we’ve been arguing about internally as we consider options. I have a couple of additional thoughts and some questions.

First, in our research we have found only one vendor (Peplink) who offers the ability to bond network interfaces, not for throughput, but for reliability. Their “Speed Fusion” technology will send data over multiple links (LTE, Wifi or whatever) at once to increase the probability that your packets will get through the first time. We think this is particularly important for field robots, particularly when one is attempting to pilot them remotely in tight spaces or at speed (learned this the hard way). I think Alex’s blog post might mention this, but I wanted to re-iterate it and ask if anyone knows of other hardware with this capability.

Second, we do not run ROS over our telemetry links. ROS1 was notorious for handling poor or lost and subsequently restored connections poorly (or not at all). [We’re looking forward to ROS2’s capability to handle this.] In the mean time we’ve written a ROS package UDP Bridge, for passing ROS messages as UDP datagrams and Command Bridge, which provides time-stamp checking and verification over the UDP connection. In our experience, these two tools have provided a very robust command and control interface when using telemetry links that may be spotty. [Keeping the size of these messages small helps too.]

But it sounds like Freedom Networks and other may have solved this another way and since this is such a critical safety issue for any field robot, I’d be curious to learn more about those other strategies.

Finally, one of the most useful features that a radio can provide is the ability to programmatically report the SNR and throughput of the telemetry system, whether Wifi, LTE or other radio system. We plot this data in real-time during all our field operations and the ability to monitor this information is a firm requirement for any hardware we purchase. Such a plot allows an operator to anticipate losses and to immediately understand when a loss is likely an RF issue or a local network issue on either side of the link. The implications of knowing that is huge - an RF issue you can do little or nothing about, a network issue on the robot requires an immediate recovery and a local network issue you can probably immediately fix and resume operations.

Thanks again for hosting this. It’s a terrifically useful discussion.
Val

3 Likes
13

Hi Alex, I’ll check. It is a rugged piece of rail spec (-40°C to 70°C, surge protected, et al.) ADLINK gear that bonds 6x LTE-A connections for luftschnittstelle I mean wireless connection for train’s systems and passenger WiFi. We make lots of crazy gear. 56x Xeon in a box? 40,000 CUDA cores in a box? Waterproof Xeon + NVIDIA RTX GPU system? We make that.

image
image692×236 44.7 KB

image
image1000×522 70.5 KB

3 Likes
14

Hi Val,

Happy to hear you are finding the discussion useful!

I’m glad you found the PepLink Speedfusion! We highly recommend their routers for robotics applications. I have an upcoming post about some routers that we have seen customers use successfully where I talk a bit more about PepLink and Speed fusion.

It is really easy to configure and deploy a ROS 1 robot that will handle a loss of connectivity poorly in a tele-operation scenario. Too combat this we recommend adding in safety at the motor controller lever so that you motor controllers are never executing commands with a late timestamp and to always test. The UDP Bridge ROS package looks really cool! I didn’t know that existed.

That’s fantastic you are monitoring the SNR. Another member of our Robotics team at Freedom just posted about our new resource management feature that will automatically log lots of great data including the network errors and network bandwidth. If your radio publishes the SNR you could also log that along with GPS location and use our API to build a heat map of connectivity to optimize the routing of your vessels to avoid areas prone to poor signal.

Cheers,
Alex

1 Like
15

@vschmidt - what does your SNR look like on the radio? Im curious to see some graphs. how do you record and plot this in realtime and what kind of insights can you gather from it? You mentioned anticipated losses - im curious how you view and predict this in that data.

We recently released a feature that we built to track network dropouts along with CPU utilization and bandwidth in a robot specific way - we’d love to add this if its relevant! https://www.freedomrobotics.ai/blog/robotics-resource-monitor

Would love to learn more!

1 Like
16

Hi!

I’ve built a solar rover with an Rpi Zero and am running standard Raspbian and a Sierra Wireless Aircard 320U - consumer grade stuff.

I have a problem with connectivity. I power my robot up for 5 minutes on the hour, every hour. It either boots and connects to our VPN perfectly after 1 minute 45 seconds, or it never finds signal and doesn’t connect.

This seems too perfect; I’d expect a bigger spread of time taken to find signal and connect. Any pointers as to why sometimes my 'bot boots but can’t find signal? Should I change a Linux thing to make it look harder/with a lower timeout?

17

Hi @whatuptkhere, Great question!

Does you robot always power on in the same location? If that is the case then the signal strength should be relatively constant. Likely the issue resides inside your robot.

The AirCard 320U is a USB device, and many computes have trouble recognizing a USB connection source on initial startup. You are seeing a good connection establish at very consistent times so it sounds to me like there is an issue in the boot sequence where the AirCard does not start communicating at the exact same time each startup which is causing inconsistencies in establishing a connection.

One solution would be to write a simple script where after 2:30 (or at whatever time you are confident that a connection will not establish) after startup, if there is not an established connection, you could reset or power cycle the USB. This will allow the Aircard to establish a connection after the compute is up and running and should establish a connection more reliably.

If your robot is booting up in a different location each time, then the signal strength could be causing the issue, in which case I would recommend mounting the antenna as high as possible and maybe switching to a router that supports an external antenna which will boost your signal. If you are looking to stay in the consumer grade product range the Huawei E8372h-517 is a great option. It has two TS-9 connectors for an external antenna.

Cheers,
Alex

18

Hi,

You may try this new cool Pi HAT -
https://www.avnet.com/wps/portal/us/products/new-product-introductions/npi/monarch-go-pi-hat/
It shoud allow every Pi user to have cellular connectivity with almost no effort.

Gadi

1 Like
20

Thanks for the help!

It’s a combination of both. In high signal areas, the robot works on 99% of boot ups. When parked in a low signal area, it’ll miss 50% or more. I’m implementing the USB reset idea and am keen to see if it works!

Another question - if I want to track my robot in real time on a map - what web app or service would be good to use? I have the GPS data and can send it to whatever app or service, I just want a web page where I can see my robot’s little dot moving in real time.

21

Hi, this is a great discussion on a timely topic. Thanks, Alex for starting it and writing an excellent blog post.

5G core networks have a feature called the Network Exposure Function (NEF) [1]. Third-party applications can programmatically (HTTP REST) request the NEF to assign specific QoS to specific IP flows. This removes the need to use multiple SIM cards with different plans. A similar feature exists in LTE core networks too called Service Capability Exposure Function. Your operator may provide such exposure features so talk to them.

I struggle to differentiate between network QoS assigned to publishers and subscribers in communicating nodes. Say for example, a publisher P1 requires normal latency QoS while P2 requires low latency QoS, with P1 and P2 in the same node. The solution I usually adopt is to break the publishers out into dedicated applications or assign the highest quality QoS to both. What is your experience in handling such varied QoS requirements in communicating nodes via ROS interfaces?