Hey guys,
We are a student team at Siemens in Germany focusing on ROS2 under the supervision of @flo. currently stumbled across some quirks and problems with using DDS as RMW with ROS2-galactic applications.
Our attempt to get a smooth camera stream up and running smoothly even via Ethernet and 480p resolution failed so far while using a computer with limited resources.
The connection had a very high latency of roughly 2-3 seconds and wasn’t reliable at all due to constant freezes. In the use case of steering a mobile platform remotely, this was simply not sufficient.
Now we want to investigate why this problem occurs and how to fix it. On our way to evaluate the current state of the transmissions and network, we discovered some helpful tools for tracking these metrics:
- CyclonDDS: ddsperf (build-in tool ros2:galactic)
o GitHub - eclipse-cyclonedds/cyclonedds: Eclipse Cyclone DDS project - FastDDS: FastDDS-Monitor (available from eprosima)
o https://www.eprosima.com/index.php/products-all/eprosima-fast-dds-monitor - DDS-Networking Performance: ApexAI
o ApexAI / performance_test · GitLab - ROS2 latency tool:
o GitHub - rex-schilasky/ros2_latency_ipc
Unfortunately, some of the tools (except the cli ddsperf) don’t work out of the box and the documentation is very limited on how to set up the tools. DDSperf (CLI) is included by default in the ros2 galactic installation with cyclondds. But the scripts for plotting and tracking a remote connection provided in cyclonedds/examples/perfscript at master · eclipse-cyclonedds/cyclonedds · GitHub won’t run. (Tools tested so far are ddsperf, and apexai)
Our setup:
- ROS2 Galactic
- DDS: Eclipse CyclonDDS with default settings, no DDS-tuning
- Connection: LAN-1Gb
- Camera Stream: based on USB_CAM
GitHub - ros-drivers/usb_cam: A ROS Driver for V4L2 USB Cameras - Camera: OAK-D-lite
o 1080p sensor-resolution
o 25fps - Mobile platform: https://www.eduart-robotik.com/
I’ve read some other threads regarding problems with the setup of DDS to work as expected. But this topic still seems to be a huge problem with many of the ROS2 users running it outside of turtlesim and examples.
The first step to overcome this is to measure the current performance of the setup and then start tweaking the DDS to our needs. In the last step, there would be a comparison with other DDS implementations. If you have any suggestions regarding this procedure or tips and tricks for the DDS setup please feel free to share.
Our first very rough findings for the performance on different devices are as following:
Packetloss:
- 0.03% on native ubuntu 20.04 machine with ros2 galactic
- 10% on IOT2050 with docker container and osrf:ros2-galactic image
- 20% on VmWare VM ubuntu 20.04 with ros2 galactic
The test was conducted with subscriber and publisher both on the tested machine with 100Hz and 1MB Payload.
$ ddsperf pub 100Hz size 1MB -u -k 1
$ ddsperf sub -u -k 1
Some other related threads: