[Google Summer of Code 2024] Zenoh Support & Benchmarking

MoveIt
, ,
1

Hello everyone,

I’ve completed the project titled Zenoh Support & Benchmarking with my mentors @sjahr and @Henning_Kayser . In the scope of this GSoC project, i’ve implemented middleware benchmarking tool which aims to compare the middleware performance for rmw_zenoh, rmw_fastrtps and rmw_cyclonedds, tried to find undiscovered bugs in rmw_zenoh. we’ve shared the blog post about this project. Note that it’s used this branch in bechmarking.

I also have made the presentation at MoveIt GSoC 2024 Final Results: Show & Tell meeting. For those who want to take a look at the presentation, it’s accessible at this link

As you can see in the presentation, we have the middleware benchmarking tool. We’ve aimed not just to conduct benchmark for basic topic sub pub but also to conduct benchmarks for real robotic scenarios whereas ApexAI’s performance_test tool just analyses for core ros concepts. For example planning pipeline at robot, high level task planning with MoveIt etc. With this, we’ve tried to understand if rmw_zenoh is suitable for MoveIt project.
Github: moveit_middleware_becnhmark

I’ve tried the best i can do during summer and i’ve hope i’ve helped in a best way for ros2 core team to fix bugs in rmw_zenoh. I’m wondering if there are some community which analyses ros middlewares like rmw_cyclonedds, rmw_fastrtps and so on. Because It was challenging for me to develop the middleware benchmarking tool in some parts like kernel configurations for benchmark reliability and so on. If there is such a community, i at least want to join their meeting and learn about their ideas. Because i couldn’t see the active tool to compare my results.

Finally we’ve seen that rmw_zenoh can achieve same performance as rmw_fastrtps and it has potential to use in real robotic applications in the future even though rmw_zenoh still has some bugs and in development phase. We also have seen that MoveIt can be used with rmw_zenoh middleware at the moment.

Thanks goes to my mentors @sjahr and @Henning_Kayser . Other thanks goes to @JEnoch for valuable suggestions at Zenoh Community (especially discord). I couldn’t complete this project without their helps.

Please test rmw_zenoh and help for ros2 core team to fix bugs. I think that to try rmw_zenoh in other computers makes its development phase easier

PC specifics:
CPU Intel® Core™ i7-6700HQ Processor 4 core 8 thread
RAM 8 GB DDR4 2133MHz
OS Ubuntu 22.04
ROS version rolling
12 Likes
2

@jarbay51 thank you for all your work. What is your current recommendation for which RMW to choose for jazzy? Is it still too early for rmw_zenoh or can you work around its problems? And also which RMW is the runner-up? Cyclone or FastDDS?

3

@Rayman unfortunately i cannot say anything about which rmw middleware should be selected for jazzy, i conducted my benchmark tests using ros rolling. But at least i can say that rmw_zenoh is not ready for production so that we still have open reported issues like rmw_subscription_data_t sub_data_handler problem · Issue #240 · ros2/rmw_zenoh · GitHub and still has some dataraces. However you can use rmw_zenoh for navigation2 and moveit2 projects. I think the more reliable benchmark results can be produced using Apex AI’s performance_test tool for rmw_fastrtps and rmw_cyclonedds.(I didn’t say for rmw_zenoh because rmw_zenoh crashes performance_test). There are a lot of factor that affects the benchmark results like qos etc. I could test with default qos and this factors can put different middlewares to first place. (i forgot to write my PC specifics. I will update in my available time)

2 Likes
4

Great work! It’s these kind of initiatives that prevent a lot of hard-to-debug scenarios when working with real robots!

May I ask what the origin of these numbers are?

# for rmw_zenoh
sudo sysctl -w "net.ipv4.tcp_rmem=102400 102400 102400"
sudo sysctl -w "net.ipv4.tcp_wmem=102400 102400 102400"
sudo sysctl -w "net.ipv4.tcp_mem=102400 102400 102400"

# for all middlewares
sudo sysctl -w net.ipv4.udp_mem="102400 873800 16777216"
sudo sysctl -w net.core.netdev_max_backlog="30000"
sudo sysctl -w net.core.rmem_max="20971520"
sudo sysctl -w net.core.wmem_max="20971520"
sudo sysctl -w net.core.rmem_default="20971520"
sudo sysctl -w net.core.wmem_default="20971520"
5

UDP relavant kernel settings come from Apex AI’s tuning suggestions (pdf)

TCP relavant kernel settings come from the discussion at Zenoh Community’s discord server rmw_zenoh uses TCP in default.

But of course i’m open to new suggestions. I will be so appreciated for new suggestions.

2 Likes