One of the big issues with quantitative evaluation in this community is that each group/maintainer seems to have the need to create its own tools. This is summarized in here generically, but if you speak with enough maintainers you’ll notice how each has its own scripts. Most of these tools and approaches are great for development, but aren’t really usable in production systems and/or to assess complete graphs in real applications.
We at Acceleration Robotics have been doing lots of performance benchmarking over the last couple of years, across multiple ROS stacks and across hardware. This is specially relevant when involving accelerators (GPUs, FPGAs) as we’ve observed how performance is often reported wrongly, or in a misleading way by some silicon vendors. Our needs for quantitative evaluation of robotics computational graphs is only growing over-time, so we’re trying to push forward community initiatives to standardize performance benchmarking via:
- REP-2014, Benchmarking performance in ROS 2
- RobotPerf, a consortium of robotics leaders from academia, research labs, and industry whose mission is to build fair and useful robotics benchmarks that provide unbiased evaluations of robotics computing performance for hardware, software, and services—all conducted under prescribed conditions.
We’re discussing both of these topics 2022-11-03T17:00:00Z, in this meeting.
We are working on this to improve navigation2 default CPU performance and leverage hardware acceleration. We are doing so in a non-functional and grey-boxed manner (meaning of this), particularly using a benchmarking approach that leverages LTTng and ros2_tracing which put together a ROS-enabled low-overhead framework for real-time tracing of ROS 2 graphs. Also, we’re pushing our approach into a community standard via REP-2014 (see here for a readable version).
I made an attempt a while ago to start contributing this approach upstream to navigation2 but the effort never got too far. Happy to chat about this again if there’s interest.
LTTng and ros2_tracing are certainly not the simplest tools but in my experience they provide plenty of versatility, are easy to integrate in CI/CD infrastructures and allow to trace distributed ROS 2 systems.