It’s probably worth having the Intrinsic folks commenting on this. Also, this discussion might be worth a different thread.
I’d point out that profiling (closer to tracing) is not the same as benchmarking. See [1] for a discussion on this. The approach we’re following at the ROS 2 Hardware Acceleration Working group through the RobotPerf subproject (GitHub org) focuses on benchmarking whereas ros2_profiling (to my understanding at least) focus on profiling. It’s true that the definition of ros2_profiling (which says “A set of tools and techniques for evaluating performance of ROS 2 based systems”) hints more towards benchmarking, which is confusing indeed. That said, both projects seem to be build upon the same underlying principles, which is good for supporting each other down the road:
From a tech viewpoint and looking at ros2_profiling and RobotPerf’s benchmarks together, both seem to be leveraging ros2_tracing and align very well to the ongoing standardization effort for Benchmarking ROS 2 at REP 2014. ros2_tracing provides the tracing tools we both leverage. That’s already a good common baseline. From my understanding of the source code, ros2_profiling focuses a bit more on the tools whereas RobotPerf’s benchmarks on the benchmarks. We’re just getting started, but RobotPerf’s benchmarks will put lots of emphasis on how to create new benchmarks (see benchmark specs), so that others (you!) can contribute their own performance evaluation tests. We’re interested to evaluate performance of ROS 2 Nodes and Graphs (as close as possible to real applications and) of incrementally growing complexity.
The point of divergence that I can see between these two initiatives is on how each project does a) collection, b) post-processing and c) analysis. @mjcarroll any chance you can drop to one of our weekly calls in the future to discuss this and align?
@fjrodl good to read you’re looking into benchmarking. Please consider contributing to either of the approaches. Also, have a look at REP 2014 and consider contributing benchmarks built on top of ros2_tracing. That way, either of the project will benefit. If you’d like to discuss your thoughts, feel free to join one of the weekly meetings of RobotPerf.