ROS 2 benchmark open source release

, , , ,
2

Happy to see this contribution :clap: . Specially, well done contributing and disclosing this under Apache 2.0 :partying_face:.

The values you highlight above are well aligned with REP-2014 PR, and with the ones we’re using for the RobotPerf project. The major difference being that RobotPerf is that 1) a vendor-neutral benchmarking suite (to evaluate :robot: robotics computing performance in various types of compute solutions, including CPUs, GPUs and FPGAs), and 2) that it leverages ros2_tracing as a unified (community-accepted) approach to instrument graphs and measure performance.

I gave this a try and here’s some feedback at the moment:

  • The repository instructions don’t work out of the box in aarch64 (also gave it a quick try in a Dockerized x64_64 ROS container and bumped into similar issues). Also, Instructions are unnecessarily complicated and currently very tied to Nvidia tools and packages, which is unnecessary. Instead, you should consider a development flow that’s simpler and capable of recognizing/benchmarking various compute substrates without imposing additional complexity to the developer. Here’s the flow we’re aiming for in RobotPerf and you can try out today:
# Create a ROS 2 overlay workspace
mkdir -p /tmp/benchmark_ws/src

# Clone the benchmark repository
cd /tmp/benchmark_ws/src && git clone https://github.com/robotperf/benchmarks

# Fetch dependencies
source /opt/ros/humble/setup.bash
cd /tmp/benchmark_ws && sudo rosdep update || true && sudo apt-get update &&
  sudo rosdep install --from-paths src --ignore-src --rosdistro humble -y

# Build the benchmark
colcon build --merge-install --packages-up-to a1_perception_2nodes

# Source the workspace as an overlay, launch the benchmark
source install/setup.bash
RMW_IMPLEMENTATION=rmw_cyclonedds_cpp ros2 launch a1_perception_2nodes trace_a1_perception_2nodes.launch.py

(taken from the README of a1 benchmark)

  • I love the fact that Nvidia embraced the discourse that “Robots are real-time systems” however things should be coherent with that statement. The way you’re collecting timestamps should be reconsidered. You should consider using a low-overhead framework for real-time tracing of ROS 2. That’s what ros2_tracing is for (also why the whole ROS 2 stack is instrumented with it, and why we all are using it)
  • The approach implemented here (difficults and) reinvents the wheel for profiling accelerators to measure performance. You should consider leveraging directly ros2_tracing instead of pointing to Nsight systems. ros2_tracing can be extended to integrate vendor-specific profilers. This was done in the past with systems from AMD and FPGA accelerators.
  • Using JSON is great, but there’s a good reason why we sticked to CTF. Besides real-time, robots are systems of systems. Often involving multiple networks, multiple clocks, etc. You want to stick to a format that can collect and store data of robot components spread across a network. CTF is really powerful on this regard.
  • I don’t believe it’s a good idea to discarding min and max results to reduce variability. I think there’re other ways to tackle variability. Max results is what we often care the most in real-time systems.
  • It’s not clear to me whether or not you’re storing data for later analysis. Is this possible? One of the fantastic aspects of the approach we’re following at RobotPerf in alignment with REP-2014 PR is that we disconnect data collection from data analysis (we generate a series of CTF files that can later be post-processed as desired).
  • Fetching the r2b dataset is incredibly slow. Moreover, right now, the ngc-cli is somewhat buggy. Only downloads r2b_cafe for me and from what I can see.

Overall, I see this very positively but I fear it’s very tied to NVIDIA-ISAAC-ROS marketing efforts, and not that useful for the overall ROS community. I’m also a bit dissapointed this came as a separate project, and not connected/aligned to community efforts at the ROS 2 Hardware Acceleration Working Group. I will plan for a discussion around this topic in the next meeting (happening 2023-04-18T16:30:00Z, event link, past recordings).

I believe there’s much we can use from this disclosure and we’ll try tro streamline it for general consumption within RobotPerf.

5 Likes