Survey on Open-Source and Benchmarking for Robotics

General
, , ,
2

Welcome to the community @adam.norton.uml,

Exciting to see more benchmarking efforts. This appears to (some questions below would clarify that though) overlap to a big extend with other well established projects in the community, particularly with the activities of the ROS 2 Hardware Acceleration Working Group including the RobotPerf project (source code) and the REP 2014 standardization effort, so my first reaction is to encourage you to join this WG, participate and try contributing. Next meeting is very soon: (LinkedIn event) Hardware Acceleration WG, meeting #15.

RobotPerf is open reference benchmarking suite that is used to evaluate robotics computing performance fairly with ROS 2 as its common baseline, so that robotic architects can make informed decisions about the hardware and software components of their robotic systems. We surveyed the community in the past, disclosed all results during the HAWG meetings and summarized results at 2022 Hardware Acceleration Report in Robotics. You can probably use this information for your own purposes.

I went through your website, and survey and have a few questions:

  • Were you aware of RobotPerf, if so, how is your project different and more importantly, is there a need to keep two separate efforts or could we merge (note RobotPerf is also being driven by academics, with Harvard spearheading that group, particularly the team behind MLPerf in the past, quite experienced)?
  • It’s not clear at all for me whether you plan or not using ROS for benchmarking purposes. Can you clarify this?
  • Why are you not sharing back the data collected in the community with this community? It seems (based on your survey) that you’d only be sharing data with other researchers. Why should industry get involved then?
  • The headline above is generic however your project seems to be only focused on perception and manipulation. Can you clarify what’s the scope in here?
  • Of most importance, how’s your project going to benefit the ROS community? There’s no mention to expected contributions neither in the text nor in the survey. You’re querying the ROS community in here. There should be some clear expectations on this regard.

Some resources worth considering by you and your group:

1 Like