`testjob_coordinator`: a parallel integration test runner PoC

General
, , ,
1

Most ROS 2 testing pipelines run all integration tests sequentially, as running in parallel is not that well supported yet. While one can set a number of workers, in practice the usefulness of that is limited as the feasible number of parallel tests greatly depends on the host (resulting in host-specific configurations) and that this number must always be based on the most resource-demanding tests, resulting in under utilization of the host resources (and thus still far from optimal test suite run durations). Also, test contamination is barely addressed, for example using separate domain ids for each test is often not the default and as far as I’m aware, no CPU isolation is done resulting in parallel tests competing for resources which increases test flakiness.

With this in mind, I developed testjob_coordinator - a proof-of-concept package that aims to address the above to easily run launch_testing-based integration tests in parallel.

In short:

  • Register tests using the testjob_coordinator_add_launch_test CMake macro.
  • If desired: specify timeout, number of CPU cores, RAM and exclusive resources - testjob_coordinator will respect these when running the tests.
  • Run the tests with ros2 run testjob_coordinator run_tests. testjob_coordinator sorts the tests to minimize overall test duration, and while running maximizes host machine resource usage while strictly enforcing each test’s limits. Also, each parallel test automatically receives a yet unused domain id through the domain coordinator.
  • Test output is available as xunit files, just like as if they were run with colcon test.

Code here, including example tests: GitHub - abaeyens/testjob_coordinator: A ROS 2 package for a PoC parallel test runner

For now the aim is to see whether this approach makes sense, before putting in further development effort (such as integration with colcon) and doing a proper system design. So please don’t already expect clean code and a polished interface.

Does this fit in your workflow? What do you like / don’t like? Which features would you like to see in such a tool? Looking forward to your feedback.

(To the moderators: If this doesn’t belong in the ‘General’ category, please move this.)

4 Likes