This is excellent! Thank you for writing up the process. The big problem with flakiness is, of course, that it makes CI uninformative and unactionable. If the user expects green CI, then red CI or a newly failed test is an obvious issue. But if CI is always red or yellow, there’s not an obvious way to know when a change is concerning.
-
How does Apex track when a failure is handled? Is there a way to make it visible from the CI status report whether a failure has an owner (e.g. a name, a link to a github issue, or an in-progress PR)? It’s often impossible to know even what repo to the failure is in let alone whether a failure has a known cause.
-
How does Apex suppress or deal with non-test failures? e.g. persistent failures that are picked up in warnings plugins and not associated with a test case, like those mentioned in https://github.com/ros2/rclcpp/pull/1047#issuecomment-613499118 ?