Greetings ROS users,
With this post we’d like to update you on a change to the Package Summary section on the ROS wiki that was recently merged and has been rolled out for packages with a devel job in Melodic.
Overview
As you may have noticed, for packages in Melodic the Continuous Integration badge does not only show whether the source repository has been registered in the distribution.yaml, but it will also show the CI status of the repository. The colour of the badge and icons shown on the badge reflect the status and show a summary of the results of tests run during the last build.
Clicking on the Continuous Integration badge shows a drop-down menu with up to five previous runs of the devel job. Clicking on any of these entries will take you to the page of the respective build for that particular repository.
The idea behind this change is that the CI badge will reflect the actual state of a package like this, and that in turn this will facilitate assessing package state by giving the Jenkins job build status and test results a more prominent place on the wiki.
Badges
There are three different icon-colour combinations, representing three different categories of packages.
Build ok, tests ok
Repositories for which all tests are succeeding and the build ended successfully will get a green badge and a check-mark:
All tests pass and Jenkins is happy about the build. So are we.
Build ok, tests skipped
Repositories with packages that have skipped tests will show a grey badge with a dash:
While skipped tests do not necessarily indicate something is wrong with a package, they are typically skipped for a reason (they are expected to fail otherwise, require some elaborate setup, are in the process of being updated, etc). Packages with grey badges are most likely perfectly fine to use, but cautious users might want to verify why the tests are skipped.
Build failed, tests failed
Repositories with failed tests will get a red badge and a cross:
A badge like this is an indication that something is not right and one or more tests are failing. Inspecting the build history might be a good idea to try and understand what failed and why.
Status and future work
The current implementation shows this information per repository, and is enabled or disabled per ROS release (and currently only enabled for Melodic). The former is both a limitation of the way test results are gathered by the buildfarm as well as a limitation of the current implementation. The latter was determined to be a good first step, with a potential extension to allow opt-outs added in a future PR.
Comment & Questions
How and whether to take this effort further is one of the topics being discussed in the Quality Working Group meetings held each month, as part of the ROSIN project. Comments, criticism and suggestions are more than welcomed: please open a topic in the Quality Discourse group to discuss.
Links
PR against ros-infrastructure/roswiki: roswiki#246.
PR against ros-infrastructure/ros_buildfarm: ros_buildfarm#541.