Upgrade to Jenkins 2.277.3 causing job failures across ROS build farms

Buildfarm
1

A security upgrade to Jenkins 2.277.3 has started causing job failures across each of our build farms: build.ros.org, build.ros2.org, and ci.ros2.org. We’re currently tracking the issue here Dev jobs fail with "java.lang.IllegalArgumentException: The tools section is required" · Issue #871 · ros-infrastructure/ros_buildfarm · GitHub and each farm is currently in shutdown mode during the initial investigation.

1 Like
2

Preliminary investigation has revealed at least a possible hotfix which is being proposed to ci.ros2.org in Update xunit configuration tag name for report types to tools. by nuclearsandwich · Pull Request #569 · ros2/ci · GitHub

Before deploying the change to ros_buildfarm I’m going to investigate the underlying issue further. Since the failures are affecting dev jobs only we are considering steps to restore service to the rest of the build farm with scoped agent labels.

3

In order to allow the rest of the buildfarm to continue operation we’ve disabled Devel and PR jobs across both build.ros.org and build.ros2.org. I’ll be working tomorrow to restore those jobs as soon as possible. The ROS Status page has an incident that can be subscribed to for updates to the service availability for those who don’t want to follow the underlying issues. ROS Status - Devel and PR jobs broken after recent Jenkins upgrade.

4

A play-by-play summary of the events so far. In bullet points:

  • ROS buildfarms are updated to Jenkins 2.277.3 to address security issues.
  • 2.277.1 also includes breaking changes relative to the previous LTS releases.
  • A change in the serialized xml schema for the xunit plugin caused the configuration to appear invalid causing builds to fail when the xunit plugin runs at the end.
  • A minor update to the xunit (2.3.9 → 2.4.0) plugin restored compatibility with the new changes.
  • Jobs which were reconfigured during the window when Jenkins 2.277.3 was running with xunit 2.3.9 were left in an invalid configuration state.
  • Plugin updates deployed to build.ros2.org do not appear to resolve the issue due to the undetected configuration change.
  • Once reconfigured, plugin updates on build.ros2.org resolved the issue in all known cases.
  • The same plugin updates have been deployed to build.ros.org and all Devel and PR jobs have been reconfigured to make sure that the configurations are all in a working state.

For those of you running build farms, if your Jenkins instance is not accessible on the public internet, I would advise discretion in updating as there is still at least one significant unresolved issue as the result of the migration https://github.com/ros-infrastructure/cookbook-ros-buildfarm/pull/90. If you are running publicly then the plugin version updates made by Update xunit plugin for 2.277 compatibility. by nuclearsandwich · Pull Request #89 · ros-infrastructure/cookbook-ros-buildfarm · GitHub should get your devel and PR jobs working again as well.