Security issue on ROS build farm

nuclearsandwich · June 4, 2019, 3:59am

We are continuing to investigate the compromise of build.ros.org, including working with independent third-party security experts. We’ll report back here as we learn more from them. So far we have no indication that the intrusion was anything more than a commodity attack by a group looking to hijack CPU cycles. But we are unlikely to ever be able to completely rule out malicious interference in the ROS binary packaging pipeline. So in an abundance of caution we are (i) continuing to rebuild everything that we reasonably can and (ii) relocating the rest.

Summary of breaking changes

These changes are coming Thursday or Friday (2019-06-06 or 2019-06-07) this week.

Signing key used for the primary ROS repositories will change.
Packages for unsupported rosdistros will move to snapshots.ros.org.
The testing repository is being relocated to ros-testing from ros-shadow-fixed.

Rebuilding supported distros

We are currently working on full rebuilds of Indigo, Kinetic, Lunar, and Melodic across all architectures on their supported Ubuntu Long Term Support distributions. We expect to have these builds completed by Thursday and available on packages.ros.org.

As we fill them in, packages.ros.org will come to contain only packages built from the bottom up on the newly deployed build farm. The packages will comprise currently supported ROS distributions (plus Indigo) and only on their respective Ubuntu LTS platform. Debian Stretch packages for Lunar and Melodic will become available some time later.

Relocating unsupported distros

Packages for the unsupported ROS distributions will be moved to snapshots.ros.org. Those package indexes will be resigned with a GPG key used specifically for the snapshots.ros.org host (this was always a different key from the one used for packages.ros.org). But we will not be rebuilding binary packages for unsupported ROS distros. Users should make their own risk assessments regarding whether to use those packages (including the risk of using unsupported software in the first place).

We are currently hoping to make these packages available on snapshots.ros.org tomorrow or Wednesday (2019-06-04 or 2019-06-05) so that there is no time when they are completely unavailable but if we have to make a choice between making unsupported packages temporarily unavailable or delaying the deployment of the newly built ROS packages, we’ll choose to deploy the newly built packages.

Renaming the testing repository

While we’re making these changes to our repository structure we’re also taking the opportunity to rename the testing repository which has been carrying the name “ros-shadow-fixed” to avoid breaking anyone using it. The new name will be “ros-testing” to match the “ros2-testing” repository already available.

How to transition

Adding the new ROS repository key

Who should do this? Everyone who installs ROS packages from packages.ros.org

When should this be done? You can do it right now!

When must this be done? Exact time to be determined, but not later than Friday this week (2019-06-07)

What to do?
Set up the new repository key

sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

Removing the old ROS repository key

Who should do this? Everyone.

When should this be done? After the ROS repositories have been redeployed with the new signing key. Exact time to be determined, but not later than Friday this week (2019-06-07)

When must this be done? You should remove it as soon as the ROS repositories are redeployed. Leaving the key trusted won’t cause anything to break but we won’t be using the key to sign anything new in the future so leaving it just leaves you open to possible misuse in the future.

What to do?
Remove the key from your apt keyring

sudo apt-key del 421C365BD9FF1F717815A3895523BAEEB01FA116

Migrating to the snapshots repository for unsupported distributions

Who should do this? Anyone who needs to access ROS distributions other than Indigo, Kinetic, Lunar, and Melodic or the Debian Jessie packages for Kinetic

When should this be done? I’ll update everyone here when I’ve created snapshots for all the old ROS releases. After that announcement you can switch to the snapshot repositories.

When must this be done? After the ROS repositories have been redeployed with the new signing key. Exact time to be determined, but not later than Friday this week (2019-06-07)

What to do?

We’re making some changes to the snapshots repository layouts to support this. I am waiting to add docs until I know they’ll be accurate. In brief the steps will be

Add the ROS Snapshots repository key (different from the key above)
Add the snapshot repository for your target ROS distribution

Updating the testing repository url

Who should do this? Anyone who is currently using the ros-shadow-fixed repository to test ROS packages before a sync.