Security issue on ROS build farm

General
1

The machine hosting build.ros.org, which is the build farm for ROS 1, was recently compromised. We took that machine offline and are in the process of deploying a new farm, with the known exploit patched.

The compromise included local privilege escalation sufficient to access the GPG private key used for signing Debian packages and to push Debian packages into the public-facing repository. We have no reason to believe that any such malicious activity occurred, but at this point we are unable to rule it out.

So, in an abundance of caution, we are using a newly generated GPG key pair for the new build.ros.org. Because the same GPG key is used to build ROS 2 packages we also swapped the key on build.ros2.org.

As a result of the farm redeployment and key change, users who are installing or updating ROS packages may encounter service disruptions.

We are working diligently to get back to normal operation as quickly as possible and will provide more updates as we have them.

8 Likes
2

Thanks for the heads up @gerkey. Given the inability to rule out malicious uploads, can we safely assume the new repo will only consist of newly-rebuilt packages? In which case I expect it’ll be quite a while before everything is back up and running.

3

@kyrofa, your supposition is correct. We’re in the process of rebuilding the core of ROS 1 Kinetic and will fan out from there once we’re sure that everything is working well. The newly built binaries are going into a new apt repo that will be made available when it’s ready for public consumption.

1 Like
4

For those of us running private instances (but perhaps publicly accessible): could you provide a link to a CVE ID or other information about the specific exploit?

5

Excellent. The new key is C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654, correct? Let us know if we can help update docs once the repo is ready.

1 Like
6

We believe that the infiltration point was via this vulnerability in the Jenkins Groovy Plugin:

All Jenkins users should update their Groovy Plugin to patch that exploit.

7

That’s correct, and we’ll definitely take you up on the kind offer for help! But I’d advise against updating anything just yet while things are in flight.

8

We are continuing to investigate the compromise of build.ros.org, including working with independent third-party security experts. We’ll report back here as we learn more from them. So far we have no indication that the intrusion was anything more than a commodity attack by a group looking to hijack CPU cycles. But we are unlikely to ever be able to completely rule out malicious interference in the ROS binary packaging pipeline. So in an abundance of caution we are (i) continuing to rebuild everything that we reasonably can and (ii) relocating the rest.

Summary of breaking changes

These changes are coming Thursday or Friday (2019-06-06 or 2019-06-07) this week.

  • Signing key used for the primary ROS repositories will change.
  • Packages for unsupported rosdistros will move to snapshots.ros.org.
  • The testing repository is being relocated to ros-testing from ros-shadow-fixed.

Rebuilding supported distros

We are currently working on full rebuilds of Indigo, Kinetic, Lunar, and Melodic across all architectures on their supported Ubuntu Long Term Support distributions. We expect to have these builds completed by Thursday and available on packages.ros.org.

As we fill them in, packages.ros.org will come to contain only packages built from the bottom up on the newly deployed build farm. The packages will comprise currently supported ROS distributions (plus Indigo) and only on their respective Ubuntu LTS platform. Debian Stretch packages for Lunar and Melodic will become available some time later.

Relocating unsupported distros

Packages for the unsupported ROS distributions will be moved to snapshots.ros.org. Those package indexes will be resigned with a GPG key used specifically for the snapshots.ros.org host (this was always a different key from the one used for packages.ros.org). But we will not be rebuilding binary packages for unsupported ROS distros. Users should make their own risk assessments regarding whether to use those packages (including the risk of using unsupported software in the first place).

We are currently hoping to make these packages available on snapshots.ros.org tomorrow or Wednesday (2019-06-04 or 2019-06-05) so that there is no time when they are completely unavailable but if we have to make a choice between making unsupported packages temporarily unavailable or delaying the deployment of the newly built ROS packages, we’ll choose to deploy the newly built packages.

Renaming the testing repository

While we’re making these changes to our repository structure we’re also taking the opportunity to rename the testing repository which has been carrying the name “ros-shadow-fixed” to avoid breaking anyone using it. The new name will be “ros-testing” to match the “ros2-testing” repository already available.

How to transition

Adding the new ROS repository key

Who should do this? Everyone who installs ROS packages from packages.ros.org

When should this be done? You can do it right now!

When must this be done? Exact time to be determined, but not later than Friday this week (2019-06-07)

What to do?
Set up the new repository key

sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

Removing the old ROS repository key

Who should do this? Everyone.

When should this be done? After the ROS repositories have been redeployed with the new signing key. Exact time to be determined, but not later than Friday this week (2019-06-07)

When must this be done? You should remove it as soon as the ROS repositories are redeployed. Leaving the key trusted won’t cause anything to break but we won’t be using the key to sign anything new in the future so leaving it just leaves you open to possible misuse in the future.

What to do?
Remove the key from your apt keyring

sudo apt-key del 421C365BD9FF1F717815A3895523BAEEB01FA116

Migrating to the snapshots repository for unsupported distributions

Who should do this? Anyone who needs to access ROS distributions other than Indigo, Kinetic, Lunar, and Melodic or the Debian Jessie packages for Kinetic

When should this be done? I’ll update everyone here when I’ve created snapshots for all the old ROS releases. After that announcement you can switch to the snapshot repositories.

When must this be done? After the ROS repositories have been redeployed with the new signing key. Exact time to be determined, but not later than Friday this week (2019-06-07)

What to do?

We’re making some changes to the snapshots repository layouts to support this. I am waiting to add docs until I know they’ll be accurate. In brief the steps will be

  1. Add the ROS Snapshots repository key (different from the key above)
  2. Add the snapshot repository for your target ROS distribution

Updating the testing repository url

Who should do this? Anyone who is currently using the ros-shadow-fixed repository to test ROS packages before a sync.

When should this be done? After the ROS repositories have been redeployed with the new signing key. Exact time to be determined, but not later than Friday this week (2019-06-07)

When must this be done? Once the deployment is complete, the shadow-fixed repository will be purged of release contents and you’ll see an error message when updating apt repositories until you update the repository url.

What to do?

Check your /etc/apt/sources.list and /etc/apt/sources.list.d/*.list for http://packages.ros.org/ros-shadow-fixed and replace it with http://packages.ros.org/ros-testing

4 Likes
9

This is definitely help we’ll need going into the weekend. I’ve opened https://github.com/ros-infrastructure/roswiki/issues/276 to track documentation that will need updating after the changes above.

10

Just adding this as I don’t see it mentioned anywhere: anyone who is building custom Docker images (or using any other sort of containerisation technology) and does not start from any of the official ros or osrf/ros images will also have to update their Dockerfiles (or whatever it is called for the tool you are using) to use / import the new keys.

3 Likes
11

Also don’t forget to update your .travis.yml / .gitlab-ci.yml etc. files if they contain the string 421C365 or ros-shadow-fixed.

12

The Snapshots repository now has final snapshots of all unsupported ROS distributions and distributions with unsupported platforms (i.e. Kinetic on Jessie) with the exception of Indigo on Trusty which is being rebuilt and will therefore be on packages.ros.org for a while longer as well as having a final snapshot made when it’s done.

If you’re using one of these unsupported rosdistros. I’d recommend updating your repositories now so you can let us know if you run into any difficulties. We are still on track to sync the updated repositories this Friday (2019-06-07).

Information on using the snapshot repository can be found on the ROS wiki http://wiki.ros.org/SnapshotRepository

1 Like
13

Wouldn’t this be a great candidate to feature on the ROS news section also? There may be a great set of users from old times that may not be regularly checking the discourse threads…

Thank you for your hard work guys!

14

We have rebuilt all debian packages for indigo, kinetic, lunar, and melodic. And the packages have been redeployed with the updated GPG key to packages.ros.org see the announcement: New GPG keys deployed for packages.ros.org

1 Like
15

That is great news.

I’d like to take this chance to extend a huge thank you to my colleagues who moved heaven and earth over the past week and a half to respond to this security compromise and get our systems back into a healthy state for the entire community. In particular, @tfoote, @nuclearsandwich, and @jrivero put in many many hours over nights and through the weekend, so cheers to them!

Also a big thanks to @Thomas_Moulard and his colleagues at AWS who helped along the way with diagnosis and mitigation of the initial problem and planning for the redeployment, and also served as our early testers of the outputs from the new farm.

And thank you to everyone in the community for your patience and support as we’ve worked through an unfortunate incident.

9 Likes
16

Here a jenkins security advice: http://turnoff.us/geek/securing-jenkins/