ROS GPG Key Expiration Incident

ROS GPG Key Expiration Incident

This evening the ROS GPG keys inadvertently expired and caused apt failures for a number of users. In our response to a security incident two years ago we deployed a new GPG key with a 2 year expiration however; we neglected to set a reminder to extend the expiration date of a GPG key prior to its expiration. A full description of the security breach and the remediation can be found in this ROS Discourse post.

The expired GPG key is used in a number of places, including the ROS 1 and ROS 2 build farm. A quick solution to this problem is to simply extend the expiration date of the offending key, however, this also requires issuing a new public key for the extended private GPG key. This is where the issue impacts users, re-issuing the public GPG key requires that users also download, install, and use the public key. Thankfully, this issue represents only a temporary service outage, and has no significant security implications.

Due to the necessity of updating the public key several other ROS services were impacted by this issue. Public keys had to be updated on build.ros.org, build.ros2.org, ROS infrastructure, ROS buildfarm, ROS distro, packages.ros.org, and the Ubuntu key server. While these changes were trivial, issuing and reviewing the pull requests, redeploying the services, and allowing the changes to propagate takes some time.

At this time all of the changes have been deployed and tested and are now working. The new GPG key is set to expire in five years (and more than one reminder has been set). To fix this issue users need to update the public key used for ROS apt repositories. To do this for ROS 1 installations you need to run a single command:

curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | sudo apt-key add -

For ROS 2 installations you will need to run this command:

sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg

Complete example instructions for Rolling can be found here.

We apologize for any inconvenience this may have caused ROS users over the long holiday weekend. While the impact of this issue is broad, we hope the remedy for our end users is fairly straightforward. In the coming days we will post a full post-mortem of the incident along with the corrective actions we plan to take to prevent this from happening again. The Open Robotics team values our users, and we strive to provide safe, secure, and reliable binary packages to the ROS community.

23 Likes

Is there an ETA for when the official Docker images will be updated/rebuild?

9 Likes

Just letting you know that this alone didn’t work for me. I had to remove and re-add the ros2.list file in addition.

Thanks for the update.
I tested on ROS Foxy with Ubuntu 20.04, but the following command did not worked.

sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg

The command you describe as for ROS1 worked fine.

curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | sudo apt-key add -

Here are the commands and the results.

ubuntu@c540a473ddac:~$ uname -a
Linux c540a473ddac 5.4.0-64-generic #72~18.04.1-Ubuntu SMP Fri Jan 15 14:06:34 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@c540a473ddac:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.1 LTS
Release:	20.04
Codename:	focal
ubuntu@c540a473ddac:~$ echo $ROS_DISTRO 
foxy
ubuntu@c540a473ddac:~$ sudo apt update
Get:1 http://dl.google.com/linux/chrome/deb stable InRelease [1811 B]
Get:2 http://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1081 B]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]      
Get:4 http://mirrors.ubuntu.com/mirrors.txt Mirrorlist [368 B]                 
Get:5 http://ftp.jaist.ac.jp/pub/Linux/ubuntu focal InRelease [265 kB]         
Get:8 http://packages.ros.org/ros2/ubuntu focal InRelease [4670 B]             
Get:7 http://ftp.riken.jp/Linux/ubuntu focal-backports InRelease [101 kB]      
Get:6 http://ftp.jaist.ac.jp/pub/Linux/ubuntu focal-updates InRelease [114 kB] 
Err:8 http://packages.ros.org/ros2/ubuntu focal InRelease           
  The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
Get:9 http://ftp.jaist.ac.jp/pub/Linux/ubuntu focal-backports/universe amd64 Packages [4305 B]
Get:10 http://ubuntutym.u-toyama.ac.jp/ubuntu focal/main amd64 Packages [1275 kB]
Get:11 http://ftp.tsukuba.wide.ad.jp/Linux/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:14 http://www.ftp.ne.jp/Linux/packages/ubuntu/archive focal-updates/multiverse amd64 Packages [29.8 kB]
Get:15 http://ubuntutym.u-toyama.ac.jp/ubuntu focal-updates/restricted amd64 Packages [299 kB]
Get:13 https://linux.yz.yamagata-u.ac.jp/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [727 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:19 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [274 kB]
Get:20 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [834 kB]
Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [27.6 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1247 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [974 kB]
Reading package lists... Done                              
W: GPG error: http://packages.ros.org/ros2/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
E: The repository 'http://packages.ros.org/ros2/ubuntu focal InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
ubuntu@c540a473ddac:~$ apt-key list
/etc/apt/trusted.gpg
--------------------
pub   dsa1024 2007-03-08 [SC]
      4CCA 1EAF 950C EE4A B839  76DC A040 830F 7FAC 5991
uid           [ unknown] Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>
sub   elg2048 2007-03-08 [E]

pub   rsa4096 2016-04-12 [SC]
      EB4C 1BFD 4F04 2F6D DDCC  EC91 7721 F63B D38B 4796
uid           [ unknown] Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
sub   rsa4096 2019-07-22 [S] [expires: 2022-07-21]

pub   rsa4096 2019-05-30 [SC] [expired: 2021-05-29]
      C1CF 6E31 E6BA DE88 68B1  72B4 F42E D6FB AB17 C654
uid           [ expired] Open Robotics <info@osrfoundation.org>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid           [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>

ubuntu@c540a473ddac:~$ sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg
ubuntu@c540a473ddac:~$ apt-key list
/etc/apt/trusted.gpg
--------------------
pub   dsa1024 2007-03-08 [SC]
      4CCA 1EAF 950C EE4A B839  76DC A040 830F 7FAC 5991
uid           [ unknown] Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>
sub   elg2048 2007-03-08 [E]

pub   rsa4096 2016-04-12 [SC]
      EB4C 1BFD 4F04 2F6D DDCC  EC91 7721 F63B D38B 4796
uid           [ unknown] Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
sub   rsa4096 2019-07-22 [S] [expires: 2022-07-21]

pub   rsa4096 2019-05-30 [SC] [expired: 2021-05-29]
      C1CF 6E31 E6BA DE88 68B1  72B4 F42E D6FB AB17 C654
uid           [ expired] Open Robotics <info@osrfoundation.org>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid           [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>

ubuntu@c540a473ddac:~$ sudo apt update
Hit:1 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease                                                                                        
Get:3 http://packages.ros.org/ros2/ubuntu focal InRelease [4670 B]                                                                                      
Err:3 http://packages.ros.org/ros2/ubuntu focal InRelease
  The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
Get:4 http://mirrors.ubuntu.com/mirrors.txt Mirrorlist [368 B]
Hit:7 http://www.ftp.ne.jp/Linux/packages/ubuntu/archive focal-backports InRelease
Hit:5 http://ftp.riken.jp/Linux/ubuntu focal InRelease                
Hit:6 http://ftp.jaist.ac.jp/pub/Linux/ubuntu focal-updates InRelease 
Reading package lists... Done
W: GPG error: http://packages.ros.org/ros2/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
E: The repository 'http://packages.ros.org/ros2/ubuntu focal InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
ubuntu@c540a473ddac:~$ curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | sudo apt-key add -
OK
ubuntu@c540a473ddac:~$ apt-key list
/etc/apt/trusted.gpg
--------------------
pub   dsa1024 2007-03-08 [SC]
      4CCA 1EAF 950C EE4A B839  76DC A040 830F 7FAC 5991
uid           [ unknown] Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>
sub   elg2048 2007-03-08 [E]

pub   rsa4096 2016-04-12 [SC]
      EB4C 1BFD 4F04 2F6D DDCC  EC91 7721 F63B D38B 4796
uid           [ unknown] Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
sub   rsa4096 2019-07-22 [S] [expires: 2022-07-21]

pub   rsa4096 2019-05-30 [SC] [expires: 2025-06-01]
      C1CF 6E31 E6BA DE88 68B1  72B4 F42E D6FB AB17 C654
uid           [ unknown] Open Robotics <info@osrfoundation.org>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid           [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>

ubuntu@c540a473ddac:~$ sudo apt update
Hit:1 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease                                                          
Get:3 http://packages.ros.org/ros2/ubuntu focal InRelease [4670 B]                                                        
Get:4 http://packages.ros.org/ros2/ubuntu focal/main amd64 Packages [660 kB]
Get:5 http://mirrors.ubuntu.com/mirrors.txt Mirrorlist [368 B]
Hit:6 http://mirror.fairway.ne.jp/ubuntu focal InRelease         
Hit:7 http://mirror.fairway.ne.jp/ubuntu focal-updates InRelease        
Hit:8 http://ftp.tsukuba.wide.ad.jp/Linux/ubuntu focal-backports InRelease               
Fetched 665 kB in 3s (247 kB/s)                                                          
Reading package lists... Done
Building dependency tree       
Reading state information... Done
517 packages can be upgraded. Run 'apt list --upgradable' to see them.

1 Like

The wiki needs to be updated as well. For example, the ROS Noetic installation instructions still refer to the old key:

http://wiki.ros.org/noetic/Installation/Ubuntu

1 Like

Thanks for the quick response.

@Katherine_Scott I think you mean four years. Unless you’re aggressively rounding up.

I’m having trouble reproducing this error, @Tiryoh. On my focal machine, I see the same error initially, but apt update is successful after running the apt-key add command you referenced.

Please try the command again with debug output enabled, which should give you an idea of where apt is finding the reference to the old public key file: sudo apt -oDebug::pkgAcquire::Worker=1 update

Thanks @Martin_Guenther, but I’m not sure that’s true. The fingerprint of the key didn’t change, so key server commands like that shouldn’t need to be any different, as long as the full public key has been updated on the key server (which is the case for keyserver.ubuntu.com).

Relevant issue tracking the official docker images: Something broken in ros:melodic-ros-base image · Issue #535 · osrf/docker_images · GitHub

For those having apt problems because of an existing bad pubkey, you can locate the offending keyring with apt-key list:

$ docker container run -it --rm ros:melodic apt-key list
/etc/apt/trusted.gpg
--------------------
pub   rsa4096 2019-05-30 [SC] [expired: 2021-05-29]
      C1CF 6E31 E6BA DE88 68B1  72B4 F42E D6FB AB17 C654
uid           [ expired] Open Robotics <info@osrfoundation.org>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid           [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>


I stumbled upon this as well
@Tiryoh:Did you add the signed-by option to the ros2 list (recent documentation change)?

1 Like

It’s Memorial day on Monday in the USA, so essentially a long weekend, and I don’t expect the Docker Hub librarians to be online over the holiday. So as for an ETA, this might only get resolved as soon as Tuesday. An unfortunate timing of events. :<

@cottsay

As @ipa-mdl said, it seems to be caused by installing without using the signed-by option.
I didn’t realize that. Thank you for letting me know, @ipa-mdl!
The environment was set up in 2020 and apt upgraded since then, so that’s probably the cause.

How about guiding ROS 2 users who set up before April 2021 to use “apt-key add”, which is the same approach as ROS1?

This is the result of sudo apt -oDebug::pkgAcquire::Worker=1 update 2>&1 | tee log.txt.

As noted in the documentation for newer Ubuntu or Debian versions, you need to use the updated method to install/update a key

Thanks to everyone who pulled an all-nighter fixing the issue, good work everyone :clap:

I almost had a heart attack when my CI pipelines broke. Thank you for keeping life exciting :stuck_out_tongue:

2 Likes

How can I update the GPG key in github-actions? The ros-tooling/setup-ros@0.2.0 command (which is the latest version 0.2.0) stops with this error:

  /usr/bin/sudo apt-get update
  Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease
  Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
  Get:3 http://packages.ros.org/ros/ubuntu focal InRelease [4,676 B]
  Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Get:5 http://packages.ros.org/ros2/ubuntu focal InRelease [4,670 B]
  Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Err:3 http://packages.ros.org/ros/ubuntu focal InRelease
    The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
  Err:5 http://packages.ros.org/ros2/ubuntu focal InRelease
    The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
  Reading package lists...
  W: GPG error: http://packages.ros.org/ros/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
  E: The repository 'http://packages.ros.org/ros/ubuntu focal InRelease' is not signed.
  W: GPG error: http://packages.ros.org/ros2/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
  E: The repository 'http://packages.ros.org/ros2/ubuntu focal InRelease' is not signed.
Error: The process '/usr/bin/sudo' failed with exit code 100

Link to corresponding ci.yml

Link to failing github-action:

@JanStaschulat this PR will update the key: Update the GPG key to the latest by Tiryoh · Pull Request #405 · ros-tooling/setup-ros · GitHub

Then we’ll need to create another release.

2 Likes