ROS GPG Key Expiration Incident

Thanks for the quick response.

@Katherine_Scott I think you mean four years. Unless you’re aggressively rounding up.

I’m having trouble reproducing this error, @Tiryoh. On my focal machine, I see the same error initially, but apt update is successful after running the apt-key add command you referenced.

Please try the command again with debug output enabled, which should give you an idea of where apt is finding the reference to the old public key file: sudo apt -oDebug::pkgAcquire::Worker=1 update

Thanks @Martin_Guenther, but I’m not sure that’s true. The fingerprint of the key didn’t change, so key server commands like that shouldn’t need to be any different, as long as the full public key has been updated on the key server (which is the case for keyserver.ubuntu.com).

Relevant issue tracking the official docker images: Something broken in ros:melodic-ros-base image · Issue #535 · osrf/docker_images · GitHub

For those having apt problems because of an existing bad pubkey, you can locate the offending keyring with apt-key list:

$ docker container run -it --rm ros:melodic apt-key list
/etc/apt/trusted.gpg
--------------------
pub   rsa4096 2019-05-30 [SC] [expired: 2021-05-29]
      C1CF 6E31 E6BA DE88 68B1  72B4 F42E D6FB AB17 C654
uid           [ expired] Open Robotics <info@osrfoundation.org>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid           [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>


I stumbled upon this as well
@Tiryoh:Did you add the signed-by option to the ros2 list (recent documentation change)?

1 Like

It’s Memorial day on Monday in the USA, so essentially a long weekend, and I don’t expect the Docker Hub librarians to be online over the holiday. So as for an ETA, this might only get resolved as soon as Tuesday. An unfortunate timing of events. :<

@cottsay

As @ipa-mdl said, it seems to be caused by installing without using the signed-by option.
I didn’t realize that. Thank you for letting me know, @ipa-mdl!
The environment was set up in 2020 and apt upgraded since then, so that’s probably the cause.

How about guiding ROS 2 users who set up before April 2021 to use “apt-key add”, which is the same approach as ROS1?

This is the result of sudo apt -oDebug::pkgAcquire::Worker=1 update 2>&1 | tee log.txt.

As noted in the documentation for newer Ubuntu or Debian versions, you need to use the updated method to install/update a key

Thanks to everyone who pulled an all-nighter fixing the issue, good work everyone :clap:

I almost had a heart attack when my CI pipelines broke. Thank you for keeping life exciting :stuck_out_tongue:

2 Likes

How can I update the GPG key in github-actions? The ros-tooling/setup-ros@0.2.0 command (which is the latest version 0.2.0) stops with this error:

  /usr/bin/sudo apt-get update
  Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease
  Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
  Get:3 http://packages.ros.org/ros/ubuntu focal InRelease [4,676 B]
  Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Get:5 http://packages.ros.org/ros2/ubuntu focal InRelease [4,670 B]
  Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Err:3 http://packages.ros.org/ros/ubuntu focal InRelease
    The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
  Err:5 http://packages.ros.org/ros2/ubuntu focal InRelease
    The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
  Reading package lists...
  W: GPG error: http://packages.ros.org/ros/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
  E: The repository 'http://packages.ros.org/ros/ubuntu focal InRelease' is not signed.
  W: GPG error: http://packages.ros.org/ros2/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 Open Robotics <info@osrfoundation.org>
  E: The repository 'http://packages.ros.org/ros2/ubuntu focal InRelease' is not signed.
Error: The process '/usr/bin/sudo' failed with exit code 100

Link to corresponding ci.yml

Link to failing github-action:

@JanStaschulat this PR will update the key: Update the GPG key to the latest by Tiryoh · Pull Request #405 · ros-tooling/setup-ros · GitHub

Then we’ll need to create another release.

2 Likes

Done. Simply bump to ros-tooling/setup-ros@0.2.1. There is nothing to do for people using @v0.2.

3 Likes

Wonderful, works like a charme. Thanks!

1 Like

+1, worked fine. Thanks for the prompt update.

Any idea when are the ROS 2 Docker images going to be usable again?

Thanks!

8 Likes

@christophebedard Thanks a lot. Works great!

1 Like

Affected images should be rebuilt soon after this gets mered upstream by the librarians:

2 Likes

:+1:

In the meantime you can setup the new key in your container before installing more packages.
For example:

docker run -it --rm ros:foxy
apt update || apt install -y curl && curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | apt-key add -
3 Likes

Just noticed the docker images were updated, thanks to everyone involved in fixing this!
Much appreciated.