ROS signing key migration guide

We are again upon the brink of the ROS signing key expiration (June 1st) and from the Infrastructure PMC we are happy to announce a new package for ROS key and repository management.
The set of packages ros-apt-source and ros2-apt-source for Ubuntu and ros2-release for RHEL provide both the signing key and the repository configuration for ROS and ROS 2 :tada:.

This a big step towards a simpler key update path allowing the ROS project to rotate keys while keeping ROS users in sync as long as the they keep their system up to date! From a security stand-point, having the ability to rotate keys without breaking ROS users has been a longtime goal for the Infrastructure PMC and this is the first step towards it.

The are two flavors of the package for Ubuntu/Debian distributions:

For RHEL there is a single package ros2-release that configures both main and testing repositories and enables the main repository as the default behavior.

How to migrate (before June 1st)

Given a setup that has the ros key and repository configuration following the former installation steps, you need to perform the following steps

Ubuntu/Debian

Install new package using the previous repository configuration

# if ROS 1 
sudo apt update && sudo apt install ros-apt-source  
# if ROS 2 
sudo apt update && sudo apt install ros2-apt-source  

Remove old repository configuration files otherwise apt will be broken due to key conflict (these files might be different in your system if you did not follow the installation steps):

# If using ROS 1
sudo rm /etc/apt/sources.list.d/ros-latest.list
sudo apt-key del "C1CF 6E31 E6BA DE88 68B1  72B4 F42E D6FB AB17 C654"
# If using ROS 2
sudo rm /etc/apt/sources.list.d/ros2.list
sudo rm /usr/share/keyrings/ros-archive-keyring.gpg

Test that packages are still discovered and thus installable:

sudo apt update
sudo apt search ^package_name (for example ros-rolling-desktop-full)

RHEL

Install new package

sudo dnf install ros2-release

Due to name conflict you will see that the new configuration is created under /etc/yum.repos.d/ros2.repo.rpmnew
Replace the old configuration with the new one :

sudo rm /etc/yum.repos.d/ros2.repo 
sudo mv /etc/yum.repos.d/ros2.repo.rpmnew /etc/yum.repos.d/ros2.repo

Test that packages are still discovered and thus installable:

sudo dnf update 
sudo dnf search package_name #for example ros-rolling-desktop

How do I migrate (after June 1st)?

There are two ways you can get the new package after June 1st:

  • Update manually to the new key and run the migration guide above. To get the updated key run:
sudo rm /usr/share/keyrings/ros-archive-keyring.gpg
sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg
  • Remove the previous configuration by running the following commands and follow the new installation instructions for your ROS distribution.
# If using ROS 1
sudo rm /etc/apt/sources.list.d/ros-latest.list
sudo apt-key del "C1CF 6E31 E6BA DE88 68B1  72B4 F42E D6FB AB17 C654" # This is the old key fingerprint, you should see by running apt-key list 
# If using ROS 2
sudo rm /etc/apt/sources.list.d/ros2.list 
sudo rm /usr/share/keyrings/ros-archive-keyring.gpg

FAQ

  • Does it support ROS 1?
    The package supports both ROS 1 and ROS 2 for the declared supported platforms on REP-2000. The list of ROS-OS pairs supported can be found here.

  • What happens if I want to use a mirror of packages.ros.org?
    For debian based systems (Ubuntu, Debian)
    The package configures the source on etc/apt/sources.list.d/ directory but does not set a preference file. It’s possible to add an additional source file (copying from the one provided by the package and modifying the source) that configures a different repository (the mirror url) with a higher preference (See apt_preferences documentation) so that packages are being sourced from the mirror.

  • I have already downloaded the package for the tutorial party/kilted testing, do I need to do anything else?
    If you downloaded the package for the release party then you have the testing repository configured. You will need to:

    Debian

    sudo apt remove ros2-testing-apt-source
    

    Follow the instructions to setup from scratch Ubuntu (deb packages) — ROS 2 Documentation: Kilted documentation

    RHEL

    In RHEL the package configures both repositories, you will need to disable the ros2-testing repo and enable the main one

    sudo dnf config-manager --disable ros2-testing
    sudo dnf config-manager --enable ros2
    
  • Can I still source the key the old way?
    Even though this is the new recommended way of getting the ROS key it does not prevent ROS users from using the previous way of setting up ROS repositories. The goal of this package is to provide an ergonomic way of configuring ROS repositories and an improved update pathway.

There is still some pending work to update the installation instructions across the docs that will be addressed during this week as well as updates to the key hosted on rosdistro repository and packages.ros.org, but we encourage you to start the migration as soon as possible.

As always feel free to ask questions here or in the #infra-help channel on Discord.

12 Likes

ROS/Installation/UbuntuMirrors - ROS Wiki will need to be updated, too (Mirrors).

1 Like

Hmm:

From Jammy onwards the key is embedded in that source file

what is the suggested path when using mirrors? Because the instructions say to copy the source file. But the copied file will not get updated key when the time comes again, right?

@peci1 That’s correct, the key is embedded from Jammy onwards, however the gpg key file is maintained in ‘/usr/share/keyrings/ros2-archive-keyring.gpg’ to ease other usages outside of the package.
You can change the Signed By of the mirror source file to point to the gpg key path to get the key update when using mirrors.

1 Like

Would it be pertinent to have the signing key in a separate package, such as the ros-archive-keyring package proposed by the PMC in 2024?

Our production machines have their own apt source configuration and for example don’t need the deb-src packages.

1 Like

I agree this would be a better solution for us, too.

@claraberendsen Could you also cross-post to https://community.gazebosim.org/ ? Many people install Gazebo from ROS vendor packages..

1 Like

Not entirely true. AFAIK people were using ros2-testing repo for T&T party. I don’t think they will want to continue using that.


If anyone is interested into how to newly install ROS 2 on a fresh system, here’s the current guide: Ubuntu (deb packages) — ROS 2 Documentation: Kilted documentation .

@claraberendsen The tutorial currently points to https://ftp.osuosl.org/pub/ros/packages.ros.org/ros2/ubuntu/pool/main/r/ros-apt-source/ros2-apt-source_1.0.0~$(. /etc/os-release && echo $VERSION_CODENAME)_all.deb . This seems to be very unmaintainable (even now there has already been a version bump to 1.1.0 which hasn’t yet propagated into the docs page, so currently, the install instructions lead to a 404 link). Two questions:

  1. Why does it point to ftp.osuosl and not “directly” to packages.ros.org ?
  2. Would it be possible to provide versionless symlinks to the debs? This would really help automated workflows like Docker, Ansible etc.

Mirrors

Okay, this seems to work for me to enable a mirror:

$ cat /etc/apt/preferences.d/ros-prefer-mirrors 
Package: *
Pin: origin "packages.ros.org"
Pin-Priority: 499

And

$ cat /etc/apt/sources.list.d/ros2-mirror.sources 
Types: deb deb-src
URIs: http://ftp.tudelft.nl/ros2/ubuntu
Suites: noble
Components: main
Signed-By: /usr/share/keyrings/ros2-archive-keyring.gpg

Now I have (the mirror has not yet picked up version 1.1.0 of the package and I’ve locally installed 1.0.0).

$ apt-cache policy ros2-apt-source 
ros2-apt-source:
  Installed: 1.0.0~noble
  Candidate: 1.0.0~noble
  Version table:
     1.1.0~noble 499
        499 http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
 *** 1.0.0~noble 500
        500 http://ftp.tudelft.nl/ros2/ubuntu noble/main amd64 Packages
        100 /var/lib/dpkg/status
1 Like

This is because due to how the hosting is setup packages.ros.org doesn’t have https. The documentation drift is part of the update to docs I need to make across the docs I ran out of time yesterday and preferred to post this so people can start migrating. Those updates are incoming.
The new url is going to link to our Github releases for bootstrapping the package and it will point to something like https://github.com/ros-infrastructure/ros-apt-source/releases/latest/download/ros2-testing-apt-source_1.1.0.jammy_all.deb.

This is something we can take a look at :thinking:. I don’t expect this package to be updated often so documentation drift should be minimal (minus this transition in which the drift is big). I will bring this up to the next Infra PMC as discussion it could be a nice quality of life improvement though we would need to correctly map the symlinks for each OS distribution.

You are right, thanks for catching this up :sparkling_heart:. I will update the migration guide to say they need to download the main repository.

1 Like

Well, the title doesn’t reflect it but: Ubuntu derivatives support by Timple · Pull Request #5675 · ros2/ros2_documentation · GitHub

But I agree, a symlink to the newest one would be nice!

2 Likes

A real case that just happened to my colleague:

He copied this as a command, figured he needs sudo, so he added sudo to the beginning, but didn’t notice it also needs to be added after the &&. This resulted in not adding the ros2-apt-source, which, when you continue with the sudo rm commands, yields a broken system. Interestingly, the rm commands have sudo.

@claraberendsen could you please convert all these commands in the initial post to actual working commands with sudo? (also, the cited command needs a # before the comment).

1 Like

Updated across the docs :slight_smile:. Sorry for the oversight :grimacing:

1 Like

Is the apt index on the server currently outdated? Neither Ubuntu (deb packages) — ROS 2 Documentation: Kilted documentation nor sudo apt install ros2-apt-source works for me at the moment.

The generated URL for cURL in the Kilted instructions (https://ftp.osuosl.org/pub/ros/packages.ros.org/ros2/ubuntu/pool/main/r/ros-apt-source/ros2-apt-source_1.0.0~noble_all.deb) points to a non-existing file (The requested URL was not found on this server.) and sudo apt install ros2-apt-source results in

Err:1 http://packages.ros.org/ros2/ubuntu noble/main amd64 ros2-apt-source all 1.0.0~noble
  404  Not Found [IP: 64.50.233.100 80]
E: Failed to fetch http://packages.ros.org/ros2/ubuntu/pool/main/r/ros-apt-source/ros2-apt-source_1.0.0%7enoble_all.deb  404  Not Found [IP: 64.50.233.100 80]

Both try to download ros2-apt-source_1.0.0~noble_all.deb but the pool only has the newer ros2-testing-apt-source_1.1.0~noble_all.deb and the old version was already removed. Isn’t it possible to store multiple package versions in a repo to avoid such issues?

Hosting two versions of the package would be difficult.
Do I understand correctly that you can pull ros packages but not ros2-apt-source?
As for the link we are updating the docs today.

Yes. I only have the http://packages.ros.org/ros2/ubuntu repo (as per jazzy instructions) set up and apt tries to download ros2-apt-source from there. I assume the repo index is outdated:

$ apt-cache policy ros2-apt-source 
ros2-apt-source:
  Installed: (none)
  Candidate: 1.0.0~noble
  Version table:
     1.0.0~noble 500
        500 http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages

It’s possible that there is some CDN propagation delay across OSUOSL mirrors. I just tested it on a fresh docker container and the apt-cache policy returns the new package:

apt-cache policy ros2-apt-source  
ros2-apt-source:
  Installed: (none)
  Candidate: 1.1.0~noble
  Version table:
     1.1.0~noble 500
        500 http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages

I had cleared the apt package cache via sudo rm -rf /var/lib/apt/lists/* and now get the newer version 1.1.0~noble.

1 Like

The Ansible role ctu_vras.ros is ready with the freshest changes to key handling. It handles ROS 1 (at least since Melodic, misusing a little the ros-apt-source for Ubuntu Focal), ROS One and all ROS 2 distros including Rolling.

3 Likes

Where is the appropriate place to report a bug in the documentation? When running the curl command as described in the link above, I found the server was responding with HTTP error code 302 (temporarily relocated). This can be addressed by allowing curl to follow the relocation by adding -L:

curl -L -o /tmp/ros2-apt-source.deb "https://github.com/ros-infrastructure/ros-apt-source/releases/download/${ROS_APT_SOURCE_VERSION}/ros2-apt-source_${ROS_APT_SOURCE_VERSION}.$(. /etc/os-release && echo $VERSION_CODENAME)_all.deb"
1 Like

Where is the appropriate place to report a bug in the documentation?

On the ros2_documentation github repo that generates the documentation. In the top right corner there’s an “Edit on GitHub” button that will take you to the source content in the repo.

This sort of concrete suggestion would be even better as a PR. This looks like a helpful fix.

3 Likes