Devise a key rotation schedule for the deployment subkeys based on either calendar time or target platform in order to reduce the blast radius of a possible compromise of one of the deployed subkeys.
Update our internal incident response procedures based on employee feedback from our internal debriefing of this incident.
We hope these measures will prevent all future key expiration issues.
So, did the Intrinsic transition break some of these measures? I thought this is still the work of Open Robotics, so I hoped these processes will not be affected.
The same key is used to sign everything in the snapshots repository.
So all snapshots (from ROS 1 or ROS 2 distros, from EOL or current distros) will face the same issue
Since the keys for packages.ros.org work as expected one can remove snapshot.ros.org and use packages.ros.org (see here) as a workaround until the key issue is fixed. This at least fixes my CI build for the time beeing.
e.g. like this:
FROM ros:galactic
RUN rm /etc/apt/sources.list.d/ros2-snapshots.list
RUN apt update && apt install curl
RUN curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg
RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null
Cross-reference: the key has been updated. In short: update the key in your installations and for new installations do nothing (be careful with your cache).
In fact the best place to report these issues is the Infrastructure repository https://github.com/osrf/infrastructure which has issue templates to guide reporters in collecting information that help the Infrastructure team triage and respond to such reports.
In short, no. The repository signing keys are in many ways the most important individual assets that OSRF manages, since they provide the trust for OSRF distributed binaries. Additionally, just about any change to the keys themselves is likely to require manual changes for every configured system in the community that utilizes the ROS or OSRF repositories. These two facts have led to a bit of paralysis on the Infrastructure team’s side as we have striven to make exactly one “perfect” migration. Furthermore, the understanding of hardware-backed keys and managed subkeys implied by that outline also turned out to be quite incomplete when the infrastructure team attempted to actually put it into practice.
The transition did occupy infrastructure staff who may have otherwise continued the key handling work with fewer or milder interruptions but the transition also re-emphasized the need to improve our rotation and handling of repository keys now that infrastructure staff need not be employed or contracted directly by OSRF.
Perhaps ironically, perhaps fatefully, @jrivero and I actually had the first conversations about resuming the work on updating our GPG key rotation practices at the end of last week with an eye toward the Ubuntu 24.04 release next April.
During the prior GPG key expiration incident, we intentionally set the expiration date to survive the full LTS length of Ubuntu 20.04 but we did not look at the ROS Snapshots repository which runs on separate infrastructure. The snapshots repository has always been operated on a best-effort basis rather than considered one of the “core” infrastructure services provided by OSRF.
I’d like to remind people that they generally should not be relying on EOL software remaining available past the support deadline. We all do use EOL software but when you do you need to know that you need to take responsibility to make sure to maintain your own copies and be ready to do whatever is necessary to make sure it stays available yourself if there’s an issue so that you can rely on it.
We set up the snapshots repository as a way to provide some buffer cushion for people who fall behind in their migrations. It had an original target of 6 months. And there has turned out to be a significant benefit in terms of being able to provide reproducibility which is very valuable for research and historical purposes especially convenient for older docker images.
Above in this thread there’s somewhat an expectation of maintaining this access continuously. And I’ve even had people suggest to me that the snapshots should be promoted to being a full production level infrastructure offering from the project. But I would like to strongly push back on scope creep in the expectations of what is provided as a service by the project. We have chosen to provide specific service lifetimes such that we can choose how to allocate our resources. It takes resources to monitor and proactively maintain resources, and the higher the level of production the more effort it takes.
We could choose to allocate more resources to make the hosting of the EOL packages more robust. Although no one is going to say that’s bad in itself, it’s inherently a choice not to spend those resources and energy to improve our current and future versions. In light of this opportunity cost I would like to both acknowledge the team’s work currently keeping some 2 dozen different services operational as well as advocate to not increase the scope of support for the EOL snapshots.
As I was installing ROS Noetic on my Jetson Nano today which had Ubuntu 20.04 based Linux-Tegra, I was struck with installation when I tried to run the command curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | sudo apt-key add - as described in the official documentation, it returned
gpg: no valid OpenPGP data found.
After that when you run sudo apt update you get the following error
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY <pub_key>
I was looking for solutions everywhere and found nothing that worked. I read a similar topic where @marguedas commented
The same key is used to sign everything in the snapshots repository.
So all snapshots (from ROS 1 or ROS 2 distros, from EOL or current distros) will face the same issue
But for the time-being I found a workaround to install ros-noetic successfully. After you encounter the error “The following signatured couldn’t be verified …”, run the command sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys <pub_key>
After this I was able to successfully run sudo apt install ros-noetic-desktop-full
For people who are facing the same issue, you can follow my method because it works for now.
But it would be great if someone can help me resolve the issue if there is any problem on my end and not the key.
Note: I was not behind any university proxy networks or any such restrictive networks. The installation was performed using my personal Mobile Hotspot.
This post is a bit out of topic, but I feel it’s important to stress out the user perspective. I strongly disagree with the following position:
I’m pretty sure that no one is really willing to use EOL software, but ROS2 is simply not yet fully mature (even though most major packages have been ported to ROS2) and may take a couple more years to be ready for widespread adoption.
ROS gained its popularity due to the widespread availability of community-contributed packages for almost everything. Unfortunately, most of them are still not available for ROS2 and might never be. At least not, until there is an easy way to write packages for both ROS1 and ROS2. Let’s not forget that ROS is predominantly utilized in academia, where numerous labs have entire stacks of legacy hardware and software that continue to function effectively for educational purposes or for specific research areas in which each lab specializes. Whichever way you look at it, the two generations of ROS will coexist for a bit while…
While there are communication bridges for sending messages between ROS1 and ROS2 nodes, it would be beneficial if further releases of ROS2 could improve backward compatibility at least by providing wrapper libraries. Learning from the Python community’s struggles with the transition from Python 2 to 3, packages like ‘six’ should be made available. There’s really no reason for renaming things like rospy into rclpy…
Therefore, even after the end of support deadlines, it is important to ensure that software repositories remain accessible. Failing to do so (in combination with the lack of easy tools for porting), it will be hard to prevent the loss of ROS user base.
I find this statement amusing. I guess you are too young to have experienced pre-ROS time, when every research group had to develop their own robot software, and interoperability was non-existent.
The silly thing is that people take this for granted. But there is no such thing as a free lunch.
I agree that there are a lot of open issues, see the other currently active discourse thread on ‘ROS 1 vs ROS 2’. But if everyone keeps expecting ‘the developers’ to solve their issues, it will not happen.
If more people would be actively porting, facing specific issues and discussing those on RSE and here, then maybe we’d finally get to a point where best practices and solutions get more widely known. And more people would stop hiding behind excuses and start porting.
As i have written in the other thread: documentation is not in anyone’s things-to-get-my-work-done path. It will not auto-emerge.
In any case: if I’d have to choose between assigning resources to maintaining ROS 1 vs to documenting ROS 2 functionality and best-practices, I’d know what to choose.
