Deployment approaches for new gpg key

Hi Everyone,

We’ve been using the ROS gpg key for many years now and unfortunately it’s now considered obsolescent due to advancements in encryption technology.

At OSRF we use the ROS key to sign the debian repositories. This is the key that you need to import before you can apt-get install ROS packages. We are planning to generate a new more modern key following the new standards.

There are several different ways to deploy the key however. The two methods that we have come down to are as follows.

  1. Replace the existing key and resign the repos as they stand w/o changing anything.
  2. Create a new repository next to the existing repository for new packages (Kinetic and newer) and sign the new repository with the new key.

There are tradeoffs to both. Here are some of the highlights as follows.

Approach 1 Tradoffs

  • The existing repositories will be more secure.
  • All existing users will be required to add the new key on any existing system.
  • New users will simply use the new key when installing.

Approach 2 Tradeoffs

  • The existing repositories for jade and older will be unchanged.
  • Existing users will not have to do anything.
  • The security of the old repository will not be improved.
  • New users will follow slightly different installation instructions for kinetic. (new source path and gpg key)
  • Kinetic beta testers will need to update their apt sources and install the new key.
  • We would take this opportunity to follow a more standard repository directory structure and naming conventions.

Request for feedback

Overall 1 is more secure, but more disruptive. Option 2 will change things going forward in a non-disruptive way but the cost of that lack of disruption is to not make installation on existing systems more secure.

We’d like to know your thoughts on whether we should use approach 1 or 2.

Resources

If you’d like to know more about the underlying issue here are some links:

I have been installing ROS, usually on new VMs, for years, Have I been
using the ROS gpg key all along without knowing it?

Yup: http://wiki.ros.org/jade/Installation/Ubuntu#jade.2BAC8-Installation.2BAC8-Sources.Set_up_your_keys

In Approach 1, can’t the new key be signed by the old key to minimize disruption?

The user already trusts the old key, so a newer key signed by the old one should also be trusted.
New installs should use the new key, and get the security benefits. Installs not migrated to the new key will be more vulnerable, but should continue to work.

Apache GPG Key Transition Page: http://www.apache.org/dev/key-transition.html.

1 Like

@rohbotics That would be a great solution. I’ll see if I can demonstrate apt supporting that approach.

So after some testing signing the new key with the old key has a problem that the user does not have the new key signed by the old key in their keyring. As such they cannot verify the repo without downloading the new key.

Signing the new key means that they will implicitly trust the new key but they cannot trust the new key without knowing about the new key and that it’s signed by the old key.

Unfortunately the overhead of downloading the new key is basically the same whether or not it’s signed.

To minimize disruption of existing install base I’m going to suggest approach #2.

I must admit I don’t understand this completely, but perhaps the following note from debian/wiki/Teams/Apt/Sha1Removal applies (under “Fixing half-broken repositories”)?

Migrating from DSA to RSA is best done by signing the repository with
two keys (old and new one) and shipping the new one to the users. A
relatively safe way to ship the key would be to embed it in the package.
Some months after those changes, it is OK to drop the old key from the
repository and the users machines (if shipped with a package).

Would a dual-signing approach be a viable approach in our case? New keys can be delivered to systems using a debian package, I’ve seen this done a number of times. Dual-sign the repository for a while, distribute the new key as a package and after a while switch everything over to the new key.

If dual signing + shipping a package with the new key works, that would be great.

@tfoote
The only thing about using option 2 is that Debian wants to kill off SHA1 by January 1, 2017. (https://wiki.debian.org/Teams/Apt/Sha1Removal)
This will probably be distributed as a security update that could make it’s way into Ubuntu 14.04, meaning indigo and jade repositories could before their EOLs.

@gavanderhoorn That’s a great completely undocumented feature that we can dual sign the repositories. Every single reference to the configuration file element SignWith defaults to singular key. I’ve validated it on a test repository. We’ll just need to setup our systems to propogate both finger prints.

@rohbotics We can definitely think about potentially bundling the new gpg key into a package to be distributed. It’s not clear which package we should put it into. We’ve thought about having a configuration package which sets up your sources for you as an installer which this might bundle with nicely.

However I think that we may just want to maintain the old key for as long as older platforms are supported. And once the other platforms are EOL we can just drop the old key.

If you do not have the new key it just gives a warning like below since the old key is still valid it passes.

W: There is no public key available for the following key IDs:
2DEE5E6E4BB3D3AE

If you have the new key only it will give a similar warning with the old key, but also work.

In the intermediate term if you want to suppress the warnings we a user can install both keys.

We’ll have to do more investigation to figure out how to fully deprecate the old key gracefully.

I would propose to ship the key in a package like debian-archive-keyring.
See this mail as well: https://lists.debian.org/debian-devel/2016/05/msg00287.html

Yes, that is what I suggested as well.

Great news, the key does not need to be changed!

After many rounds of debugging and testing I determined that it’s actually the gpg settings on the server which need to be updated not the key itself. And the new gpg settings are backwards compatible too. So with these changes on the server, there are no installation instructions changes, new repositories, or key changes. And the higher security will hold for all the older distros.

There were lots of github issues opened in the process, but this is the one to implement the change repeatably: https://github.com/ros-infrastructure/buildfarm_deployment/issues/130

Great news on not changing the keys.

Reading your OP, a tangential question came up for me.

We would take this opportunity to follow a more standard repository directory structure and naming conventions.

Can you briefly describe what that would look like, or do you have a resource on best-practices that you planned to follow here? I am currently setting up internal APT repos for our company. For now it would be only custom packages + some ROS packages on top of indigo for Ubuntu 14.04, but I would like to have a future-proof layout for more to come down the road. What is wrong with packages.ros.org/ros/ubuntu/dists/<dist-name>/main?

Looking at the debian and ubuntu repos. The standard urls are

Canonical urls:
http://{mirror.}archive.ubuntu.com/{os_name}/

Debian urls:
http://<mirror.>.debian.org/debian

Launchpad PPAs:
http://ppa.launchpad.net/{username}/{repo_nam}/ubuntu

Ubuntu Old Releases:
http://old-releases.ubuntu.com/ubuntu/

CentOS:
http://{mirror.}centos.org/centos/{version}/{reponame}/{arch}

Fedora:
http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/

The biggest non-standard name we use is ros-shadow-fixed. And we have the inverted higherarchy with the named repository before the platform. As we consider adding fedora etc. It makes more sense to put the os name at the top level instead of intermingling the different repositories. Such that we could have top level directories ubuntu, debian, fedora, …

The other practice that we would like to pick up from Canonical is to move the EOL’d distros to a different server both to differentiate the support level as well as differentiate the bandwidth requirements. It will also make it easier to host mirrors as they won’t continuously grow in size as new distros are added and the EOL’d distros stick around and require more mirrored disk space. Especially as we’ve gotten much better at packaging and our new packages are much smaller than our earliest releases despite newer releases having many more packages.

Thanks for the reply. What you say about EOL’d distros makes sense. And what I take away from the hierarchy point is that I should consider http://myserver/ubuntu/ros/ over http://myserver/ros/ubuntu/.