There is an established workflow for new releases within a ROS distribution. As improvements are made, changes are periodically synced into the main ROS repositories and those changes are announced via Discourse. While we strive to avoid regressions and compatibility breaks in new syncs they can occur, causing disruption in your workflow if it happens in packages you rely on. Even without regressions syncs can affect users who install new packages without upgrading installed packages between syncs. Or your build system may rely on storing and retrieving packages with exact version numbers which become unavailable after being replaced by new versions during a sync.
With the aim of improving overall reproducibility in the ROS community we’ve set up a new apt repository host that will provide persistent hosting for snapshots of each sync for active ROS distros. The snapshots will be taken just after every sync of the ROS stable repository and will be available for at least six months from the sync date.
For teams that want to more closely control when they update to the latest sync, these snapshot repositories will make it easier to access the packaged binaries from earlier syncs.
Snapshot repositories for the following ROS distributions are available now and will remain available for at least 6 months from their sync date.
Snapshots do not receive bug fixes or security updates so it is still a good idea to follow ROS releases and upgrade periodically.
ABI compatibility between snapshots is not tested or guaranteed so when switching a system to a new snapshot updating all ROS packages is recommended.
Should you use the snapshot repositories?
We think most ROS developers should continue to use ROS from the primary repositories.
The snapshot repository is still in preview so waiting for us to stabilize it before using it on production robots is prudent.
But if your build and deploy process involves bundling ROS packages before transferring a comprehensive artifact, like a Docker or OCI container image, then the snapshot repository is worth checking out today. And after you do, reply to this thread with any feedback or to let us know you found it useful.
Awesome work! I’ve seen few folks who have been ticking the OSRF docker image repo for similar snapshotting capabilities. Although the DockerHub’s registry allows you to Pull an image by digest (immutable identifier), we’ve still encountered a few cases that would’ve been better served with these snapshot repositories. Perhaps folks could now use docker registries in courses with the snapshot repositories to archive installed binary images beyond the six months from the sync date.
For those who don’t know, the Offical Library images for ROS are also archived in DockerHub’s registry, the digests of which can be found by navigating the commit history of the Library’s repo-info:
Not sure how far back the archive stretches, but I can still pull ros:indigo-ros-core tag from 2016. Note: the offical library images themself however are still configured to use the primary repositories.
I’m glad to learn about the snapshot repos. A few months ago I was burned by a regression in the main repo and didn’t know where to find the old debs. This looks like a great resource.
Is the reproducilbility of ROS being tracked somewhere? I’ve been working on it some myself and was wondering if there were ways to contribute or learn what other people might be doing.
Because of the regression I ran into I started caching the ROS packages locally using aptly. I noticed @Mike_Purvis published the aptly docker container so I wondered if clearpath is doing something similar.
I also posted this, Rosdep sources list a short while ago because I was concerned about reproducibility when using rosdep Being able to get the old debs if required is a large step, but if you’re using the rosdep db in github, it could shift underneath you even if you have stable deb packages.
I don’t mean to get too off track with that, but the more reproducible the better. Thanks for the announcement!
Not that I’m aware of, but it’s a concern I’ve been pondering and iterating on since I joined Open Robotics. As I’ve had my hands in different elements of the release process it’s given me a pretty deep view (although I’m sure it’s still not complete) of all the ingredients that make up a ROS release and this discussion adds to the motivation to carve out time to document what I think I know.
Snapshots will be available for at least six months from the sync date.
Just a clarification; What about after a distro went EOL, e.g. would Indigo snapshot(s) remain available after it goes EOL this year, or would the snapshot(s) of Indigo be removed 6+ months after EOL?
Either way this is absolutely great. I’ll test this out in a (pre-)production setting at our team (probably combined with container approach @ruffslreferred to for more portability). Thank you!
I’m not making any extended commitments for the snapshot repositories yet so for now let’s assume that final snapshots will expire just like any other. Because we don’t remove EOL distros from the main repositories, once a distro goes EOL it’s final snapshot would be nearly equivalent to the state of the main repositories (except releases of the python utilties: rosdep, rosdistro, bloom, etc). There are two overarching reasons we’re not making an extended commitment out the gate.
I want to leave myself room to iterate, including breaking changes like urls, and the six month lifespan allows me to do that gracefully.
The cost to maintain these snapshots indefinitely, both in storage and bandwidth is not something we can responsibly commit to without having a better idea of what that cost will be. Before releasing this I did some back-of-the-envelope estimates but none of it replaces real-world data. So as we add more snapshots I’ll be looking at the growth rate of our storage consumption and at the bandwidth consumption and that will inform how this service evolves.
The Snapshots repository is now being used to archive unsupported ROS distributions which are about to be removed from packages.ros.org (see Security issue on ROS build farm for details) using a datestamp of final.
In order to allow for future snapshots to include repositories beyond just Ubuntu or Debian repositories, we’ve also moved the apt repositories into an ubuntu subdirectory symlinking to debian when appropriate. The earlier snapshots are available both at the datestamp root and the ubuntu subidrectory but future snapshots will be published only to the ubuntu subdirectory.
Looking into using the snapshots repo for EOL distros docker images, I noticed that there is no debian subdirectory for ROS Lunar/Stretch, but there is one for Kinetic/Jessie.
Will the symlink be created in the future? Or is it recommended to use the ubuntu subdirectory for Debian distros?
Edit: Is there a recommended repo / place for reporting issues with the snapshots repositories?
This was an oversight on my part and has been corrected.
Not at the moment. I think for now questions about the snapshots repository can go on ROS Answers and reporting issues like this is probably best done replying to this thread, which I’m watching.
Someday I would like to get the Chef recipes and shell scripts used to create the snapshots repositories cleaned up, extracted, and published which would create a natural home for issues with the canonical deployment as well. But I don’t have time to do that work on the visible horizon.
@nuclearsandwich Now that you probably have even better insight for the maintenance cost of snapshots, do you have any update for the plan? E.g. Is 6-month still the plan? I see for Kinetic there’s even 2018-12, more than 1.5 years ago, which is great.
Also, if possible, what turned out to be the most costly if you don’t mind talking about? Disk, labor or else?
I found the snapshot very helpful for building our production system. It helps us to remove one big unknown for sure. It’s just 6-month period is a little bit of mental burden as it might be a bit too quick for my team (but from security standpoint it may be practically sufficient. Discussion is active in my team).
Thanks for the report. I’ve investigated the issue and it should now be resolved.
The infrastructure provider for the snapshot repository had to stop the backing host due to problems with the underlying hardware. The host has been migrated to new hardware and is back online. For the time being please report similar issues with the Snapshots repository with a new thread on the Buildfarm category on ROS Discourse.
I’ll post follow-up questions on ROS Answers