Proposed changes to the ROS releases

[clalancette]

First, apologies for being so quiet on this after the initial post; I was busy getting Melodic out. Now that that is done, I’m going to try and address some of the comments all at once here. Feel free to follow up.

@gbiggs

Overall I think the new plan sounds great.

I do think, though, that the full potential of the rolling release isn’t being realised. It should be seen by package maintainers as a place to shake down their packages before they go into an LTS release. While at the start of a cycle this would probably involve lots of new features being introduced and APIs broken, over time the package would stabilise and bugs would be wrung out (while still providing room for new features if desired). The reason I think this is a benefit is that it gives users of the LTS release more confidence that the software has been well used before release rather than just being the head of the repository dropped in before the release goes out. If the wringing out happens before release then a company has to do less wringing out of their own before using the release in a product.

As you mention, this is closer to the Debian model, and has similar benefits along the testing/unstable/stable axis. I do like the ability to get some additional testing on packages before they go out, though I will point out that ros-shadow-fixed does provide that capability today. I will also point out that even if we don’t decide to do that right away, it seems like a natural extension later on.

@gbiggs

To do this, I think that the rolling release plan needs two changes:

1. Clearly defined guidelines for the purpose of the rolling release (providing new features while aiming to stabilise the package for the next LTS), for feature freezes (e.g. from six months prior to the planned LTS release, only patch version bumps are allowed for packages), and for allowing a new package or package version into the rolling release (it must compile and its dependencies must be available, for a start).

2. Regularly automatically built binaries available from a repository. Otherwise no one will use it and it won’t get the testing it needs.

For ROS it would also require more trust in package maintainers to obey the rules, since ROS is more distributed than something like Debian.

We can certainly have some guidelines, that makes sense (this proposal wasn’t meant to be all of the details, just to get the initial conversation started and gauge community interest). However, as you point out in several places, we don’t really have mechanisms to enforce any of the guidelines, we just have to trust the maintainers to do the correct thing.

@qnetjoe

Overall, I think the new plan is a step forward in the right direction.
One of the things that concerns is me is by removing non-LTS Ubuntu distributions and limiting support to only one Debian version, how much automated testing are you removing in the process? What are the downstream effects of this? How does it affect downstream projects like meta-ros?

It is kind of tautological, but the removal of packages on non-LTS Ubuntu distributions removes testing on Ubuntu non-LTS, and doesn’t really have much other impact on automated testing. The rolling release is meant to fill in that gap. As for how it affects things like meta-ros, I don’t really know.

@qnetjoe

One of the hardest things to do is support and encourage a large and diverse user base. Would it make sense to have a required support/testing for the Ubuntu and Debian LTS but then have recommend support/testing for non-LTS, RHEL/Fedora, Gentoo and perhaps even Yocto/open-embedded?

We could go in that direction, but honestly, given the numbers we’ve seen, there is just no reason to support the Ubuntu non-LTS versions. So few people use them that the amount of testing we get there is negligible.

@qnetjoe

In the context of the non-LTS release, can it be replaced by a prolonged beta system of the next LTS release? Or perhaps something similar to the Debian “testing” branch (or the Debian testing and unstable branches as @gbiggs described). This could serve as way for people to get the cutting edge while reducing the work to maintain.
Also having binaries is a huge benefit of the non-LTS release so please do not abandon it.

Overall, I’m not usually a huge fan of prolonged testing cycles. My experience has been that you get a flurry of activity right at the beginning or right at the end (sometimes both), but the middle sees little use and thus you get little value out of it. Since maintainers have the ability to update their packages after a release, I’d rather see the release out there and then make improvements to it along the way.

@peci1

How’d the fat binary archives get system/pip dependencies resolved? Would they contain one loooong apt install command? Or by running rosdep?

Good question. I think either could work, though if we are making users download a tarball and extract it anyway, I would lean towards using rosdep for this.

@kyrofa

And how will deprecations work with the rolling release strategy?

There are a few different things we could do with deprecations. If a package is marked as deprecated/orphaned, we could immediately remove it from the rolling release, which would be a clear signal to downstream consumers that it won’t be available in the next LTS. Or we could let it continue to build in the rolling distro until it starts failing, and only remove it then. I’m generally in favor of failing loud and early, so I’d lean towards the former, but we haven’t come up with any of those details yet.

@NikolausDemmel

• • The buildfarm would not produce any Debian packages for the rolling ROS distro (in order to not require running bloom for future Ubuntu distros).

The justification sounds like a technical limitation of the tooling, rather than a desired limitation in the process. I know too little of the technical details of bloom to judge how and with how much effort it could be realized, but would it not be possible to update bloom to directly support this, meaning that a release in the rolling ROS distro would automatically be “transferred” / “made available” to the latest Ubuntu version, once that is added, such that the build farm could build binaries?

I also don’t know all of the details here, so I can’t give a direct answer. I agree that this seems like a limitation that could be overcome, but I don’t know how much work that would be.

@NikolausDemmel

Releasing a package to the rolling distro would probably mean blooming it. How exactly would packages be transferred from the rolling release to the new LTS and the rolling release after that? I.e. would maintainers every two years take the latest version of the package in the rolling release and explicitly release it for the LTS, or would that be automatic based on the current rolling release? Would the new rolling release be based on the now newly created LTS release (after some specific time?) or would the rolling release simply continue like before the LTS? In the latter case, what happens to old / abandoned packages? They would continue to be the rolling release even if they are no longer working / supported?

The intent is that the packages will be taken from the last LTS into the new rolling distribution immediately after the LTS is released. See my earlier answer about abandoned packages; there are a few different things we could do here.

@NikolausDemmel

As mentioned, most users are using Ubuntu LTS. In order to ensure test coverage of the rolling release, would it not make sense to support the last Ubuntu LTS at least for some time (> 6 months)? I guess the issue is that at some point packages once Ubuntu LTS+1 is out, some packages might want to switch to require newer system dependencies. Would it make sense to at least have the guideline that the core packages should try to continue to work on the latest LTS as long as possible, maybe for 12 or 18 months, before braking ties and switching to newer dependencies in preparation for the new Ubuntu / ROS LTS? If there was any way to make binary packages available at least for the compatible subset of packages for the rolling ROS release on the current Ubuntu LTS for as long as possible, that would probably help to increase test coverage as well. However, I understand that this might add quite a bit of complexity to processes and tools.

The initial proposal does call for using the latest Ubuntu LTS for at least 6 months (basically, until the next Ubuntu non-LTS is released). The main driver for the rolling distribution is to give maintainers advanced notice of platform changes that are coming, so we would want to switch to the non-LTS as soon as it was available. The last ROS LTS is still available, and can still get package updates, so users who need stability can still use that. If we did something more like @gbiggs testing/unstable/stable idea, that could potentially bridge that gap a bit more, though it is more work both to develop the infrastructure and for the maintainers long-term. It is something we will have to consider.