Proposed changes to the ROS releases

General
1

Based on the results of our recent survey (ROS1 2018 Version Survey: The results are in), as well as the download statistics that were posted last year (http://download.ros.org/downloads/metrics/metrics-report-2017-07.pdf), Open Robotics is proposing some changes to the ROS release schedule that should reduce the support burden on maintainers and on Open Robotics itself. The proposals below are roughly ordered from having the least impact to the most impact:

Continue releasing a ROS LTS every 2 years

  • Conclusion: There is high use and demand for the ROS LTS releases.
  • Proposal: Keep releasing the ROS LTS on a 2 year schedule, as is currently done.

Stop targeting non-LTS Ubuntu distributions with ROS LTS

  • Conclusion: There is very low adoption of the non-LTS Ubuntu distributions by users. The non-LTS Ubuntu versions are only supported for a short time after the ROS LTS release.
  • Proposal: Stop targeting non-LTS Ubuntu distributions with ROS LTS releases.
  • Consequences:
    • No ROS LTS binaries for non-LTS Ubuntu releases.
    • No need to support non-LTS Ubuntu releases for packages in ROS LTS.
    • Instead of using non-LTS Ubuntu releases users can use the Ubuntu enablement stacks (HWE), e.g. 16.04.4.
    • May be more difficult for users to build from source on non-LTS Ubuntu distributions since it is not tested in any automated way.

Only support one version of Debian with one ROS LTS

  • Conclusion: Having a Debian release span two ROS LTS releases prevents us from using newer system dependencies.
  • Proposal: Only support one version of Debian with one ROS LTS.
  • Consequences:
    • A Debian release would only be associated with the first ROS LTS released while the Debian release is also available.
    • Some ROS LTS’s may have no Debian version associated with it.

Replace the non-LTS ROS release with a rolling ROS distro

  • Conclusion: There is very light usage of the ROS non-LTS releases.
  • Proposal: Only release ROS LTS’s, and create a rolling ROS distro based on the last ROS LTS to exist between ROS LTS’s. The goal is to reduce the workload on both maintainers and Open Robotics by removing a distribution which has only a few users. To compensate for the large gap between releases, the rolling ROS distro will allow maintainers to depend on newer system dependencies and get advanced notice if the next Ubuntu release will have API breaking changes that packages need to deal with.
    • As a starting point the rolling ROS distro would take package releases from the latest ROS LTS.
    • Maintainers could optionally opt-in to specifying newer versions of their packages (newer than the version in the LTS) for the rolling ROS distro by providing additional information.
    • The buildfarm (http://build.ros.org) will periodically attempt to build packages in the rolling ROS distro only on the latest released version of Ubuntu (in the first 6 month the Ubuntu LTS, later only the latest non-LTS Ubuntu), notifying maintainers on failure.
    • (optional) The result of these periodic jobs could be made available as a monolithic archives, e.g. a .zip or .tar.gz file, which would contain all the packages which built successfully. This avoids making users build all packages from source.
  • Consequences:
    • No future ROS non-LTS releases will be created.
    • Users that want to use the latest code must build from source to do so, unless we elect to make “fat archives” of the periodic builds available.
    • The buildfarm would not produce any Debian packages for the rolling ROS distro (in order to not require running bloom for future Ubuntu distros).

Please let us know what you think about each of these proposals.

Your Friendly ROS team

Options discussed for non-LTS ROS releases

While discussing the option of dropping the non-LTS ROS releases, we came up with a number of different things we could do. The following is a complete list of the options we discussed, with some details of the pros and cons we see on each approach.

  • Have a rolling ROS distro targeting the latest Ubuntu release (this is the option we recommend).

    • Pros
      • Ability to automatically detect changes in Ubuntu dependencies that may break ROS packages when updating to a newer Ubuntu release.
      • (optional) Zip archive allows people to use binaries.
      • Packages can use the new functionality from Ubuntu dependencies.
    • Cons
      • Could make ROS LTS less stable - maintainers may push more stuff into ROS LTS to get binary packages.
      • Few users want to use non-LTS Ubuntu releases.

  • Have a rolling ROS distro, targeting the latest Ubuntu LTS only.

    • Pros
      • The base Ubuntu platform is used by many users.
      • Ability to automatically detect changes in ROS dependencies that may break ROS packages.
      • Maintainers can release an unstable version of their packages on a base Ubuntu platform used by many users.
      • Easier for users to test the rolling ROS distro since it is the latest Ubuntu LTS.
    • Cons
      • No ability to automatically detect changes in Ubuntu dependencies that may break ROS packages when updating to a newer Ubuntu release.
      • Users or packages can’t easily use new functionality from new Ubuntu releases.

  • Have a rolling ROS distro, targeting both the latest Ubuntu non-LTS and the latest Ubuntu LTS.

    • Pros
      • Packages tested against stable Ubuntu dependencies and against newest Ubuntu dependencies.
    • Cons
      • If users want to use newer versions of dependencies, their code needs to build for both the current Ubuntu LTS and the latest Ubuntu.
      • Might be painful for maintainers to maintain code handling all Ubuntu versions as specified above.

  • Do nothing, keep the ROS non-LTS releases as is currently being done.

    • Pros
      • Keeps the status quo.
    • Cons
      • Maintainers have to release/re-bloom their packages every year for a small userbase.
      • Some maintainers do not release into the ROS non-LTS releases, so those releases are always less complete.
      • Open Robotics needs to do work every year for very small userbase.

  • Just do a ROS LTS release every two years, with no additional infrastructure provided in-between.

    • Pros
      • Package maintainers and Open Robotics only have to do work every 2 years.
    • Cons
      • Large Ubuntu platform delta between ROS releases, possibly leading to lots of API breaks due to Ubuntu platform dependencies changing.
      • A full tick-tock cycle of API deprecation/removal would take 4 years (2 full ROS releases).
6 Likes
2

Overall I think the new plan sounds great.

I do think, though, that the full potential of the rolling release isn’t being realised. It should be seen by package maintainers as a place to shake down their packages before they go into an LTS release. While at the start of a cycle this would probably involve lots of new features being introduced and APIs broken, over time the package would stabilise and bugs would be wrung out (while still providing room for new features if desired). The reason I think this is a benefit is that it gives users of the LTS release more confidence that the software has been well used before release rather than just being the head of the repository dropped in before the release goes out. If the wringing out happens before release then a company has to do less wringing out of their own before using the release in a product.

To do this, I think that the rolling release plan needs two changes:

  1. Clearly defined guidelines for the purpose of the rolling release (providing new features while aiming to stabilise the package for the next LTS), for feature freezes (e.g. from six months prior to the planned LTS release, only patch version bumps are allowed for packages), and for allowing a new package or package version into the rolling release (it must compile and its dependencies must be available, for a start).

  2. Regularly automatically built binaries available from a repository. Otherwise no one will use it and it won’t get the testing it needs.

For ROS it would also require more trust in package maintainers to obey the rules, since ROS is more distributed than something like Debian.

The ability to have the above and the latest and greatest and breakiest (while still being in a release and so finding incompatibilities) is why Debian has two rolling releases, one that is intended to become the next stable and one that will probably step on your cat. I think if ROS were to add an unstable release, it wouldn’t need binaries built automatically.

I’m aware that many people don’t consider this approach compatible with ROS, but I think it is and that it’s worth pursuing, even with the extra infrastructure development it would require.

2 Likes
3

I think the new plan sounds good too.

Also, on the rolling release approach, I’d like to point that there is an effort on having ROS on Gentoo, which is a rolling distro. There we find already things breaking by using the very latest. And it’s not that bad! @allenh1

4

Overall, I think the new plan is a step forward in the right direction.
One of the things that concerns is me is by removing non-LTS Ubuntu distributions and limiting support to only one Debian version, how much automated testing are you removing in the process? What are the downstream effects of this? How does it affect downstream projects like meta-ros?
One of the hardest things to do is support and encourage a large and diverse user base. Would it make sense to have a required support/testing for the Ubuntu and Debian LTS but then have recommend support/testing for non-LTS, RHEL/Fedora, Gentoo and perhaps even Yocto/open-embedded?
In the context of the non-LTS release, can it be replaced by a prolonged beta system of the next LTS release? Or perhaps something similar to the Debian “testing” branch (or the Debian testing and unstable branches as @gbiggs described). This could serve as way for people to get the cutting edge while reducing the work to maintain.
Also having binaries is a huge benefit of the non-LTS release so please do not abandon it.

5

How’d the fat binary archives get system/pip dependencies resolved? Would they contain one loooong apt install command? Or by running rosdep?

6

And how will deprecations work with the rolling release strategy?

7

This all sounds good to me, but I definitely hear the concerns raised above with respect to non-binary releases. Fortunately, I have some ideas to solve the lack of binary packaging problem.

@awesomebytes and I have been working on stabilizing the ROS Gentoo prefix installation, which, in case of the new release type, seems like a good solution. The Gentoo prefix can install itself in any existing linux system, and it can be done outside of a root environment (though not all Gentoo packages support this method, @awesomebytes has been working hard to fix the incompatible packages that are relevant to ROS).

One can actually create binary packages with Gentoo for installation. Moreover, it’s possible to host a server for your binary packages built by portage (see here).

So, with a Gentoo prefix installation, we can have a community based server to build packages for your favorite robot, then host them on a community server.

The other advantage of this rolling release is that one can freeze a copy of the ROS-Overlay (the Gentoo install’s repository) by forking it, or even creating your own repo using Superflore. Superflore pulls changes from rosdistro and then creates the installers for a given ros release. This is a great use case for Superflore, actually, as it could actually create these binary packages (if we write the generator script for it) and push the packages to the package server.

8

Hi, it seems like the right discussion to expose something that has always been on my mind. Not about the rolling release schedule, but how these releases are developed and maintained using git. I’m not super expert on the matter, and not seeing anyone say anything about it makes me think I might be wrong or not seeing the obvious. But hey, you never know, so here are is my concern.

I’ve always found the ROS branching-per-devel, and accepting development on those branches, philosophy kind of poor regarding guarantee new features and bug fixes in new releases. Let me explain why.

I myself (remember, not being a super expert), found a missing feature in releases ahead that was available in previous releases. Here is the issue: https://github.com/ros/ros_comm/issues/575 If you follow the issue, there was timing issue between accepting a PR and creating a branch for a new release. In fact, you can see Dirk Thomas in the end adding commits manually in order to have the feature in subsequent releases. Which I don’t know for you, but to me it seems kind of awkward, right? I mean, having to cherry-pick or merge features into future releases does not sound good to me in general (there might be rare situations though). Luckily it was a not-critical thing, but what if it were? Whence my concern.

So I wonder: Why most of ROS development don’t use a unique branch, say master or unstable or default, where you mostly accept PRs for new features and bug fixes, and then tag releases from there with any of the suggested rolling schedules proposed above, (or even branch from there, if you like a release-per-branch style to cherry-pick compatible bug fixes and features into previous releases)? This way, you guarantee all new features and bug fixes will be available in every new release since they are in a single development branch. At the same time, this is a bit of a push for people to update to new releases in order to get an improved and cleaner version of the software, as well as reducing efforts for maintainers in keeping track of new code being added for new releases IMHO.

Anyone sharing the same concern or with any reason why this is not a good practice?

1 Like
9

Depending on the requirements for your packages your suggestion has the problem of containing “contrary” goals:

  • have new features available in every
  • aim for stability since every code change has the potential to introduce regressions

The balance between these two is the same in any software development. If you think about Linux distributions it is the same there: released distros don’t get new features but only important bug fixes. The same rational applies to ros_comm: stability is very important and “weighted higher” than the availability of the latest feature.

For more “leaf” packages which are not that heavily relied on the weighting would be different. And I think that is what we see also in the branching style of ROS package. Packages further on the “lower” level (in the dependency hierarchy) and heavily used often use distro specific branches and more “higher” level packages often use a single master / development branch spanning multiple ROS distros.

10

Thanks for your reply. I understand the point of the requirements of each package.

Precisely my point, why do you need to put efforts in regression of those changes? Unless you want to be stable in all distros, which can also be potentially hard to achieve anyway, right?

Perhaps this scheme clarifies both my concern and my suggestion:

Currently in most of ROS repositories (comm and leaves):

           C5 <- C6 <- C7 <- [melodic]
          / 
  C3 <- C4 <- [lunar]
 /
C1 <- C2 <- C3 <- [kinetic]

For some reason, the bug fixed C3 was accepted in kinetic, but then cherry-picked/merged into lunar manually, that is, into a future release. So melodic and lunar will have a bug fixed that was manually added. But, melodic will have it only if the branching took place after C3 on lunar, otherwise it needs to be added manually too.

My suggestion would be (release-per-branch or tags, but keeping a single line of development):

  C3 <- C6 <- [kinetic]    [lunar]     [melodic]
 /                        /           /
C1 <- C2 <- C3 <- C4 <- C5 <- C6 <- C7 <- [master]

Where C3 is still a bug fixed, and add C6 if you want, merged into a previous release, namely kinetic. These fixes do not need to be manually added to either lunar and/or melodic, since they will have them by construction.

Correct me if I’m wrong, but wouldn’t the later makes it easier and safer to maintain for both comm and leaf packages? And still comply with both goals, have new features for new releases and still able to deal with regression stability?

Side note: Just interested in your points of view as a consumer of comm packages and developer of leaf ones.

11

I don’t see this often tbh. It seems a bit forced.

MoveIt is one example of the sort of setup I’ve seen more: PRs target the newest supported ROS release and then when it makes sense, those fixes get backported.

12

That is certainly not the normal flow. I am not sure of that is a real-world example (can you provide a link to a PR?) or just illustrating.

I can’t speak for all ROS repos but in ros_comm any PRs are only merged into the latest devel branch. The only very rare exception is if a patch is only necessary for an older distro but not the latest one.

And then in a second step all changes on the latest devel branch are being considered for backport to the previous devel branch (and then from there to the second previous etc.). See changes between 1.12.12 and 1.13.6 for backporting by dirk-thomas · Pull Request #1323 · ros/ros_comm · GitHub for an example of such a backport.

So as far as I understand you proposal that is exactly what is happening in ros_comm with the only exception that the “single line of development” is not named master but after the latest ROS distro.

I would suggest to continue this specific conversation in a separate topic / thread since it gets a little bit away form the scope of the original topic.

13

I don’t see this often tbh. It seems a bit forced.

Not seeing it often does not mean it can not happen, but certainly accountable.

(can you provide a link to a PR?) or just illustrating.

I already gave you one in my first post above. Not exactly a PR, but an issue finding commits in previous releases not available in future releases. That was the one that triggered me to question the current branching strategy. So just wanted to know your reasoning behind it.

So as far as I understand you proposal that is exactly what is happening in ros_comm with the only exception that the “single line of development” is not named master but after the latest ROS distro.

Not exactly the same thing then, since a branch is created every release cycle to continue the development.

I would suggest to continue this specific conversation in a separate topic / thread since it gets a little bit away form the scope of the original topic.

Sure, I’ll create it.

14

I know I’m a bit late to the party. In principle the idea of LTS release + rolling release makes a lot of sense to me. However, I have some questions about the proposed release process, test coverage and availability of debian packages. (You mentioned the fat tar of compiled packages for the latest Ubuntu version, but I wounder how much use that would actually be.)

The justification sounds like a technical limitation of the tooling, rather than a desired limitation in the process. I know too little of the technical details of bloom to judge how and with how much effort it could be realized, but would it not be possible to update bloom to directly support this, meaning that a release in the rolling ROS distro would automatically be “transferred” / “made available” to the latest Ubuntu version, once that is added, such that the build farm could build binaries?

Releasing a package to the rolling distro would probably mean blooming it. How exactly would packages be transferred from the rolling release to the new LTS and the rolling release after that? I.e. would maintainers every two years take the latest version of the package in the rolling release and explicitly release it for the LTS, or would that be automatic based on the current rolling release? Would the new rolling release be based on the now newly created LTS release (after some specific time?) or would the rolling release simply continue like before the LTS? In the latter case, what happens to old / abandoned packages? They would continue to be the rolling release even if they are no longer working / supported?

As mentioned, most users are using Ubuntu LTS. In order to ensure test coverage of the rolling release, would it not make sense to support the last Ubuntu LTS at least for some time (> 6 months)? I guess the issue is that at some point packages once Ubuntu LTS+1 is out, some packages might want to switch to require newer system dependencies. Would it make sense to at least have the guideline that the core packages should try to continue to work on the latest LTS as long as possible, maybe for 12 or 18 months, before braking ties and switching to newer dependencies in preparation for the new Ubuntu / ROS LTS? If there was any way to make binary packages available at least for the compatible subset of packages for the rolling ROS release on the current Ubuntu LTS for as long as possible, that would probably help to increase test coverage as well. However, I understand that this might add quite a bit of complexity to processes and tools.

15

I’m a little concerned about this, too. I would prefer tools be made to fit the process rather than the other way around.

I think that these sorts of issues are why there need to be explicitly defined rules for how packages move between the rolling release and an LTS release, similar to how Debian has rules on things like how long a package must go without any critical bug reports to move into testing. I agree that this is harder to achieve in ROS due to the more distributed nature, but I think that the value in having well-defined criteria for allowing a package into the LTS makes it worth the effort.

16

First, apologies for being so quiet on this after the initial post; I was busy getting Melodic out. Now that that is done, I’m going to try and address some of the comments all at once here. Feel free to follow up.

As you mention, this is closer to the Debian model, and has similar benefits along the testing/unstable/stable axis. I do like the ability to get some additional testing on packages before they go out, though I will point out that ros-shadow-fixed does provide that capability today. I will also point out that even if we don’t decide to do that right away, it seems like a natural extension later on.

We can certainly have some guidelines, that makes sense (this proposal wasn’t meant to be all of the details, just to get the initial conversation started and gauge community interest). However, as you point out in several places, we don’t really have mechanisms to enforce any of the guidelines, we just have to trust the maintainers to do the correct thing.

It is kind of tautological, but the removal of packages on non-LTS Ubuntu distributions removes testing on Ubuntu non-LTS, and doesn’t really have much other impact on automated testing. The rolling release is meant to fill in that gap. As for how it affects things like meta-ros, I don’t really know.

We could go in that direction, but honestly, given the numbers we’ve seen, there is just no reason to support the Ubuntu non-LTS versions. So few people use them that the amount of testing we get there is negligible.

Overall, I’m not usually a huge fan of prolonged testing cycles. My experience has been that you get a flurry of activity right at the beginning or right at the end (sometimes both), but the middle sees little use and thus you get little value out of it. Since maintainers have the ability to update their packages after a release, I’d rather see the release out there and then make improvements to it along the way.

Good question. I think either could work, though if we are making users download a tarball and extract it anyway, I would lean towards using rosdep for this.

There are a few different things we could do with deprecations. If a package is marked as deprecated/orphaned, we could immediately remove it from the rolling release, which would be a clear signal to downstream consumers that it won’t be available in the next LTS. Or we could let it continue to build in the rolling distro until it starts failing, and only remove it then. I’m generally in favor of failing loud and early, so I’d lean towards the former, but we haven’t come up with any of those details yet.

I also don’t know all of the details here, so I can’t give a direct answer. I agree that this seems like a limitation that could be overcome, but I don’t know how much work that would be.

The intent is that the packages will be taken from the last LTS into the new rolling distribution immediately after the LTS is released. See my earlier answer about abandoned packages; there are a few different things we could do here.

The initial proposal does call for using the latest Ubuntu LTS for at least 6 months (basically, until the next Ubuntu non-LTS is released). The main driver for the rolling distribution is to give maintainers advanced notice of platform changes that are coming, so we would want to switch to the non-LTS as soon as it was available. The last ROS LTS is still available, and can still get package updates, so users who need stability can still use that. If we did something more like @gbiggs testing/unstable/stable idea, that could potentially bridge that gap a bit more, though it is more work both to develop the infrastructure and for the maintainers long-term. It is something we will have to consider.

17

I think that you can, in that you can control what gets accepted into the release. But it would be more effort on your part to verify that maintainers are actually meeting their obligations. I think that Debian’s use of its own bug tracker for all its packages is an aide to this for them. They can easily check how many open tickets a package has and what type they are. It’s possible that some of the work coming out of the current quality assurance effort might provide something that can be used here. For example, the CI badge functionality can be used to automatically check that a package has CI, and is passing that CI, before it gets accepted into the LTS release.

But as you say, ultimately with the current nature of the ROS community, trusting the maintainers is necessary.

I don’t think the purpose of Debian’s testing branch existing for two years before it becomes stable is to have two years of testing a single version of the package. It provides enough time for users to work with software that isn’t blowing up but provides up-to-date features with further updates possible and gives them time to update their own stuff to match.

Maintainers upating their packages in an LTS release should have very very strict requirements on what is accepted (bug fixes only being an obvious one) to ensure that the LTS remains the same stable target it is at the start.

I think that this is in conflict with the goal of the LTS release being a stable platform on which products can be built. I think it’s also part of why Debian has separate unstable and testing rolling releases: testing is considered usable for non-critical applications (I know many people who use testing for their daily-use computer, but only one probably insane person who uses unstable for daily use) so it gets the package out there as you want, without making the stable release risky.

I strongly agree with failing early. Users of the next release need as much advance notice as possible that something they depend on is going to disappear so they can plan and enact changes in their own stuff.

I would prefer the opposite approach: the rolling release should be on-going forever, and the LTS release should be a snapshot of it at a point in time. I do not see a benefit to restarting the rolling release every time a new LTS release is made. Is the intention here that testing for the LTS release is done separately from the rolling release and the rolling release becomes a new-features branch of the LTS release?

I think that much of this discussion is being driven by an unstated assumption about what the LTS release is intended to be. I would like the OSRF to make that assumption clear so we know what the goal is. Is the LTS release intended to be a stable platform for products to be built on and for use in critical applications, or is it a snapshot of package versions at a point in time that may be updated as package maintainers feel like it?

Put another way, does the OSRF see its position as like the Linux kernel developers, who release new kernels regularly but do not pay much attention to the testing necessary to ensure usability in critical applications and leave the production of releases for critical applications to others (the Debians and Red Hats), or does the OSRF see itself as like the Debian Foundation, providing a well-tested base on which applications can be built?

I’m not criticising either position because both are perfectly valid, it just needs to be clear. It will impact the release process requirements, how many resources the OSRF needs to invest, and what companies would then expect from a release provided by the OSRF. I hope to one day see a Red Hat emerge for ROS so if the OSRF does not want to be doing this work I do not think it is a disaster for ROS.

2 Likes