| ROS Resources: ROS Homepage | Media and Trademarks | Documentation | ROS Index | How to Get Help | Q&A Help Site | Discussion Forum | Service Status |
Hello Noetic maintianers!
Starting now, I will be holding new Noetic ros/rosdistro release PRs with a plan to sync ROS Noetic packages to the main apt repo on 2020/06/05. Please comment here if there are any issues I should know about before performing the sync.
There are currently no regressions and 184 packages waiting to sync.
In the future, would it be possible for us to know in advance when the hold starts (If I missed an announcement I’m sorry)?
The sync is out!
In the future, would it be possible for us to know in advance when the hold starts
I can think about doing that. Ideally syncs would be on a regular schedule, but if I understand correctly regressions can delay it. May I ask why it would be helpful to know ahead of time that a freeze is coming?
I have the same request. Reason: I don’t want to bump the version number of all packages in a repository for every new fix/feature and kick off a release.
Doing so just before a sync makes this less work and people still get the latest and greatest binaries.
This discussion has happened several times before. Usually release managers argue that they do not want people to release their packages just before the sync deadline. Instead they want you to release a package as soon as a sufficient amount of feature lands, because this gives the released binaries more time in the “testing” repository, giving everyone more time to test it (does this actually happen?), and it gives the package maintainers more time to fix regressions before everything’s synced to the main package repo.
Indeed as a release manager I generally prefer not to to have a rush of releases right before the sync. It gives the highest likelyhood of regressions. The longer since something has been released the more likely someone will have caught a regression.
There is not a lot of manual testing. However all the PR builds and devel job builds are run against the testing repositories. This means that any package that has a commit or a PR submitted will be tested against the new release. In addition if there has been enough time to complete a rebuild then it’s at least clear that all downstream packages at least build.
If there’s a flurry of releases just before the freeze then a large fraction of the freeze will be spent with the farm just rebuilding and it will require urgent fixes during the closing freeze window which might even be blocking others from testing. We can only get a few iterations of rebuilds in during a sync window. So if there’s multiple things to triage that interfere it’s possible that one regression prevents the testing or debugging of another package entirely. Whereas if we’re in an already validated good state with no issues we only have to deal with the unusual regressions.
We are working to keep the syncs on a regular cadence. And that should allow maintainers to batch up changes in their preferred manner.
Also note that rolling back in the testing repository is possible. Once it’s been deployed to the the main repository any regression that requires a rollback then requires a new higher numbered release with the changes reverted and an extra sync to distribute it generally. The freeze is considered the minimum testing time and generally for core packages we will often hold packages even before a freeze and land them at the beginning of a testing cycle to make sure that they get as much soak time in the testing repository as possible. If users can’t wait for an extra sync cycle to get the latest fixes/features I generally recommend that they use the testing repository anyway as they are preferring the speed over the stability. Trying to game the system and have the minimum testing time has the tradeoff of increasing the risks of regressions for everyone and makes the main repository users experience more like the testing repository user’s experience.
Lets see if I understand. It sounds like there are two pressures on releases: one to give people the latest and greatest binaries, and another to reduce the maintenance burden caused by making a new release each change. If the bugs/features aren’t important enough to make an immediate release, then releasing right before the freeze for a sync gives you as a maintainer a reminder to make a release with a batch of changes?
@tfoote points out a concern about releases right before a sync reducing the likelihood of catching a regression before syncing with the main apt repo. I wonder, do we have different expectations about the main apt repo? I wouldn’t expect it to satisfy people who need the latest and greatest changes because that wouldn’t leave a repo for users who value stability.
What if the sync itself being completed was the reminder to release? I think that has a lot of good qualities. It would be a defined point in time for a maintianer to make a release, make binaries available in the testing repo for users that need the latest and greatest, and give the most testing time before hitting the main repo for users needing stability.
This is a very good point, I didn’t think about that. So that means there really is some testing involved during the pre-sync period.
I think that there are differing expectations.
Many users of ROS want the latest and greatest features. Others value stability above all else. And, if your’re building a product, you may not even want the latest bug fixes, because every bug fix has a chance to introduce new bugs. For many companies it’s preferable to have an older version that everyone’s been using for a while and has known bugs with known workarounds, than a newer version with less experience in use and more potential unknown bugs. This is especially true in safety-critical applications.
If users want the latest and greatest, the new rolling distribution should satisfy them in the ROS 2 world, leaving the released distributions to fulfill the stable role. We don’t have anything similar for ROS 1.
Since I have had broken packages in testing before and solved it using a version bump + release: How does “rolling back” work? Can that be triggered by package maintainers or do we need a release manager to do that? And does it mean we can somehow patch a released version (in testing) without changing the version number?
I’m always embarrassed by the “PR noise” for the release maintainers when I’m fixing bugs in packaging / on certain distributions which are hard to reproduce locally, so if there’s a better solution I’d be glad to know 
This is opening a PR against the rosdistro with the previous released version.
The package maintainer can open the PR on rosdistro to do it or the release manager.
No you cannot patch a released version without changing the version number. There is the debian increment number which can be used to patch something in the release repository, but it still requires a version change in the rosdistro. And this is more relevant for the standard debian release pipeline where the debian maintainer is separate from the upstream package maintainer and may be restructuring contents to meet the debian packaging guidelines. Unless you know exactly what you’re doing this is not recommended.
Almost everyone has to take a few tries at their first release. This is expected as there’s a learning curve. We know that and it’s ok. The best ways to validate before testing is to run a prerelease test and you can also add your package to the index prior to releasing a binary to get the devel job tests to validate.
With all that the debian packaging job is still the most strict build environment that the tests can’t fully replicate without actually just running the builds. So sometimes there are things that aren’t caught.
@tfoote Thanks for the detailed answer 