I would also like to add that that change violated one promise (API/ABI) in order to remedy another (No GUI dependencies in the robot metapackage) - Kinda stuck between a rock and a hard place, you’re violating a promise one way or another.
In this case, it’s violating a (slightly less rigid, as @emersonknapp pointed out) promise to remedy usability. I’m not saying the situations are entirely dissimilar, but they are slightly different.
I don’t know the ROS2 landscape well enough to give a well informed opinion here, but given the leafy-ness of the package, my opinion leans towards prefering the break.
I disagree, I think @gavanderhoorn is correct that fragmentation is a worse evil here. Given how currently unusable rosbag2 is and the (reasonable?) assumption that the package is fairly leaf-like, an API break seems more reasonable to me.
I’m in favour of the backport even if it involves breaking API/ABI.
@tomoyafujita mentioned data corruption as one reason to break. I think that missing topics and messages are arguably data corruption issues that need to be fixed.
@gavanderhoorn mentioned bugs. rosbag2 doesn’t work properly, which, like him, I think counts as a bug.
And as @emersonknapp said, the API has not yet reached version 1.0 so the version number doesn’t block breaking changes.
The only thing blocking a breaking change is the rule in ROS that API/ABI does not break within a distribution. I think we need to accept that this rule is not 100% absolute. If there is a strong enough reason, as agreed by the community (which for all intents and purposes can only be defined as “those interested enough to read Discourse and respond”), then in my opinion the break is justified.
I’ll also throw in here that I’m investigating how much of a patch would be needed to break no public APIs. It’s much harder to make that promise about the ABI - but I think we may be in a much better position if we can remove the possibility of breaking anybody’s build.
The only thing blocking a breaking change is the rule in ROS that API/ABI does not break within a distribution. I think we need to accept that this rule is not 100% absolute.
atm, i was thinking that this is hard guarantee to support the products. but now i understand API/ABI occasionally breaks if necessary. and we can detect that with checking the major version of the packages.
I’m investigating how much of a patch would be needed to break no public APIs.
What about approaching this from the packaging side? I.e. create a new set of packages with the same package name (or a similar one), like ros2-foxy-rosbag2-fixed, and configure these packages to conflict with the original rosbag2 packages and replace them. Then users could choose whether they stay with the broken but compatible version, or whether they install the fixed but incompatible version. And the old version could issue a loud warning in the console telling the users what can be done to get a better rosbag.
I’m not sure yet how this would work with rosdep, which might try to always install the old version, though…
Or another approach - you could set up an additional PPA that would contain just the fixed rosbag. Everybody could then decide what versions of the packages are better for each specific use-case.
Current Foxy rosbag2 also does not work for autonomous cargo delivery and other autonomous driving pilots underway.
Indy Autonomous Challenge and other autonomous driving uses is why Apex.AI, TierIV, Bosch, ADLINK worked with Robotec.AI, ROS Tooling WG to contribute the huge improvements that are now available to Foxy users if backported.
Says who? But to be fair that is what most people assume. @spiderkeys and many others run Foxy on Jetson without issue when built from source. And NVIDIA has released ros2 foxy dockers for Jetson. Yes, this would be good for us to make clear in ros2 docs.
The road to stability is sometimes a wee bit bumpy.
I’d be very happy to have a fully functioning set of common tools in an LTS.
In lack of that, people will keep rushing onto new versions, now having to migrate to stuff they never intended to in the new distro or worse, go full “Not invented here”…
Perhaps what may help reasoning about the severity of this change is the current size estimate of the Foxy userbase?
Long time ROS1 user here. By now we have accumulated terabytes of ROS1 bagfiles for debugging, future testing and future possibilities with the lidar data (learning?).
We’re on the verge of using ROS2 for our newest project. Foxy would be obvious choice here since Galactic is still a year away.
Now reading that the rosbag2 system is flaky is really worrisome. I am glad I read it here first.
This could potentially have cost lost of engineering hours to figure out that we lost data, how we lost data and how we should not lose data again.
So for the sake of ROS2 newbies, please allow the current release to be not a regression from ROS1.
This would be great to get into FOXY. This is the designated ROS2 version for the Indy Autonomous Challange and the autonomous vehicle has lots of cameras, lidars, radars, GPS/IMU, bunches of sensors not to mention a very underdeveloped software stack. All this means lots and lots of data flowing around and needed to be captured at 200 mph. It is our sense the current RosBag2 just won’t be able to keep up and we’ll have to make some tough decisions not to try to get all the data we’d like for post-processing onto the disk. Univ of Hawaii Autonomous racing team - AIracingTech
Given how easy it is with rosbag2 in its old state to silently drop messages and even entire topics I’m in favour of a backport.
So far the workarounds (in various stages of maturity) I’ve seen are
separate data recording solutions
a custom DDS implementation with its own recording feature
strictly limited rosbag use for small debug purposes on select topics (still shaky and annoying)
All of these are quite a burden. So it’s likely this LTS’s rosbag will haunt people for another year, if they are unable to always jump to the latest release.
Hello everyone - thanks for the lively discussion, this has really helped to shake out lots of good points. We’ll be coming to a final decision on this in tomorrow’s Tooling WG Meeting. I invite you to join if you’d like to discuss (gently and respectfully, if so, please!)
The frontrunner solution right now is to release a new package into Foxy with a new name rosbag2-future which contains all latest development (as @peci1 and others have brought up). This fragmentation is not ideal (as @matthew-reynolds mentioned) - but I do counter that it is by no means worse than nothing. This approach avoids the scream-test, which @kyrofa has wisely reminded us only works when you get screamed at. While no solution is ideal, this gets the code to those who really need it, while avoiding the rather large downside of breaking promises to users that they get stability within a distribution. While some such as @gbiggs have made an argument that dropping data should be considered a breaking bug, there will always be some upper limit where this happens - the performance improvements just pushed that bar much closer to the disk write limit.
A quick summary of the reasoning around this
Pros:
No API breakage (or the even more sinister ABI breakage) - and therefore no promises broken
No tricky work to make a “clean backport”
Cons (with counterpoints):
Users have to explicitly know there is an alternative
we will call it out clearly on the rosbag2 README on the latest and foxy branches
this thread probably reached a lot of the key parties who want the improvement
Foxy will only last for one more year, and Galactic is almost out, so this situation won’t last that long
We don’t want users to think that this will happen again. This is not a precedent to be followed, it’s a symptom of the immaturity of rosbag2 in early 2020
we will also mention this in the README
Double installation (rosbag2 and rosbag2-future) increases installed size
Users fine-tuning insall size should avoid ros2/variants (e.g. ros-base) anyways, and will not have rosbag2 installed at all.
ros2 bag verb overlaps, so we need to register ros2 bag-future, requiring a change in user’s recording scripts, and have to switch back in a future distro
users who have installed rosbag2-future will at least know they have done so and can make a note to switch back on the next distro
Basically the mitigation to the downsides comes down to: be very communicative on Discourse and the rosbag2 README about this approach
Glad to hear progress on this. I think overall this approach is good (Even if I would prefer an ABI break ), certainly better than leaving things as they are.
I can’t make it to the meeting, but I also have no strong objections. A couple small points:
and therefore no promises broken
As mentioned, technically there wasn’t a promise as this isn’t yet 1.0, but I concede this is a technicality and in practical terms the compatibility is expected.
Foxy will only last for one more year, and Galactic is almost out, so this situation won’t last that long
Unless I’m mistaken, Foxy goes EOL in May 2023. Galactic goes EOL May 2022 (Or is is Nov 2022? REP 2000 says both “1.5 years” and “until May '22”, not sure which to believe). In any case, Foxy is LTS and has 2 more years of life, and will last beyond Galactic.
ros2 bag verb overlaps, so we need to register ros2 bag-future , requiring a change in user’s recording scripts, and have to switch back in a future distro
I have no idea how the entry points work, but rather than introducing a new verb, would it be possible to have the -future package take precedence over the existing ros2 bag verb if it’s installed? My thinking is this would be much more seamless for users and scripts, and would be similar to how import __future__ in Python2 overrides keywords. Even if it means a one-off special case in the upstream ros2cli package, I think it’s worth consideration given the uniqueness of this situation.
OK - in the Tooling WG meeting today we have decided definitively that yes, we will be going with the rosbag2-future approach.
Some extra notes on this after the discussion:
We plan to use the <conflicts> package.xml tags so that only one of rosbag2 or rosbag2-future can be installed at a time in Foxy.
Becaue of this exclusive installation, there will not be a separate verb, rosbag2-future can be invoked via ros2 bag
The specific commit that will be released has not yet been decided, but please keep an eye out for updates over the next couple weeks - rosbag2-future in Foxy will likely be very close, if not identical, to the Galactic release of rosbag2
The maintenance story isn’t yet decided, but we will probably weigh more heavily towards updating rosbag2-future for the remainder of Foxy support, instead of backporting anything new to rosbag2
We do not plan to ever do this again! So it’s a one-distro-time thing to install/depend on a different rosbag2 package.
We’ll do a new Discourse post, and update the rosbag2 READMEs for foxy, foxy-future, and master to reflect the situation, once this is released.
Thanks again to everyone for the discussion here, it really helped consolidate the thinking around this decision. We understand that no solution was going to make everybody perfectly happy, but are predicting that this one causes the least trouble, while getting everyone what they need.
I’m glad you chose this way which keeps trustworthiness of the API stability claims about ROS
Did you also consider adding a runtime warning to the old rosbag2 telling the user that the installed version may be flawed and what can be done to get the better version? This wouldn’t be the first time such a thing happened during a distro’s lifetime. See e.g. Xacro and its --inorder flag.
thanks
), certainly better than leaving things as they are.