Fast Forward Merging rosbag2 master API to Foxy

I’ll also throw in here that I’m investigating how much of a patch would be needed to break no public APIs. It’s much harder to make that promise about the ABI - but I think we may be in a much better position if we can remove the possibility of breaking anybody’s build.

4 Likes

The only thing blocking a breaking change is the rule in ROS that API/ABI does not break within a distribution. I think we need to accept that this rule is not 100% absolute.

atm, i was thinking that this is hard guarantee to support the products. but now i understand API/ABI occasionally breaks if necessary. and we can detect that with checking the major version of the packages.

I’m investigating how much of a patch would be needed to break no public APIs.

:+1: thanks

2 Likes

What about approaching this from the packaging side? I.e. create a new set of packages with the same package name (or a similar one), like ros2-foxy-rosbag2-fixed, and configure these packages to conflict with the original rosbag2 packages and replace them. Then users could choose whether they stay with the broken but compatible version, or whether they install the fixed but incompatible version. And the old version could issue a loud warning in the console telling the users what can be done to get a better rosbag.

I’m not sure yet how this would work with rosdep, which might try to always install the old version, though…

Or another approach - you could set up an additional PPA that would contain just the fixed rosbag. Everybody could then decide what versions of the packages are better for each specific use-case.

I was able to get the API/ABI report running - https://build.ros2.org/job/Fpr__rosbag2__ubuntu_focal_amd64/126/API_5fABI_20report/

1 Like

Yes, backport to Foxy because racecar! Foxy’s current as-is rosbag2 performance is completely unusable for the 500 students and 28 universities in the Indy Autonomous Challenge. ROS 2 Foxy is the recommended software but Foxy rosbag2 can not cope with the data rates for the 3x flash lidar, 6x cameras, 4x radars, GNSS w IMU, drive-by-wire shown here

Current Foxy rosbag2 also does not work for autonomous cargo delivery and other autonomous driving pilots underway.

Indy Autonomous Challenge and other autonomous driving uses is why Apex.AI, TierIV, Bosch, ADLINK worked with Robotec.AI, ROS Tooling WG to contribute the huge improvements that are now available to Foxy users if backported.

3 Likes

Says who? But to be fair that is what most people assume. @spiderkeys and many others run Foxy on Jetson without issue when built from source. And NVIDIA has released ros2 foxy dockers for Jetson. Yes, this would be good for us to make clear in ros2 docs.

The road to stability is sometimes a wee bit bumpy. :+1:

I’d be very happy to have a fully functioning set of common tools in an LTS.
In lack of that, people will keep rushing onto new versions, now having to migrate to stuff they never intended to in the new distro or worse, go full “Not invented here”…

Perhaps what may help reasoning about the severity of this change is the current size estimate of the Foxy userbase?

People “assume” this because it’s clearly documented in REP 2000 -- ROS 2 Releases and Target Platforms (ROS.org)

Thanks for the heads up that Foxy might work!

+1 to backport :smile:

1 Like

Long time ROS1 user here. By now we have accumulated terabytes of ROS1 bagfiles for debugging, future testing and future possibilities with the lidar data (learning?).

We’re on the verge of using ROS2 for our newest project. Foxy would be obvious choice here since Galactic is still a year away.

Now reading that the rosbag2 system is flaky is really worrisome. I am glad I read it here first.
This could potentially have cost lost of engineering hours to figure out that we lost data, how we lost data and how we should not lose data again.

So for the sake of ROS2 newbies, please allow the current release to be not a regression from ROS1.

2 Likes

Just to be clear; Galactic is 3 months away, not a year. The timeline is here: <no title>

1 Like

You are absolutely right: ROS Clock :slight_smile:

If this backport doesn’t make it, Galactic might be the way to go instead of the recommended Foxy.

1 Like

This would be great to get into FOXY. This is the designated ROS2 version for the Indy Autonomous Challange and the autonomous vehicle has lots of cameras, lidars, radars, GPS/IMU, bunches of sensors not to mention a very underdeveloped software stack. All this means lots and lots of data flowing around and needed to be captured at 200 mph. It is our sense the current RosBag2 just won’t be able to keep up and we’ll have to make some tough decisions not to try to get all the data we’d like for post-processing onto the disk. Univ of Hawaii Autonomous racing team - AIracingTech

1 Like

Given how easy it is with rosbag2 in its old state to silently drop messages and even entire topics I’m in favour of a backport.

So far the workarounds (in various stages of maturity) I’ve seen are

  • separate data recording solutions
  • a custom DDS implementation with its own recording feature
  • strictly limited rosbag use for small debug purposes on select topics (still shaky and annoying)

All of these are quite a burden. So it’s likely this LTS’s rosbag will haunt people for another year, if they are unable to always jump to the latest release.

1 Like

Hello everyone - thanks for the lively discussion, this has really helped to shake out lots of good points. We’ll be coming to a final decision on this in tomorrow’s Tooling WG Meeting. I invite you to join if you’d like to discuss (gently and respectfully, if so, please!)

The frontrunner solution right now is to release a new package into Foxy with a new name rosbag2-future which contains all latest development (as @peci1 and others have brought up). This fragmentation is not ideal (as @matthew-reynolds mentioned) - but I do counter that it is by no means worse than nothing. This approach avoids the scream-test, which @kyrofa has wisely reminded us only works when you get screamed at. While no solution is ideal, this gets the code to those who really need it, while avoiding the rather large downside of breaking promises to users that they get stability within a distribution. While some such as @gbiggs have made an argument that dropping data should be considered a breaking bug, there will always be some upper limit where this happens - the performance improvements just pushed that bar much closer to the disk write limit.

A quick summary of the reasoning around this

Pros:

  • No API breakage (or the even more sinister ABI breakage) - and therefore no promises broken
  • No tricky work to make a “clean backport”

Cons (with counterpoints):

  • Users have to explicitly know there is an alternative
    • we will call it out clearly on the rosbag2 README on the latest and foxy branches
    • this thread probably reached a lot of the key parties who want the improvement
    • Foxy will only last for one more year, and Galactic is almost out, so this situation won’t last that long
  • We don’t want users to think that this will happen again. This is not a precedent to be followed, it’s a symptom of the immaturity of rosbag2 in early 2020
    • we will also mention this in the README
  • Double installation (rosbag2 and rosbag2-future) increases installed size
    • Users fine-tuning insall size should avoid ros2/variants (e.g. ros-base) anyways, and will not have rosbag2 installed at all.
  • ros2 bag verb overlaps, so we need to register ros2 bag-future, requiring a change in user’s recording scripts, and have to switch back in a future distro
    • users who have installed rosbag2-future will at least know they have done so and can make a note to switch back on the next distro

Basically the mitigation to the downsides comes down to: be very communicative on Discourse and the rosbag2 README about this approach

4 Likes

Glad to hear progress on this. I think overall this approach is good (Even if I would prefer an ABI break :wink: ), certainly better than leaving things as they are.

I can’t make it to the meeting, but I also have no strong objections. A couple small points:

and therefore no promises broken

As mentioned, technically there wasn’t a promise as this isn’t yet 1.0, but I concede this is a technicality and in practical terms the compatibility is expected.

Foxy will only last for one more year, and Galactic is almost out, so this situation won’t last that long

Unless I’m mistaken, Foxy goes EOL in May 2023. Galactic goes EOL May 2022 (Or is is Nov 2022? REP 2000 says both “1.5 years” and “until May '22”, not sure which to believe). In any case, Foxy is LTS and has 2 more years of life, and will last beyond Galactic.

ros2 bag verb overlaps, so we need to register ros2 bag-future , requiring a change in user’s recording scripts, and have to switch back in a future distro

I have no idea how the entry points work, but rather than introducing a new verb, would it be possible to have the -future package take precedence over the existing ros2 bag verb if it’s installed? My thinking is this would be much more seamless for users and scripts, and would be similar to how import __future__ in Python2 overrides keywords. Even if it means a one-off special case in the upstream ros2cli package, I think it’s worth consideration given the uniqueness of this situation.

1 Like

OK - in the Tooling WG meeting today we have decided definitively that yes, we will be going with the rosbag2-future approach.

Some extra notes on this after the discussion:

  • We plan to use the <conflicts> package.xml tags so that only one of rosbag2 or rosbag2-future can be installed at a time in Foxy.
  • Becaue of this exclusive installation, there will not be a separate verb, rosbag2-future can be invoked via ros2 bag
  • The specific commit that will be released has not yet been decided, but please keep an eye out for updates over the next couple weeks - rosbag2-future in Foxy will likely be very close, if not identical, to the Galactic release of rosbag2
  • The maintenance story isn’t yet decided, but we will probably weigh more heavily towards updating rosbag2-future for the remainder of Foxy support, instead of backporting anything new to rosbag2
  • We do not plan to ever do this again! So it’s a one-distro-time thing to install/depend on a different rosbag2 package.
  • We’ll do a new Discourse post, and update the rosbag2 READMEs for foxy, foxy-future, and master to reflect the situation, once this is released.

Thanks again to everyone for the discussion here, it really helped consolidate the thinking around this decision. We understand that no solution was going to make everybody perfectly happy, but are predicting that this one causes the least trouble, while getting everyone what they need.

6 Likes

I’m glad you chose this way which keeps trustworthiness of the API stability claims about ROS :slight_smile:

Did you also consider adding a runtime warning to the old rosbag2 telling the user that the installed version may be flawed and what can be done to get the better version? This wouldn’t be the first time such a thing happened during a distro’s lifetime. See e.g. Xacro and its --inorder flag.

2 Likes

Happy to hear that you guys seem to have figured out a solution satisfying both the promise ob API stability as well as allowing better rosbag usability in foxy! Thanks for the work you are putting into this!

1 Like

Have to be careful adding warnings - that can break ci/test scripts that are overly reliant on parsing stdout