Rosbag Backward Compatibility

Next Generation ROS
1

I’m not sure if ROS 2 has reached this point in its development, but has there been any discussion about maintaining some level of backwards compatibility with bag files from ROS 1? I’m specifically thinking about the scenario where I have a lot of data collected using ROS 1 that I would like to replay into ROS 2 systems.

2

ROS 1 bags store the actual message data with the same format as it is being used on the wire (plus some additional header and meta information). This is done so that recording / replaying bags is efficient.

Since ROS 2 will use a different marshaling / transport its wire format will be different. Therefore I would expect the ROS 2 “bags” to be stored in a different format.

But there are two ways moving forward:

  • You can simply use the ROS 1 rosbag application to playback a ROS 1 bag file and use the ros1_bridge to bridge the messages to a ROS 2 system.
  • As soon as a ROS 2 specific bag solution exists someone could write a “plugin” to enable also reading files with the ROS 1 bag format.
3

I’d also say that, as you pointed out, we’re not to the point where we’ve decided on the bag format. In ROSBag version 2 (which is the current version used in ROS 1: http://wiki.ros.org/Bags/Format/2.0), it assumes ROS 1’s serialization and ROS 1’s topic properties (like latching = true/false). One option would be to do a ROSBag version 3 which would have the ability to store arbitrary connection information and serialized data. Then you’d just need the right plugin to read or write a particular format for the data. If we did that, then we’d probably consider trying to backport this to ROS 1 and create migration tools like we did with ROSBag version 1 -> version 2 (http://wiki.ros.org/rosbag/migration).

Another option would be to use some other existing data format scheme like hdf5 (https://support.hdfgroup.org/HDF5/) or something like that. We’re keeping an open mind about reusing more broadly used solutions, but the rosbag format has familiarity and commonality with ROS 1 in its favor. Until we come up with a concrete solution for ROS 2, I think you have to assume the format might be different from what’s in ROS 1. Either way, we’re aware that people have loads of rosbags already and will need to either have them work natively in ROS 2 (through the bridge or a plugin maybe) or have them easily converted to the ROS 2 format.

4

Thanks for the quick response, it sounds like there’s a number of viable solutions.

5

Several years ago I did a bit of work on developing and implementing a possible new format for ROSBag heavily based on the Matroska container format (.mkv). It uses EBML and is very flexible, supporting pretty much any data and metadata you can dream of, chapters, streaming, chunking across multiple files, etc. It has compatibility with the tools you can use to modify and manage Matroska files. You could even conceivably store a ROSBag 2 format file in this container format, although I doubt that would be very efficient or worthwhile.

Although I finished the format design, I never finished the implementation (changes at work got in the way). The half-finished implementation is still online:

The specification is probably more useful than half an implementation:

I’m not suggesting that this format is the way ROSBag should go in ROS 2. Never finishing the implementation meant I never tested its viability and speed, for example. But I thought I’d mention it if anyone is interested. @iluetkeb was also involved in this work.

@iluetkeb also did work in his previous lab with a variation of the ROSBag format. I’m sure he will chime in with the results of that work.

At the same time as the above work, I also looked at HDF5. I haven’t looked at it since, but back then my impression was that it’s great for certain kinds of data, like lots of raw numbers coming from things like strain sensors and temperature sensors, but it was not an ideal fit for more structurally complex data. However, that may have changed and the library and tools were good. Being able to make use of existing tools would be nice. The people at KULeuven have done a lot of work with HDF5.

6

Phew, been a while :wink:

What I can say about my own work on the format @gbiggs mentions is that it was motivated not completely, but significantly, by internal politics. At my old lab, we had a different middleware developed internally, and it was not acceptable (politically) to use ROS. I thought that maybe others had the same problem, but exchanging data would be very useful, so we tried to create something which didn’t have “ROS” in the name, but was conceptually quite close, so ROS could eventually migrate. There was not a lot of interest either in the ROS camp, nor in the other camp, though, so this went nowhere, unfortunately.

Otherwise I probably wouldn’t have felt the need to change anything fundamentally :wink:

  • The use of textual data-type definitions in rosbag2 was the most problematic thing, and we discussed about that quite a bit.
  • I recall I found some of the arbitrarily sized meta-data structures to be a bit annoying to parse.
  • Other than that, yes, the format is non-standard, but it’s not overly complicated. To my knowledge, there are no generic data-writing libraries out there, because it appears that almost everybody his or her own format. This is a bit annoying, but it also suggests that finding the right abstractions is not trivial.

My colleagues here also tell me that they some of the use cases of rosbag2 are problematic, e.g., when a rosbag is split, there is no information about this in the bag file. You have to infer either using external information, or a heuristic based on the file-name. We would prefer some information on this to be in the library.

Maybe I’ll post some more later, if I remember anything more.

7

Having spoken with a number of colleagues about this now, we primarily see a few necessary feature extensions. This would actually apply to the current rosbag already:

  • continuation information. when a bag is split, there is no information about this in the file. You have to guess from the file-name. It would be awesome if there would be a marker at the end of the file, to let readers know where to continue.
  • information on parameters: currently completely missing, so you can’t reconstruct parameter settings, nor can you detect run-time changes
  • latching. there is no information that I know of that a topic is latched
  • playing multiple bags: when loading multiple bags, I’m told that rosbag indexes all of them in a time-consuming fashion to determine whether they are to be played sequentially or interleaved. maybe we could avoid that.
8

The Matroska-based format I mentioned above can do all those things with ease. With the exception of the last point, I expect it wouldn’t be too hard to add then to the existing format, too.

9

Hi Dirk,

I am trying the first approach i.e getting rosbag data in ros2 using ros1 bridge.
But I am facing some issues here.
The callback for the subscription of rosbag data is not consistently happening.
I suspect that this is somewhat related to the clock. Please let me know your suggestions on the same.
I did not play bag with --clock as I am not sure that how to use use_sim_time with my ros2 nodes.

steps i followed in each terminal:-
roscore
start the dynamic bridge :- ros2 run ros1_bridge dynamic_bridge
rospbag play
ros2 run ros2-node

Thanks,
Poonam