Hello friends of rosbag2.
Over the last couple of months, we’ve been working hard to bring rosbag2 to a level which significantly improves the write performance and made it easier to work with compression.
Together with the help of Robotec.AI we have been focusing on improving the storage backend to deal with high volume data traffic, allowing us to record multiple hundreds of megabytes per second with our default sqlite3 storage backend. Before the improvements, users were experiencing significant message loss even with throughputs of 100 Mb/s or less, especially when recording many topics, and we have received feedback that for some of them, this was a serious limitation for their use cases. With our optimizations, data is recorded as expected up to the hardware limitation of disk write speed, and rosbag2 seems to be on par with rosbag (ROS 1). For more details about the improvements please find a full report on Github. You can also refer to an initial metaticket of performance evaluation to understand the motivation behind the improvements as well as analysis of Foxy rosbag2 bottlenecks.
Similarly, Amazon put some effort into improving the performance of working with compressed rosbags. That is, we’ve introduced a streaming interface which allows reading the compressed bagfile without loading it as a whole beforehand into memory. We’ve also implemented multithreaded compression which brings an improved performance when writing bagfiles.
While these improvements work great on the current master branch, as well as the rolling release, we’d love to provide the improved functionality to most of you as a backport to Foxy.
Unfortunately, these performance improvements come with a price. And that price is a breaking ABI/API compatibility with the currently released Foxy version. The changes are affecting the public API in rosbag2_cpp, rosbag2_storage as well as the sqlite3 storage plugin, however will not break with the overall usage of the rosbag2 command line interface (though there may be some new arguments available).
As a basic rule of thumb, it holds that during the lifespan of a ROS distro, released packages maintain ABI compatibility. While we generally consistently aim for this, we are hereby asking your permission to exceptionally break with this rule.
We consider the performance improvements important enough to justify an API breakage within the same distro release. The two main reasons for it are that Foxy has the largest user base and thus most people will profit from a binary release. Second, we believe that rosbag2 (and its derivative packages such as rosbag2_storage, rosbag2_cpp, …) are high enough in the stack to be considered leaf packages with the hope that not too many applications are compiling/linking against its API. Obviously, we are trying to minimize the changes required by client code.
We are thus considering fast forwarding the current API as available on the master branch to Foxy in a one-time shot PR, adapting for all changes to dependencies such as rcutils, rcpputils, etc. We’ve opened a first draft of this PR which shows promising results, having only a few minor adaptations.
We’d love to hear your feedback on this before we take action. We’re internally leaning towards the API break, but if you - the community - have strong arguments why an API change is not an option, we’re all ears. As mentioned above, we will respect ABI compatibility for future enhancements for rosbag2 and consider this an exception.
your ros-tooling team.