PSA: Default ROS 2 bag storage format is changing to MCAP in Iron

General
,
1

Hi all! This is a post to let anyone interested know that there is now a draft pull request open to switch the default storage plugin for recording in Rolling from SQLite3 to MCAP. There is still work to complete before this merges, but we’re aiming to have it merged by the end of this year (with plenty of time to test before ROS 2 Iron is released).

TL; DR

  • This PR adds rosbag2_storage_mcap to the rosbag2 meta-package dependencies, and changes default --storage-id argument to “mcap” when recording with ros2 bag record.

  • MCAP file format offers several benefits over the current default, SQLite3.

  • This change affects Iron and future distributions only. No changes for users of Foxy, Galactic or Humble

  • If you rely on SQLite3-specific features such as:

    • querying bags directly with SQL statements
    • Using message-level compression

    You can preserve existing behavior when upgrading to Iron by explicitly setting --storage-id sqlite3.

  • There is a TODO list in the PR description of issues to be addressed before merging - if you know of anything not on that list, please comment on the PR.

Summary of changes

  • Add rosbag2_storage_mcap as a dependency of rosbag2_default_plugins
    • This makes the mcap storage plugin a transitive dependency of the rosbag2 package, which means it will be installed by default when installing ROS2 Iron.
    • The SQLite3 storage plugin will continue to be installed by default as well.
  • Switch default argument for ros2 bag record –storage-id from sqlite3 to mcap
    • This means that if you don’t specify a storage plugin to record with, your bags will be recorded in the MCAP format instead of SQLite3.
    • If you explicitly specify your storage plugin with --storage-id, nothing will change for you.

Reason for change

MCAP is now technically superior to SQLite3 in most ways for bag recording. @john-at-foxglove.dev gave a talk at ROSCon 2022 which goes through some of the major advantages:

User Impact

  • This does not affect users of Foxy, Galactic, or Humble. This change would only merge into the Rolling tree and would appear in the Iron distro next year.
    • Users of Foxy, Galactic, and Humble may opt in to using MCAP by installing the rosbag2_storage_mcap package and passing the --storage-id mcap argument to ros2 bag.
  • Rosbag2 is designed to automatically select the correct storage plugin for the file when reading, so any tools for reading bags that use the rosbag2 API will continue to work seamlessly.
  • Most external tools for examining SQLite3 bag contents (such as PlotJuggler and Foxglove Studio) can read MCAP.
  • If you have workflows for reading bag contents that directly use SQL statements, these will not work with MCAP files. If you need to keep using those, make sure to explicitly specify the SQLite3 storage plugin when recording with the --storage-id sqlite3 argument.
  • MCAP is not compatible with --compression-mode message. The reason is that MCAP is designed to be self-describing, and message-level compression breaks this (see this github issue for more details).
    • MCAP instead supports chunk compression, which batches several messages in each compressed block. This dramatically increases compression efficiency for bags with many small messages.
    • If your workflow depends on using message-level compression, make sure to explicitly specify the SQLite3 storage plugin with --storage-id sqlite3.

What’s happening now?

There is a TODO list in the PR description of issues to be addressed before merging - if you know of anything not on that list, please comment on the PR.

12 Likes
2

The link you provided shows benchmarking for write performance only. How does the read performance compare to sqlite3? Sometimes it’s mighty convenient to read back data directly from a ROS bag into an application and in this situation read performance matters.

3

Hi Bernd, informally read performance is similar, but I agree it’s a gap that we don’t have a published benchmark comparison. I can take a look at publishing something to cover this :+1:

1 Like
4

To expand on this, MCAP files store a message index by default. This index allows seeking, iterating messages in timestamp order, and range-based queries, similar to how SQLite indexes work. That’s why we see similar read performance from both SQLite and MCAP.

1 Like
5

I actually notice a small read performance degradation when using mcap bag for map generation with cartographer offline node. I believe it is due to the fact that mcap bag are by default compressed ?

6

As of version 0.5.0, the default configuration is to write uncompressed, chunked MCAP files.

1 Like
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.