MCAP is a modular container file format for pub-sub messages, intended to be used in robotics and designed as a successor to rosbag and rosbag2 SQLite files. It supports arbitrary serializations, with specifications published for storing ROS1 and ROS2 data as well as Protobuf-based data. The goal is to provide self-contained recordings that contain message definitions, per-run metadata, and attachments such as crash logs in addition to the message stream. The format is optimized for append-only writing with lower overhead and includes a variety of tuning knobs to balance performance and write throughput such as compression, CRCs, indexing, and variable chunk size.
You currently need to build from source until the first release makes its way into the Galactic or Humble package ecosystem, but the README includes instructions on how to build and start using the storage plugin.
Feedback is greatly appreciated: does this format meet your use cases, file size and performance measurements from real-world systems or synthetic tests, build woes, etc. Benchmarks will be published soon.