Rosbag dataset hub

avatar042 · 2021-05-10T16:41:18.831Z

Hi all,

Is there a well-known repository of rosbag datasets that is publicly available? I’m curious if this is something people would be interested in contributing to, if it doesn’t exist today? Something like a “dataset hub for robots”. I’ve found some links (added below for completeness), but nothing large enough to call it a repository of diverse robot logs. I’d love to put together a set of logs across domains/applications to build an evaluation benchmark of sorts.

Some related topics/links I was able to dig-up:

https://marvhub.com/
self-driving-car/datasets at master · udacity/self-driving-car · GitHub
Rosbag sharing place

Katherine_Scott · 2021-05-14T19:13:44.323Z

I’ve talked about building a list previously. I might be able to put sometime on this later this year.

I did grep through weekly udpates and found these.

Dataset of Datasets
Stereo Event Camera Vehicle Autonomy Dataset
NuScenes Autonomy Data Set Gets Bigger
Shanghai Tech University SLAM Research and High-Res Data Sets
Boston Dynamics Spot Robot Bag Data

avatar042 · 2021-06-01T00:39:03.394Z

Thanks for the links @Katherine_Scott.

For the folks in industry: I’m curious how you manage your collection of rosbags (presumably on S3, GFS). I’ve heard of in-house solutions for this, are there open-source resources for managing and exploring rosbags in a DB?

Timple · 2021-06-01T07:57:53.028Z

SWRI has a nice tool for this:

GitHub

swri-robotics/bag-database

A server that catalogs bag files and provides a web-based UI for accessing them. - swri-robotics/bag-database

jatowler · 2021-06-01T16:04:30.807Z

Thanks for pointing out that tool, @Timple. We built it specifically to have an on-prem bag database. We just dump bags in a directory on an internal shared drive and let the database parse them all out for us. (If you love your hierarchies, it maintains those, too, so you can view the tree in the web interface if you want.)

bonolo · 2021-06-02T11:21:10.031Z

jatowler:

t it specifically to have an on-prem bag database. We just dump bags in a directory on an internal shared drive and let the database parse them all out for us. (If you love your hierarchies, it maintains those, too, so you can view the tree in the web interface if you want.)

This looks like a very nice tool. Thanks for sharing

avatar042 · 2021-06-10T18:18:28.258Z

Thanks @Timple for the link. I noticed that link a while back; @jatowler how does this tool scale with more rosbags (>10K) / larger bags (>300GB / rosbag) in your internal shared drive? Also, was this tool meant more for web visualization or for general metadata cataloging and bag management?

daniel_dsouza · 2021-06-11T18:15:53.135Z

@avatar042 currently we have roughly 12k bags indexed in our bag database instance without any adverse effects. Most of these bags are under 10 GB. For large bags over 20 GB, you will need have speedy storage for timely processing.

Our current setup stores about 20TB of bags on a (dated) Synology NAS, mounted as an NFS share over a 10 gig connection to the VM running the bag database. The limit on individual bag sizes right now is the NAS, which is on my wishlist of hardware upgrades.

Overall, this tool is focused more towards “general metadata cataloging and bag management” over “web visualization.”