Hey, everyone! It’s your favorite Autoware developer Josh here! Just wanted to kick up a discussion on something I’ve noticed in the ROS community lately regarding “standardized” messages. In this context, I don’t mean messages which are part of “Core ROS” but more in the sense of “used by much of the community and/or have had considerable cross-industry work done to come to sensible conclusions.”
There appears to have been a good amount of thought and work put into many of the message packages that are scattered across Github organizations and if people don’t know about them, they won’t consider them for usage in their next project! Some examples include:
So, my proposal is to create a ros-messages Github organization which will provide a more “de-facto” location for messages which are widely used, heavliy discussed, and considered “common” across the community just like the other ros- Github orgs. Additionally, I feel like the message structures are a very important API for node/module interoperability and having only the lowest-level, core messages (e.g. common_interfaces) be considered “standard” does not fulfill the needs of the community. I am also volunteering to help maintain this Github org, should the community agree on it’s necessity.
In general a great idea for maintaining the ROS message base. In my projects we plan to transform messages into other IDL formats like google protobuf. Such a central repository would simplify to track changes and synchronize those interfaces.
Would this repo cover ALL ROS message types including the “built in ones” (std_msgs, sensor_msgs …) ?
I think it’s likely that ros2/common_interfaces would remain as the set of “core messages” for ROS. The messages I’m recommending for inclusion are more “second-level” messages. This would hold them to a higher standard than they currently are, but wouldn’t be quite as stringent as those that are considered “core” to ROS.
Per RER 2005, that’s pretty much the new definition of what counts as “ROS2”. I like this idea as long as the interface packages included are all REP 2005-able or on an imminent path towards it. All the examples you mention I think count.
I think Josh means an organization with repos made of the individual packages. I would not like to add all interfaces to a single repo, building that would be a pain. They should all be separate unless being pulled into common_interfaces for shipping with the distributions. I think of this as being a good staging ground for some potential common_interfaces inclusions.
I like this name better, but technically I think we refer to these as interfaces, so ros-interfaces might help to re-enforce that vocabulary.
Agreed. Additionally, “interfaces” also covers service and action definitions, which I think would be good to have here too if they fit. We can come up with a more “formal” definition of what would go in this Github org after we have several examples migrated.
Maybe we should have a guide in the ROS 2 docs pointing people in the direction of the “soft standards” everyone is using for messages. I could see cross referencing the message types with supported hardware as being handy (e.g. Sensor A, Sensor B, and Sensor C are interchangeable at the message level).
Having a central organisation for interfaces that are on the path to becoming commonly-accepted is, I think, a useful idea. We also need a maintained list of useful interfaces that includes ones not in that repository, though. Not everyone will want to hand theirs over to an org, or they might want to keep them close to their own source.
I completely agree that the shared interfaces are one of the most important aspects facilitating code reuse in the ROS ecosystem. We should find ways to make recommended ones more visible and discoverable. However I don’t believe that putting them into one organization is the best way to do that, and it also has significant downsides.
Clearly it makes them more visible by having one place to go find them. However, this adds a lot of overhead and administrative challenges for managing the maintenance access etc. It also ends up with a gatekeeping aspect where there’s a special team who decides what goes in and what doesn’t. If a developer wants to make up my their own messages and share them when do they have to transfer ownership for others to reuse otherwise they would be considered 3rdparty? And as others have mentioned sometimes is better to keep the messages near the code, especially if there might be evolution down the line.
Also as a side note, the location of the repository is moot for most users who will access the package via their package manager and read the centralized documentation.
One of the things that we’ve learned through the development of message interfaces in ROS is that the best way to validate a design is to successfully build on top of it. Early in the process there may be several different approaches and until you try them and test them it’s hard to tell what exactly would be best. And at this stage having a few different options is useful. You can try them both and then use the one that works best for you. With the community exploring in this manner it’s usual that one of the versions reaches critical mass and becomes the defacto standard.
I believe that the best way to promote any of these messages is to generate libraries and tutorials for them that makes it easier for users to use them. It will help the messages gain critical mass. And then discoverability is no longer a problem, because all searches for that topic will come back with examples of using the messages, tutorials, or questions about how to use it. And if there’s a good ecosystem of libraries building on top of it even better as that functionality will then also be reused. If there’s not an ecosystem of tools and libraries using a perfectly designed message, that message itself is not very valuable to an individual user.
I agree with the issues you mention with steady state additions.
However, I think there is still value in having the existing interfaces centralized in a place so that there is a single place to look for them (though search-ability is less important to me than just being organized in a separate mental bin). There are alot of *_msgs packages in ros-perception, ros-planning, and ros2 that would be nice to stick in one place and for *_msgs that don’t naturally fit into those orgs. This seems like a really important thing to not to have a clear home.
I don’t think we should be developing new *_msgs packages there, but I think its a good place to aggregate the existing ones in other ROS orgs and then add new ones as they are needed, somewhat mature, and been around for a little while.
Besides a central place for ROS message packages it would be great to have a kind of decoupled toolset and documentation
design of the message format like that one for protobuf (protobuf overview)
a standalone message compiler that can be easily extended by plugins to transform messages into other languages or message formats
a c++ ROS message serialization / deserialization library that can be used for none DDS rmw implementation to not invent the wheel again and again for dynamic type support
Maybe most of this is existing and I need to google better but would be nice to open the ROS message format to third party projects that need to interface with ROS and to use ROS message like all the other formats out there (protobuf, flatbuffers, capnproto, bond, msgpack …).
The message format is documented in the Concepts overview: <no title>
That’s exactly how we’ve implemented it. There is a generator system for which you can write plugins to generate for any language. A community example can be seen for java and rust and is used the same way for our core languages c++ and python There’s also a complementary typesupport system to support different implementations. There’s a good overview in this answer
We provide a cmake based toolchain to discover and select the rosidl extensions for which you would like to generate. The typesupport and rosidl generation is all separated from any of the runtime requirements and can be used without installing any of the communication implementations.
For ROS 2 we have chosen not to design or implement our own serialization solution but instead default to using the Common Data Representation, CDR, standard maintained by OMG which is also the standard for DDS thus there are implementations available from all those implementations. A quick google search finds open source implementations for serialization in multiple langues such as c++ and Rust
We already generate idl datatypes from the msg format. Implementing generators that provide convenient bridges to the other formats would be a great contribution. Since this is going off topic for this thread. If you or anyone would like to follow up, I’d suggest starting a new thread in this category.
but would be nice to open the ROS message format to third party projects that need to interface with ROS and to use ROS message like all the other formats out there (protobuf, flatbuffers, capnproto, bond, msgpack …).