Most of the DDS vendors support the idea of a primary key in their message structures. Have we discussed enabling that from the ROS msg format? What are the issues here? A “key” keyword seems fairly straightforward.
We’ve discussed it before, but it’s largely seen as overlapping in functionality with topic namespaces.
If you have a use case that is made possible with keys, then that would be interesting to hear about it.
However, there are quite a few little details for implementing the key field which are not straightforward to me, like:
- each vendor has its own way of defining them
- dealing with key fields in nested messages
- use of them has to be made visible from all of the ROS tools
Probably other things I cannot even think of now. So unless there is a big use case which is not possible with namespaced topics, I think we’d probably not expose it.
I just found this discussion when looking for keyed topic support. At Houston Mechatronics we have been using a long ago branched version of ROS2. We are now moving to Ardent and saw that keys still aren’t there. We use keys for several different things. One of the things we have implemented is a parameter server purely in DDS which uses keyed topics so there is only one Parameter topic. The key is the name of the parameter and the value is in the other part of the topic data. This gives us great flexibility with the underlying DDS software because it handles the history. If I put my history at keeping only 1 the DDS vendor software we are using (RTI) will intelligently give me the last of each instance of the key. I only need to worry about one topic, but I can get all of the latest values and nothing else.
RTI at least has suggested using keyed topics as one of the primary ways to efficiently use DDS in numerous parts of our system. We’ve had to fork rosidl and rosidl_dds to have this DDS feature on our end. It supports the RTI version of keys in IDLs, but something similar could likely be extended for the other vendor solutions if they differ.
Using keys for this use case has one drawback. Keys are not “introspectable” without receiving the actual data. You can’t answer the question what parameters exists - without actually receiving all the parameter values first.
So if you would like to implement a user interface which should show the name of all parameters you can’t do that efficiently. Imagine some parameters contain large data but the user only wants to see the names of all parameters and then decides to get only the “small” value of one of the parameters.
We’ve been talking at length to RTI about this topic as well as some of the other vendors, and they suggested two concrete ways of using keys in ROS 2:
- using keys implicitly under the hood to make our topics which have namespaces more efficient (sharing data readers and writers between otherwise unrelated subscriptions and publishers)
- exposing the feature to users like yourself so you can choose to use keys in specific situations
For the first point, I think we decided that this would be very difficult to do correctly and efficiently. There are a lot of technical points as to why, but @dirk-thomas alluded to one which is that keys are not part of discovery, so something like
rostopic list becomes a lot less useful.
In the end, we imagined a new feature that DDS could have which provides the performance benefits of
instances which you get when using keyed fields on a topic, but without having to have the key itself in the message definition. This new kind of key is something between keys and partitions, or another imperfect analogy is partitions which create instances. This is by no means a sure thing to happen, and even if it did we’d have to wait on the DDS standard to add it and adopt it. But what I took away from it is that: keys are not ideal in all cases and we would love to use instances to reduce overhead and discovery traffic but not at the expense of requiring key fields in our messages.
For the second point, there’s no reason (as long as we’re not using keys under the hood) that we could not allow users to use keys in their own way, perhaps in the way you described. The only hesitation we had was that this is a sophisticated feature, which really ties our API to DDS much more strongly. To be clear, we don’t intend to support anything other than DDS for the foreseeable future, but we did take a lot of care to insulate our robot specific code from the communication system underneath. Perhaps this is not a realistic abstraction, but in our reference implementation of ROS 2 we’re trying to provide it. Therefore, we’re (or at least I am) wary of taking more and more complicated features into our middleware abstraction layer (rmw, see: https://github.com/ros2/rmw). It’s not a deal breaker for key fields as a feature of ROS 2, but a consideration we have.
To disagree or correct my self, I previously said:
I can now express a use case where keys would be much more efficient, which is the case of the parameter events topic.
I’m speaking here about parameters as described for ROS 2, see: http://design.ros2.org/articles/ros_parameters.html, where they are decentralized and owned by nodes, and there is no “special” centralized parameter server. Global parameters are realized with a node that has a conventional name (like
global_parameter_server) and accepts all changes.
So, every node publishes to the “parameter event topic” anytime a parameter it owns is changed (CRUD), then a tool can subscribe to the topic and monitor all changes to the parameter system.
Now, this could either be “every node publishes to the same
/parameter_events topic” or “every node publishes to its own
~/parameter_events topic”. The benefit of the first is that a tool that wants to monitor all changes only needs to subscribe to one topic (this is the same use case as
rosout, many producers, one consumer of all producers). However, if you want to monitor one node, or only a few nodes, then you need to receive all the data and discard what you don’t want, which is inefficient. For that use case, the second option is better, since you can subscribe to only the nodes you want to monitor, but is very inefficient if you want to monitor them all, because you need to subscribe to N topics, where N is the number of nodes with parameters.
This is a perfect case where keys are optimal, because it gives granularity but also efficiency when doing a mass subscription. However, I think simply exposing the keyed fields feature would allow us to do this, we don’t need to put it under the hood of our namespaced topics. Also, after thinking about it a lot, and looking for common patterns in ROS 1 which would benefit from keys, I come up with very few. Some of them are important, but I don’t get the feeling that it would be a feature that could be applied all over the place to strict benefit. But that last part is just my opinion.
I agree changing other ROS features to use keyed topics is a much larger undertaking. The only thing we are specifically looking for is the ability to create a msg file which will cause them to be generated at this point. RTI’s implementation looks for the (@)key at the end of the line, so realistically just passing along comments from a msg file to the IDL would make this work. For example, our msg file is below:
string<=512 key #//(@)key
We made an update to basically pass this along so the IDL file now includes the following:
string<512> key_; (@)key string<512> value_;
}; // struct Parameter_
Dragging up an old thread, but I was curious about this part of your reply @wjwwood:
Without using keys, would this (ie: individual topics vs single topic with client-side filtering) not be something that could be worked-around (almost) by using content-based subscriptions?