I propose adding an optional tag in front of the field type in the .msg format.
I know users generally assign a sentinel value (i.e. NaN or 0xFFFF), when there no value is present; but it would be more semantic to build it into the message.
In C++14, it could be implemented in the experimental optional implementation. Then, when the code base moves to C++17, the experimental tag could be removed.
It might be useful for making different measurements in a compatible format. For example, my 1D gyro could produce a 9D imu message with just that one optional measurement field filled in.
Can you clarify what you goals for optional values are. Do you want to distinguish an unset value from any other value? If the value is not set do you want it to not use any space on the wire? Do you want an older message definition (without the optional field) be compatible with a newer variation of the message (with the optional field added)?
I donāt understand how C++14 / C++17 is related to this and what you mean with āexperimental optional implementationā. Maybe you can clarify that part.
Do you want to distinguish an unset value from any other value?
Yes, the goal is to specify an unset or unused value from another value, but maintain compatibility.
If the value is not set do you want it to not use any space on the wire?
I donāt care about the wire representation at this point. I was concentrating on the user perspective.
Do you want an older message definition (without the optional field) be compatible with a newer variation of the message (with the optional field added)?
That would be nice and could be another way of getting the same thing. For example, somehow relating the 1D gyro to the 9D IMU message without much overhead or specification. I was mostly thinking about the inherent compatibility with one message type.
I donāt understand how C++14 / C++17 is related to this and what you mean with āexperimental optional implementationā. Maybe you can clarify that part.
Sorry, I started to get into the weeds. I like the use of std::array and std::vector in the C++ message API. I thought it would be nice to use std::optional, which is available in C++14 as a pseudo experimental feature: std::experimental::optional - cppreference.com. Basically, some compilers added it in the std::experimental namespace before it was officially released in C++17.
I would also appreciate support for optional fields in ROS2. The best use case I can currently think of is to get rid of the geometry_msgs/FooWithCovariance message types and merge them into one geometry_msgs/Foo message with an optional covariance field (not considering backwards compatibility in this case). A node that is only interested in the actual value and not in the covariance information would not need to implement different subscribers for the two types or apply shape shifting. Other tools like rviz could optionally visualize the covariance, if available.
In general, having native optional field support in many cases removes the need to define different, but very similar messages and - depending on how the presence of optional fields is encoded in the wire format - can also help to reduce wire size because unavailable data does not have to be represented by a full 8-byte NaN double.
As for a direct use case, we ran into one recently when wanting to write a ROStful (ROS-REST conversion interface) client. As a client to a REST server api, it may need to set outgoing optional fields and parse incoming optional fields. A fixed ros message is not rich enough to transmit/receive this kind of information.
The best use case I can currently think of is to get rid of the geometry_msgs/FooWithCovariance message types and merge them into one geometry_msgs/Foo message with an optional covariance field
While I agree that there may be valid use cases for optional fields, I donāt think this is one of them. In a different robot framework that I regularly use, there is no PoseStamped message, only a RigidBodyState, which is the logical conclusion of what you propose and which has these fields:
position, cov_position
orientation, cov_orientation
velocity, cov_velocity
angular_velocity, cov_angular_velocity
In contrast, I like the ROS messages (Pose, PoseStamped, PoseWithCovariance ā¦) better. The signature of the topic tells me immediately which fields are used or supposed to be filled. I donāt have to do a rostopic echo or read the code to find out, and if they later decide to use a covariance, the API changes, so my code breaks and I know I have to adjust it.
Thanks for the pointer to the changes between Protobuf 2 and Protobuf 3. A few other searches found some other interesting opinions about proto2 vs proto3 such as āall fields are optionalā and that the proto3 version is "simplerā.
Although the answer that āeverything is optionalā is a bit tongue in cheek. I think that it actually has a valuable insight. In particular for the API, some fields may not need to be filled. And this depends completely on the message documentation. Messages are both a datatype as well as a documented standard on how to fill them out. Some messages provide multiple fields but only some are considered valid based on a flag. Similarly some messages use variable length arrays as optional fields where there are some configurations where some arrays may be empty. Which is basically using the implicit flag of the length of the array, which is an introspectable property of the message.
Using the empty arrays gains the benefit of not needing to send the empty data fields. (Less the overhead inherent to communicating that the array is empty.)
Two other advantages of āoptionalā data fields are the ability to not send the data if itās not used and ābackwardsā compatibility between two versions of a message where an āoptionalā field has been added and the receiver can automatically drop the extra field that it does not know about.
Not actually sending blank data can be covered as above by using variable length arrays. And the linked article above from the Capn Proto FAQ covers how the ācompatibilityā breaks down quickly.
The other major change that supporting optional fields as protobuf does is that it requires us to switch to method based access. Such that you must query whether or not a field exists before you ask for it. Our community has shows a strong preference for having member based access to the datatypes for the ability to implement high performance methods using things like iterators without going through accessors or making a local copy that would cause significant overhead.
With the new bounded arrays, we could pick up a new common message pattern in the case that you want to have an optional field.
...
sensor_msgs/Image optional_image_field[<=1] # Optional image in this message.
...
Though when designing messages like this it needs to be kept in mind that adding this complexity will force all subscribers to handle both cases. Or if thereās multiple optional fields the number of cases can grow combinatorically.
Iām not sure that optional fields are the best approach for this. In this case Iād recommend that using the full 9D message with the appropriate zero entries in the covariance. This means that all the tools can be written against a single message. Weāve developed the same policy for 2D vs 3D poses. Where if youāre working in 2D you still use the 3D points. And then the messages can be rendered following the same code path in all the tools instead of needing to have two different code paths. This approach has proven very powerful as primatives used by the 2D navigation stack which only supports 2D navigation can be reused when people are starting to do 3D free space navigation and almost all the tools still work.
Can you clarify what you mean by this? Are you trying to represent an arbitrary REST message?
To clarify what @Daniel_Stonier was talking about, it is serializing/deserializing from json to rosmsg and vice versa (since most REST apis use json messages), in order to interface ROS systems (a robot) with a potentially useful REST API (a server somewhere).
Note the ROS system can be a client, but also a server, such as when someone writes a WebApp for a Robot UI, interfacing with ROS, using some ārobot REST APIā derived from existing ROS services.
As a Reference, here is how one would specify a message for a REST API : http://swagger.io/specification/#simple-model-87 by default all fields are optional. required fields need to be specified explicitly. Same as current protobuf design from what i understood.
An unset field has a different meaning than a field set to a default value automatically by the message type, without explicit user intentā¦
And I would personally add a even more obvious usecase : remote function call in python.
A function can have optional arguments, and since python works with references and implicit typing, a safe simple default for any type is None
Can you clarify what you mean by this? Are you trying to represent an arbitrary REST message?
Alexandre is the technical guy
More simply, yes we were trying to represent or parse an arbitrary REST message. I wanted our ROS guys to talk to an external REST service without having to know or use REST. Once the REST API is known, in most cases we can spec the ROS engineers a set of messages and services they can use that will talk to the REST service via ROStful. Unfortunately that REST service is not always something we design/control, so it will often have optional fields in it.
Vice versa also happens - if we do have control over the design of the REST api, then the REST lads are then forced to become aware of the non-optional constraint.
By the way, Iām not necessarily arguing for the increase in complexity such a feature would bring, just presenting a use case where it would have been useful.
Iām have trouble coming up with a good pub/sub use case for optional. In my example, Instead of a gyro driver producing an IMU message, it probably makes more sense to create a converter. Maybe, type masquarading and an ease of building small message converters has more value right now. Type masquerading is already on the roadmap: https://github.com/ros2/ros2/wiki/Roadmap.
I agree that remote function calls could use an optional field. Remote function calls are less common.
The protobuf history is interesting. I didnāt really get their problem. It seemed to be a testing problem, which they didnāt address. In the Capān Proto FAQ, the author mentioned protobuf 2ās required field caused parsing failures when that part of the message definition was modified. DDS is strict on message types as well and will not allow reading part of a message. So the main point seems less relevant here.
There are a lot of ways to implement optional. Mathematically, setting the covariance to 0 may work if it is a final output; but it will not work if that data needs to be fused. This post is about treating optional as a first class message feature so that the interface is explicit. When it is explicit, the user is more likely to deal with it not being set.
As a little bit of context, my main motivation for proposing this feature was this talk called ā
CppCon 2016: Ben Deane āUsing Types Effectively"
ā at https://www.youtube.com/watch?v=ojZbFIQSdl8&t=2753s. The presenter argues why C++17ās std::optional and std::variant are fundamental to types. I thought it would apply to ROS messages as well. Iād argue that we should add something like variant, but I donāt quite understand it yet.
I think that good cases both for an against optional fields have been made here and elsewhere.
To add to them, I also think that the Python experience shows the value of optional fields for function call type communication. It can give an API more expressiveness and make more explicit how it is intended to be used correctly. I agree that it can also add a lot of complexity, but at least for services that complexity may be worth it. I think it depends on how the majority of ROS users are creating services. Are they being used as remote procedure calls, or are they more often used as two-way or pull-style pub/sub communication (e.g. pull a copy of the map only when wanted)? If it is the former, then perhaps optional fields should be considered for service definitions. Of course, that raises the prospect of reducing the compatibility between .svc and .msg definitions, which I think is probably not desirable.
I have a use case for optional fields. I store a vector map. This map has a name, a few other fields, and list of polygons. The polygons are large. If I just want to rename the map, I want to be able to publish an update message that leaves the polygons unset. That would in turn be interpreted as ākeep the current polygons I already know aboutā at the receiver. I donāt want a separate update message for each map field. It seems that I would still need a separate message to update individual polygons, but I would still hope to leave fields that shouldnāt change out of the message. JSON typically utilizes this principle: if a property is null for the serializer the property name isnāt included and, vice-versa, when deserializing absent properties donāt change the existing value in the target.
I think that this can relatively easily be covered by an update message with variable length arrays of changed elements for each type. Empty would imply no updates. There could also be defined additional flags for each type to tell the merge algorithm the expected policy, āmergeā, āreplaceā, ādropā etc.
This seems to be conflating the message and the merge logic. If you have an update message and a data type youāre merging it with, the policy of keeping existing values is defined by your merging function and not the message itself or the transport of that message. And as mentioned above the merge logic could be informed by message elements.
I think part of the challenge is that there are two concepts of optional fields. One is that you can state if an element is set or unset. The other is that you can send and receive somewhat compatibly using a partial definition if the other side only has additional optional fields. (Ala what protobuf2 had but has been cut from protobuf3)
Rereading this thread it sounds like a lot of what people are looking for is not necessarily the different version compatibility but actually improved API for communicating an element is set or not. Which can be done using the [<=1] idiom, but requires you to be pretty verbose and dereference into the list etc.
For which we could consider adding syntactic sugar to make the user faceing API a little bit easier to use.
Clearly something like std::optional would be nice. However thatās not available until C++17. And we would want to replicate this across all languages so keeping it unified would be valuable. But however itās implemented Iād suggest that this is mostly client library syntactic sugar instead of a need to necessarily change the primitives if we use this as an extended API on top of the basic primitives we already have.
