ROS Resources: Documentation | Support | Discussion Forum | Service Status | Q&A answers.ros.org

Optional Fields in Message

I was taking a look at https://github.com/ros2/ros2/wiki/Beta1-Overview and http://design.ros2.org/articles/interface_definition.html. I noticed that there were additions to the .msg format and I started thinking of another addition.

I propose adding an optional tag in front of the field type in the .msg format.

I know users generally assign a sentinel value (i.e. NaN or 0xFFFF), when there no value is present; but it would be more semantic to build it into the message.

In C++14, it could be implemented in the experimental optional implementation. Then, when the code base moves to C++17, the experimental tag could be removed.

It might be useful for making different measurements in a compatible format. For example, my 1D gyro could produce a 9D imu message with just that one optional measurement field filled in.

Can you clarify what you goals for optional values are. Do you want to distinguish an unset value from any other value? If the value is not set do you want it to not use any space on the wire? Do you want an older message definition (without the optional field) be compatible with a newer variation of the message (with the optional field added)?

I don’t understand how C++14 / C++17 is related to this and what you mean with “experimental optional implementation”. Maybe you can clarify that part.

1 Like

Do you want to distinguish an unset value from any other value?

Yes, the goal is to specify an unset or unused value from another value, but maintain compatibility.

If the value is not set do you want it to not use any space on the wire?

I don’t care about the wire representation at this point. I was concentrating on the user perspective.

Do you want an older message definition (without the optional field) be compatible with a newer variation of the message (with the optional field added)?

That would be nice and could be another way of getting the same thing. For example, somehow relating the 1D gyro to the 9D IMU message without much overhead or specification. I was mostly thinking about the inherent compatibility with one message type.

I don’t understand how C++14 / C++17 is related to this and what you mean with “experimental optional implementation”. Maybe you can clarify that part.

Sorry, I started to get into the weeds. I like the use of std::array and std::vector in the C++ message API. I thought it would be nice to use std::optional, which is available in C++14 as a pseudo experimental feature: http://en.cppreference.com/w/cpp/experimental/optional. Basically, some compilers added it in the std::experimental namespace before it was officially released in C++17.

+1

I would also appreciate support for optional fields in ROS2. The best use case I can currently think of is to get rid of the geometry_msgs/FooWithCovariance message types and merge them into one geometry_msgs/Foo message with an optional covariance field (not considering backwards compatibility in this case). A node that is only interested in the actual value and not in the covariance information would not need to implement different subscribers for the two types or apply shape shifting. Other tools like rviz could optionally visualize the covariance, if available.

In general, having native optional field support in many cases removes the need to define different, but very similar messages and - depending on how the presence of optional fields is encoded in the wire format - can also help to reduce wire size because unavailable data does not have to be represented by a full 8-byte NaN double.

Protobuf 2 also has optional (an required) fields, but apparently this feature has been removed in Protobuf 3 for a good reason…

The Capn Proto FAQ has a rant about the required field that elucidates (in part) the history behind it.

Note also that both Capn Proto and Protocol Buffers 3 still support the notion of optional - Capn Proto via unions and Protocol Buffers 3 via OneOf.

As for a direct use case, we ran into one recently when wanting to write a ROStful (ROS-REST conversion interface) client. As a client to a REST server api, it may need to set outgoing optional fields and parse incoming optional fields. A fixed ros message is not rich enough to transmit/receive this kind of information.

The best use case I can currently think of is to get rid of the geometry_msgs/FooWithCovariance message types and merge them into one geometry_msgs/Foo message with an optional covariance field

While I agree that there may be valid use cases for optional fields, I don’t think this is one of them. In a different robot framework that I regularly use, there is no PoseStamped message, only a RigidBodyState, which is the logical conclusion of what you propose and which has these fields:

  • position, cov_position
  • orientation, cov_orientation
  • velocity, cov_velocity
  • angular_velocity, cov_angular_velocity

In contrast, I like the ROS messages (Pose, PoseStamped, PoseWithCovariance …) better. The signature of the topic tells me immediately which fields are used or supposed to be filled. I don’t have to do a rostopic echo or read the code to find out, and if they later decide to use a covariance, the API changes, so my code breaks and I know I have to adjust it.

1 Like

Thanks for the pointer to the changes between Protobuf 2 and Protobuf 3. A few other searches found some other interesting opinions about proto2 vs proto3 such as “all fields are optional” and that the proto3 version is "simpler’.

Although the answer that “everything is optional” is a bit tongue in cheek. I think that it actually has a valuable insight. In particular for the API, some fields may not need to be filled. And this depends completely on the message documentation. Messages are both a datatype as well as a documented standard on how to fill them out. Some messages provide multiple fields but only some are considered valid based on a flag. Similarly some messages use variable length arrays as optional fields where there are some configurations where some arrays may be empty. Which is basically using the implicit flag of the length of the array, which is an introspectable property of the message.

Using the empty arrays gains the benefit of not needing to send the empty data fields. (Less the overhead inherent to communicating that the array is empty.)

Two other advantages of “optional” data fields are the ability to not send the data if it’s not used and “backwards” compatibility between two versions of a message where an “optional” field has been added and the receiver can automatically drop the extra field that it does not know about.

Not actually sending blank data can be covered as above by using variable length arrays. And the linked article above from the Capn Proto FAQ covers how the “compatibility” breaks down quickly.

The other major change that supporting optional fields as protobuf does is that it requires us to switch to method based access. Such that you must query whether or not a field exists before you ask for it. Our community has shows a strong preference for having member based access to the datatypes for the ability to implement high performance methods using things like iterators without going through accessors or making a local copy that would cause significant overhead.

With the new bounded arrays, we could pick up a new common message pattern in the case that you want to have an optional field.

...
sensor_msgs/Image optional_image_field[<=1]  # Optional image in this message.
...

Though when designing messages like this it needs to be kept in mind that adding this complexity will force all subscribers to handle both cases. Or if there’s multiple optional fields the number of cases can grow combinatorically.

I’m not sure that optional fields are the best approach for this. In this case I’d recommend that using the full 9D message with the appropriate zero entries in the covariance. This means that all the tools can be written against a single message. We’ve developed the same policy for 2D vs 3D poses. Where if you’re working in 2D you still use the 3D points. And then the messages can be rendered following the same code path in all the tools instead of needing to have two different code paths. This approach has proven very powerful as primatives used by the 2D navigation stack which only supports 2D navigation can be reused when people are starting to do 3D free space navigation and almost all the tools still work.

Can you clarify what you mean by this? Are you trying to represent an arbitrary REST message?

1 Like

To clarify what @Daniel_Stonier was talking about, it is serializing/deserializing from json to rosmsg and vice versa (since most REST apis use json messages), in order to interface ROS systems (a robot) with a potentially useful REST API (a server somewhere).

Note the ROS system can be a client, but also a server, such as when someone writes a WebApp for a Robot UI, interfacing with ROS, using some “robot REST API” derived from existing ROS services.

As a Reference, here is how one would specify a message for a REST API : http://swagger.io/specification/#simple-model-87 by default all fields are optional. required fields need to be specified explicitly. Same as current protobuf design from what i understood.

An unset field has a different meaning than a field set to a default value automatically by the message type, without explicit user intent…

And I would personally add a even more obvious usecase : remote function call in python.
A function can have optional arguments, and since python works with references and implicit typing, a safe simple default for any type is None

if I have a function :

def do_that(fibnum, timeout = None): 
  return fibonnacci_compute(fib_num, timeout)

and I want to expose it to other nodes via a service :

float timeout
---
int fib_result
  • sending timeout == 0 (how the ROS1 msg type __init__ function initialize it) means return right away.
  • sending timeout == None (how the python api is built) means no timeout, ie. run for ever.

Nullable default arguments are core part of the language so it is not unusual to see such APIs in python.

So I think having some standard way to represent a nullable/optional field in a message would be very useful.

Can you clarify what you mean by this? Are you trying to represent an arbitrary REST message?

Alexandre is the technical guy :slight_smile:

More simply, yes we were trying to represent or parse an arbitrary REST message. I wanted our ROS guys to talk to an external REST service without having to know or use REST. Once the REST API is known, in most cases we can spec the ROS engineers a set of messages and services they can use that will talk to the REST service via ROStful. Unfortunately that REST service is not always something we design/control, so it will often have optional fields in it.

Vice versa also happens - if we do have control over the design of the REST api, then the REST lads are then forced to become aware of the non-optional constraint.

By the way, I’m not necessarily arguing for the increase in complexity such a feature would bring, just presenting a use case where it would have been useful.

1 Like

I’m have trouble coming up with a good pub/sub use case for optional. In my example, Instead of a gyro driver producing an IMU message, it probably makes more sense to create a converter. Maybe, type masquarading and an ease of building small message converters has more value right now. Type masquerading is already on the roadmap: https://github.com/ros2/ros2/wiki/Roadmap.

I agree that remote function calls could use an optional field. Remote function calls are less common.

The protobuf history is interesting. I didn’t really get their problem. It seemed to be a testing problem, which they didn’t address. In the Cap’n Proto FAQ, the author mentioned protobuf 2’s required field caused parsing failures when that part of the message definition was modified. DDS is strict on message types as well and will not allow reading part of a message. So the main point seems less relevant here.

There are a lot of ways to implement optional. Mathematically, setting the covariance to 0 may work if it is a final output; but it will not work if that data needs to be fused. This post is about treating optional as a first class message feature so that the interface is explicit. When it is explicit, the user is more likely to deal with it not being set.

As a little bit of context, my main motivation for proposing this feature was this talk called ‘
CppCon 2016: Ben Deane “Using Types Effectively"
’ at https://www.youtube.com/watch?v=ojZbFIQSdl8&t=2753s. The presenter argues why C++17’s std::optional and std::variant are fundamental to types. I thought it would apply to ROS messages as well. I’d argue that we should add something like variant, but I don’t quite understand it yet.

I think that good cases both for an against optional fields have been made here and elsewhere.

To add to them, I also think that the Python experience shows the value of optional fields for function call type communication. It can give an API more expressiveness and make more explicit how it is intended to be used correctly. I agree that it can also add a lot of complexity, but at least for services that complexity may be worth it. I think it depends on how the majority of ROS users are creating services. Are they being used as remote procedure calls, or are they more often used as two-way or pull-style pub/sub communication (e.g. pull a copy of the map only when wanted)? If it is the former, then perhaps optional fields should be considered for service definitions. Of course, that raises the prospect of reducing the compatibility between .svc and .msg definitions, which I think is probably not desirable.

I have a use case for optional fields. I store a vector map. This map has a name, a few other fields, and list of polygons. The polygons are large. If I just want to rename the map, I want to be able to publish an update message that leaves the polygons unset. That would in turn be interpreted as “keep the current polygons I already know about” at the receiver. I don’t want a separate update message for each map field. It seems that I would still need a separate message to update individual polygons, but I would still hope to leave fields that shouldn’t change out of the message. JSON typically utilizes this principle: if a property is null for the serializer the property name isn’t included and, vice-versa, when deserializing absent properties don’t change the existing value in the target.

I think that this can relatively easily be covered by an update message with variable length arrays of changed elements for each type. Empty would imply no updates. There could also be defined additional flags for each type to tell the merge algorithm the expected policy, ‘merge’, ‘replace’, ‘drop’ etc.

This seems to be conflating the message and the merge logic. If you have an update message and a data type you’re merging it with, the policy of keeping existing values is defined by your merging function and not the message itself or the transport of that message. And as mentioned above the merge logic could be informed by message elements.

I think part of the challenge is that there are two concepts of optional fields. One is that you can state if an element is set or unset. The other is that you can send and receive somewhat compatibly using a partial definition if the other side only has additional optional fields. (Ala what protobuf2 had but has been cut from protobuf3)

Rereading this thread it sounds like a lot of what people are looking for is not necessarily the different version compatibility but actually improved API for communicating an element is set or not. Which can be done using the [<=1] idiom, but requires you to be pretty verbose and dereference into the list etc.

if (msg.element.length() == 1)
{
  my_value = msg.element[0];
}
else 
{
  my_value = "default_value"
}

For which we could consider adding syntactic sugar to make the user faceing API a little bit easier to use.

Clearly something like std::optional would be nice. However that’s not available until C++17. And we would want to replicate this across all languages so keeping it unified would be valuable. But however it’s implemented I’d suggest that this is mostly client library syntactic sugar instead of a need to necessarily change the primitives if we use this as an extended API on top of the basic primitives we already have.

1 Like