Optional Fields in Message

[tfoote]

@BrannonKing

I have a use case for optional fields. I store a vector map. This map has a name, a few other fields, and list of polygons. The polygons are large. If I just want to rename the map, I want to be able to publish an update message that leaves the polygons unset. That would in turn be interpreted as “keep the current polygons I already know about” at the receiver. I don’t want a separate update message for each map field.

I think that this can relatively easily be covered by an update message with variable length arrays of changed elements for each type. Empty would imply no updates. There could also be defined additional flags for each type to tell the merge algorithm the expected policy, ‘merge’, ‘replace’, ‘drop’ etc.

@BrannonKing

JSON typically utilizes this principle: if a property is null for the serializer the property name isn’t included and, vice-versa, when deserializing absent properties don’t change the existing value in the target.

This seems to be conflating the message and the merge logic. If you have an update message and a data type you’re merging it with, the policy of keeping existing values is defined by your merging function and not the message itself or the transport of that message. And as mentioned above the merge logic could be informed by message elements.

I think part of the challenge is that there are two concepts of optional fields. One is that you can state if an element is set or unset. The other is that you can send and receive somewhat compatibly using a partial definition if the other side only has additional optional fields. (Ala what protobuf2 had but has been cut from protobuf3)

Rereading this thread it sounds like a lot of what people are looking for is not necessarily the different version compatibility but actually improved API for communicating an element is set or not. Which can be done using the [<=1] idiom, but requires you to be pretty verbose and dereference into the list etc.

if (msg.element.length() == 1)
{
  my_value = msg.element[0];
}
else 
{
  my_value = "default_value"
}

For which we could consider adding syntactic sugar to make the user faceing API a little bit easier to use.

Clearly something like std::optional would be nice. However that’s not available until C++17. And we would want to replicate this across all languages so keeping it unified would be valuable. But however it’s implemented I’d suggest that this is mostly client library syntactic sugar instead of a need to necessarily change the primitives if we use this as an extended API on top of the basic primitives we already have.