How to gracefully deprecate old interface (msg/srv) in favor of a new interface?

General
1

When we find that a specific message/service can’t be scaled, it’s time to move to a new format. Current approach is:

  1. Create a new file, name it <old_name><increment_version>.{msg,srv,action}. Eg: MyMessage.msg → MyMessage2.msg
  2. Add a new publisher/server in the existing name under some namespace. Eg: my_node/my_topic → my_node/my_topic/format2
  3. Move the old publisher/server to a similar namespace. Eg: my_node/my_topic → my_node/my_topic/format1
  4. Add a parameter to launch file to rename one of them to the old resource location: my_node/my_topic, default: old format
  5. Wait for a while, get everyone to switch to new format, and toggle the launch behavior
  6. Wait longer and remove the old format, the launch file toggle, and the old interface file

This works quite well, BUT has some glaring gaps:

  • Manual notice required for migration
  • Toggling the launch file behavior needs to be done in a full replica of the production environment
  • The launch toggle isn’t easily discoverable by the consumers of the API
  • To discover the change in format, consumers need to run ros2 topic list | grep 'format[0-9]'

This brings me to my questions:

  1. Is there a better way to deprecate interfaces in ROS2? :pray:
  2. Is there a better way to inform the consumers of an API that the interface format is deprecated? Eg: at compile time for the compile happy folks
  3. Is there a way to discover (at runtime/code analysis) all connections (clients+servers and pub+sub) which are using the deprecated format? :thinking: This will allow creating some tooling/automated detection

Out of scope:

  • Can we add a way to version in the interface file itself? (That’s protobuf land)
  • How do we convert between old and new format? (:sparkles: magic :sparkles: )

(Question is valid for ROS1, but it isn’t expected to see new development)

4 Likes
2

In the subscriber case there may be a clean solution by changing the interface to use the rospy.AnyMsg functionality, which would allow connections of any kind so that you can then check for outdated interface usage (to print a warning) before converting to the appropriate message class.

So in ROS1 Python, that might be something like:

import rospy
from roslib.message import get_message_class

class MyNode:
    def __init__(self):
        rospy.Subscriber("topic", rospy.AnyMsg, self.callback)

    def callback(self, msg):
        msg_class = get_message_class(msg._connection_header["type"])
        deserialized_msg = msg_class().deserialize(msg._buff)
        if type(msg_class) is old_msg_class:
            rospy.logerr_once("Please use new_msg_class")
            self.old_callback(deserialized_msg)
        else:
            self.new_callback(deserialized_msg)

    def old_callback(self, msg):
        return # some deprecated functionality

    def new_callback(self, msg):
        return # some new functionality

This allows you to keep the same topic for two different message types, which is neat. And it happens once at call-time, which has the advantage of not requiring a compile to check: even libraries that are installed via a package manager or python will display the appropriate error.

But, I’ll admit that my approach here has a handful of problems:

  1. This is ROS1, you asked about ROS2 (although I’m sure there are ways of doing this in ROS2 that I’m just less familiar with)
  2. This only works for subscribers, it’s not clear to me what the equivalent “AnySrv” might be
  3. This works for when you’re the destination, but what if you’re the source? If we were publishing here, do we need to reach out to the existing subscribers, figure out what they want, and conform to that?

But, I think those are all solvable problems. There might be room for a handy little library that performs this functionality… If I get a chance to do some hacking on that later this week I’ll follow up.

1 Like
3

We can definitely solve 1 issue:

  • Monitor in production if the setup is still using old interface

If we have a config of topics to monitor (so manual…) and some conventions, it could be used in tests to monitor migration progress.

But I hope you get time to hack a little since usually the interface break is driven by the publisher. ROS1 solution is fine too, since it would provide the general direction at least. Thanks for the detailed breakdown :slight_smile:

4

Versioning of the interfaces can be done with the semantic versioning in a package.xml file: increment major version for breaking backwards compatibility or increment minor version to introduce new interface or extend current ones.

In case of a roadblock with current interfaces my team decided just to extend them (by literally adding new fields to their definitions), marking them for cleanup for next major release.

1 Like
5

I noticed that the package.xml based versioning isn’t enforced? Or was that just colcon_catkin_interface which has the issue and not ament? (Or was I using a super old version while testing?)

6

This + versioning would work in most cases where ROS API is the only one that matters, but doing so is a breakage of ABI (for sure) and API (in case this is used in public headers) so would necessitate a major release of the msgs package + update in all rdeps due to that. Did you face issues with the API/ABI due to this strategy?
(In my usecase, I have a feeling that careful restructuring of the msgs might be needed for this to be a policy)

How do you propagate this information to the ROS clients? I can think of just 1 way:

  • Update the major version and wait for them to find out :laughing:
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.