Restricting communication between robots

How will ros2 work with multipel robots and restricting communication between the robots?

Is there any mechanism now that can be used? Is this supposed to be solved by the Quality of Service specification?

I want certain topics to just be available inside the robot were the network is on cable.

2 Likes

You can set the DDS Domain ID. On ROS2 it’s called ROS_DOMAIN_ID (Not sure about the spelling)
I think it should be enough to export it as an environment variable.

For that to work I think you need to be able to set the domain id on a topic. Is that possible? Or have different namespace for internal and external messages and possibility to set domain id and QoS per namespace and topic.

We also have the caser were data sent intenrlly have different QoS specification than data sent externally. Is that supported in any way?

Ahh sry, didn’t see you wanted certain topics for both robots available. In this case I think using namespaces for the other topics should be the thing you are looking for.

In theory DDS supports applications that use multiple DDS Domain IDs but I don’t think ROS2 allows this.

But even if I use different topics somebody can start a subscription by mistake and kill the wifi.

The solution I am thinking of is:

  • Use namespace topics inside the robot
  • For all inter robot communication use topics without namespace

I then just have to trust that nobody subscribes on the namespace topics between robots. What I wantes was a techical way to make it impossible.

If you want to enforce connection limits we are in the process of adding support for the DDS Security specification. That would allow you to prevent connections between the robots if you deploy certificates appropriately.

1 Like

A DDS person would tell you to use partitions, probably. For us, since we’re currently using DDS partitions in our implementation and don’t expose them through the ROS 2 api, I would advise you to just use ROS namespaces. We’re planning on having more sophisticated at runtime and during runtime topic manipulation to help with this.

I gained a lot of insight on what would be needed to deal with situations like this when working on the capabilities package as part of the rocon project. I think with some more options to configure the namespaces (remapping and aliasing) it will be a more tractable problem, but we don’t have those features yet.

A stronger separation would be to have different domains and have some bridges to expose parts of one domain to another.

What about a capability to specify that a specific namespace uses a specific domain? It sounds clunky, but if a neat way could be found to do it, that would give proper separation.

Has there been any advancement in this regard?

Specifically I believe there are two issues here, how to handle multiple independent robots on a same physical network, and how to have some sort of multirobot communication.

Regarding the first issue, from the comments above it it looks like either setting a different ROS_DOMAIN_ID or using namespaces might be enough.

But each seems to be either limited or cumbersome when compared to the old “use different ROS_MASTER_URI”.

If we need a separate domain id per physical robot, as more robots are distributed, having non-conflicting domain ids seems extremely difficult.

On the other hand, using a different namespace for each robot’s nodes and topics might do the trick, but in my opinion it needs to be implicitly supported in ROS2, so that it can be transparent to the user after configuring it.

Let’s say you want to distribute an rviz2 configuration for visualizing the tf data of the Robot Foo, you add a display of /tf.

If you have two of these robots in your office, foo-1 and foo-2, in ROS1 you’d export the ROS_MASTER_URI to the corresponding hostname and open rviz.

There needs to be a way of doing this so that when the user opens rviz (or any other tool, or launches a node of it’s own), it doesn’t need to change the topic manually.

Is there anything that I’m missing?

As a DDS person and someone who built distributed industrial robotics systems I can tell you that partitions are by far the best way to go when trying to build such a system. DDS partitions are hierarchical and support wildcards.

So the general approach is to publish all robot internal data on a partition called ‘robot-X’.

It gets more complicated when other systems like a global TF is involved. So a robot might want to know all local TFs and TFs provided by a external map server but not TFs from other robots. In this case the robot can subscribe to its local partition and to a global partition.

Also, the use case of remote diagnostic systems is covered as they can on demand access all topics of a specific robot by just accessing the proper partition.

I think it is really important to remove dependency of ROS 2 from DDS partitions and open them up for usage of the end user.

To be clear, as of the Bouncy release of ROS2, we have stopped using partitions for ROS2 internal use. That means they now should be available for end-user usage (though there are no provisions for them in the RMW API right now, so it would have to be configured “outside” of ROS2).

Great news! Thank you!

What is the current status of this? We’re starting to need to commit to an approach for our work on this, and it’s not clear what is best between namespaces, partitions, and domains. Given our specific needs, our instinct is to use a separate domain for inter-robot vs intra-robot communication, but this is not easily possible as I don’t think the ROS layer currently exposes the ability to participate in more than one domain from a single process (though it looks like CycloneDDS does support it as of v0.5, released Nov 2019).

What are others doing?

2 Likes

This is a very common thing that comes up in Navigatoin2-land as people are deploying fleets and I’d also be interested in knowing some best practices around this. Perhaps some folks at Open Robotics that considered multi-robot requirement in the design phase would be able to chime in. @Katherine_Scott could you connect us with the right people?

Your suggestion sounds like the most reasonable thing I’ve considered as well, but it would be awesome to be able to have an approved list of topics to share with other domains (ex. /tf /robot_pose /task_id /path etc).

If we can create some common design patterns here, it would be invaluable to developing standards and capabilities in the mobile robot ecosystem around multirobot tooling (which we can’t do much about today because there’s no recommendd pattern most people follow).

2 Likes

Hi all,

I’d like to bring our point of view from the Fast DDS development team. I apologize if the post is a bit long, but I think there are many things to consider here.

DDS domains concern the DomainParticipant entities, and they physically limit the discovery of peer DomainParticipants when using Simple Discovery Protocol (the standard discovery protocol in the DDS specification) by assigning different multicast ports for DomainParticipants in different domains. This way, announcements from DomainParticipants of one domain are not received by DomainParticipants in other domains. The way this applies to ROS 2 (since Foxy) is that each context has one DDS DomainParticipant entity, meaning that the domains apply to ROS 2 contexts (not nodes, which “live” inside a context).

Partitions on the other hand represent a logical isolation within a domain at the DDS Publisher/Subscriber level. This means that all the entities in the domain discover each other, but publishers and subscribers do not match unless they share at least one partition. Whereas the domain is set on DomainParticipant creation and it is not mutable, partitions can change at run time, meaning that a publisher/subscriber can be included in any partition at any time, which will trigger the matching mechanism with the entities in that new partition.

With these two concepts in place, it is possible to create different “sub-networks” using domains, and different “allowed” connections within a domain using partitions.

While domain configuration can be done on ROS 2 using an environment variable, partitions are not that straightforward to set up since there is no API (to my knowledge) to do that. However, Fast DDS offers a “tangencial” way of configuring the underlying DDS entities using XML configuration files (where you can configure partitions). This is explained here. However, this does not come without limitations that have yet to be worked out:

  1. Since ROS 2 does not offer the possibility to create the context, publisher, or subscription with a user defined name, the only thing that can be done is to set a default XML profile for each entity kind (mind that it is possible to load different XML files for different applications, since the file is defined using an environment variable FASTRTPS_DEFAULT_PROFILES_FILE). This of course limits the number of entities that can be configured this way, since the call to ros2 run in one terminal session will configure all the DomainParticipants (ROS 2 context), DataWriters (ROS 2 publishers), and DataReaders (ROS 2 subscriptions) in the same manner.

  2. Starting in ROS 2 Foxy, there is only one DomainParticipant per context (before there was one per node). This means that using XML profiles to configure nodes can only be done for the case of one node per context.

From my point of view, we are in deep need of APIs to fully configure the underlying DDS entities from the application layer, since it is often the case that users would benefit from DDS functionalities that are already available on the DDS implementations, but are not accessible from ROS 2.

Regarding the good old ROS_MASTER_URI, we have been collaborating with OpenRobotics to bring a similar functionality to ROS 2 using our own alternative discovery protocol that we call “Discovery Server”. This functionality is already available on ROS 2 Eloquent testing branch and will make its way into Foxy in the upcoming weeks. With it, you can instantiate a Server much as the ROS 1 master, and then configure your nodes as clients using an environment variable. Even without the full mechanism in place in Foxy (executable + environment variable), you can still use the Discovery Server protocol, since Foxy’s Fast DDS version has it. I gave a presentation on how to do that for ROS 2 Eloquent in the last ROS Developers Day and you can see the video here (from 5:30:10 to 6:13:00). Stay tuned!

5 Likes

A recent-ish comment by @spiderkeys on ros2/design#261 offers some insight into how they are setting up multi-robot systems:

1 Like

Doing this would require first deciding that we want to lock ROS 2 to only use DDS as its middleware.

The alternative would be to decide what concepts we want at the ROS 2 level (such as the ability to do hard and soft partitions of the graph) and then design the mapping of those to DDS concepts for provision by each RMW.

The former is less work but the latter offers greater flexibility and capability to use non-DDS middlewares. However the latter would have to involve some level of compromise and feature reduction.

Agreed, but at some point I think there hits a point where the direction we’re going indicates a distinctly DDS approach, especially if making that decision can make ROS2 more easily feature complete (as this topic would want) or faster (that @joespeed thinks we can make the ROS2 layers much more streamline that way).

A middle compromise would be to have the RMW’s be required to be DDS but also supply a shim RMW that a less-complete communication middleware could use to implement the most basic things and the shim provides the rest. That way we can push that overhead out of the ROS 2 stack for most users and the overhead is only seen by users that have to use it. We do this in navigation alot by having plugins loading plugins. We have a non-DDS plugin that fits inside of a generic RMW container.

1 Like

I think you bring a very good point, it would be ideal to do this in a way that all kinds of middlewares can benefit from it, and it is definitely a very interesting discussion we should have.

There is still the issue of taking full advantage of the specific middleware implementation the application is using. All implementations have their different tweaks for different use-cases (apart from the standard ones). Maybe we can think of a way to configure those from the application layer in a way that is common to all of them. We could for example have key-value pairs that are pushed down and interpreted by each RMW implementation. We could standardize some of the keys to the concepts we deem common, but leave the implementation to decide on what to add there. Of course, this approach would require documentation for each specific RMW, but I think advanced users and industrial deployments would greatly benefit from being able to configure every single aspect of the middlewares.

With that approach (which is probably one of many possibilities), we could configure middleware implementations at build time, or even at run time loading some JSON file, which would make very easy for users to test different configurations without rebuilding their application every time.

We’ve discussed a similar idea in a few recent design discussions, but not a shim as such.

What we’ve discussed is providing default implementations for some of the more specialised features that we want to use - content-based filtering being a particularly good example. An RMW can then either provide support for that feature using its middleware, load in the ROS-provided implementation, or provide that feature in some other way of its choosing.

This approach is more flexible than a monolithic shim RMW, but it requires designing some APIs.