Discovery problems with the discovery server ;-)

A while ago, a new discovery server was announced, which aims to reduce discovery traffic and improve reliability of discovery over WiFi.

However, this servers appears to come with some limitations which are quite serious in practice and do seem unintentional. A typical issue is reported here:
ros2 topic empty with fastdds discovery server - ROS Answers: Open Source Q&A Forum To sum this up, ros2 topic list is empty when using the discovery server, despite topics being published successfully. Similarly, when calling ros2 node info on an existing node, you get no topic data for this node.

It is interesting to observe that when you do ros2 topic echo /chatter while the listener/talker example is running, you get a “this topic does not seem to be published” message, but when you do a ros2 topic echo /chatter std_msgs/msg/String (that is, in addition you provide the type of the topic), you successfully retrieve the data. This explains why the listener/talker examples work, but ros2 topic list doesn’t.

All of this leads me to suspect that the filtering approach that has been added in December 2020 is enabled by default, instead of being turned off by default as most people would expect.

It would be great if someone from eprosima, maybe @EduPonz or @Katrin_Kellner, could comment on this, because it greatly reduces the practical utility of the discovery server.

4 Likes

@Ingo_Lutkebohle thanks for posing information. btw, do we have issue for these problems on github? i’d like to keep my eyes on those.

For what it is worth, https://github.com/ros2/ros2_documentation/pull/1028 describes how to setup the ROS 2 daemon to work appropriately with the Fast-DDS Discovery Server. We haven’t merged it yet since it is a draft, but it should give some further information.

It would be good to know if the procedures in that pull request address @Ingo_Lutkebohle concerns.

2 Likes

Hi @Ingo_Lutkebohle,

As @clalancette correctly points out, ROS 2 CLI commands query (not using DDS) the ROS 2 daemon to get the information they display. The daemon is a ROS 2 node that monitors the discovery information in the domain to keep a record and maintain the node graph. Because of that, the daemon itself needs to be configured properly in order for the ROS 2 CLI to work as expected. Apart from the PR to ROS 2 documentation, Fast DDS documentation has a detailed guide on how to set ROS 2 up for Discovery Server usage: 16.2. Use ROS 2 with Fast-DDS Discovery Server — Fast DDS 2.3.0 documentation. You’d be specially interested in the ROS 2 instrospection.

Please do give it a go a tell us whether everything works and whether the documentation is detailed enough to follow.

Thanks!

1 Like

Hey @EduPonz and @clalancette , thanks for the additional information.

Using the SUPER_CLIENT profile, it works but only on rolling!

On foxy, I get this error instead when running ros2 topic list --no-daemon with the super_client profile:

2021-04-14 09:18:05.453 [XMLPARSER Error] Node 'discoveryProtocol' with bad content -> Function getXMLEnum
2021-04-14 09:18:05.453 [XMLPARSER Error] Error parsing participant profile -> Function parseXMLParticipantProf
2021-04-14 09:18:05.453 [XMLPARSER Error] Error parsing profile's tag participant -> Function parseProfiles
2021-04-14 09:18:05.453 [XMLPARSER Error] Error parsing 'super_client_configuration_file.xml' -> Function loadXMLFile

In my foxy install, which I just updated, libfastrtps.so.2.0.2 is used, while rolling uses fastrtps.so.2.3. That probably explains the difference. libfastrtps 2.0.2 doesn’t seem to understand the option.

If I may offer a suggestion here: It would make people’s life much easier if SUPER_CLIENT mode would be the default. This already reduces discovery traffic and moreover, the discovery server also helps with multicast-denied networks (which is my current motivation for using it). Then, people who really need to reduce discovery traffic even further, they could still enable filtered mode.

2 Likes

Hi @Ingo_Lutkebohle,

As you pointed out, SUPER_CLIENT is only present in ROS 2 Rolling (for now). It was first introduced in Fast DDS v2.3.0, and then it was backported to Fast DDS 2.1.x branch. Currently, it is included in Fast DDS v2.2.1, which is intended to be shipped with ROS 2 Foxy in the next patch release (Upgrade Fast DDS to v2.1.1 by EduPonz · Pull Request #1114 · ros2/ros2 · GitHub). This means that SUPER_CLIENT will be available in ROS 2 Foxy soon.

On the mean time, it is still possible to use ROS CLI with discovery server by taking a different approach. The solution is explained here. It entails either setting the ROS 2 daemon itself as the SERVER (which is what that tutorial does), or setting the ROS 2 daemon as a SERVER which connects to your main SERVER (you can see and example on how to do that here). I personally recommend setting a main SERVER elsewhere, and then configure the daemon as a SERVER that connects to the main one. This is because the ROS 2 daemon automatically shuts down after 2 hours of inactivity, which means that it can become unavailable after some time.

This is an open internal discussion for us. Right now it is possible to instantiate SERVERs using Fast DDS CLI, and the CLIENTS can be configured using a simple environment variable, no XML needed. Instead of making SUPER_CLIENT the default, what we had in mind is to use this env variable to somehow indicate that the participant should be a SUPER_CLIENT instead of a CLIENT. We are more than open to suggestions from your side on how to specify this in the easier, most understandable manner.

Hi @EduPonz,

great to hear that we can expect this in foxy as well.

Instead of making SUPER_CLIENT the default, what we had in mind is to use this env variable to somehow indicate that the participant should be a SUPER_CLIENT instead of a CLIENT. We are more than open to suggestions from your side on how to specify this in the easier, most understandable manner.

I appreciate that you’re trying to make this easier, but I’m not sure that overloading that environment variable is ideal. People will have to know about this in the first place, which is where most of the problem usually is :wink: I mean, I did try to look at the docs first, and I didn’t find the right ones. What I did find was a ROS answers question with no answer, leading me to believe that this is an unsolved issue.

The point of my suggestion to make SUPER_CLIENT default is that for people who don’t bother to read everything, it will still work as expected and only people who want to optimize totally will have to read up more. I mean, I’m not trying to advocate laziness, just the principle of least surprise.

4 Likes