ROS graph information tools implementation discussion

marguedas · 2016-10-19T18:00:16.402Z

Glad to see that many people manifested interest in this discussion.
As expected these tools are needed by everybody and it resulted on a lot of prototype solutions being created by each and everyone of us. Hence the need for this discussion to provide a global rmw agnostic solution for these tools.

Thanks @gbiggs for this great summary.

A couple notes / open questions on top of what has been stated:

Agreed on @dirk-thomas comment that the daemon should be seen as a way to provide information faster rather than as a single-point of failure. The feature should hence work without it (at a performance cost)
on the fact that the ROS side of things should be rmw_implementation agnostic and allow users to develop their own tools or daemons using the protocol of their choice.
Protocol-wise I’d be curious to know what the preference of all of you is. It seems to me that REST services became very popular now that most people provide web interfaces to their systems but would it be your preferred choice ?
In the case of multiple juxtaposed ROS systems, it comes down to how do we identify “different” ROS systems. An approach could be to have the daemon aggregates all graph information for a given DDS Domain. And thus create a deamon for each DDS Domain used. What other approach for differentiating ROS systems could be used ?
Another point being addressed is the tradeoff between duplicating daemons (and information storage) on each host vs having a single global daemon and service. If the default protocol allows it, this could be left up to the end user by providing an option to specify when and how to launch the daemon. And the way to query daemon information should be adapted accordingly. We will still need to specify the default behavior for this.

gbiggs · 2016-10-20T06:48:51.492Z

dirk-thomas:

Imo all of the following cases should work:

The daemon got started before (by launch, the system, wherever, this should be an implementation detail).

The daemon gets started on demand by the first invocation of a command line tool needing it (slower on first call).

No daemon is there but the tool should still provide reasonable results (trade off between wait time and completeness).

The command line tools are only one use case of that interface. It should be possible to write code on-top of it which accesses the information without caring about the internal optimization of having a daemon. So all the functionality should be exposed in an API which is then also being used to implement the cli.

Thanks for enumerating those. I agree whole-heartedly with all of them. (Especially the last! I implement my tools like that and it’s been very useful.)

dirk-thomas:

I am not convinced that the daemon should have a ROS-based interface.In order to call its service the client would still need to wait for some discovery phase. Even if that is local it will take additional time which we want to avoid.
Therefore I think the daemon should provide its interface using a different protocol. That protocol needs to work without any “discover” and be available across all targeted platforms.

I strongly support having a non-ROS based interface. You’ve listed the advantages already. However perhaps it could be useful to have a ROS-based interface in case someone wants to use the functionality from a node? Perhaps this should be a lower-priority feature unless we find an actual use case for it.

dirk-thomas:

With the choice of the protocol comes also the decision how that daemon is being identified. It could be one single daemon per ROS graph or it could be one daemon per host to make the queries “local”. If the daemon would e.g. expose its interface via d-bus you would probably run it on each host. Someone mentioned a REST service it could be either a single global one or a local one per host. Anyway we need to determine how the daemon is being referred to in the context of running to separate ROS systems on a single machine.

Yes, there are several ways that this could go. I think that if we use an environment variable that we can build an automatic mechanism without too much trouble. For example, the rostopic tool/library could look for an environment variable defining the graph information daemon’s address, and if it isn’t defined attempt to contact one on the local machine. If that doesn’t work then it could request the daemon be launched on the local machine.

dirk-thomas:

Any of these tools and interfaces should be independent of any specific rmw implementation. If the rmw interface is being implemented by the newest and hotest discovery / marshalling / transport solution the tools should continue to work. The rmw interface might need to be extended to provide all the necessary information.

Absolutely agree. This is another benefit of hiding it behind a well-known service description, I think: It becomes easier to abstract away the implementation. Then we can look into using existing technologies such as those @Barrett_Strausser mentioned and see if they do what we need already.

marguedas:

Protocol-wise I’d be curious to know what the preference of all of you is. It seems to me that REST services became very popular now that most people provide web interfaces to their systems but would it be your preferred choice ?

I’m in favour of a REST service as the initial goal. I’ve also been told by someone locally who does a lot of tool implementations, using the browser as an interface, that he wants “everything” to be available over REST. I’m assuming he’s talking about introspection facilities, not absolutely everything.

However, I think the information should be available on other protocols from the same daemon if someone wants it that way. If we specify the data structure and the protocol used to access it separately then it should be relatively easy to add additional access methods.

marguedas:

In the case of multiple juxtaposed ROS systems, it comes down to how do we identify “different” ROS systems. An approach could be to have the daemon aggregates all graph information for a given DDS Domain. And thus create a deamon for each DDS Domain used. What other approach for differentiating ROS systems could be used ?

Being able to configure which domains the daemon aggregates data for, and run different daemons for different domains, sounds like a sensible use case. Especially with the potential for using domains for security zone partitioning.

iluetkeb · 2016-10-20T08:45:26.373Z

I would check whether requiring a non-ROS (which, in this case I take to mean a non-DDS) interface is strictly necessary. Maybe we can avoid the discovery phase by other means, e.g. by pre-supplying the necessary information, assuming that the daemon runs locally (which other means would also have to assume).

My reason for that is to avoid duplicating communication code and functionality. While at first it might appear simple to open a TCP connection and exchange some data, you have to handle edge cases, like when the port is blocked by something else, or supporting in-process communication, and so on. I think the reliance on XML-RPC in ROS1 was a cause of complexity, and one of the advantages of using DDS is that we can avoid that.

It would still be possible, of course, to offer a non-ROS interface in addition, e.g. for use by web-based services. This would not necessarily have to run on every machine, however, but only when necessary.

iluetkeb · 2016-10-20T08:52:31.564Z

@dirk-thomas

Demand starting is a tricky business. I would prefer that, in the first instance, the launch procedure starts the daemon, the client implementations falls back to DDS discovery when the daemon is not present. Also, I would prefer that when demand-starting is implemented, we do this through a minimal daemon process which loads the “real” process upon first connection. This again keeps demand starting functionality out of the core libraries.

Regarding independence of the specific rmw implementation: The interface should be independent, but the internal implementation could be vendor-specific. Several DDS vendors offer this functionality already (including FastRTPS, as far as I can tell), and this would be very good to re-use. I would be worried that we’re inefficient otherwise, when we bypass the vendor.

gbiggs · 2016-10-20T23:50:17.647Z

iluetkeb:

I would check whether requiring a non-ROS (which, in this case I take to mean a non-DDS) interface is strictly necessary. Maybe we can avoid the discovery phase by other means, e.g. by pre-supplying the necessary information, assuming that the daemon runs locally (which other means would also have to assume).

So we’ve got a vote here for the ROS interface being the only interface.

I think it is reasonable to have the same information available over different interfaces. Certain tools (e.g. web-based tools) would benefit from not needing to use the DDS stack.

Which interfaces are required, and which interface is considered the “default” interface by tools, are the questions that we need to settle.

How fast a tool using a ROS topic can start up and get the information it wants is something I hope to test today.

iluetkeb:

My reason for that is to avoid duplicating communication code and functionality. While at first it might appear simple to open a TCP connection and exchange some data, you have to handle edge cases, like when the port is blocked by something else, or supporting in-process communication, and so on. I think the reliance on XML-RPC in ROS1 was a cause of complexity, and one of the advantages of using DDS is that we can avoid that.

I think this overstates the difficulty. If we implement using Python, as an example, there are good libraries available for providing web services, such as Django, CherryPy and several lower-level-but-still-abstracted libraries included in Python 3.

The counter-point there is that, like XML-RPC in ROS1, we would be increasing the number of dependencies if we use something like Django or CherryPi.

iluetkeb · 2016-10-21T07:30:41.196Z

@gbiggs

I think this overstates the difficulty.

I meant not just implementation difficulty, but also use-time complexity. While not entirely comparable, consider in ROS1, the use of XML-RPC was the root cause of all the mess with ROS_NAME/ROS_IP, and the various DNS resolution issues that led to. At least in my experience, lots of people struggled with that.

You’re taking in an entirely new communications stack, with all the side effects that has.

Sure, you can try and avoid that, for this special case, but that’s just why I’m suggesting that we investigate whether it’s really necessary first, instead of barging headlong into an implementation.

gbiggs · 2016-10-21T07:53:19.395Z

You are right about use-time difficulty; I hadn’t thought of that. The less that can go wrong, the better, from that perspective.

I agree that investigation is necessary. I originally envisioned the ROS tools like rostopic using the DDS stack to get the information, with the REST interface being a convenience thing for makers of tools that benefit from not using the whole ROS stack. @dirk-thomas would prefer it be the other way around, I think.

Anyway, I spent some time today hacking a wholely-unscientific benchmark into the add_two_ints_client example. Here are my results, based on 100 runs on a 2015 macbook pro (because for some reason alpha8 just point blank doesn’t run on my Ubuntu desktop and I haven’t had time to figure out why yet). Times are in seconds.

                         minimum   first quartile median    mean      third quartile  maximum
rclcpp::init             0.000041  0.000049       0.000051  0.000052  0.000053        0.000071
rclcpp:node::make_shared 0.005342  0.005714       0.005867  0.005948  0.006078        0.007536
node->create_client      0.005635  0.005979       0.006139  0.006234  0.006385        0.008588
make_shared request      0.005643  0.005987       0.006147  0.006243  0.006394        0.008600
wait_for_service         0.008758  0.009588       0.010619  0.030617  0.011594        1.011910
async_send_request       0.008900  0.009737       0.010776  0.030780  0.011735        1.012290
receive result           0.224500  0.232855       0.235795  0.251124  0.238558        1.015980
rcpcpp::shutdown         0.224662  0.233006       0.235900  0.251237  0.238677        1.016050

jfabry · 2016-10-21T14:36:13.214Z

I would like to second the argument of @iluetkeb about use-time complexity of multiple communication mechanisms, from a different point of view: I have a student working on developer understanding of ROS and the current variety of communication mechanisms, and the consensus about the persons being interviewed is that there are too many of them.

I am all in favor of keeping it simple, and implementation-wise, the original proposal using a daemon and the existing ROS communication mechanism looks very attractive to me. The daemon always runs and has the information available as a ROS service call, plus uses a topic to broadcast changes to the graph such that long-running tools are informed of this.

If later on it turns out that more is needed (e.g. REST API), this can simply be implemented as extra functionality on top of this layer.

dirk-thomas · 2016-10-21T14:51:05.535Z

I just want to reiterate: the reason why the current command line tools are slow is because they have to wait for the discovery phase to finish before they can query the desired information. The idea of the daemon is that it is already running before and it has already accumulated the information.

If the command line tool wants to use the ROS interface to request information from the daemon it again needs to wait for the discovery phase to finish before it can do so. It might only need to wait for the availability of that daemon but still this implies a significant overhead in waiting time for the user. I don’t think a command line tool (which e.g. is often also used for completion) can have that time penalty. While I am certainly not a big fan of having a different transport mechanism I don’t see how the requirement of the lowest latency possible can be fulfilled with using a distributed peer-to-peer system like DDS.

NikolausDemmel · 2016-10-21T15:05:15.334Z

I guess @iluetkeb point was to use DDS communication, but find a way to avoid the discovery phase in the specific case of connecting to the daemon. It seems to me that letting a tool like rostopic know how to directly connect to the daemon with DDS is not a much different problem from letting the tool know how to connect to the daemon through some other protocol.

iluetkeb · 2016-10-21T15:12:08.720Z

Thanks to @NikolausDemmel for making my point better than I did Yes, that is exactly what I was tryint to suggest.

if the command line tool wants to use the ROS interface to request
information from the daemon it again needs to wait for the discovery
phase to finish before it can do so.

Not necessarily. See http://eprosima-fast-rtps.readthedocs.io/en/latest/advanced.html#matching-endpoints-the-manual-way, for example.

dirk-thomas · 2016-10-21T15:20:52.163Z

This functionality would need to be exposed through the rmw interface in an abstract way. And it needs to be implementable with all current vendors. I am not sure that Connext / OpenSplice provide a similar API and if yes how their configuration option differ.

NikolausDemmel · 2016-10-22T10:21:28.989Z

FWIW, http://design.ros2.org/articles/discovery_and_negotiation.html talks about static vs dynamic discovery. The way I understood it static discovery is exactly what we are talking about here (just in this case specifically only for one connection, from tool --> daemon). Is static discovery not within the scope of ROS2? (I believe static discovery is relevant not only for this use case).

An additional advantage of relying on ROS2 communication is that other consumers of the same API that don’t necessarily worry about short start-up time will just as well work using dynamic discovery.

iluetkeb · 2016-10-22T15:05:11.879Z

As for the general question, maybe @Jaime_Martin_Losa could shed some light on this.

For RTI Conext, you can pre-supply discovery peers, at least.

See “add_peer” in https://community.rti.com/static/documentation/connext-dds/5.2.0/doc/manuals/connext_dds/html_files/RTI_ConnextDDS_CoreLibraries_UsersManual/index.htm#UsersManual/ConfigPeersListUsed_inDiscov.htm#discovery_507287096_336417 This seems to be a Conext-specific API.

CoreDX (which you don’t use, but just as another datapoint) goes even further and implements the whole daemon-based central discovery solution for you.

Therefore, it might actually make sense to provide the topic list (or something from which it can directly be derived) in the rmw API, and let the vendor-specific solution use the best vendor-specific approach for supplying it. We would then need to fall back to our own daemon only when necessary.

btw, in general this is not such an outlandish feature, and such an obvious optimization, that I would be very surprised if a vendor did not support some means of realizing it.

gbiggs · 2016-10-24T07:51:15.414Z

It isn’t necessarily an optimisation from the point of view of a lot of DDS use cases. The lack of need for a central discovery daemon was a major design goal in the way DDS works.

However, there are use cases where it is an optimisation, as you say. Perhaps using that optimisation, when it’s available, via rmw, and providing our own daemon as part of the rmw implementation when it’s not, is a valid implementation.

Thoughts on this approach?

Does anyone (@iluetkeb) have any data on which DDS implementations provide a list of available topics in near-instant time, and which do not? Remember that providing access to the list of topics seen by the participant since it started is not the same thing as what we need. We need something that accumulates the list of topics currently available, including those that the rostopic tool may not have seen itself, and can provide that list on demand. So far the CoreDX implementation that @iluetkeb mentioned sounds like it provides this functionality already, but I’m not aware of any others that do.

One of the reasons for choosing a well-known topic was so that discovery can be short-circuited in DDS implementations that support it.

Jaime_Martin_Losa · 2016-11-04T09:37:57.951Z

Hi guys,

On Fast RTPS you can listen to discovery data:

http://eprosima-fast-rtps.readthedocs.io/en/latest/advanced.html#subscribing-to-discovery-topics

Also, you can set the endpoints for discovery:

http://eprosima-fast-rtps.readthedocs.io/en/latest/pubsub.html#defining-input-and-output-channels

gbiggs · 2017-02-02T02:01:17.758Z

I started writing a prototype of the approach discussed in this thread. But then Work^tm had other ideas, as it so often seems to.

The code is up on Github. It’s nowhere near complete, but perhaps there is enough to see where I was going. It’s not just rostopic, it’s intended to be a complete set of command line tools for working with the ROS graph.

I still want to keep working on it myself, but I think progress would be faster as a group effort.

GitHub

gbiggs/ros2graphtools

Libraries and tools for introspecting the ROS 2 graph - gbiggs/ros2graphtools

Having said that, there has been activity recently on the ros2 github to implement a rosnode tool directly, so perhaps this work is moot.

karsten · 2017-02-03T01:07:00.309Z

@gbiggs: Are you referring to this [0] repo? If so, it’s true that we have two scripts, rosnode_list and rostopic_list, which however are not meant to be in any releasable or complete state. These are mainly the result of debugging tools needed during the current development on our side (they are neither built on the buildfarm nor listed in the repos file) . We are currently not actively working on implementing these tools, given the discussion of this thread and the remaining open questions.
We may add more scripts in that repository, which then can be helpful to others as well, we would though recommend not just yet open PRs against it.

[0] https://github.com/ros2/cli_tools

gbiggs · 2017-02-03T01:15:57.908Z

Yes, that’s the stuff I was talking about, @karsten. Thanks for explaining their purpose.

ross_2 · 2018-10-29T17:49:52.045Z

Hi all, I’d like to kick this thread back up.

Problem:

We are only able to access all nodes and all topics/types.
No equivalent rosnode info exists for ROS2 and these daemon’s are still not available.

Goal

Expose the node graph for gui and cli tools to show the user.
Expose the following per node via rcl API:

services
subscriptions
publications
actions

Solution

Regardless of the daemon or on-demand approach to accessing node graph details, exposing the node graph through the rcl layer for tools to use would be advantageous to developers.

These interfaces will be fulfilled via the Simple Discovery Protocol, which uses well-defined unicast and multicast ports for each participant to listen to meta-traffic.

Are there any objections to using SDP to discover the node graph?