Scalability issues with large number of nodes

General
21

It will be regenerated at any case, thats what python SingleThreadedExecutor does. doesnt it?

22

I just look into the python implementation, indeed, the entities are recollected on every wait call before creating the Waitset.

Another thing you should keep in mind, if you create service clients on the fly is, that they take some time to connect.

Just to know what numbers we are talking about, what is the max numbers of clients you have in ONE node ?

23

This is our biggest node (state machine node)
clients 100
services 32
publishers 8
subscribers 13

24

Okay, that is somewhat sane. Now back to your initial problem, does the node go into 100% CPU load, and nothing is processed any more, or is the CPU load low, but everything is slow for some other reason ?

25

We get close to 100% CPU load at the beginning, when launching the whole stack, but after that the CPU keeps around 30-40% all the time, so must be some other reason

26

I guess its profiler time :wink:

27

Any recommended tool?

28

callgrind + kcachegrind did a very good job for me in the past.

29

There is also instrumentation potentially built in via ros2_tracing.

https://docs.ros.org/en/iron/Tutorials/Advanced/ROS2-Tracing-Trace-and-Analyze.html

3 Likes
30

Perf and flamegraphs

31

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.