Looks good. Can you also please share the meeting link.
@ishugoel and I are also interested in joining this meeting. We have been looking at the current implementation of the SingleThreadedExecutor to investigate the high CPU overhead. We have an ongoing discussion here: SingleThreadedExecutor creates a high CPU overhead in ROS 2 . As a first step we are starting work on creating a static version of the executor to investigate how much performance can be gained (POC). We think this is a separate issue from the scheduling “problem” discussed in Tobias’ work. Both issues are important to us and probably to others who want to run a “normal” ROS2 stack on an embedded board.
@Dejan_Pangercic I’d like to join the WG as a listener for the time being
Yes, that should be enough.
Very nice work on the CPU overheads!. I agree that the two problems are not related.
I’d like to join, thanks for taking care of this thread.
Is it the same Zoom meeting like last time?
I’d like to join it.
We might want to spend some more time thinking about this, before someone goes off and starts on it. I’m not sure 10 minutes is enough here – maybe have it as the main topic of an upcoming meeting?
I think there might be a connection to tracing here… Just saying
In principle yes, but I’m not available on the 21st of August, as I’m on vacation. Let’s connect afterwards.
the meeting details are here:
Topic: My Meeting
Time: Aug 21, 2019 07:00 AM Pacific Time (US and Canada)
Join Zoom Meeting
One tap mobile
+16699006833,264081213# US (San Jose)
+19294362866,264081213# US (New York)
Dial by your location
+1 669 900 6833 US (San Jose)
+1 929 436 2866 US (New York)
Meeting ID: 264 081 213
Find your local number: https://zoom.us/u/aAmxdSnyw
Yes - we absolutely should not do this without the proper design document.
Wednesday, August 21, 2019 2:00 PM
This is the same time as an AWF board meeting so I may or may not be able to join. Don’t wait for me if I’m not there.
I would like to join.
Sorry guys I was stuck on a plane while this meeting was happening and I only just got clear. Are there any meeting minutes for this?
@Dejan_Pangercic Can you please share MoM or meeting recording
@lbegani and all:
here the recordings of the 2nd meeting (I will follow-up with the short transcription tomorrow):
Topic: 2nd ROS Real-time working group meeting
Start Time : Aug 21, 2019 06:51 AM
I would like to join this group and attend the next meeting please.
@brettpac see the invite to the next meeting: ROS2 Real-time Working Group Online Meeting 3 - Oct 2, 2019, 7AM PDT (UTC-7)
@wjwwood I transcribed your part of the meeting. Please quickly review if I got anything wrong:
This was a 2 pass work:
- finding and fixing unneeded memory allocations in rmw, rcl, rclcpp
- make changes in the rmw interface and make changes to allow middleware to manage the memory
For 1 memory_testing_tools has been developed. Looked for the tool to to not have to write it (similar to valgrind ,…) but could not find one that would also be portable to different platforms. It is implemented such that it injects the shared library and e.g. overwrites malloc.
Install hook for malloc as in gtest framework. They found lots of cases with unexpected memory allocation. You also get a stack trace and unwinding is setup. Using this you can exclude memory allocations from the shared libraries that you do not care about, e.g. some tests will only care if there is memory allocation within an rcl shared library.
One thing they found is the way thread local storage is implemented in most compilers is that the first time you access the thread local variable in a new thread, it allocates the memory then and not when you create the thread.
They have something similar to err-no in rmw which allows to set errors in a thread safe way and they had to find ways to work around that. Most of the memory allocations that is still happening in the common functions such as publish and take, are happening in the middleware. There are still memory allocations happening in the rclcpp - needs to still be audited.
There are still improvements to be made in the intra-process communication.
For 2: one problems in e.g. Connext Pro, we are not able to serialize ROS m sg to CDR and publish that with Connext. DDS prefers to use the DDS type so they can control serialization and de-serialization. What OSRF does is convert ROS to DDS msg before publishing it. So when the user publishes the ROS msg, we need to copy the data from ROS datastruct to DDS datastruct, write the DDS msg and free it again. This is bad because you call create and delete data functions from DDS once every publish. To avoid that they introduced a new API called pre-allocation and the idea is that the client lib (rclcpp) they can create a pre-allocation which with Connext Pro is an object opaque to the rclcpp. This object is created once and then re-used.
More work going on now:
- loan data from middleware
- mechanisms to specify additional information
I just wanted to say thank you for both organizing the meeting earlier today and for inviting me as well. It was very interesting.
I look forward to attending future meeting (especially since they will be more frequent) and contributing.
That all looks fine.