ROS2 Real-time Working Group Online Meeting 2 - Aug 21st, 2019 between 7AM and 8AM PDT (UTC-7)

Next Generation ROS
1

Hi all,
this is a call for our 2nd ROS2 Real-time Working Group meeting: Aug 21st, 2019 between 7AM and 8AM PDT (UTC-7).

I would like to propose the following agenda constructed from the action items of the previous meeting: ROS2 Real-time Working Group Online Meeting - May 20th, 2019 between 7AM and 8AM PDT (UTC-7) - #8 by Dejan_Pangercic

  1. [10min] Create an rmw implementation which works for a single process. Once that works connect with the underlying data link (e.g. TSN). Measure: malloc/realloc calls, page faults, context switches, memory consumption, blocking calls, … RTI Connext Micro is a suitable DDS implementation for something like this due it its staticity and intra-process communication feature.

    1. AI : find someone that will lead this work.
    2. I propose to ask one of the following folks to lead this
      1. GitHub - ros2/rmw_fastrtps: Implementation of the ROS Middleware (rmw) Interface using eProsima's Fast RTPS. (Jaime, Miguel) or
      2. GitHub - ros2/rmw_cyclonedds: ROS 2 RMW layer for Eclipse Cyclone DDS (Eric, Joe)
      3. @lbegani also volanteered

  2. [10min] Performance testing on target hardware

    Buildfarm
    Buildfarm1655Ă—1119 175 KB

    This is the topic that I am adding in addition and it was originally proposed by Acutronic Robotics. The proposal is to create a buildfarm with couple of select computers (ECUs) on which to run the GitHub - ApexAI/performance_test: **This project is deprecated** Go to https://gitlab.com/ApexAI/performance_test and measure performance regressions. @Dejan_Pangercic realized that several parties are trying to work on this (Amazon, iRobot, Apex, eProsima, Open Robotics, …)
    We would need to figure out the following:

    1. target HW => could Amazon host?
    2. target (RT)OS => Linux with RT-PREEMPT, QNX, VxWorks, …
    3. how to automate flashing of (RT)OS and provisioning of DDS, ROS 2 and GitHub - ApexAI/performance_test: **This project is deprecated** Go to https://gitlab.com/ApexAI/performance_test => Apex works on this
    4. how to configure and automate running of GitHub - ApexAI/performance_test: **This project is deprecated** Go to https://gitlab.com/ApexAI/performance_test in a clean environment => Apex works on this
      1. how to run Alpha release of ROS 2 tracing tools
    5. how to store the results => Apex works on this
    6. how to visualize the results (e.g. What is Munin ? — Munin 2.999.16-detached-2023-06-14-c98-ga24eda76 documentation) such that we can compare historical runs, view results for different configurations of HW, RTOS, DDS, ROS 2 => ???
    7. which additional metrics do we want => ???

  3. [10min] Invest in the tools for static and dynamic code analysis and tracing and carefully scan the code.

    1. AI: create a shared repository for above tools. Bosch will e.g. open-source their integration of LTTng in ROS2. Silexica is integrating they dynamic analyser.
    2. Bosch already made the first steps: ROS2 Real-time Working Group Online Meeting - May 20th, 2019 between 7AM and 8AM PDT (UTC-7) - #11 by christophebedard
    3. Silexica dynamically analysed Autoware.Auto: Merge requests · AutowareAuto / AutowareAuto · GitLab
    4. @Ingo_Lutkebohle can you take a lead on this?

  4. [20min]

    1. AI : present the paper in detail in one of the next meetings.
    2. @tobiasblasscan you do this in 1/2h?

  5. [10min]

    • perform memory audit in rmw, rcl and rclcpp (remove unneeded memory allocations)
    • split memory allocation in init and runtime phases, avoid memory fragmentation
    • remove all blocking calls (or replace with timed calls, e.g. mutex vs timed_mutex )

    Document current state of the art. Someone remarks that there were still mutexes used in e.g. rmw or rclcpp. Memory is still being allocated/de-allocated when receiving messages.

    1. AI2 : get William to present his findings
    2. @Dejan_Pangercic will get William to present this

LMK if you agree with the agenda and if you can join.

D.

3 Likes
2

@vmayoral @tobiasblass @Ingo_Lutkebohle @davecrawley @aohsato @gbiggs @lbegani @carlossv @nburek See above.

3

@wjwwood @Jaime_Martin_Losa @mkhansen @anup.pemmaiah @joespeed see above.

4

One remark - regarding the following action item:

  • Propose a use case => requirements => architecture => hardware
  • Proposed canonical use case: a vehicle/robot encounters an obstacle and has to stop safely for it
  • Proposed other use case: autonomous driving
  1. AI : Dejan to provide a use case description and requirements
  2. I did not have time to provide this. @gbiggs and one of our engineers are working on this however.
5

Looks good. Can you also please share the meeting link.

1 Like
6

@eboasson @joespeed will join, possibly @niels.kortstee too

1 Like
7

@ishugoel and I are also interested in joining this meeting. We have been looking at the current implementation of the SingleThreadedExecutor to investigate the high CPU overhead. We have an ongoing discussion here: SingleThreadedExecutor creates a high CPU overhead in ROS 2 . As a first step we are starting work on creating a static version of the executor to investigate how much performance can be gained (POC). We think this is a separate issue from the scheduling “problem” discussed in Tobias’ work. Both issues are important to us and probably to others who want to run a “normal” ROS2 stack on an embedded board.

1 Like
8

@Dejan_Pangercic I’d like to join the WG as a listener for the time being

9

Yes, that should be enough.

Very nice work on the CPU overheads!. I agree that the two problems are not related.

1 Like
10

@Dejan_Pangercic

I’d like to join, thanks for taking care of this thread.

tomoya

1 Like
11

Hi @Dejan_Pangercic,

Is it the same Zoom meeting like last time?
I’d like to join it.

Kind Regards,
Andrei

1 Like
12

We might want to spend some more time thinking about this, before someone goes off and starts on it. I’m not sure 10 minutes is enough here – maybe have it as the main topic of an upcoming meeting?

I think there might be a connection to tracing here… Just saying :wink:

In principle yes, but I’m not available on the 21st of August, as I’m on vacation. Let’s connect afterwards.

13

Hi all,
the meeting details are here:

Topic: My Meeting
Time: Aug 21, 2019 07:00 AM Pacific Time (US and Canada)

Join Zoom Meeting

One tap mobile
+16699006833,264081213# US (San Jose)
+19294362866,264081213# US (New York)

Dial by your location
+1 669 900 6833 US (San Jose)
+1 929 436 2866 US (New York)
Meeting ID: 264 081 213
Find your local number: https://zoom.us/u/aAmxdSnyw

1 Like
14

Yes - we absolutely should not do this without the proper design document.

Correct. Added.

OK.

15

Wednesday, August 21, 2019 2:00 PM

This is the same time as an AWF board meeting so I may or may not be able to join. Don’t wait for me if I’m not there.

16

I would like to join.

Sincerely,
Kosuke

17

Sorry guys I was stuck on a plane while this meeting was happening and I only just got clear. Are there any meeting minutes for this?

David

18

@Dejan_Pangercic Can you please share MoM or meeting recording

19

@lbegani and all:
here the recordings of the 2nd meeting (I will follow-up with the short transcription tomorrow):

Topic: 2nd ROS Real-time working group meeting
Start Time : Aug 21, 2019 06:51 AM

Meeting Recording:
https://zoom.us/recording/share/DY2qaOx4qqzUyJaIxsgdLEhfIdLdl4nPc0forr7e2S-wIumekTziMw

1 Like
20

Hello,
@Dejan_Pangercic
I would like to join this group and attend the next meeting please.

Thank you,