I am looking into using ros2 and have done some testing on sending images between nodes using intraprocess communication. It seems like the memory usage is slowly increasing for all publishing nodes where intra-process communication is activated. This is regardless of any intra-process subscribers.
The same nodes with intra-process communication turned off does not have increasing memory usage.
I am currently using the default Fast-rtps implementation. I first noticed the problem in the “latest-release”, but checked out the master branch yesterday to see if the problem was fixed. (it wasn’t).
It can be reproduced by running the intra-process-demo image nodes.
Is it possible to fix this?
Hi there,
I just did a run on the latest master code. There do seem to be a couple of leaks, so then I ran image_pipeline_with_two_image_view under valgrind (to be totally explicit, I ran valgrind --leak-check=full ./build_isolated/intra_process_demo/image_pipeline_with_two_image_view. Looking at the output from that, it looks like the leaks are coming from somewhere down in the OpenCV code. From a brief glance at it, it is not clear to me whether the leaks are in OpenCV itself, or how we are using it in the image demo.
All of that being said, this seems to be a problem with the interaction with OpenCV, not something fundamental to intra-process communication itself. I probably won’t have too much more time to spend on it right now, but if you have time to look at it, the code is in ros2/demos/intra_process_demo/include/image_pipeline/image_view_node.hpp
My program is actually not using OpenCV, and valgrind does not report memory leaks (even if the memory usage is increasing). However when I look at the memory usage in htop, the RES tab for my program it increases (quite slowly). I ran a test over the weekend, and the memory usage had continued to increase the whole weekend. When i turned off intra process communication and ran it over night, the RES tab showed no increase in memory usage…
I suspect that the pointers used in for the intra process communication are stored somewhere, or something like that.
@eirikhex can you provide code to reproduce this ?
I tried to reproduce the described behavior with a basic intra-process talker/listener and the reserved memory stayed steady (32124) after 20000 publications/subcriptions.
I also did a small test with a basic talker/listener. I used the “two_node_pipeline” from the intra process demo code:
ros2/demos/intra_process_demo/include/two_node_pipeline/two_node_pipeline.cpp
The only change I made was to change the time interval in the timer on line 49. from 1s to 10ms to get results faster. I got the following results:
Starting memory 21 420
after 80 000 publications: 28 100
after 100 000 publications: 29 948
after 175 000 publications: 37 724
after 200 000 publications: 39 308
And continues to rise.
I then changed both nodes by turning off intra-process communication, to verify that it in fact was something with the intra process communication causing the increased memory usage:
Starting memory: 18036
after 200 000 publications: 18036
My original program sending a jpeg image over ros2 15 times per second has now been running for 6 days. It had a starting reserved memory of 33M, and has monotonically increased its reserved memory to 981M. I did the same test on the program with intra-process communication turned off and the memory did not increase.
Thanks @eirikhex for the additional details. I guess I didnt let it run for long then. I’ll give it another shot
The leaked memory from within cvCreateMat is constant - two times (one per window) independent how long the process is running. So this is not responsible for the increasing memory reported.