Hi I just wanted to share something we discovered recently, not sure how much it’ll help others but it helped us big time (hence the YMMV). We had some code (and it seems like some parts of Gazebo itself also) that depends on the OpenBLAS linear algebra library. After a long time of our sim running at a RTF of 0.2-0.4 or so, one of our developers discovered that the defaults for this library are set to use all available cores for parallel processing. This means even throwing more CPU at the problem does not help. Their readme specifies you can change this default behavior by setting an environment variable OPENBLAS_NUM_THREADS=4 (or something appropriate for your system, we set it to 1). And that change alone took our simulation from 0.2ish to over 1.0 RTF. Without a lot more deep diving it’s difficult to tell if this was just mostly our code or also gazebo (and plugins), but this library seems to get used everywhere so I at least wanted to share in hopes that it’d help someone else. We saw no performance degredation in any of the code or sim as a result of the setting.
Thanks for sharing!
I’m curious, are you using Gazebo-classic or the new Gazebo?
Gazebo classic 11.10.2
Thanks for sharing. We have exactly the same experience from SubT challenge (Ignition Dome). It seems the management of the extra threads, locking and context switching is often more costly than the speedup you get. More importantly, if you squeeze the max of the computer (all cores 100%), it does make much less sense to parallellize everything.
This is a good candidate for a check Gazebo could do during runtime and hint the user to change the variable.
Or Gazebo could just set a better default? OpenBLAS: Setting the number of threads at runtime
Seems like a small change that could go a long way in usability.
One more thing I just wanted to mention in case it wasn’t painfully obvious. We did not have to recompile anything, just set that environment variable in the terminal you run from.
I remember seeing/fixing these issues with different python ML codes (I think numpy uses openBLAS, or some other tools in ML toolchains).
Amazing it affects so much else.
This is probably the worst chosen default setting I have seen.
So I tried to look for where OpenBLAS is used in Gazebo classic using apt-rdepends libgazebo11 and I could only see it in the simbody physics engine, through libgdal for heightmaps, and through libavilter7 for recording videos. Notably, it’s not used by the physics engines ODE, Bullet, or DART.
Could be plugins and not just the main executable?
I did the same with gazebo11-plugins-base, and had the same result. It’s possible I might have missed something.