OpenVDB Compile Size

smac · 2024-07-23T18:08:36.590Z

That explains some of the non-determinism on the Humble / Iron jobs that seem to pass 20-30% of the time.

Is there a not-too-painful way to get more memory for this job? For most packages, I might choose to not release them if this is going to be an issue, but this is a somewhat core package to the Nav2 userbase that we addressed with the vendorization by popular demand and multiple OS’ where the openvdb binaries were not suitable. I don’t expect this package to be rebuilt often (this package has very infrequent updates).

nuclearsandwich · 2024-07-23T18:31:08.635Z

smac:

Is there a not-too-painful way to get more memory for this job?

Do you know how much memory is needed? 8GiB is a hard limit since we don’t have bigger machines than that in the worker pool.

Most of these beefy packages succeed when retried after everything else has already built and they can use an entire worker.

smac · 2024-07-23T23:43:27.191Z

I don’t have a hard number, I’ll say that my developer PC has issues with it with 16GB, but Iron / Humble seem to be able to do it in the build farm with 8GB (not sure why Jazzy is different beyond 24.04). @Timple were you compiling this locally previously, can you chime in?

tfoote · 2024-07-24T06:13:54.765Z

smac:

Is there a not-too-painful way to get more memory for this job?

Part of the way that we size the machines on the build farm is that it’s also basically setting a requirement for all users to compile.

smac:

For most packages, I might choose to not release them if this is going to be an issue, but this is a somewhat core package to the Nav2 userbase that we addressed with the vendorization by popular demand and multiple OS’ where the openvdb binaries were not suitable. I don’t expect this package to be rebuilt often (this package has very infrequent updates).

Because of this being a core package it’s going to effectively be a minimum requirement for any Nav2 user to compile from source. If we bump that up to 8GB it will make life a lot harder for SBC users and other embedded dev environments.

borongyuan · 2024-07-24T07:39:06.510Z

Hi, @smac, thank you for providing openvdb_vendor, which makes it easy to use OpenVDB with cmake. We plan to add OpenVDB support to RTAB-Map for better map representation. New features are expected to be launched in half a year. We hope this will be fixed so that we can use OpenVDB from the ROS repository as a dependency directly.

Timple · 2024-07-24T12:04:09.302Z

It took some iterations on Humble and Iron as well. But it succeeded in the end. And as you mentioned, there hasn’t been a need for re-compilation due to the minimal amount of dependencies.

If there isn’t a maximum duration for the jobs, we might be able to enforce something like MAKEFLAGS=-j1 for the buildfarm.
Does anyone here know how to pass this to the buildfarm without hardcoding it in the package?

clalancette · 2024-07-24T12:11:49.224Z

Timple:

If there isn’t a maximum duration for the jobs, we might be able to enforce something like MAKEFLAGS=-j1 for the buildfarm.

There is a maximum duration (2 hours), but I don’t think that matters anyway; the buildfarm by default already uses MAKEFLAGS=-j1, as far as I understand (@nuclearsandwich can confirm).

nuclearsandwich · 2024-07-24T15:35:25.978Z

Timple:

If there isn’t a maximum duration for the jobs, we might be able to enforce something like MAKEFLAGS=-j1 for the buildfarm.

The default is already -j1 inherited from Debian defaults. We have never changed this specifically because we intend for the build farm to parallelize primarily at the package build level so that individual package builds are as deterministic as possible to aid debugging.

smac · 2024-07-24T17:56:45.359Z

@Timple what’s your compiling experience been? Admittedly I don’t use STVL on a day to day anymore in my current role.

While suboptimal, we could compile a debian and link it to a release in github that people can download the debian from to install. I’m not 100% sure how we’d make STVL apt-installable using that except by having the STVL CMake manually get that file and download/undeb it. This is a bit of a security issue, but that way STVL can be apt installed but openvdb_vendor can be compiled once where we don’t have the limitations.

I was also thinking perhaps github actions could be used here to compile it… that way there’s at least some visibility into the workflow that created the binary.

tfoote:

If we bump that up to 8GB it will make life a lot harder for SBC users and other embedded dev environments.

Without the build farm / provided binaries by some mechanism, I’m pretty sure SBC users can’t compile openvdb on their own hadware anyway.

Timple · 2024-07-24T17:59:02.403Z

I took a quick peek through openvdb/CMakeLists.txt but it seems that only the core and the binaries are build by default.
The binaries are disabled here.

Which leaves the core, which builds only two libraries, a static and a shared. But since -j1 is already present, these should build sequential.

So not sure how to reduce the memory footprint for this compilation…

(We might gain a factor 0.5 in time by disabling the static libraries though, depending on how good ccache does it’s work).

smac · 2024-07-24T18:09:30.024Z

I don’t think time is the issue here, its memory. 03:37:43 is when the build starts and 04:41:31 is when its c++: fatal error: Killed signal terminated program cc1plus.

Admittedly, that’s suspiciously close to 1 hour, if the build duration maximum was actually 1 hour in stead of 2…

Another job failed with the following, which I’m not 100% sure how to read into, but I could believe that’s another instantiation of “I’m out of memory” unless someone knows its due to something else → I see jobs where this happens after 30 minutes - 1 hour. Most jobs are actually failing this way (not c++: fatal error: Killed signal terminated program cc1plus), but only for bin jobs.


03:37:42 FATAL: command execution failed
03:37:42 java.nio.channels.ClosedChannelException
03:37:42 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
03:37:42 	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
03:37:42 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:817)
03:37:42 	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:179)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:281)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:246)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:198)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:211)
03:37:42 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:785)
03:37:42 	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:172)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
03:37:42 	at hudson.remoting.Channel.close(Channel.java:1502)
03:37:42 	at hudson.remoting.Channel.close(Channel.java:1455)
03:37:42 	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:884)
03:37:42 	at hudson.slaves.SlaveComputer.access$100(SlaveComputer.java:110)
03:37:42 	at hudson.slaves.SlaveComputer$2.run(SlaveComputer.java:765)
03:37:42 	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
03:37:42 	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
03:37:42 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
03:37:42 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
03:37:42 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
03:37:42 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
03:37:42 	at java.lang.Thread.run(Thread.java:750)
03:37:42 Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from ip-10-0-1-232.us-west-1.compute.internal/10.0.1.232:51244' is disconnected.

clalancette · 2024-07-24T18:20:37.645Z

smac:

Admittedly, that’s suspiciously close to 1 hour, if the build duration maximum was actually 1 hour in stead of 2…

The maximum duration is definitely two hours: ros_buildfarm_config/jazzy/release-build.yaml at 9985e2e406678e41ff026373e52b0d2ce5541bf3 · ros2/ros_buildfarm_config · GitHub

Timple · 2024-07-25T06:20:59.466Z

On Humble it succeeded at once (once the dependencies were in order). However the second build took 7 iterations.

On Iron, both times it took 4 iterations.

So it seems indeed like a non-consistency in the buildfarm were sometimes the runner is lucky in that it’s not competing with other runners.

I’m not sure if the “8gb should be a limit to guard for dev requirements” holds. My experience with ROS 2 in general is that memory usage is factors more than the same code in ROS 1. But this is just what it is, not sure if we want to enforce these specifications for the future to come. I just popped more memory in my laptop

clalancette · 2024-07-25T11:15:23.822Z

Timple:

I’m not sure if the “8gb should be a limit to guard for dev requirements” holds

The problem is cost. All of the workers run in the cloud, and going from 8gb to something larger costs more. Given how many workers we use across the infrastructure, this adds up fast. So its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

gavanderhoorn · 2024-07-25T11:53:58.213Z

For this instance specifically (so not a proper solution): what about building it in a PPA / some other context and then setting up an import job @nuclearsandwich? Similar to how this is done for other packages (like Colcon, Gazebo and catkin_lint) in ros-infrastructure/reprepro-updater?

That’s probably what @smac was thinking of in his comment.

clalancette:

The problem is cost. […] its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

Would there be a way to donate cycles instead of money?

tfoote · 2024-07-25T18:31:49.767Z

gavanderhoorn:

Would there be a way to donate cycles instead of money?

This is a great concept, but the increased complexity of managing heterogeneous systems causes significantly increased labor/time to support it which can potentially offset any cost savings for the compute. And depending on the mechanism of delivery there’s significant trust and security complexities too. Not to mention bandwidth costs, latency concerns, uptime monitoring too.

gavanderhoorn:

For this instance specifically (so not a proper solution): what about building it in a PPA / some other context and then setting up an import job @nuclearsandwich? Similar to how this is done for other packages (like Colcon, Gazebo and catkin_lint) in ros-infrastructure/reprepro-updater ?

We’re working really hard to get away from this process. You’ll note that there’s been no new packages added there recently that aren’t directly developed by the core team, and those are mostly updates of existing packages.

If someone sets up a PPA for a package, they then need to also setup a build for every other potentially supported platform and arch combination. REP 2000 -- ROS 2 Releases and Target Platforms (ROS.org) I don’t think that we want to limit Nav2 to only run on specific Ubuntu arches in a PPA. And there’s also significant issues with reproducibiliity and trust for these packages too. What happens in the case of a security vulnerability. We would need to be able to respond and have pathways to patch and rebuild. We’ve had issues in the past when PPA maintainers were unavailable and we had to look at creating our own copies of it to reproduce with a fixed version. And suddenly it looks like we’re effectively recreating what we’re already doing on the ROS Buildfarm, but with more effort for a subset of platforms.

peci1 · 2024-07-26T01:54:40.921Z

clalancette:

So its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

I know this would be another exception to maintain, but what about creating a (manually composed?) list of packages that would build on a larger worker? In such case, it would be sufficient to pay only for one large worker.

Rayman · 2024-07-26T08:44:43.374Z

I have a potential solution, on jazzy the Ubuntu openvdb is version 10 so we could already use that:

github.com/SteveMacenski/spatio_temporal_voxel_layer

Avoid compilation on jazzy or higher

SteveMacenski:ros2 ← nobleo:feature/avoid-compilation-on-jazzy

opened 08:55AM - 26 Jul 24 UTC

Rayman

+7 -1

On jazzy the released openvdb version is already 10.0.1, so we don't have to com…pile the vendor package. Compilation on the build farm was already quite flakey because the package is quite heavy, so I'd be best to avoid this. I don't know if openvdb is also released for ARM or any other ubuntu platforms, but we'll have to see on the buildfarm.

clalancette · 2024-07-26T11:44:03.563Z

Rayman:

I have a potential solution, on jazzy the Ubuntu openvdb is version 10 so we could already use that:

Nice. That is the best solution.

I’ll point out that there are other solutions here, mostly involving modifications to the OpenVDB source code:

Compile with a lower optimization level, which may or may not help. This also has the downside that things will run slower.
Split the offending source file into two or more smaller files.
Move more implementation out of the header files and into the C++ files. This can make each compilation unit use slightly less memory.
Reduce the amount of templating in the header files. This is the hardest to do, but also will probably (along with 2) have the most benefit.

smac · 2024-07-31T05:57:47.532Z

Rayman:

I have a potential solution, on jazzy the Ubuntu openvdb is version 10 so we could already use that:

I believe I pointed out in the PR that this needs to be tested and not simply compiled. Openvdb’s binaries always compiled but didn’t work in practice with the compiler flags that whomever does the releases seems to use for the last couple of OS’.

Another option that just struck me is Davide’s bonxai. While not available in binaries or supported releases, if it works ‘well enough’, we can pull the headers into STVL directly and ditch Openvdb (which has been the bane of my existence since 2018 with release issue after logistical issue). I’d be willing to take a slight performance hit in exchange for easier release and deployment processes so it can be distributed wider and easier - and I think it might be a little bit faster since it doesn’t contain all the openvdb additional complexity.