OpenVDB Compile Size

smac · 2024-07-23T17:25:46.508Z

For some reason, I didn’t get emails about this one, so I didn’t notice to address sooner! Sorry about that. It looks like openvdb_vendor doesn’t have my email in the package.xml.

So probably the best thing to do is to try to profile the compilation

How much memory is the build farm configured with by default? I’ll say even I struggle to compile OpenVDB on my laptop due to the same reason. This was built successfully on Humble/Iron, though its about 30/70 if it gets through the build.

The source jobs turn over reliably, which is interesting. I expect the difference there is release flags.

clalancette · 2024-07-23T17:44:43.865Z

smac:

How much memory is the build farm configured with by default?

I’m not sure. @nuclearsandwich can probably chime in here.

nuclearsandwich · 2024-07-23T17:51:43.989Z

smac:

How much memory is the build farm configured with by default? I’ll say even I struggle to compile OpenVDB on my laptop due to the same reason. This was built successfully on Humble/Iron, though its about 30/70 if it gets through the build.

Each build agent is a compute-optimized “xlarge” instance in EC2 spread between c5.xlarge, c6i.xlarge, or m5.xlarge which all have 4vCPUs and 8GiB of memory.

Each individual package build is expected to use 1/4th of those resources: 1vCPU and 2GiB of RAM. These are soft limits that are not enforced, but since agents run up to four jobs at a time, packages using more than the available memory will be subject to pressure from the OS.

smac · 2024-07-23T18:08:36.590Z

That explains some of the non-determinism on the Humble / Iron jobs that seem to pass 20-30% of the time.

Is there a not-too-painful way to get more memory for this job? For most packages, I might choose to not release them if this is going to be an issue, but this is a somewhat core package to the Nav2 userbase that we addressed with the vendorization by popular demand and multiple OS’ where the openvdb binaries were not suitable. I don’t expect this package to be rebuilt often (this package has very infrequent updates).

nuclearsandwich · 2024-07-23T18:31:08.635Z

smac:

Is there a not-too-painful way to get more memory for this job?

Do you know how much memory is needed? 8GiB is a hard limit since we don’t have bigger machines than that in the worker pool.

Most of these beefy packages succeed when retried after everything else has already built and they can use an entire worker.

smac · 2024-07-23T23:43:27.191Z

I don’t have a hard number, I’ll say that my developer PC has issues with it with 16GB, but Iron / Humble seem to be able to do it in the build farm with 8GB (not sure why Jazzy is different beyond 24.04). @Timple were you compiling this locally previously, can you chime in?

tfoote · 2024-07-24T06:13:54.765Z

smac:

Is there a not-too-painful way to get more memory for this job?

Part of the way that we size the machines on the build farm is that it’s also basically setting a requirement for all users to compile.

smac:

For most packages, I might choose to not release them if this is going to be an issue, but this is a somewhat core package to the Nav2 userbase that we addressed with the vendorization by popular demand and multiple OS’ where the openvdb binaries were not suitable. I don’t expect this package to be rebuilt often (this package has very infrequent updates).

Because of this being a core package it’s going to effectively be a minimum requirement for any Nav2 user to compile from source. If we bump that up to 8GB it will make life a lot harder for SBC users and other embedded dev environments.

borongyuan · 2024-07-24T07:39:06.510Z

Hi, @smac, thank you for providing openvdb_vendor, which makes it easy to use OpenVDB with cmake. We plan to add OpenVDB support to RTAB-Map for better map representation. New features are expected to be launched in half a year. We hope this will be fixed so that we can use OpenVDB from the ROS repository as a dependency directly.

Timple · 2024-07-24T12:04:09.302Z

It took some iterations on Humble and Iron as well. But it succeeded in the end. And as you mentioned, there hasn’t been a need for re-compilation due to the minimal amount of dependencies.

If there isn’t a maximum duration for the jobs, we might be able to enforce something like MAKEFLAGS=-j1 for the buildfarm.
Does anyone here know how to pass this to the buildfarm without hardcoding it in the package?

clalancette · 2024-07-24T12:11:49.224Z

Timple:

If there isn’t a maximum duration for the jobs, we might be able to enforce something like MAKEFLAGS=-j1 for the buildfarm.

There is a maximum duration (2 hours), but I don’t think that matters anyway; the buildfarm by default already uses MAKEFLAGS=-j1, as far as I understand (@nuclearsandwich can confirm).

nuclearsandwich · 2024-07-24T15:35:25.978Z

Timple:

If there isn’t a maximum duration for the jobs, we might be able to enforce something like MAKEFLAGS=-j1 for the buildfarm.

The default is already -j1 inherited from Debian defaults. We have never changed this specifically because we intend for the build farm to parallelize primarily at the package build level so that individual package builds are as deterministic as possible to aid debugging.

smac · 2024-07-24T17:56:45.359Z

@Timple what’s your compiling experience been? Admittedly I don’t use STVL on a day to day anymore in my current role.

While suboptimal, we could compile a debian and link it to a release in github that people can download the debian from to install. I’m not 100% sure how we’d make STVL apt-installable using that except by having the STVL CMake manually get that file and download/undeb it. This is a bit of a security issue, but that way STVL can be apt installed but openvdb_vendor can be compiled once where we don’t have the limitations.

I was also thinking perhaps github actions could be used here to compile it… that way there’s at least some visibility into the workflow that created the binary.

tfoote:

If we bump that up to 8GB it will make life a lot harder for SBC users and other embedded dev environments.

Without the build farm / provided binaries by some mechanism, I’m pretty sure SBC users can’t compile openvdb on their own hadware anyway.

Timple · 2024-07-24T17:59:02.403Z

I took a quick peek through openvdb/CMakeLists.txt but it seems that only the core and the binaries are build by default.
The binaries are disabled here.

Which leaves the core, which builds only two libraries, a static and a shared. But since -j1 is already present, these should build sequential.

So not sure how to reduce the memory footprint for this compilation…

(We might gain a factor 0.5 in time by disabling the static libraries though, depending on how good ccache does it’s work).

smac · 2024-07-24T18:09:30.024Z

I don’t think time is the issue here, its memory. 03:37:43 is when the build starts and 04:41:31 is when its c++: fatal error: Killed signal terminated program cc1plus.

Admittedly, that’s suspiciously close to 1 hour, if the build duration maximum was actually 1 hour in stead of 2…

Another job failed with the following, which I’m not 100% sure how to read into, but I could believe that’s another instantiation of “I’m out of memory” unless someone knows its due to something else → I see jobs where this happens after 30 minutes - 1 hour. Most jobs are actually failing this way (not c++: fatal error: Killed signal terminated program cc1plus), but only for bin jobs.


03:37:42 FATAL: command execution failed
03:37:42 java.nio.channels.ClosedChannelException
03:37:42 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
03:37:42 	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
03:37:42 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:817)
03:37:42 	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:179)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:281)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:246)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:198)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:211)
03:37:42 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:785)
03:37:42 	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:172)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
03:37:42 	at hudson.remoting.Channel.close(Channel.java:1502)
03:37:42 	at hudson.remoting.Channel.close(Channel.java:1455)
03:37:42 	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:884)
03:37:42 	at hudson.slaves.SlaveComputer.access$100(SlaveComputer.java:110)
03:37:42 	at hudson.slaves.SlaveComputer$2.run(SlaveComputer.java:765)
03:37:42 	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
03:37:42 	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
03:37:42 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
03:37:42 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
03:37:42 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
03:37:42 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
03:37:42 	at java.lang.Thread.run(Thread.java:750)
03:37:42 Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from ip-10-0-1-232.us-west-1.compute.internal/10.0.1.232:51244' is disconnected.

clalancette · 2024-07-24T18:20:37.645Z

smac:

Admittedly, that’s suspiciously close to 1 hour, if the build duration maximum was actually 1 hour in stead of 2…

The maximum duration is definitely two hours: ros_buildfarm_config/jazzy/release-build.yaml at 9985e2e406678e41ff026373e52b0d2ce5541bf3 · ros2/ros_buildfarm_config · GitHub

Timple · 2024-07-25T06:20:59.466Z

On Humble it succeeded at once (once the dependencies were in order). However the second build took 7 iterations.

On Iron, both times it took 4 iterations.

So it seems indeed like a non-consistency in the buildfarm were sometimes the runner is lucky in that it’s not competing with other runners.

I’m not sure if the “8gb should be a limit to guard for dev requirements” holds. My experience with ROS 2 in general is that memory usage is factors more than the same code in ROS 1. But this is just what it is, not sure if we want to enforce these specifications for the future to come. I just popped more memory in my laptop

clalancette · 2024-07-25T11:15:23.822Z

Timple:

I’m not sure if the “8gb should be a limit to guard for dev requirements” holds

The problem is cost. All of the workers run in the cloud, and going from 8gb to something larger costs more. Given how many workers we use across the infrastructure, this adds up fast. So its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

gavanderhoorn · 2024-07-25T11:53:58.213Z

For this instance specifically (so not a proper solution): what about building it in a PPA / some other context and then setting up an import job @nuclearsandwich? Similar to how this is done for other packages (like Colcon, Gazebo and catkin_lint) in ros-infrastructure/reprepro-updater?

That’s probably what @smac was thinking of in his comment.

clalancette:

The problem is cost. […] its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

Would there be a way to donate cycles instead of money?

tfoote · 2024-07-25T18:31:49.767Z

gavanderhoorn:

Would there be a way to donate cycles instead of money?

This is a great concept, but the increased complexity of managing heterogeneous systems causes significantly increased labor/time to support it which can potentially offset any cost savings for the compute. And depending on the mechanism of delivery there’s significant trust and security complexities too. Not to mention bandwidth costs, latency concerns, uptime monitoring too.

gavanderhoorn:

For this instance specifically (so not a proper solution): what about building it in a PPA / some other context and then setting up an import job @nuclearsandwich? Similar to how this is done for other packages (like Colcon, Gazebo and catkin_lint) in ros-infrastructure/reprepro-updater ?

We’re working really hard to get away from this process. You’ll note that there’s been no new packages added there recently that aren’t directly developed by the core team, and those are mostly updates of existing packages.

If someone sets up a PPA for a package, they then need to also setup a build for every other potentially supported platform and arch combination. REP 2000 -- ROS 2 Releases and Target Platforms (ROS.org) I don’t think that we want to limit Nav2 to only run on specific Ubuntu arches in a PPA. And there’s also significant issues with reproducibiliity and trust for these packages too. What happens in the case of a security vulnerability. We would need to be able to respond and have pathways to patch and rebuild. We’ve had issues in the past when PPA maintainers were unavailable and we had to look at creating our own copies of it to reproduce with a fixed version. And suddenly it looks like we’re effectively recreating what we’re already doing on the ROS Buildfarm, but with more effort for a subset of platforms.

peci1 · 2024-07-26T01:54:40.921Z

clalancette:

So its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

I know this would be another exception to maintain, but what about creating a (manually composed?) list of packages that would build on a larger worker? In such case, it would be sufficient to pay only for one large worker.