OpenVDB Compile Size

clalancette · 2024-07-24T12:11:49.224Z

Timple:

If there isn’t a maximum duration for the jobs, we might be able to enforce something like MAKEFLAGS=-j1 for the buildfarm.

There is a maximum duration (2 hours), but I don’t think that matters anyway; the buildfarm by default already uses MAKEFLAGS=-j1, as far as I understand (@nuclearsandwich can confirm).

nuclearsandwich · 2024-07-24T15:35:25.978Z

Timple:

If there isn’t a maximum duration for the jobs, we might be able to enforce something like MAKEFLAGS=-j1 for the buildfarm.

The default is already -j1 inherited from Debian defaults. We have never changed this specifically because we intend for the build farm to parallelize primarily at the package build level so that individual package builds are as deterministic as possible to aid debugging.

smac · 2024-07-24T17:56:45.359Z

@Timple what’s your compiling experience been? Admittedly I don’t use STVL on a day to day anymore in my current role.

While suboptimal, we could compile a debian and link it to a release in github that people can download the debian from to install. I’m not 100% sure how we’d make STVL apt-installable using that except by having the STVL CMake manually get that file and download/undeb it. This is a bit of a security issue, but that way STVL can be apt installed but openvdb_vendor can be compiled once where we don’t have the limitations.

I was also thinking perhaps github actions could be used here to compile it… that way there’s at least some visibility into the workflow that created the binary.

tfoote:

If we bump that up to 8GB it will make life a lot harder for SBC users and other embedded dev environments.

Without the build farm / provided binaries by some mechanism, I’m pretty sure SBC users can’t compile openvdb on their own hadware anyway.

Timple · 2024-07-24T17:59:02.403Z

I took a quick peek through openvdb/CMakeLists.txt but it seems that only the core and the binaries are build by default.
The binaries are disabled here.

Which leaves the core, which builds only two libraries, a static and a shared. But since -j1 is already present, these should build sequential.

So not sure how to reduce the memory footprint for this compilation…

(We might gain a factor 0.5 in time by disabling the static libraries though, depending on how good ccache does it’s work).

smac · 2024-07-24T18:09:30.024Z

I don’t think time is the issue here, its memory. 03:37:43 is when the build starts and 04:41:31 is when its c++: fatal error: Killed signal terminated program cc1plus.

Admittedly, that’s suspiciously close to 1 hour, if the build duration maximum was actually 1 hour in stead of 2…

Another job failed with the following, which I’m not 100% sure how to read into, but I could believe that’s another instantiation of “I’m out of memory” unless someone knows its due to something else → I see jobs where this happens after 30 minutes - 1 hour. Most jobs are actually failing this way (not c++: fatal error: Killed signal terminated program cc1plus), but only for bin jobs.


03:37:42 FATAL: command execution failed
03:37:42 java.nio.channels.ClosedChannelException
03:37:42 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
03:37:42 	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
03:37:42 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:817)
03:37:42 	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:179)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:281)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:246)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:198)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:211)
03:37:42 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:785)
03:37:42 	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:172)
03:37:42 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
03:37:42 	at hudson.remoting.Channel.close(Channel.java:1502)
03:37:42 	at hudson.remoting.Channel.close(Channel.java:1455)
03:37:42 	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:884)
03:37:42 	at hudson.slaves.SlaveComputer.access$100(SlaveComputer.java:110)
03:37:42 	at hudson.slaves.SlaveComputer$2.run(SlaveComputer.java:765)
03:37:42 	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
03:37:42 	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
03:37:42 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
03:37:42 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
03:37:42 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
03:37:42 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
03:37:42 	at java.lang.Thread.run(Thread.java:750)
03:37:42 Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from ip-10-0-1-232.us-west-1.compute.internal/10.0.1.232:51244' is disconnected.

clalancette · 2024-07-24T18:20:37.645Z

smac:

Admittedly, that’s suspiciously close to 1 hour, if the build duration maximum was actually 1 hour in stead of 2…

The maximum duration is definitely two hours: ros_buildfarm_config/jazzy/release-build.yaml at 9985e2e406678e41ff026373e52b0d2ce5541bf3 · ros2/ros_buildfarm_config · GitHub

Timple · 2024-07-25T06:20:59.466Z

On Humble it succeeded at once (once the dependencies were in order). However the second build took 7 iterations.

On Iron, both times it took 4 iterations.

So it seems indeed like a non-consistency in the buildfarm were sometimes the runner is lucky in that it’s not competing with other runners.

I’m not sure if the “8gb should be a limit to guard for dev requirements” holds. My experience with ROS 2 in general is that memory usage is factors more than the same code in ROS 1. But this is just what it is, not sure if we want to enforce these specifications for the future to come. I just popped more memory in my laptop

clalancette · 2024-07-25T11:15:23.822Z

Timple:

I’m not sure if the “8gb should be a limit to guard for dev requirements” holds

The problem is cost. All of the workers run in the cloud, and going from 8gb to something larger costs more. Given how many workers we use across the infrastructure, this adds up fast. So its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

gavanderhoorn · 2024-07-25T11:53:58.213Z

For this instance specifically (so not a proper solution): what about building it in a PPA / some other context and then setting up an import job @nuclearsandwich? Similar to how this is done for other packages (like Colcon, Gazebo and catkin_lint) in ros-infrastructure/reprepro-updater?

That’s probably what @smac was thinking of in his comment.

clalancette:

The problem is cost. […] its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

Would there be a way to donate cycles instead of money?

tfoote · 2024-07-25T18:31:49.767Z

gavanderhoorn:

Would there be a way to donate cycles instead of money?

This is a great concept, but the increased complexity of managing heterogeneous systems causes significantly increased labor/time to support it which can potentially offset any cost savings for the compute. And depending on the mechanism of delivery there’s significant trust and security complexities too. Not to mention bandwidth costs, latency concerns, uptime monitoring too.

gavanderhoorn:

For this instance specifically (so not a proper solution): what about building it in a PPA / some other context and then setting up an import job @nuclearsandwich? Similar to how this is done for other packages (like Colcon, Gazebo and catkin_lint) in ros-infrastructure/reprepro-updater ?

We’re working really hard to get away from this process. You’ll note that there’s been no new packages added there recently that aren’t directly developed by the core team, and those are mostly updates of existing packages.

If someone sets up a PPA for a package, they then need to also setup a build for every other potentially supported platform and arch combination. REP 2000 -- ROS 2 Releases and Target Platforms (ROS.org) I don’t think that we want to limit Nav2 to only run on specific Ubuntu arches in a PPA. And there’s also significant issues with reproducibiliity and trust for these packages too. What happens in the case of a security vulnerability. We would need to be able to respond and have pathways to patch and rebuild. We’ve had issues in the past when PPA maintainers were unavailable and we had to look at creating our own copies of it to reproduce with a fixed version. And suddenly it looks like we’re effectively recreating what we’re already doing on the ROS Buildfarm, but with more effort for a subset of platforms.

peci1 · 2024-07-26T01:54:40.921Z

clalancette:

So its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

I know this would be another exception to maintain, but what about creating a (manually composed?) list of packages that would build on a larger worker? In such case, it would be sufficient to pay only for one large worker.

Rayman · 2024-07-26T08:44:43.374Z

I have a potential solution, on jazzy the Ubuntu openvdb is version 10 so we could already use that:

github.com/SteveMacenski/spatio_temporal_voxel_layer

Avoid compilation on jazzy or higher

SteveMacenski:ros2 ← nobleo:feature/avoid-compilation-on-jazzy

opened 08:55AM - 26 Jul 24 UTC

Rayman

+7 -1

On jazzy the released openvdb version is already 10.0.1, so we don't have to com…pile the vendor package. Compilation on the build farm was already quite flakey because the package is quite heavy, so I'd be best to avoid this. I don't know if openvdb is also released for ARM or any other ubuntu platforms, but we'll have to see on the buildfarm.

clalancette · 2024-07-26T11:44:03.563Z

Rayman:

I have a potential solution, on jazzy the Ubuntu openvdb is version 10 so we could already use that:

Nice. That is the best solution.

I’ll point out that there are other solutions here, mostly involving modifications to the OpenVDB source code:

Compile with a lower optimization level, which may or may not help. This also has the downside that things will run slower.
Split the offending source file into two or more smaller files.
Move more implementation out of the header files and into the C++ files. This can make each compilation unit use slightly less memory.
Reduce the amount of templating in the header files. This is the hardest to do, but also will probably (along with 2) have the most benefit.

smac · 2024-07-31T05:57:47.532Z

Rayman:

I have a potential solution, on jazzy the Ubuntu openvdb is version 10 so we could already use that:

I believe I pointed out in the PR that this needs to be tested and not simply compiled. Openvdb’s binaries always compiled but didn’t work in practice with the compiler flags that whomever does the releases seems to use for the last couple of OS’.

Another option that just struck me is Davide’s bonxai. While not available in binaries or supported releases, if it works ‘well enough’, we can pull the headers into STVL directly and ditch Openvdb (which has been the bane of my existence since 2018 with release issue after logistical issue). I’d be willing to take a slight performance hit in exchange for easier release and deployment processes so it can be distributed wider and easier - and I think it might be a little bit faster since it doesn’t contain all the openvdb additional complexity.

peci1 · 2024-07-31T10:32:38.280Z

smac:

Another option that just struck me is Davide’s bonxai.

Bonxai is currently under development and I am building this mostly for fun and for educational purposes. Don’t expect any API stability for the time being.

smac · 2024-07-31T15:44:27.407Z

peci1:

Bonxai is currently under development and I am building this mostly for fun and for educational purposes. Don’t expect any API stability for the time being.

I know, but its also not going to be released into binaries. In his ROSCon talk, he mentioned just copy + pasting headers into your project, so we don’t really need API stability as long as it works

Timple · 2024-08-28T07:39:29.693Z

So the solution of @Rayman works . Saves a lot of buildfarm CPU cycles.

However for arm, there is no openvdb binary package, thus the buildfarm falls back to compiling openvdb from source. Leading to memory issues again.

Personally I don’t mind that arm64 fails, but maybe someone has some good suggestions? Otherwise my mailbox will keep being flooded. . And the buildfarm will probably keep trying…

clalancette · 2024-08-28T11:50:11.828Z

Timple:

Personally I don’t mind that arm64 fails, but maybe someone has some good suggestions?

If you don’t care about arm64, then you can add it to the blacklist at ros_buildfarm_config/jazzy/release-ubuntu-arm64-build.yaml at ros2 · ros2/ros_buildfarm_config · GitHub (you’ll have to do the same for each release that you want to blacklist it from).

Timple · 2024-08-30T08:42:49.685Z

If @smac has no objections, I’ll go and do just that.

Rayman · 2024-08-30T09:03:03.844Z

I have a fix for the ARM64 build so you don’t have to blacklist: Fix arm64 build by Rayman · Pull Request #303 · SteveMacenski/spatio_temporal_voxel_layer · GitHub