OpenVDB Compile Size

[tfoote]

@gavanderhoorn

Would there be a way to donate cycles instead of money?

This is a great concept, but the increased complexity of managing heterogeneous systems causes significantly increased labor/time to support it which can potentially offset any cost savings for the compute. And depending on the mechanism of delivery there’s significant trust and security complexities too. Not to mention bandwidth costs, latency concerns, uptime monitoring too.

@gavanderhoorn

For this instance specifically (so not a proper solution): what about building it in a PPA / some other context and then setting up an import job @nuclearsandwich? Similar to how this is done for other packages (like Colcon, Gazebo and catkin_lint) in ros-infrastructure/reprepro-updater ?

We’re working really hard to get away from this process. You’ll note that there’s been no new packages added there recently that aren’t directly developed by the core team, and those are mostly updates of existing packages.

If someone sets up a PPA for a package, they then need to also setup a build for every other potentially supported platform and arch combination. REP 2000 -- ROS 2 Releases and Target Platforms (ROS.org) I don’t think that we want to limit Nav2 to only run on specific Ubuntu arches in a PPA. And there’s also significant issues with reproducibiliity and trust for these packages too. What happens in the case of a security vulnerability. We would need to be able to respond and have pathways to patch and rebuild. We’ve had issues in the past when PPA maintainers were unavailable and we had to look at creating our own copies of it to reproduce with a fixed version. And suddenly it looks like we’re effectively recreating what we’re already doing on the ROS Buildfarm, but with more effort for a subset of platforms.

[peci1]

@clalancette

So its probably not viable to increase the memory size of the workers just to compile one or two packages (unless someone dumps a bunch of money on OSRF to support that).

I know this would be another exception to maintain, but what about creating a (manually composed?) list of packages that would build on a larger worker? In such case, it would be sufficient to pay only for one large worker.

[Rayman]

I have a potential solution, on jazzy the Ubuntu openvdb is version 10 so we could already use that:

Avoid compilation on jazzy or higher

[clalancette]

@Rayman

I have a potential solution, on jazzy the Ubuntu openvdb is version 10 so we could already use that:

Nice. That is the best solution.

I’ll point out that there are other solutions here, mostly involving modifications to the OpenVDB source code:

1. Compile with a lower optimization level, which may or may not help. This also has the downside that things will run slower.
2. Split the offending source file into two or more smaller files.
3. Move more implementation out of the header files and into the C++ files. This can make each compilation unit use slightly less memory.
4. Reduce the amount of templating in the header files. This is the hardest to do, but also will probably (along with 2) have the most benefit.

[smac]

@Rayman

I have a potential solution, on jazzy the Ubuntu openvdb is version 10 so we could already use that:

I believe I pointed out in the PR that this needs to be tested and not simply compiled. Openvdb’s binaries always compiled but didn’t work in practice with the compiler flags that whomever does the releases seems to use for the last couple of OS’.

Another option that just struck me is Davide’s bonxai. While not available in binaries or supported releases, if it works ‘well enough’, we can pull the headers into STVL directly and ditch Openvdb (which has been the bane of my existence since 2018 with release issue after logistical issue). I’d be willing to take a slight performance hit in exchange for easier release and deployment processes so it can be distributed wider and easier - and I think it might be a little bit faster since it doesn’t contain all the openvdb additional complexity.

[peci1]

@smac

Another option that just struck me is Davide’s bonxai.

Bonxai is currently under development and I am building this mostly for fun and for educational purposes. Don’t expect any API stability for the time being.

[smac]

@peci1

Bonxai is currently under development and I am building this mostly for fun and for educational purposes. Don’t expect any API stability for the time being.

I know, but its also not going to be released into binaries. In his ROSCon talk, he mentioned just copy + pasting headers into your project, so we don’t really need API stability as long as it works

[Timple]

So the solution of @Rayman works . Saves a lot of buildfarm CPU cycles.

However for arm, there is no openvdb binary package, thus the buildfarm falls back to compiling openvdb from source. Leading to memory issues again.

Personally I don’t mind that arm64 fails, but maybe someone has some good suggestions? Otherwise my mailbox will keep being flooded. . And the buildfarm will probably keep trying…

[clalancette]

@Timple

Personally I don’t mind that arm64 fails, but maybe someone has some good suggestions?

If you don’t care about arm64, then you can add it to the blacklist at ros_buildfarm_config/jazzy/release-ubuntu-arm64-build.yaml at ros2 · ros2/ros_buildfarm_config · GitHub (you’ll have to do the same for each release that you want to blacklist it from).

[Timple]

If @smac has no objections, I’ll go and do just that.

[Rayman]

I have a fix for the ARM64 build so you don’t have to blacklist: Fix arm64 build by Rayman · Pull Request #303 · SteveMacenski/spatio_temporal_voxel_layer · GitHub

[smac]

I’m cutting another release now for ARM