The repository is under the GitHub organization of the ROS 2 tooling working group. Unfortunately, most of the original authors have since moved on. The tool is in need of a bit of love, bug fixes, reviews, etc., so we are looking for help maintaining it!
If you’ve used this package and would like to help maintain it – or if this looks like a cool tool to you and you’d like to try it out and contribute – let us know below!
I have not used this tool before, but looking over it I would like to point out two things:
First: the name is misleading. It is not doing cross compiling in the sense of doing a build for one architecture on another architecture without emulation. This project uses QEMU to run emulated builds, which I wouldn’t call “cross compiling”. Emulated builds will not be as performant as a native build, although true cross compiling may approach it.
Second: this project scripts together different features of Docker and QEMU. However, these days, docker buildx has QEMU support out of the box. Why maintain QEMU integration for docker when docker is already doing it? I’m all for making ROS software easier to build, but I don’t think this project is the right starting point to rally around. Just my opinion.
cross_compile is a nice integration tool but maintaining it is lots of work. We evaluated this while designing the architecture for the ROS 2 Hardware Acceleration Working group (we considered various options, cross_compile package being one of them) and as part of REP-2008 effort but decided to go a different direction. The tool does the following as explained in here:
- Create a Docker image that has rosdep
- Run the rosdep image against your target workspace to output a script that describes how to install its dependencies
Create “sysroot image” that has everything needed for building target workspace
- Use a base image for the target architecture (aarch64, armhf, …)
- Install build tools (compilers, cmake, colcon, etc)
- Run the dependency installer script collected in Step 1 (if dependency list hasn’t changed since last run, this uses the Docker cache)
- Runs the “sysroot image” using QEMU emulation
- colcon build
(Optional) Create runtime image
- Creates a docker image that can be used on the target platform to run the build. See “Runtime Image” section.
The main issues with the tool I experienced are the following ones:
Complexity: There’re lots of moving parts in here and abstractions used. Technically, it’s using two types of virtualization (OS-virtualization and then hardware-virtualization) chained together to build the artifacts. It’s first using OS-virtualization (typically referred to as ”containers” in Linux) through Docker to build a clean/reproducible environment from wherein to start operating (, ). Then, it’s using hardware-virtualization through QEMU type two hypervisor (that’s used from within the Docker container) to build the ROS workspace natively ().
Incompatible with other frameworks/libraries that require x86 support: There’re many tools and frameworks that aren’t available on aarch64 (or other) architecture and only have support for x86, so you can’t use them with QEMU. cross_compile approach does not match many of these frameworks and would require splitting the generation of artifacts, which complicates the overall process significantly and easily leads to errors. One good example is Vitis, which is used to produce accelerators for FPGAs, AI Engines and other compute resources.
The approach we took instead is extending the ROS 2 build system (ament) (see here for an explanation) adding CMake logic with cross-compilation and any other step necessary to generate the build artifacts in one go.
For completeness, here is how you might do an emulated build using docker buildx, although I will note once more that this is not very performant: if you actually have a target that you can compile on, unless it is extremely resource constrained, it will probably just be faster to compile on that.
Make a Dockerfile to build your project. Example: Nav2
This is rather difficult to do in an efficient way, so this is an area where some community love could really help. But essentially you need to just base off a ROS base image, install rosdep dependencies, install any additional dependencies, and invoke colcon build.
Ensure you have docker buildx installed.
This comes pre-packaged with Docker Engine 20+ in some distributions, but you may have to install it manually
Create a new “builder” for buildx that has the docker-container driver and supports your target arch. For example, if you are building on an x86 host, but you want to build ARM64 images, you would do the following:
Since no one expressed willingness to volunteer and support ros_cross_compile package.
Also giving in to consideration that multiple people expressed opinion that this package outdated by design, over-complicated, inefficient and almost no one using it.
I am going to deprecate/archive it and remove from supporting packages list in “Tooling working group”.
@ahemwe, refer to the ros_cross_compile package for help on this regard. Maybe filing an issue there will help.
As for our other approach explained here initially and now turned into an official REP (REP 2008), refer to the reference implementation section for an example. There’s no further official documentation on this (that I know) but if you’re looking for professional help you can get it reaching out here.
I assume micro ROS documentation might also provide relevant info wrt cross compilation (did not check it).
However, reading through this thread makes me think twice before going through the cross-compile route, at least for a relatively performant platform as a raspberry pi 4.
Also relevant: dist-cc is mentioned as a possible alternative to cross-compiling.
@vmayoral I assume the extension of ament as you propose rather is meant to seamlessly integrate host-native compilers (typically FPGA-brand specific) into the ROS2 build system, as opposed to setting up a full build environment with all target platform specific library dependencies etc?
To answer your question, no, ament_acceleration(with extensions such asament_vitis) doesn't just integrate host-native compilers but it also aligns with the rest of the firmware artifacts (see firmware` extensions to ROS 2 workspaces) necessary to cross-compile for target embedded CPUs. See REP 2008 for details, including a reference implementation.
As discussed in past threads (and also in some GitHub issues which aren’t handy right now), we benchmarked native cross-compilation vs OS-virtualized (e.g. via Docker) cross-compilation obtaining that the former was about 10 times faster. When developing and iterating in cycles (e.g. when building accelerators), this time factor is very relevant. That’s why we decided to not build on top of ros_cross_compile tool and instead extend directly ament.