Call for help: maintainership of the ros_cross_compile tool

The ros_cross_compile tool automates the compilation of ROS and ROS 2 workspaces for other architectures. See the repository: GitHub - ros-tooling/cross_compile: A tool to build ROS and ROS2 workspaces for various targets

The repository is under the GitHub organization of the ROS 2 tooling working group. Unfortunately, most of the original authors have since moved on. The tool is in need of a bit of love, bug fixes, reviews, etc., so we are looking for help maintaining it!

If you’ve used this package and would like to help maintain it – or if this looks like a cool tool to you and you’d like to try it out and contribute – let us know below!

4 Likes

I have not used this tool before, but looking over it I would like to point out two things:

First: the name is misleading. It is not doing cross compiling in the sense of doing a build for one architecture on another architecture without emulation. This project uses QEMU to run emulated builds, which I wouldn’t call “cross compiling”. Emulated builds will not be as performant as a native build, although true cross compiling may approach it.

Second: this project scripts together different features of Docker and QEMU. However, these days, docker buildx has QEMU support out of the box. Why maintain QEMU integration for docker when docker is already doing it? I’m all for making ROS software easier to build, but I don’t think this project is the right starting point to rally around. Just my opinion.

4 Likes

Unfortunately, I’m not one of the authors and have never really used the tool, but I believe it was written before docker buildx was available.

Fortunately, though, this is exactly the kind of feedback and comments I was looking for, so thank you. IIRC, other people have also mentioned this, and there’s a PR for “proper” cross compilation here: [WIP] Enable true cross compilation workflow by emersonknapp · Pull Request #242 · ros-tooling/cross_compile · GitHub. I think @TSC21 was interested in this, but anyone could pick it back up.

If the community/current users think this project should be deprecated, then that’s fine. However, there are currently no alternatives that I’m aware of.

1 Like

I second @aposhian’s view above.

cross_compile is a nice integration tool but maintaining it is lots of work. We evaluated this while designing the architecture for the ROS 2 Hardware Acceleration Working group (we considered various options, cross_compile package being one of them) and as part of REP-2008 effort but decided to go a different direction. The tool does the following as explained in here:

1. Collect dependencies
- Create a Docker image that has rosdep
- Run the rosdep image against your target workspace to output a script that describes how to install its dependencies
2. Create “sysroot image” that has everything needed for building target workspace
- Use a base image for the target architecture (aarch64, armhf, …)
- Install build tools (compilers, cmake, colcon, etc)
- Run the dependency installer script collected in Step 1 (if dependency list hasn’t changed since last run, this uses the Docker cache)
3. Build
- Runs the “sysroot image” using QEMU emulation
- colcon build
4. (Optional) Create runtime image
- Creates a docker image that can be used on the target platform to run the build. See “Runtime Image” section.

The main issues with the tool I experienced are the following ones:

  • Complexity: There’re lots of moving parts in here and abstractions used. Technically, it’s using two types of virtualization (OS-virtualization and then hardware-virtualization) chained together to build the artifacts. It’s first using OS-virtualization (typically referred to as ”containers” in Linux) through Docker to build a clean/reproducible environment from wherein to start operating (1, 2). Then, it’s using hardware-virtualization through QEMU type two hypervisor (that’s used from within the Docker container) to build the ROS workspace natively (3).

  • Speed: Use cross-compilers to compile workspace instead of an emulated compilation · Issue #69 · ros-tooling/cross_compile · GitHub reported that this approach is about 8 times slower than true cross-compilation. We validated this experimentally obtaining numbers above that in more complex ROS workspaces. While developing, you perform this many times. While it’s true the tooling can be modified to get an interactive CLI to the emulated architecture with QEMU. The (over emulation) native build times are simply very slow in comparison to using a cross-compiler directly in your workstation.

  • Incompatible with other frameworks/libraries that require x86 support: There’re many tools and frameworks that aren’t available on aarch64 (or other) architecture and only have support for x86, so you can’t use them with QEMU. cross_compile approach does not match many of these frameworks and would require splitting the generation of artifacts, which complicates the overall process significantly and easily leads to errors. One good example is Vitis, which is used to produce accelerators for FPGAs, AI Engines and other compute resources.

The approach we took instead is extending the ROS 2 build system (ament) (see here for an explanation) adding CMake logic with cross-compilation and any other step necessary to generate the build artifacts in one go.

4 Likes

For completeness, here is how you might do an emulated build using docker buildx, although I will note once more that this is not very performant: if you actually have a target that you can compile on, unless it is extremely resource constrained, it will probably just be faster to compile on that.

  1. Make a Dockerfile to build your project. Example: Nav2

    • This is rather difficult to do in an efficient way, so this is an area where some community love could really help. But essentially you need to just base off a ROS base image, install rosdep dependencies, install any additional dependencies, and invoke colcon build.
  2. Ensure you have docker buildx installed.

    • This comes pre-packaged with Docker Engine 20+ in some distributions, but you may have to install it manually
  3. Create a new “builder” for buildx that has the docker-container driver and supports your target arch. For example, if you are building on an x86 host, but you want to build ARM64 images, you would do the following:

docker buildx create --use --platform=linux/arm64,linux/amd64 --name emulated-multiarch --driver docker-container
  1. Do this magic incantation to setup QEMU:
    docker run --privileged multiarch/qemu-user-static:latest --reset -p yes

  2. Build:

docker buildx build . --platform=linux/arm64
2 Likes

Since no one expressed willingness to volunteer and support ros_cross_compile package.
Also giving in to consideration that multiple people expressed opinion that this package outdated by design, over-complicated, inefficient and almost no one using it.
I am going to deprecate/archive it and remove from supporting packages list in “Tooling working group”.

3 Likes

Sorry for re-triggering this old thread once again.

I’m referring to this statement:

I can’t find any documentation on how to use this approach.
Where can I find more information about the cmake logic for cross compilation which is not using ros-cross-compile? (Foxy)

There is actually a question on that on answer.ros.org but without any answer.

Additionally, this has been asked on stackoverflow.

@vmayoral

@ahemwe, refer to the ros_cross_compile package for help on this regard. Maybe filing an issue there will help.

As for our other approach explained here initially and now turned into an official REP (REP 2008), refer to the reference implementation section for an example. There’s no further official documentation on this (that I know) but if you’re looking for professional help you can get it reaching out here.

@ahemwe I think it might be relevant to read through this: Use cross-compilers to compile workspace instead of an emulated compilation · Issue #69 · ros-tooling/cross_compile · GitHub.

This discussion references to:

  1. This branch of the repository, which seems out of date,
  2. The cross compile tools of epuck_ros2, which is probably a good starting point for people looking into cross-compiling for raspberry pi, and
  3. ccws, which seems targeted mainly to ROS1 (?).

I assume micro ROS documentation might also provide relevant info wrt cross compilation (did not check it).

However, reading through this thread makes me think twice before going through the cross-compile route, at least for a relatively performant platform as a raspberry pi 4.
Also relevant: dist-cc is mentioned as a possible alternative to cross-compiling.

@vmayoral I assume the extension of ament as you propose rather is meant to seamlessly integrate host-native compilers (typically FPGA-brand specific) into the ROS2 build system, as opposed to setting up a full build environment with all target platform specific library dependencies etc?

Regards,
Johan

ament extensions can accomplish both things.

To answer your question, no, ament_acceleration(with extensions such asament_vitis) doesn't just integrate host-native compilers but it also aligns with the rest of the firmware artifacts (see firmware` extensions to ROS 2 workspaces) necessary to cross-compile for target embedded CPUs. See REP 2008 for details, including a reference implementation.

As discussed in past threads (and also in some GitHub issues which aren’t handy right now), we benchmarked native cross-compilation vs OS-virtualized (e.g. via Docker) cross-compilation obtaining that the former was about 10 times faster. When developing and iterating in cycles (e.g. when building accelerators), this time factor is very relevant. That’s why we decided to not build on top of ros_cross_compile tool and instead extend directly ament.

1 Like