How to cross-compile ROS2 for arm64 architecture

I am trying to cross-compile ROS2 for arm64 ubuntu box. It was going well until hitting an issue in ‘resource_retriever’.

Scanning dependencies of target resource_retriever
[ 50%] Building CXX object CMakeFiles/resource_retriever.dir/src/retriever.cpp.o
[100%] Linking CXX shared library libresource_retriever.so
/home/sirius/projects/ros2/build_isolated/libcurl_vendor/libcurl_install/lib/libcurl.so: error adding symbols: File in wrong format
collect2: error: ld returned 1 exit status
CMakeFiles/resource_retriever.dir/build.make:96: recipe for target 'libresource_retriever.so' failed
make[2]: *** [libresource_retriever.so] Error 1
CMakeFiles/Makefile2:99: recipe for target 'CMakeFiles/resource_retriever.dir/all' failed
make[1]: *** [CMakeFiles/resource_retriever.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

Here is the command for cross-compiling

./src/ament/ament_tools/scripts/ament.py build --isolated \
--skip-packages rviz_assimp_vendor rviz_ogre_vendor \
                rviz_yaml_cpp_vendor rviz_rendering rviz_rendering_tests \
                rviz_common rviz2 rviz_default_plugins ros1_bridge \
--symlink-install \
--cmake-args -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \
             -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \
             -DCMAKE_LINKER=aarch64-linux-gnu-ld \
             -DCMAKE_TOOLCHAIN_FILE=${PWD}/JETSON_TX2_AARCH64_CMAKE_TOOLCHAIN_FILE

The reason of the failure seems to be related to ‘ExternalProject_Add’ macro for CONFIGURE_COMMAND doesn’t take cross-compiling arguments into ‘autoconf/automake’ style packages.

ExternalProject_Add(curl-7.57.0
  URL https://github.com/curl/curl/releases/download/curl-7_57_0/curl-7.57.0.tar.gz
  URL_MD5 c7aab73aaf5e883ca1d7518f93649dc2
  CONFIGURE_COMMAND
    <SOURCE_DIR>/configure
    CFLAGS=${extra_c_flags}
    CXXFLAGS=${extra_cxx_flags}
    PKG_CONFIG_PATH=$PKG_CONFIG_PATH:${PKGCONFIG_FOR_OPENSSL}
    --prefix=${CMAKE_CURRENT_BINARY_DIR}/libcurl_install
    --with-ssl
  BUILD_COMMAND ${MAKE}
  INSTALL_COMMAND make install
  TIMEOUT 600
)

It makes me think if the method for cross-compiling is right…

How could I cross-compile ROS2 for arm64? Is there a step-by-step guide on this?

By-passing the issue on libcurl_vendor and resource_retriever by ‘–skip-packages’. However, it seems now I am blocked by another strange issue.

[ 82%] Building C object CMakeFiles/std_srvs__rosidl_typesupport_c__pyext.dir/rosidl_generator_py/std_srvs/_std_srvs_s.ep.rosidl_typesupport_c.c.o
In file included from /usr/include/python3.5m/Python.h:8:0,
                 from /home/sirius/projects/ros2/build_isolated/std_srvs/rosidl_generator_py/std_srvs/_std_srvs_s.ep.rosidl_typesupport_introspection_c.c:4:
/usr/include/python3.5m/pyconfig.h:9:12: fatal error: aarch64-linux-gnu/python3.5m/pyconfig.h: No such file or directory
 #  include <aarch64-linux-gnu/python3.5m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

It is complaining that the header file is missing. However, I do have the file install with multiple-arch solution in my Ubuntu 16.04 box.

sirius@Sirius-vm:~/projects/ros2$ find /usr/ -iname "pyconfig.h"
/usr/include/python3.5m/pyconfig.h
/usr/include/aarch64-linux-gnu/python3.5m/pyconfig.h
/usr/include/aarch64-linux-gnu/python2.7/pyconfig.h
/usr/include/python2.7/pyconfig.h
/usr/include/x86_64-linux-gnu/python3.5m/pyconfig.h

It seems that ‘ament’ doesn’t take ‘/usr/include’ into its INCLUDE_PATH. I have also tried out to add the path in by define ‘-DCMAKE_INCLUDE_PATH=/usr/include’ which was rejected by ‘amend’.

Does ROS2 support cross-compilation for now?

I think this guy - @lmayencourt upstreamed the ROS2 cross-compilation can help you. :slight_smile:

Yes ROS2 can be cross-compiled :slight_smile:

You might want to have a look to:
https://github.com/ros2/ros2/tree/master/arm_crosscompilation
(but python support has been removed. Only CPP support is present)

There is also the following link that may help:

1 Like

By these, cross-compilation is working.

sed -e '/py/ s/^#*/#/' -i src/ros2/rosidl_typesupport/rosidl_default_generators/CMakeLists.txt
sed -i -r 's/<build(.+?py.+?)/<\!\-\-build\1\-\->/' src/ros2/rosidl_typesupport/rosidl_default_generators/package.xml

touch \
  src/ros2/demos/AMENT_IGNORE \
  src/ros2/examples/rclpy/AMENT_IGNORE \
  src/ros2/geometry2/AMENT_IGNORE \
  src/ros2/orocos_kinematics_dynamics/AMENT_IGNORE \
  src/ros2/rclpy/AMENT_IGNORE \
  src/ros2/rcl_interfaces/test_msgs/AMENT_IGNORE \
  src/ros2/robot_model/AMENT_IGNORE \
  src/ros2/robot_state_publisher/AMENT_IGNORE \
  src/ros2/ros1_bridge/AMENT_IGNORE \
  src/ros2/rosidl/rosidl_generator_py/AMENT_IGNORE \
  src/ros2/system_tests/AMENT_IGNORE \
  src/ros2/urdfdom/AMENT_IGNORE \
  src/ros2/urdfdom_headers/AMENT_IGNORE \
  src/ros2/urdf/AMENT_IGNORE \
  src/ros2/urdfdom/AMENT_IGNORE \
  src/ros2/kdl_parser/AMENT_IGNORE

src/ament/ament_tools/scripts/ament.py build \
    --force-cmake-configure \
    --skip-packages resource_retriever \
            rviz_assimp_vendor rviz_ogre_vendor \
            rviz_yaml_cpp_vendor rviz_rendering \
            rviz_rendering_tests rviz_common \
            rviz2 rviz_default_plugins \
    --cmake-args \
    -DCMAKE_TOOLCHAIN_FILE=`pwd`/aarch64_toolchainfile.cmake \
    -DTHIRDPARTY=ON --

However, ‘ExternalProject_Add’ macro could not take toolchain settings into CMake based vendor package. That is the reason why ‘resource_retriever’ package is skipped. I may spend some time to look into the issue.

It is helpful. Thank you, @pokitoz

Another question is about the statement in https://github.com/ros2/ros2/tree/master/arm_crosscompilation

No target filesystem with Python libraries is provided here, so Python is not supported (only C++).

Does that mean if I could get a ‘rootfs’ working for aarch64 I may enable python support?

Yes, I did that earlier with an arm64 Ubuntu 16.04 rootfs and it was working fine with a simple python publisher.

Yes, that’s exactly what GitHub - esteve/ros2_raspbian_tools: Tools for crosscompiling ROS2 for the Raspberry Pi does to cross compile ROS2 for the Raspberry Pi. The scripts in the repo fetch a Raspbian image and prepare a rootfs so that CMake will find all the dependencies and compile/link against them.

The scripts in GitHub - esteve/ros2_raspbian_tools: Tools for crosscompiling ROS2 for the Raspberry Pi are generic enough that they can be adapted to cross-compile for aarch64, let me have a look at it and I’ll flesh them out. What’s your target aarch64 distribution? Ubuntu?

I am working on Jetson TX2 board, which has Ubuntu 16.04 arm64 installed. I think I can ‘rsync’ to fetch back the rootfs and try to cross-compile against.

I am not family with ‘polly’ and ‘docker’ but ‘ros2_raspbian_tools’ looks pretty good. If it could work with any rootfs rather than raspbain only that will be great.

Hmmmm, if it is about a python publish, will it work with ‘multiarch’ enabled Ubuntu box. What I am trying to do is about to install ‘libpython3-dev:arm64’ package for python dev headers and libs. However, it seems ‘ament’ cannot find them in ‘/usr/include’ or ‘/usr/lib’.

Yeah, you’ll need python3-dev:arm64 because it contains the pkgconfig files for Python3 that ament uses to “discover” the location of the Python headers and libraries.

The instructions in https://github.com/ros2/ros2/tree/master/arm_crosscompilation use the changes I made to ROS2 to cross-compile it for Android and iOS (i.e. Removed Boost and ported to C++11. Added support for Android and iOS by esteve · Pull Request #26 · eProsima/Fast-DDS · GitHub, https://github.com/ros2/rmw_fastrtps/pull/27 and a bunch of others). But if you want to target a Linux distro, it’s best to use a rootfs instead since you’ll end up with a complete ROS2 environment, I’m going to update the guide in the wiki to reflect that.

To setup the python support, I used an already created ubuntu-base:
http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz

And then I installed the needed packages using chroot + qemu-aarch64-static

In the toolchain file I used the variable CMAKE_SYSROOT to specify the path of my rootfs to ament/cmake and compile.

Thank you, @esteve and @pokitoz. I got cross-compilation done for Jetson TX2.

Here is a brief for who is interesting on this.

    set(CMAKE_SYSTEM_NAME Linux)
    set(CMAKE_SYSTEM_VERSION 1)
    set(CMAKE_SYSTEM_PROCESSOR aarch64)
    set(CMAKE_SYSROOT $ENV{SYSROOT})
    # specify the cross compiler
    set(CMAKE_C_COMPILER $ENV{CROSS_COMPILE}gcc)
    set(CMAKE_CXX_COMPILER $ENV{CROSS_COMPILE}g++)
    # where is the target environment
    set(CMAKE_FIND_ROOT_PATH ${CMAKE_CURRENT_LIST_DIR}/install)
    set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
    set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
    set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
    set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

    # This assumes that pthread will be available on the target system
    # (this emulates that the return of the TRY_RUN is a return code "0"
    set(THREADS_PTHREAD_ARG "0"
      CACHE STRING "Result from TRY_RUN" FORCE)
  • Apt ‘sources.list’
    deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial main restricted
    #deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial main restricted
    deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main restricted
    #deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main restricted
    deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted
    #deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted
  • Install ‘libpython3-dev:arm64’

  • Ignore ‘rviz’ and ‘opencv’ related packages

    src/ament/ament_tools/scripts/ament.py build \
        --force-cmake-configure \
        --skip-packages resource_retriever \
                        rviz_assimp_vendor \
                        rviz_ogre_vendor \
                        rviz_yaml_cpp_vendor \
                        rviz_rendering \
                        rviz_rendering_tests \
                        rviz_common \
                        rviz2 \
                        rviz_default_plugins \
        --cmake-args \
            -DCMAKE_TOOLCHAIN_FILE=`pwd`/aarch64_toolchainfile.cmake \
            -DTHIRDPARTY=ON \
            -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
            -DPYTHON_INCLUDE_DIR=$(python3 -c "from distutils.sysconfig import get_python_inc; print(get_python_inc() + ';' + '/usr/include')") \
            -DPYTHON_LIBRARY=$(python3 -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))")/aarch64-linux-gnu/libpython3.5m.so --
1 Like

Hi jcadam, esteve,

I try to reproduce the cross-compilation of the complete stack and get some issue during the process.
I try to follow esteve approach with docker to build the rootfs based on a arm64v8 ubuntu docker image. The cmake used is similar to the one above, and I’m using esteve “build_ros2.bash” script to trigger the build.
During the process, I got the following error:
fatal error: Eigen/Core: No such file or directory
#include <Eigen/Core>

CMake ignore the manually specified location : “DEIGEN3_INCLUDE_DIR=”${RASPBERRYPI_CROSS_COMPILE_SYSROOT}/usr/include/eigen3"
and use the default “/usr/include” path:
– Using Eigen3 include dirs: /usr/include/eigen3

I’m struggling to understand why CMake ignore the specified path… Any idea ?

Hi…i am a new user here. As per my knowledge the 64-bit ARM environment is relatively new. This means that most applications, including ROS, need to be ported to the new processor. For a complicated application like ROS, this may necessitate a variety of code and environment changes to support the new computer architecture.

board assembly

Hi ChesLans,

There is no issue to build ROS2 for aarch64. You can follow the wiki instruction on your aarch64 machine and build the complete stack, it will just take some time =)
For cross-compilation, you can build a subset of ROS2 by following the instruction here.
@jcadam and @esteve already have successfully cross-compile the complete ROS2 for raspi (arm 32) and the Jetson TX2 (arm 64).

FYI to the community, none of the above mentioned approaches work with the release of Crystal without a good deal of digging into the scripts to address moved or added dependencies like python-catkin-pkg-modules, and others. At least I haven’t been able to find a working solution…so I’m going at it from scratch. As soon as I have it, I’ll post.

1 Like

Hi @binary42,

A new documentation page just get merged explaining how to achieve cross-compilation of the Crystal release, based on the assets available at ros2/cross_compile

Please let me know if you have any issue with it !