ROS Resources: Documentation | Support | Discussion Forum | Service Status | Q&A answers.ros.org

How to cross-compile ROS2 for arm64 architecture


#1

I am trying to cross-compile ROS2 for arm64 ubuntu box. It was going well until hitting an issue in ‘resource_retriever’.

Scanning dependencies of target resource_retriever
[ 50%] Building CXX object CMakeFiles/resource_retriever.dir/src/retriever.cpp.o
[100%] Linking CXX shared library libresource_retriever.so
/home/sirius/projects/ros2/build_isolated/libcurl_vendor/libcurl_install/lib/libcurl.so: error adding symbols: File in wrong format
collect2: error: ld returned 1 exit status
CMakeFiles/resource_retriever.dir/build.make:96: recipe for target 'libresource_retriever.so' failed
make[2]: *** [libresource_retriever.so] Error 1
CMakeFiles/Makefile2:99: recipe for target 'CMakeFiles/resource_retriever.dir/all' failed
make[1]: *** [CMakeFiles/resource_retriever.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

Here is the command for cross-compiling

./src/ament/ament_tools/scripts/ament.py build --isolated \
--skip-packages rviz_assimp_vendor rviz_ogre_vendor \
                rviz_yaml_cpp_vendor rviz_rendering rviz_rendering_tests \
                rviz_common rviz2 rviz_default_plugins ros1_bridge \
--symlink-install \
--cmake-args -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \
             -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \
             -DCMAKE_LINKER=aarch64-linux-gnu-ld \
             -DCMAKE_TOOLCHAIN_FILE=${PWD}/JETSON_TX2_AARCH64_CMAKE_TOOLCHAIN_FILE

The reason of the failure seems to be related to ‘ExternalProject_Add’ macro for CONFIGURE_COMMAND doesn’t take cross-compiling arguments into ‘autoconf/automake’ style packages.

ExternalProject_Add(curl-7.57.0
  URL https://github.com/curl/curl/releases/download/curl-7_57_0/curl-7.57.0.tar.gz
  URL_MD5 c7aab73aaf5e883ca1d7518f93649dc2
  CONFIGURE_COMMAND
    <SOURCE_DIR>/configure
    CFLAGS=${extra_c_flags}
    CXXFLAGS=${extra_cxx_flags}
    PKG_CONFIG_PATH=$PKG_CONFIG_PATH:${PKGCONFIG_FOR_OPENSSL}
    --prefix=${CMAKE_CURRENT_BINARY_DIR}/libcurl_install
    --with-ssl
  BUILD_COMMAND ${MAKE}
  INSTALL_COMMAND make install
  TIMEOUT 600
)

It makes me think if the method for cross-compiling is right…

How could I cross-compile ROS2 for arm64? Is there a step-by-step guide on this?


ROS2 Cross compilation
#2

By-passing the issue on libcurl_vendor and resource_retriever by ‘–skip-packages’. However, it seems now I am blocked by another strange issue.

[ 82%] Building C object CMakeFiles/std_srvs__rosidl_typesupport_c__pyext.dir/rosidl_generator_py/std_srvs/_std_srvs_s.ep.rosidl_typesupport_c.c.o
In file included from /usr/include/python3.5m/Python.h:8:0,
                 from /home/sirius/projects/ros2/build_isolated/std_srvs/rosidl_generator_py/std_srvs/_std_srvs_s.ep.rosidl_typesupport_introspection_c.c:4:
/usr/include/python3.5m/pyconfig.h:9:12: fatal error: aarch64-linux-gnu/python3.5m/pyconfig.h: No such file or directory
 #  include <aarch64-linux-gnu/python3.5m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

It is complaining that the header file is missing. However, I do have the file install with multiple-arch solution in my Ubuntu 16.04 box.

sirius@Sirius-vm:~/projects/ros2$ find /usr/ -iname "pyconfig.h"
/usr/include/python3.5m/pyconfig.h
/usr/include/aarch64-linux-gnu/python3.5m/pyconfig.h
/usr/include/aarch64-linux-gnu/python2.7/pyconfig.h
/usr/include/python2.7/pyconfig.h
/usr/include/x86_64-linux-gnu/python3.5m/pyconfig.h

It seems that ‘ament’ doesn’t take ‘/usr/include’ into its INCLUDE_PATH. I have also tried out to add the path in by define ‘-DCMAKE_INCLUDE_PATH=/usr/include’ which was rejected by ‘amend’.

Does ROS2 support cross-compilation for now?


#3

I think this guy - @lmayencourt upstreamed the ROS2 cross-compilation can help you. :slight_smile:


#4

Yes ROS2 can be cross-compiled :slight_smile:

You might want to have a look to:
https://github.com/ros2/ros2/tree/master/arm_crosscompilation
(but python support has been removed. Only CPP support is present)

There is also the following link that may help:


#5

By these, cross-compilation is working.

sed -e '/py/ s/^#*/#/' -i src/ros2/rosidl_typesupport/rosidl_default_generators/CMakeLists.txt
sed -i -r 's/<build(.+?py.+?)/<\!\-\-build\1\-\->/' src/ros2/rosidl_typesupport/rosidl_default_generators/package.xml

touch \
  src/ros2/demos/AMENT_IGNORE \
  src/ros2/examples/rclpy/AMENT_IGNORE \
  src/ros2/geometry2/AMENT_IGNORE \
  src/ros2/orocos_kinematics_dynamics/AMENT_IGNORE \
  src/ros2/rclpy/AMENT_IGNORE \
  src/ros2/rcl_interfaces/test_msgs/AMENT_IGNORE \
  src/ros2/robot_model/AMENT_IGNORE \
  src/ros2/robot_state_publisher/AMENT_IGNORE \
  src/ros2/ros1_bridge/AMENT_IGNORE \
  src/ros2/rosidl/rosidl_generator_py/AMENT_IGNORE \
  src/ros2/system_tests/AMENT_IGNORE \
  src/ros2/urdfdom/AMENT_IGNORE \
  src/ros2/urdfdom_headers/AMENT_IGNORE \
  src/ros2/urdf/AMENT_IGNORE \
  src/ros2/urdfdom/AMENT_IGNORE \
  src/ros2/kdl_parser/AMENT_IGNORE

src/ament/ament_tools/scripts/ament.py build \
    --force-cmake-configure \
    --skip-packages resource_retriever \
            rviz_assimp_vendor rviz_ogre_vendor \
            rviz_yaml_cpp_vendor rviz_rendering \
            rviz_rendering_tests rviz_common \
            rviz2 rviz_default_plugins \
    --cmake-args \
    -DCMAKE_TOOLCHAIN_FILE=`pwd`/aarch64_toolchainfile.cmake \
    -DTHIRDPARTY=ON --

However, ‘ExternalProject_Add’ macro could not take toolchain settings into CMake based vendor package. That is the reason why ‘resource_retriever’ package is skipped. I may spend some time to look into the issue.


#6

It is helpful. Thank you, @pokitoz


#7

Another question is about the statement in https://github.com/ros2/ros2/tree/master/arm_crosscompilation

No target filesystem with Python libraries is provided here, so Python is not supported (only C++).

Does that mean if I could get a ‘rootfs’ working for aarch64 I may enable python support?


#8

Yes, I did that earlier with an arm64 Ubuntu 16.04 rootfs and it was working fine with a simple python publisher.


#9

Yes, that’s exactly what https://github.com/esteve/ros2_raspbian_tools does to cross compile ROS2 for the Raspberry Pi. The scripts in the repo fetch a Raspbian image and prepare a rootfs so that CMake will find all the dependencies and compile/link against them.

The scripts in https://github.com/esteve/ros2_raspbian_tools are generic enough that they can be adapted to cross-compile for aarch64, let me have a look at it and I’ll flesh them out. What’s your target aarch64 distribution? Ubuntu?


#10

I am working on Jetson TX2 board, which has Ubuntu 16.04 arm64 installed. I think I can ‘rsync’ to fetch back the rootfs and try to cross-compile against.


#11

I am not family with ‘polly’ and ‘docker’ but ‘ros2_raspbian_tools’ looks pretty good. If it could work with any rootfs rather than raspbain only that will be great.


#12

Hmmmm, if it is about a python publish, will it work with ‘multiarch’ enabled Ubuntu box. What I am trying to do is about to install ‘libpython3-dev:arm64’ package for python dev headers and libs. However, it seems ‘ament’ cannot find them in ‘/usr/include’ or ‘/usr/lib’.


#13

Yeah, you’ll need python3-dev:arm64 because it contains the pkgconfig files for Python3 that ament uses to “discover” the location of the Python headers and libraries.

The instructions in https://github.com/ros2/ros2/tree/master/arm_crosscompilation use the changes I made to ROS2 to cross-compile it for Android and iOS (i.e. https://github.com/eProsima/Fast-RTPS/pull/26, https://github.com/ros2/rmw_fastrtps/pull/27 and a bunch of others). But if you want to target a Linux distro, it’s best to use a rootfs instead since you’ll end up with a complete ROS2 environment, I’m going to update the guide in the wiki to reflect that.


#14

To setup the python support, I used an already created ubuntu-base:
http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz

And then I installed the needed packages using chroot + qemu-aarch64-static

In the toolchain file I used the variable CMAKE_SYSROOT to specify the path of my rootfs to ament/cmake and compile.


#15

Thank you, @esteve and @pokitoz. I got cross-compilation done for Jetson TX2.

Here is a brief for who is interesting on this.

    set(CMAKE_SYSTEM_NAME Linux)
    set(CMAKE_SYSTEM_VERSION 1)
    set(CMAKE_SYSTEM_PROCESSOR aarch64)
    set(CMAKE_SYSROOT $ENV{SYSROOT})
    # specify the cross compiler
    set(CMAKE_C_COMPILER $ENV{CROSS_COMPILE}gcc)
    set(CMAKE_CXX_COMPILER $ENV{CROSS_COMPILE}g++)
    # where is the target environment
    set(CMAKE_FIND_ROOT_PATH ${CMAKE_CURRENT_LIST_DIR}/install)
    set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
    set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
    set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
    set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

    # This assumes that pthread will be available on the target system
    # (this emulates that the return of the TRY_RUN is a return code "0"
    set(THREADS_PTHREAD_ARG "0"
      CACHE STRING "Result from TRY_RUN" FORCE)
  • Apt ‘sources.list’
    deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial main restricted
    #deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial main restricted
    deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main restricted
    #deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main restricted
    deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted
    #deb-src [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted
  • Install ‘libpython3-dev:arm64’

  • Ignore ‘rviz’ and ‘opencv’ related packages

    src/ament/ament_tools/scripts/ament.py build \
        --force-cmake-configure \
        --skip-packages resource_retriever \
                        rviz_assimp_vendor \
                        rviz_ogre_vendor \
                        rviz_yaml_cpp_vendor \
                        rviz_rendering \
                        rviz_rendering_tests \
                        rviz_common \
                        rviz2 \
                        rviz_default_plugins \
        --cmake-args \
            -DCMAKE_TOOLCHAIN_FILE=`pwd`/aarch64_toolchainfile.cmake \
            -DTHIRDPARTY=ON \
            -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
            -DPYTHON_INCLUDE_DIR=$(python3 -c "from distutils.sysconfig import get_python_inc; print(get_python_inc() + ';' + '/usr/include')") \
            -DPYTHON_LIBRARY=$(python3 -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))")/aarch64-linux-gnu/libpython3.5m.so --

#16

Hi jcadam, esteve,

I try to reproduce the cross-compilation of the complete stack and get some issue during the process.
I try to follow esteve approach with docker to build the rootfs based on a arm64v8 ubuntu docker image. The cmake used is similar to the one above, and I’m using esteve “build_ros2.bash” script to trigger the build.
During the process, I got the following error:
fatal error: Eigen/Core: No such file or directory
#include <Eigen/Core>

CMake ignore the manually specified location : “DEIGEN3_INCLUDE_DIR=”${RASPBERRYPI_CROSS_COMPILE_SYSROOT}/usr/include/eigen3"
and use the default “/usr/include” path:
– Using Eigen3 include dirs: /usr/include/eigen3

I’m struggling to understand why CMake ignore the specified path… Any idea ?


#17

Hi…i am a new user here. As per my knowledge the 64-bit ARM environment is relatively new. This means that most applications, including ROS, need to be ported to the new processor. For a complicated application like ROS, this may necessitate a variety of code and environment changes to support the new computer architecture.

board assembly


#18

Hi ChesLans,

There is no issue to build ROS2 for aarch64. You can follow the wiki instruction on your aarch64 machine and build the complete stack, it will just take some time =)
For cross-compilation, you can build a subset of ROS2 by following the instruction here.
@jcadam and @esteve already have successfully cross-compile the complete ROS2 for raspi (arm 32) and the Jetson TX2 (arm 64).


#19

FYI to the community, none of the above mentioned approaches work with the release of Crystal without a good deal of digging into the scripts to address moved or added dependencies like python-catkin-pkg-modules, and others. At least I haven’t been able to find a working solution…so I’m going at it from scratch. As soon as I have it, I’ll post.


#20

Hi @binary42,

A new documentation page just get merged explaining how to achieve cross-compilation of the Crystal release, based on the assets available at ros2/cross_compile

Please let me know if you have any issue with it !