Issue Reproduction
Hello, I am currently working on cross-compiling ROS 2 from x86 to ARM. In my project, I have overwritten some of the built-in ROS 2 packages and also created some custom basic packages. However, when I try to use my own packages in other packages, ROS 2 cannot handle this dependency.
When I compile natively on either the x86 or ARM platform, it succeeds without any issues. But when I attempt cross-compiling, I encounter this problem. Therefore, I suspect that ROS 2 might not be handling the import of custom packages properly.
The detailed error message and attempted resolution process are as follows:
After running /cross_root/usr/bin/colcon build --cmake-args -DCMAKE_TOOLCHAIN_FILE="/cross_root/data/toolchain.cmake" --packages-up-to mg_topic
In file included from /home/ros2_ws/src/ros2_libs/mg_tracing_manager/src/perf_record_daemon.cpp:17:
/home/ros2_ws/src/ros2_libs/mg_tracing_manager/include/mg_tracing_manager/logger.hpp:7:10: fatal error: mg/mg_logging.hpp: No such file or directory
7 | #include <mg/mg_logging.hpp>
| ^~~~~~~~~~~~~~~~~~~~~~~
(Actually located at ./install/rclcpp/include/rclcpp/mg/mg_logging.hpp)
Upon inspection, it was found that the program was only looking for header files in the target platform’s ROS 2 software directory at /cross_root/opt/ros/humble/install/include/. After copying several of my package’s header files there,
(one of the commands was cp -r ./install/rclcpp/include/rclcpp/mg/ /cross_root/opt/ros/humble/install/include/rclcpp, along with other header files),
it could find the header files, but it was unable to find implementations for the functions during linking.
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: CMakeFiles/list.dir/list.cpp.o: in function `std::enable_if<rosidl_generator_traits::is_message<statistics_msgs::msg::TopicHz_<std::allocator<void> > >::value, rosidl_message_type_support_t const&>::type rclcpp::get_message_type_support_handle<statistics_msgs::msg::TopicHz_<std::allocator<void> > >()':
list.cpp:(.text._ZN6rclcpp31get_message_type_support_handleIN15statistics_msgs3msg8TopicHz_ISaIvEEEEENSt9enable_ifIXsrN23rosidl_generator_traits10is_messageIT_EE5valueERK29rosidl_message_type_support_tE4typeEv[_ZN6rclcpp31get_message_type_support_handleIN15statistics_msgs3msg8TopicHz_ISaIvEEEEENSt9enable_ifIXsrN23rosidl_generator_traits10is_messageIT_EE5valueERK29rosidl_message_type_support_tE4typeEv]+0xc): undefined reference to `rosidl_message_type_support_t const* rosidl_typesupport_cpp::get_message_type_support_handle<statistics_msgs::msg::TopicHz_<std::allocator<void> > >()'
collect2: error: ld returned 1 exit status
(Actually, the function implementation is located in a library file under ./install)
Upon inspection, it was found that the program was only looking for library files in the target platform’s ROS 2 software directory at /cross_root/opt/ros/humble/install/lib/. After copying all .so files from ./install/ there, it could find them again.
However, this method is not a fundamental solution, and I am looking for a general solution that does not require copying.
Cross-Compilation Environment
I have set up a docker on an x86 server running Ubuntu 20.04, which comes pre-installed with native ROS 2. Then, I copied part of the root directory from an ARM device (including ROS 2) to this docker container. The directory structure of the docker’s root looks roughly as follows:
- usr
- bin
- aarch64-linux-gnu-gcc-9 (compiler for cross-compilation)
- aarch64-linux-gnu-g+±9
- …
- lib
- opt
- nvidia
- ros (host’s ROS 2)
- humble
- home
- ros2_ws
- src
- build (directories created during ROS 2 compilation)
- install
- log
- ros2_libs (all are my own projects)
- control
- driver
- …
- src
- ros2_ws
- cross_root (part of the root directory copied from the ARM machine)
- data
- toolchain.cmake (configuration for cross-compilation)
- etc
- opt
- nvidia
- ros
- humble (ROS 2 for the target platform)
- etc
- install
- bin
- COLCON_IGNORE
- include
- lib
- …
- setup.bash
- melodic
- humble (ROS 2 for the target platform)
- usr
- var
- data
Project Directory Structure
The structure of the project files to be compiled is as follows:
- ros2_libs (rewrote some built-in ROS 2 packages, including rclcpp, pcl_ros, etc., and then added some of my own custom packages)
- abseil-cpp-0.4.2
- CMakelists.txt
- package.xml
- abseil_cpp
- …
- redis++ (a pure CMake package)
- rcl_interfaces (rewritten built-in package)
- rcl_logging
- rclcpp (which calls rcl_interfaces/srv/get_logger_level.hpp)
- …
- mg_cmd (other custom packages)
- mg_trace
- …
- abseil-cpp-0.4.2
- control (for motion state control)
- controller (a ROS 2 package under the control module, which uses abseil-cpp-0.4.2)
Cross-Compiling Configuration
During cross-compilation, aside from using the host’s tools such as aarch64-linux-gnu, cmake, make, etc., I use materials designated for the target platform for everything else: colcon, various ARM architecture libraries, etc. (all placed under cross_root, directly copied from the ARM machine). For example, one compilation process goes as follows:
source /cross_root/opt/ros/humble/setup.bash
/cross_root/usr/bin/colcon build --cmake-args -DCMAKE_TOOLCHAIN_FILE="/cross_root/data/toolchain.cmake" --packages-up-to mg_topic
As mentioned in the first part regarding issue reproduction, running this command results in errors due to missing header files and function implementations.
The content of toolchain.cmake is as follows:
# Located at /cross_root/data/rostoolchain.cmake
# +++++ Set the compiler and linker paths -----
set(CMAKE_SYSROOT /cross_root)
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)
set(CMAKE_C_COMPILER /usr/bin/aarch64-linux-gnu-gcc-9)
set(CMAKE_CXX_COMPILER /usr/bin/aarch64-linux-gnu-g++-9)
set(CMAKE_FIND_ROOT_PATH
/cross_root
/home/ros2_ws/src/install
)
set(CMAKE_MODULE_PATH
/cross_root/usr/share/cmake-3.10/Modules
/cross_root/usr/lib/aarch64-linux-gnu/pkgconfig
)
# +++++ Set runtime settings for the entire cross-compilation process -----
set(CMAKE_CROSSCOMPILING ON)
set(BUILD_TESTING OFF CACHE BOOL "Disable build testing") # Essential, disable gtest
# ++++++ Set python -----
set(Python3_EXECUTABLE /usr/bin/python)
set(PYTHON_SOABI cpython-38-aarch64-linux-gnu)
# +++++ Set cuda -----
set(CMAKE_BUILD_RPATH "/cross_root/usr/lib;/cross_root/usr/lib/aarch64-linux-gnu/tegra/;/cross_root/usr/local/cuda/targets/aarch64-linux/lib/") # Add associated SO rpath, e.g., tegra, cudart
set(CMAKE_SHARED_LINKER_FLAGS "-L/cross_root/usr/lib -L/cross_root/usr/local/cuda/targets/aarch64-linux/lib/ -L /cross_root/usr/local/lib/")
set(CMAKE_EXE_LINKER_FLAGS "-L/cross_root/usr/lib -L/cross_root/usr/local/cuda/targets/aarch64-linux/lib/ -L /cross_root/usr/local/lib/")
# ++++++ Set search modes -----
# Never search for tool programs in the specified directory (for cross-compilation). (Compilation uses host tools)
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
# Only search for libraries in the specified directory (for cross-compilation)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
# Only search for header files in the specified directory (for cross-compilation)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
# Only search for packages in the specified directory (for cross-compilation)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
# ++++++ Set include directories -----
include_directories(
/cross_root/usr/local/include
/cross_root/usr/include/
/cross_root/usr/include/aarch64-linux-gnu/
/home/ros2_ws/src/system/
/home/ros2_ws/src/system/mg_reporter/include/
/home/ros2_ws/src/deps/
# Native ROS 2 can automatically handle the below, but it doesn't work after switching to cross_root
# /home/ros2_ws/src/install/rcl_interfaces/include/rcl_interfaces/
# /home/ros2_ws/src/install/statistics_msgs/include/statistics_msgs/
)
include_directories(BEFORE
/home/ros2_ws/src/ros2_libs/rcutils/include
/home/ros2_ws/src/ros2_libs/rcl_logging/rcl_logging_interface/include
)
Attempted Solutions
- I tried modifying
toolchain.cmake’sinclude_directories()to force cmake and make to find header files in the generation directory of custom packages. This indeed allowed for the correct processing of imported header files, but this method requires specifying every search path explicitly and needs to be updated with the addition of new packages, making it less scalable. Moreover, even if the header files were imported, the library files for function implementations could not be found, leading to errors. - I tried using the host’s
colcon, but encountered the same issues. - I attempted native compilation on both x86 and ARM, which succeeded without issues, but cross-compilation failed.