Humble build farm job on arm fails on first release

Buildfarm
1

Hi,
We release a first version on librealsense2 release for Humble distro.
We saw that x86 + arm Jenkins jobs failed… after a few days the x86 jobs passed (without changes on our side)
and currently the arm build still fails.
Is there a known issue on arm buildfarm or something?

https://build.ros2.org/view/Hsrc_uJ/job/Hbin_ujv8_uJv8__librealsense2__ubuntu_jammy_arm64__binary/

Thanks,
Nir

1 Like
2

In general this buildfarm category is for discussing the buildfarm infrastructure itself rather than getting help with specific builds. I’d recommend asking similar questions on ROS Answers in the future.

The output of the job that you linked fails during the build section.
It looks like this is the error which halted the build

23:37:24 cd /tmp/binarydeb/ros-humble-librealsense2-2.51.1/.obj-aarch64-linux-gnu/tools/convert && /usr/bin/cmake -E cmake_link_script CMakeFiles/rs-convert.dir/link.txt --verbose=1
23:37:24 /usr/lib/ccache/c++ -g -O2 -ffile-prefix-map=/tmp/binarydeb/ros-humble-librealsense2-2.51.1=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 -pedantic -g -Wno-missing-field-initializers -Wno-switch -Wno-multichar -Wsequence-point -Wformat -Wformat-security -mstrict-align -ftree-vectorize -pthread -rdynamic CMakeFiles/rs-convert.dir/rs-convert.cpp.o CMakeFiles/rs-convert.dir/converter.cpp.o CMakeFiles/rs-convert.dir/converters/converter-csv.cpp.o -o rs-convert  -Wl,-rpath,/tmp/binarydeb/ros-humble-librealsense2-2.51.1/.obj-aarch64-linux-gnu: ../../librealsense2.so.2.51.1 
23:37:24 /usr/bin/ld: ../../librealsense2.so.2.51.1: undefined reference to `fw_target_data'
23:37:26 /usr/bin/ld: ../../librealsense2.so.2.51.1: undefined reference to `fw_target_size'
23:37:26 collect2: error: ld returned 1 exit status

I’m not familiar enough with librealsense to say whether these symbols are from a third party library or within librealsense itself but it appears that there is a configuration issue with the build that only expresses on arm64.

Our builds run on native ARM hardware but you can also run them on your local machine if you have python3, docker and qemu-user-static installed and configured with binfmt_misc.
If you’re on Ubuntu 20.04 or 22.04 the qemu-user-static package will recommend the binfmt-support package needed and you should be good to go as long as you have both of them installed.

To run buildfarm jobs you’ll need a copy of ros_buildfarm from the master branch to replicate our CI environment. I usually recommend installing it in a virtual environment.
Once installed locally you can run the following commands

mkdir /tmp/librealsense2-humble-arm64
generate_release_script.py https://raw.githubusercontent.com/ros2/ros_buildfarm_config/ros2/index.yaml humble ujv8 librealsense2 ubuntu jammy arm64 > /tmp/librealsense2-humble-arm64/run.sh
cd /tmp/librealsense2-humble-arm64
sh run.sh

Skipping the ceremony of creating and switching to the temporary directory the generate_release_script.py command is taking our ROS 2 buildfarm confinguration via url and some further configuration values to generate a script which will run use docker to build and run the same tasks in the same contexts that they would be run on our Jenkins agent. It may run a bit slower through qemu but unless the problem only appears natively it should produce the same problem to help you investigate further.

One downside of this approach is that it’s very hard to make changes to the build context or the source code built by it since the script is exactly reproducing our build farm’s behavior to fetch and build the project.

3

Hi @nuclearsandwich , Appreciate the very informative answer.
Strangely the same code works for Foxy + Galactic (x86 + arm), failing only on Humble arm (x86 works).
We will try the advised approach and will keep in mind the note about the incorrect forum.
Thanks!

Nir

4

Hi @nuclearsandwich , @Nir-Az

As I see, these flags “-flto=auto -ffat-lto-objects” are being added to arm compilation for C files in Humble distro. This was not the case on galactic, as you can see here:
Arm - Humble - Jammy : https://build.ros2.org/view/Hsrc_uJ/job/Hbin_ujv8_uJv8__librealsense2__ubuntu_jammy_arm64__binary/47/consoleText
Arm - Galactic - Focal :
https://build.ros2.org/view/Gsrc_uF/job/Gbin_ufv8_uFv8__librealsense2__ubuntu_focal_arm64__binary/20/consoleText
(search for “common/fw/CMakeFiles/fw.dir/target.c.o” to see the differences)

I guess these flags in humble environment removes the fw_target_data and fw_target_size since they are not in use, and then when trying to link with these references, we get the error.

Do you know why these flags are being added or how can we manage disable them ?

Thanks,
Samer

5

Link Time Optimization has been turned on by default in the debian toolchain.

I repled with more details in the GitHub issue: