Leveraging ROS build infrastructure for non-ROS packages

, ,
1

I had an interesting discussion with @peci1 over on the exiv2 github and we decided to move it here for some more opinions/recommendations because it illustrates a larger issue with the standard ROS 2 build tooling when a company wants to build a larger software stack with colcon and rosdep.

First, some background

  • I work at a company that uses ROS 2 with C++
  • We use colcon to build the software we maintain
  • We use package.xml files for colcon to know build dependencies
  • We have a few forks of ROS repos (rosidl, for example) with changes that are not merged upstream
  • We also rely a lot on system packages from outside ROS
  • These system packages have known bugs or missing ABI compatible features on Ubuntu 22ā€™s apt version, so we want to use newer versions. exiv2 is one such example, but there about 15 packages in such a state. I usually upstream our patches, but in some cases the repos are also unmaintained, so we carry these patches internally
  • We do NOT worry about breaking compatibility with what the ROS distribution or Ubuntu distribution because we have a product to deliver and no time to worry about ABI outside our closed source applications
  • We donā€™t care about internal ABI, we donā€™t use SemVer on internal package, and our internal messages arenā€™t stable either. We use VCS to freeze the state of our source code with vcs export --exact when we do a release.

Constraints

  • We want to avoid duplicating building the same library multiple times with tools like FetchContent because our CI build times get too slow
  • We donā€™t have a large enough team or funding to switch to conan, switch to bazel, switch to yocto, and so far, we havenā€™t set up the ROS build farm interally thatā€™s also intimidating amount of work.
  • We want to do our best to only use one version of a library in the project (IE, donā€™t link against 4 different versions of exiv2 in four different packages). Due to the size of our project, this is a reasonable restriction
  • Maintainers of non-ROS packages like exiv2 are not interested in a package.xml in their repo
  • The ubuntu package maintainers appear to be understaffed and underfunded so long-standing bugs still exist that affect us, largely in their CMake code that show up when you cross compile. Many packages in Ubuntu distros are built with autotools even though the packages support CMake for years. Iā€™m not ready to sign up as an Ubuntu package maintainer , so this problem is probably not going away any time soon

How we currently solve the problem:

  • For packages like exiv2, we have a repo that contains shell scripts called ā€œbuild_exiv2.shā€ and ā€œinstall_exiv2.shā€.
    Originally, these were CMake builds and Cmake installs, but we recently started uploading debians and installing debians
  • In docker for our dev and deploy images, we use a RUN install_exiv2.sh which takes advantage of the docker build cache. This happens AFTER rosdep.
  • We skip rosdep keys on packages like exiv2 because we want to avoid installing the same library twice - this has bitten us multiple times. If you try compiling some of our ROS package outside of our development environment that installed exiv2, it will fail to compile. You can get around this by running install_exiv2.sh.
  • We use ccache to speed up builds. We donā€™t really care how large our source workspace gets because we can pin dependencies and ccache makes it really fast for already built packages.
  • We use vcs to keep track of all ROS-package dependencies
  • For meta-repos, we use yq to do a merge of the vcs files, this ensures we only build one version of a repo (This has some problems if two packages pin different versions, so we manually keep these in sync)
  • We cross compile all our source code with a sysroot, and carry many internal patches for libraries that do not cross compile well, as well as workarounds in our cross compile toolchain that rewrite absolute paths to relative paths when we donā€™t have time to fix upstream CMake code
    • The upstream fixes can be quite involved. Hereā€™s a recent example I did for LibKML

What we would like:

  • All packages we build from source for to be in the ROS workspace
  • Rosdep doesnā€™t need a list of keys to skip that we propograte through all our repos
  • Reduce the number of custom lines of code maintaining custom packaging
  • Spend as little time on devops as possible long term
  • Not make extra work every time we want to break internal ABI
  • master branch of everything we build to work with a cross compile toolchain, and the eventual upgrade to Ubuntu 24/26/28 to have debians that work in sysroot out of the box.

A simple use case illustrating the difficulty:

  1. Create a ROS workspace with a src directory
  2. Clone exiv2 in the workspace git clone https://github.com/Exiv2/exiv2.git
  3. Create another package that depends on it: ros2 pkg create exiv2_consumer --build-type ament_cmake --dependencies exiv2
  4. Run rosdep as normal rosdep install --from-paths src --ignore-src
  5. Oh crap, it wants to install Ubuntuā€™s old broken version: executing command [sudo -H apt-get install exiv2] even though exiv2 is in my workspace
  6. Ok, add a --skip-keys exiv2 everywhere we call rosdep. rosdep install --from-paths src --ignore-src --skip-keys exiv2
  7. Dependencies are installed, time to build! colcon build
CMake Error at cmake/Findinih.cmake:28 (message):
  inih library not found
Call Stack (most recent call first):
  cmake/findDependencies.cmake:84 (find_package)
  CMakeLists.txt:84 (include)

Sighā€¦ (proceeds to submit a PR to our pre-colcon build repo with yet another shell script that needs installs apt dependencies and exiv2 in all dockerfiles that consume exiv2_consumer)

Other approaches Iā€™ve seen (docker-based)

# Dockerfile in project/repo1
RUN apt-get update && apt-get install 
  < giant list of packages that are outside rosdep

# Dockerfile in project/repo2
RUN apt-get update && apt-get install 
  < A slightly different giant list of packages that are outside rosdep

Or, a slighty better version (ignore the actual syntax, you get the point)

# Dockerfile in project/repo1
RUN apt-get update && cat local-apt-pkgs-list.txt | xargs apt-get install 

# Dockerfile in project/repo2
RUN apt-get update && apt-get install 
  cat local-apt-pkgs-list.txt | xargs apt-get install  &&
  cat project/repo1/local-apt-pkgs-list.txt
   # And it spirals out of control with >10 repos

Or this way of treating docker as our dependency manager which you have periodic build system failures for a rolling hash or tagging nightmares every time you want to break ABI

# Dockerfile in project/repo1
FROM ros:rolling as base
COPY --from project/repo1/base /usr /usr
COPY --from project/repo2/base /usr /usr
...

Questions:

  • How are you solving this workflow issue? (If itā€™s conan, or a smattering of pre-colcon shell scripts, please share)
  • Are there any quick wins I could contribute to colcon and/or rosdep to make this workflow easier
  • Is there any chance there could be a way to supply a 3rd party package.xml for a non-ROS package so that dependencies can be managed through rosdep without requiring maintainers to accept ROS tooling into their repos?
2 Likes
2

Iā€™ll repost here part of the solution that can be used right away, and is often very overlooked: custom rosdep keys and source installer. These will both prevent autoinstallation of unwanted system packages and allow you to reference the custom build scripts in a standard way.

You can put links to custom rosdep rules in /etc/ros/rosdep/sources.list.d . Just give them a higher number than the default file so that your rules will overwrite the default ones.

One use for these custom rules is to ā€œsatisfyā€ dependencies with empty lists of packages (e.g. if you know that someone used a non-existing key in their package.xml and you want to ignore it to get a clean status running rosdep install):

/etc/ros/rosdep/sources.list.d/30-my.list:

yaml file:///usr/share/my/rosdep.yaml

/usr/share/my/rosdep.yaml:

non-existing:
  ubuntu: []

Another possibility is to use the custom rosdep rule for specifying a custom set of bash scripts that tell rosdep whether the package is installed and how to install it:

/etc/ros/rosdep/sources.list.d/30-my.list:

yaml file:///usr/share/my/rosdep.yaml

/usr/share/my/rosdep.yaml:

my:
  ubuntu:
    source:
      uri: "file:///usr/share/my/rosdep.rdmanifest"
      md5sum: deadcafe

/usr/share/my/rosdep.rdmanifest:

uri: 'https://acme.org/latest.tar'
md5sum: deadcafedead
install-script: |
  #!/bin/bash
  mkdir build && cd build
  cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local
  make
  sudo make install

check-presence-script: |
  #!/bin/bash
  [ -f /usr/local/bin/my ] && exit 0 || exit 1

depends: ['other', 'rosdep', 'keys', 'this', 'package', 'depends', 'on']
3

There were situations in which I really wanted a feature like this one. There are many pure-cmake packages which can be easily built with just ā€˜cmake ā€¦ && makeā€™ and are not released into Ubuntu. If we could just provide the package.xml for them, they could easily become parts of our workspaces without needing the extra setup work @RFRIEDM is complaining about. Or imagine these damn Pip packages with Apt dependencies which currently have no way of being properly represented for rosdep.

1 Like
4

Neat way to use rosdep! We have a few custom rules now, but we arenā€™t using them in that way. I like it.

I quite like the way of a way to add a package.xml to a standard CMake package thatā€™s not distributed on ubuntu too - we use a fair bit of those and I forgot about that use case. Itā€™s very similar.

5

tbh Iā€™m not too sad rdmanifests donā€™t get mentioned that often, as youā€™re essentially running an arbitrary Bash script as root.

If thatā€™s from a trusted source, maybe OK.

I asked a question about them (quite) some years ago: ros - What is the status of rdmanifests? - Robotics Stack Exchange.

6

Yes, I agree that itā€™s good theyā€™re not too widespread in buildfarm packages (not sure if buildfarm even supports them). But for local usage in a controlled environment, they are quite useful.

7

I might try creating a vendor package for exiv2 with the package name being exiv2 and putting that in your colcon workspace. The package.xml for the vendor package can install all the dependencies to build/run exiv2, and the -i/--ignore-src/--ignore-packages-from-source option to rosdep will prevent rosdep from installing the package given by the exiv2 rosdep key because a package with that name already exists in your workspace.

You might also consider blocking apt from installing exiv2 on your systems to prevent it accidentally getting installed by anything else.

1 Like
8

I guess one of the key points was to avoid externalproject as it comes with its own difficulties. But I agree it seems like one of the easier ways to satisfy many of the above points.

9

Wow, this ROSCon talk totally fits in this topic :slight_smile: https://roscon.ros.org/2024/talks/Building_system_packages_with_colcon_in_your_own_compact_buildfarm.pdf

2 Likes
10

Hi there! Conan developer here :smile:

We recently released an integration for ROS that makes the task of consuming Conan packages in ROS projects very easy. Any C++ library packaged with Conan can be consumed in your ROS packages without hassle. Take a look at the blog-post Enhancing ROS robotics development with C/C++ Conan packages

I am writing because it ticks many of your boxes. You can even create your own Conan recipes for libraries that are unavailable and easily plug them via Conan without the need to add a package.xml or include them via index-files in rosdep.

As described in the blog post, the steps to build packages with Conan dependencies in a Colcon workspace are nonintrusive. They require a conan install command for the dependencies and activating a script to set the Conan environment.

We are trying to gather some feedback about it, so if you give it a go please let us know! :smiley:

1 Like
11

Great options. Iā€™m extremely excited to see official Conan support for ROS 2. My previous experience with Conan1 was largely positive. The main reason weā€™re not using it now was because it wasnā€™t smooth sailing in a ROS ecosystem (oh, and Iā€™ve been waiting over 18 months for IT to set up an internal conan serverā€¦)

Regarding the environment setup instructions, when you install dependencies with:

conan install str_printer/conanfile.txt --build missing --output-folder install/conan

Is there a way to install dependencies for the whole workspace in a one-liner? Can Conan recursively find all the conanfile.txt files just like rosdep can recursively find package.xml files? I donā€™t see a recursive option in conan install yet.

In rosdep, our usual incantation is:
rosdep install --from-paths src --ignore-src -y