ROS 2 Conan Integration

[alsora]

Hi everyone,

with this post I would like to start a discussion around one of the challenges that people face when they look into transitioning to ROS 2.

As we all know, ROS 2 is an highly-federated ecosystem of packages and comes with its own build tools and package managers (e.g. colcon).
In my opinion, these tools work great when you are developing a new project from scratch and you want to completely embrace the ROS ecosystem.
On the other hand, if you are in a situation where you already have a large project and you only would like to be able to add ROS 2 to it, the aforementioned tools may not look like the best solution.

To address these issues, in the past few months Eric Riff and I, from iRobot, have developed a set of tools that allow to integrate ROS 2 into Conan, one of the most popular C++ package managers.
If you are not familiar with Conan, I invite you to have a look at the extremely well-written documentation that can be found on their website.

As a quick summary, the Conan package manager is based on the concept of “recipe”, i.e. a Python script that provide instructions for building and packaging a given library. It specifies its name, the version/s, sources and dependencies. It also provides a list of options which can be used to enable/disable functionalities or change certain aspects of the library to be compiled
Some of the great features of this approach are:

• Recipes allow to easily fetch pre-built packages and use them in your application
• There is a lot of support for cross-compilation
• You can pin exact versions of packages
• Conan is open-source and decentralized, so you can have your own repository of recipes and server of packages

We created a Github repository where you can find some Conan recipes that can be used to build ROS 2 packages, together with an example application that consumes them.

The core of the contribution consists in some base-class recipes that allow to easily write generic ROS 2 recipes.
Then it’s possible to choose any type of granularity: you may have a single recipe that contains all the ROS 2 libraries that you want to use or you may want to separate ROS 2 build tools (ament) from the ROS 2 core libraries (which need to be cross-compiled) and from the developer tools (rviz, rqt, etc).

The approach has still some areas where it can be improved: for example conan recipes require to manually list all the dependencies, which will become very tedious and error-prone the more recipes you have (especially in the extreme case where you may have one recipe for each ROS 2 library)!

We would like to hear feedbacks from the community to be able to improve this infrastructure and, hopefully, to have officially released ROS 2 Conan recipes one day!

Thank you,
Alberto

[gavanderhoorn]

for Conan in general.

I’m wondering though:

@alsora

As we all know, ROS 2 is an highly-federated ecosystem of packages and comes with its own build tools and package managers (e.g. colcon).
In my opinion, these tools work great when you are developing a new project from scratch and you want to completely embrace the ROS ecosystem.
On the other hand, if you are in a situation where you already have a large project and you only would like to be able to add ROS 2 to it, the aforementioned tools may not look like the best solution.

How does using Conan support the use-case (requirement? constraint?) you mention specifically (ie: being able to add ROS 2 to it, instead of completely embrace[ing] the ROS ecosystem)?

(I expect you know / have figured that out, but your post seems not to make it explicit)

[lyle]

Another +1 for Conan in general.

[wjwwood]

@alsora

The approach has still some areas where it can be improved: for example conan recipes require to manually list all the dependencies, which will become very tedious and error-prone the more recipes you have (especially in the extreme case where you may have one recipe for each ROS 2 library)!

This is the problem that tools like bloom (bloom - ROS Wiki) and superflore (GitHub - ros-infrastructure/superflore: An extended platform release manager for Robot Operating System) were designed to solve. We have all this information via the package.xml and rosdep, and we use it to create deb and rpm artifacts.

I experimented with generating Homebrew formulae for macOS but it stalled due to the moving target of macOS+homebrew.

If you don’t use a tool like this, then you have to maintain the recipes manually and as you suggested this would be very time consuming.

It seems to me that generating these Conan recipes from bloom and then having aggregating packages to combine them might make sense.

At the very least you could use the same api and logic bloom uses to automate the creation of your recipes, even if you end up grouping things.

In general I’m in favor of using newer tools like Conan, but I do think automating the integration is the most sustainable thing to do, even if it requires a larger initial investment.

[traversaro]

Just for the sake of providing backlings to anyone exploring this topic, in Packaging Ignition with Conan - Projects - Gazebo Community there was similar discussion on packaging Ignition libraries for conan-center-index, that is the sort of main repo for conan recipes that I guess can be used as a source for non-ROS dependencies (the one usually provided by Debian/Ubuntu repos in apt distributions).

The approach has still some areas where it can be improved: for example conan recipes require to manually list all the dependencies, which will become very tedious and error-prone the more recipes you have (especially in the extreme case where you may have one recipe for each ROS 2 library)!

In this area, for RoboStack (the packaging of ROS for conda, another user-level package manager as Conan) @Tobias_Fischer worked on rosdep integration, you can read about his work in patch rosdep to work with conda-forge/robostack packages · Issue #41 · RoboStack/ros-noetic · GitHub , perhaps it can be interesting for your use case.

By the way, with this announcement and considering existing approaches in packaging ROS on conda, nix and vcpkg I think that the main user-level package managers for which no ROS packaging as been attempted so far are Spack and Julia’s BinaryBuilder/Yggdrasil, I wonder if interest for having ROS there will emerge as well (for BinaryBuilder I was just able to find a GitHub issue: Artifacts? · Issue #90 · jdlangs/RobotOS.jl · GitHub ).

[traversaro]

For people reading mail notifications: the previous message was sent incomplete, and it is not has been fixed.

[alsora]

It’s great to see a lot of Conan support here!
Sorry for the late replies, but I just returned from vacations.

@gavanderhoorn

How does using Conan support the use-case (requirement? constraint?) you mention specifically (ie: being able to add ROS 2 to it , instead of completely embrace[ing] the ROS ecosystem )?

There are multiple reasons why we looked into doing this work.

Despite colcon being in theory a generic package manager, it is in practice used almost exclusively by the ros community. This implies that 1) there are way less libraries that already support it (e.g. that have a package.xml,etc) and 2) its features and design choices are quite specific to a ros-best-practice ecosystem.

A standard approach when starting a ROS project consists of creating one or more ROS workspaces containing all your required packages, build and source them and then build your project in a new workspace.
This may work ok with the ROS-model, where there are a lot of different repositories/packages, so the number of dependencies of each of them is not that big.
But it will quickly become problematic for example when dealing with complex packages with hundreds of dependencies (most of them non-ROS related) or with single-repositories-systems.

There are a wide variety of issues, that I think would require the creation and maintenance of complex infrastructure on top of colcon itself.
For example: being able to download pre-compiled packages with a specific version or for a specific architecture or having some dependencies be available only for certain architectures.

Conan solves most of these issues.

[ruffsl]

@alsora

Despite colcon being in theory a generic package manager, it is in practice used almost exclusively by the ros community.

I don’t think colcon was ever intended to be a “package manager”, but is considered to be purely a build tool. It seems like you may be expecting colcon to also be a dependency manager, which seems out of scope for most build tools, let alone most build systems.

@alsora

This implies that 1) there are way less libraries that already support it (e.g. that have a package.xml,etc) and 2) its features and design choices are quite specific to a ros-best-practice ecosystem.

Most plane Cmake and python packages I’ve encountered seem to build fine with colcon, and I don’t think many folks expect cmake or setup.py scripts to automatically install dependencies onto your system, in fact I prefer that they don’t. Rust & cargo perhaps being one notable exception.

@alsora

For example: being able to download pre-compiled packages with a specific version or for a specific architecture or having some dependencies be available only for certain architectures.

That is rather cool. How does conan currently automatically infer the build order of workspace dependencies; remaining OS and architecture agnostic? Is conan specific to only pure C/C++ projects with recursively pure C/C++ dependencies, or is it also applicable for building libraries and dependencies written in other programing languages? E.g. if I needed to build multiple C projects that have Fortran or Rust dependencies for numerical computing or for FFI with sensor SDKs?

[gavanderhoorn]

I sort-of agree with @ruffsl in that it looks like comparing Colcon to a package manager + build tool + dependency fetcher seems to conflate a few things. It’s expressly only 1 of those.

Separating responsibilities makes sense to me, but that doesn’t necessarily change your argument of course.

[Jason_Beach]

I would suppose it’s more appropriate to compare conan with rosdep. I do think colcon is trying to do too much (i.e traversing directories within your workspace looking for packages, running msg generation scripts, single build tool for c++ and python, etc.) I would hands down prefer to use industry (outside of ros) standard practices and tools like conan and straight cmake. Trying to switch our cmake project to using ROS2 as the middleware has been quite painful.

[wjwwood]

It’s been said elsewhere, but colcon doesn’t require you to use any kind of special cmake. You can build plain cmake stuff with colcon.

If you have more than one cmake project colcon will try to figure out the order, but if it is too complicated you might need a package.xml.

If you have just one cmake package then you don’t need colcon at all. Just source the setup file so that the cmake prefix path is set and build your project as normal.

It’s a common misconception that you have to use the ROS infrastructure to build on top of ROS but it isn’t really the case.

@Jason_Beach

I do think colcon is trying to do too much (i.e traversing directories within your workspace looking for packages, running msg generation scripts, single build tool for c++ and python, etc.)

Colcon must find packages in your workspace, but it doesn’t run anything for message generation, that happens through cmake and would happen if you just ran cmake and make on the package with messages.

There’s no industry standard that mixes cmake and python in the way that we need, as far as I’m aware. Maybe Conan does that.

@Jason_Beach

standard practices and tools like conan and straight cmake. Trying to switch our cmake project to using ROS2 as the middleware has been quite painful.

It sucks that you’ve had trouble migrating but I’m curious what the issues were because it’s likely something that can be worked around or avoided entirely if you prefer.

[russkel]

@Jason_Beach

I do think colcon is trying to do too much (i.e traversing directories within your workspace looking for packages, running msg generation scripts, single build tool for c++ and python, etc.) I would hands down prefer to use industry (outside of ros) standard practices and tools like conan and straight cmake.

I think colcon fills exactly the gap it was designed to fill. And while I think there’s a slight learning curve to using colcon, it has been an asset. Being able to do colcon build --packages-up-to XXX and have it compile all the deps and the package itself is magical.

I have had less success compiling external ‘non-ament’ packages with slightly awkward build processes with colcon. I was never able to figure out how to tell colcon the exact steps it should take to compile it.

Back to the topic at hand:
Perhaps conan support should be introduced into rosdep as well?

[alsora]

Sorry for the very late replies, but I completely missed the last comments.

@wjwwood

If you have just one cmake package then you don’t need colcon at all. Just source the setup file so that the cmake prefix path is set and build your project as normal.

Unfortunately, there are some limitations in relying on the setup shell scripts.

• They are not really relocatable, as you can only specify one path in the COLCON_CURRENT_something env variable, so this does not work if you built your ROS dependencies in multiple workspaces.
• Using environment variables may cause problems as you may mix things up (e.g. the ROS package that you are sourcing and your host environment).

On a quick note, using a single ROS workspace is not a solution:

• some packages don’t need to be cross-compiled
• the ament CMake logic automatically searches for things to determine what ROS interfaces to generate: if both fast-dds and cyclone are found, both will be built, even if you may only care about one (small problem) and similarly if rosidl_python is found, python interfaces will be built (this is a major problem because if you are cross-compiling for a platform that does not have python, then the build will fail).

@wjwwood

This is the problem that tools like bloom (bloom - ROS Wiki ) and superflore (GitHub - ros-infrastructure/superflore: An extended platform release manager for Robot Operating System ) were designed to solve. We have all this information via the package.xml and rosdep, and we use it to create deb and rpm artifacts.

@wjwood how do these tools handle the versions of dependencies? these are not specified in the package xml

[Jaron]

@russkel

Perhaps conan support should be introduced into rosdep as well?

I think the better comparison for conan is apt/dnf/vcpkg/pip rather than colcon as they include obtaining source to build dependencies. I played around briefly with integrating conan into ros and you can follow their tutorials exactly without any conflicts with ros, so the window is wide open if anyone needs this. Colcon doesn’t need to know that you use conan at all as it still sees the cmake build type. I would almost say that rosdep could be a key for rosdep except for the CMakeLists.txt changes that need to be made to integrate conan into CMake. I wonder if conan support could somehow be built into ament_cmake non-intrusively.

[HeinzBenjamin]

Hello there and thanks a lot for your efforts @alsora!

I wondered if you are still working on this topic and what the general status of the project is?

As I share a lot of your pain points, I’m trying to figure out if it was advisable to build a new ros2 development workflow around your conan integration?

Cheers

[Maghob]

Thanks for your idea @alsora
I am so interested to see this conan integration with ROS2 in the future.
In my case, getting dependencies from source to build them with specific flags depending on the ROS package is a very nice feature.
I will take a look at the repository

Cheers

[john-at-foxglove.dev]

Better conan support would be great to see. The MCAP C++ library is distributed via conan (mcap C/C++ Package - JFrog ConanCenter), but we have to use a more complex solution with submodules for statically linked vendored code when it comes to integrating with ROS2 (Build dependencies from submodules, remove conan build dependency by jtbandes · Pull Request #27 · ros-tooling/rosbag2_storage_mcap · GitHub).

[aposhian]

I think that ROS development would benefit greatly from more fleshed out dependency management. rosdep has some significant limitations. rosdep isn’t really a package manager itself: it is a meta-package manager, designed for cross-platform compatibility. However, it doesn’t expose many features that are common place in package managers, like version restrictions (it ignores version attributes in package.xml). Also, rosdep is limited to the keys that are in the published sources, so it is not helpful to install nonstandard system or pip dependencies. In contrast (as a project, not another package manager), colcon, while still designed for ROS, has a top-level design that is ROS-agnostic, and is designed with extensibility in mind.

[facontidavide]

Adding one more comments to this thread.

With BehaviorTree.CPP, I need to take into account multiple users:

1. People that do not use ROS
2. ROS users that want the package to be available with apt install

When integrating 3rd party dependencies, I would prefer to use conan; this would make happy my users in group 1. and, allow me to use a specific version of the library, instead of the one coming with my specific Ubuntu distribution.

But using conan seems to be incompatible with bloom and the ROS buildfarm, as far as I understand.
Does anyone have a solution? Does anyone actually care?

[RFRIEDM]

I haven’t yet seen a solution in the ROS community for version pinning that I would like to have.

Say, I want to release a breaking change and bump the major version of a package from 1 to 2. If there are dependent packages that want version 1 but I merge my version 2 changes, then call rosdep to install dependencies, I now get version 2 and my build breaks. The only way we’ve dealt with this is doing breaking version changes on integration branches for the entire repo, which can be cumbersome.

That said, we have consumed packages that only supply themselves as Conan packages successfully. If we try to submodule them with CMake, they do not build, because the maintainers do not support that configuration. To integrate them into our colcon build system, we utilised the cmake-conan helpers. They actually worked great and allowed us to version pin! Trouble is, the wrapper doesn’t (yet) support Conan 2.0, and Conan 2.0 is close to complete.

If it was possible to add Conan support to rosdep such that it would obey version numbers, I would absolutely be interested in using it. Until then, we’re stuck using whatever is available through submodules or debian packages, and if it’s debian, which ever release the package manager deems is appropriate for the distro.