Package Separation for Building and Running ROS Applications

[oliver485]

Hi there,

I was wondering why packages for building and running a ROS application are usually situated in a same workspace. My question is motivated by a cross-compilation issue.
I want to have a minimal ros-core for my embedded device. Therefore, I do not need for any development packages, I only need the packages for running the application. Later, when I create the application, I cross-compile a workspace, copy that to the embedded device an run it.

I already tried this and found that there is a ros-core package for a minimal installation. I used the rosinstall_generator to get all dependencies. But when I cross-compile it results in an about 4GB build. As far as I can see this, that build contains all build-tools (that I do not need on the target) and some basic packages for running an application.

So, what am I missing, is this separation possible, how can I manage this?

Kind regards
Oliver

[peci1]

You can try to figure out which packages pull in some unwanted dependencies by iterating rosdep what-needs -t exec unwanted_package and rosdep keys -t exec package. At least from ROS 1, my experience is that the non-core package.xml files are often written poorly and overdepend on many things.

[marguedas]

@peci1

my experience is that the non-core package.xml files are often written poorly and overdepend on many things.

In most setups I encountered the usual suspects that would take most of the space were boost, libpcl-all-dev and libopencv-dev. So that may be a good place to start

For most packages the fix is simply narrowing the rosdep keys in the package.xml (and some of them the fixes may already have the fix suggested or merged: Reduce boost dependency scope by stwirth · Pull Request #118 · ros/nodelet_core · GitHub )

[gavanderhoorn]

Keeping things connected:

Generating 'dev' and runtime artefacts from ROS packages

First: I’m not entirely sure this belongs here, but I couldn’t think of a better category. It’s probably something that will touch the buildfarm if we can come up with something, hence why I posted it here. I’ve been interested in reducing the size of deployments of ROS applications, both native and in Docker (and Singularity) images for some time now, and one thing that I have been curious about is whether it would be possible to generate separate runtime and -dev archives from ROS packages (…

[jspricke]

I have implemented separate -dev and -runtime packages in bloom:

Generate -runtime packages with --runtime-pkg

Also I recommend looking into Debian for cross compiling, I have started fixing the dependencies there. For example this cross-compiles roscpp-core for amd64:

sbuild -d unstable --host arm64 ros-roscpp-core

You need deb-src lines in apt/sources.list and I recommend sbuild+unshare as described here:
https://wiki.debian.org/sbuild#Using_unshare_with_mmdebstrap_.28no_root_needed.29

[robwoolley]

We have separate build and runtime environments in our OpenEmbedded cross-compiled builds with meta-ros

It isn’t trivial to maintain. It gets complicated by having to carefully manage the CMake variables as well as having to provide a native Python3 interpreter but point to the target’s modules in order to satisfy dependency checks.

However, we do have ros-image-core working for all supported combinations and soon ros-image-world will be available. We also have been working on generating SDKs so application developers can build applications without building the platform.

If you have any questions about it, I’m happy to help.

Cheers,
Rob

[oliver485]

Hey Jochen,

as far as I understood, bloom is for ROS 1? The documentation says it is for catkin?
Is this correct?
I should have been a little bit more specific. I use the humble distribution.

[jspricke]

bloom is for ROS 1 and 2 but the patch is for ROS 1 only right now but should be easy to adopt for 2.

EDIT: The patch is for both as well :D.

[tfoote]

@oliver485

as far as I understood, bloom is for ROS 1? The documentation says it is for catkin?
Is this correct?

Bloom was originally developed against catkin based packages. However it has templates for 4 different build types: catkin, ament_cmake, ament_python, cmake (this is just plain cmake) It is the release tool used for both ROS 1 and ROS 2 debian and rpm packages.

In the underlying packages we have 7 different dependency types that are designed to capture the different nuances required for the differentiation of runtime, build time, and build tool needs for cross compilation, as well as test and documentation dependencies.

These dependency declarations can be leveraged for to optimize for certain build targets. As highlighted by Rob you can make use of those actively to build optimized runtime only targets with the OpenEmbedded builds.

The debian pipline already splits out the build, runtime dependencies. And of course there are more optimizations possible such as the one Jochen has proposed. This level of complication is not in our default tutorials or workflows as the additional complications further increases the learning curve and is not adding value for most of the development use cases. The value comes in when you start looking at deployments and you’ll find people doing everything from the aforementioned OpenEmbedded, to Conda/Pixi/RoboStack to Nix to Snaps

Here’s some links to resources to get you started learning about the many different ways to potentially deploy ROS besides the main debian packages and source builds which are focused on the developer experience.

OpenEmbedded

New way of installing ROS

Here is a blog about a new way of installing ROS. We’re heavily developing a tool called pixi that will change the way you can make virtual environments using the conda ecosystem. No more apt, pip, brew, nuget, just one tool that is shared over all platforms (Windows, MacOS and Linux). Do you want multiple versions of ROS on your computer without them breaking each other installation? And not just ROS1 and ROS2 but even different versions of the same ROS. Then you should definitely give it …

Automatic Deployment of ROS2 Based System to remote devices: Dual Copy or Containers?

I would suggest to go the ROS-with-Nix guru himself: @Mike_Purvis. See his ROSCon22 presentation: “Better ROS Builds with Nix” (recording, slides).

https://snapcraft.io/docs/ros-applications

[oliver485]

Hey guys,

you are amazing, thanks for your help and your fast responses.
I think I have to look into a lot of suff. That is great.

[oliver485]

Hey there,

I try to summarize the things I learned:

To run ROS on a system, you have to consider 2 kinds of dependencies:

• a related ros-package (you can fetch them with rosinstall_generator)
• a system library/package (you can fetch them with rosdep)

The package.xml definition allows us to distinguish between different kind of dependencies (build, test, execution, system, …), but if poorly written, it can lead to too many dependencies you maybe not gonna need.
So in my case, if I use rosinstall_generator ros_core from scratch it will come up with a lot of dependencies. For instance several middleware abstractions (fastrtps, cyclonedds, …), ament build stuff, … and this creates a ros workspace of about 4GB + some system dependencies like boost.
And this creates a challenge for embedded devices with limited capacity.

There are some approaches to this:

• using meta-ros and yocto to create an image for an embedded device with ros runtime support
• using bloom to seperate development and runtime packages
  (Is this for debian systems only?)
• create your own ros-workspace with runtime packages only by your own
  (Due to the poorly written dependency issue it will be kind of an iterative try and error procedure?)

For cross-compiling:

• Resolving system dependencies with rosdep is not gonna help us because it will install the dependencies on the host. But you can set the sysroot option for colcon and add/install the dependencies for the target on your own.

• How should I resolve the ros-package dependencies? Some are on the target that I do not have to build in my workspace. I could exclude the list of packages with rosinstall_generator.

Is this the preferred approach?
Please correct me/comment if I am wrong.

Kind regards
Oliver

[haudren]

Just a small remark, but although the workspace might be really big, all you need is the install folder at runtime. For example on my machine I have a big 16GB workspace, but it’s mostly source + build. The install itself is only about 1GB, and is relocatable (you can move it to another machine, but you will need to install system dependencies there). Unfortunately, when building from source, there is no way that I know of to trim down the install folder to obtain only the required runtime dependencies of a specific package. Indeed, the install folder will contain the built artifacts of every package in your workspace, including things that you may not care about. You might want to write a little bit of automation that reads package.xml or use rosdep to figure out which bits are truly needed at runtime and only copy that to your device.