ROS packages on conda / conda-forge

General
1

Hi all,
people in the Python world (and others) have gotten used to use the conda package manager.

I think it could be an excellent venue for ROS packages as well.

Some of conda’s strength:

  • True dependency resolution with SAT solving, that makes sure that all packages have the correct packages and versions in place and
  • Built-in environment management that would make it easy to install different ROS versions side-by-side and switch between them
  • Conda ships real binaries for all major platforms (including a compiler) so it works on all Linux distros, as well as Windows and OS X
  • Low barrier of entry to package building since it’s a yaml based format that’s a lot more accessible than debian (less secure because no package signing though)

I am curious to hear the opinion of the crowd.

Also if someone else is interested in this I would be happy to team up for this.

Maybe someone knows what the best starting points would be?

5 Likes
2

That sounds like a great idea to me.
I have a lot of experience with conda and some with conda-forge (I maintain one C++/py package there).
It is a great tool.
I am relatively new to ROS, but I assume it can be built entirely from source, so it should be possible to build a conda-forge repo that will produce conda packages for ROS

3

Hi there,

I am coming from the PyData ecosystem too and know very little about ROS.

This is more a FYI than anything else: I have started some work to package pinocchio into conda-forge. The main goal is to make pinocchio easier to install.

For example, eigenpy (Python bindings between Eigen and numpy) is now available in conda-forge since a few days. Eventually, this will lead to have conda-forge packages for a few ROS packages that pinocchio depends on for example urdfdom.

At the moment, note that my main goal is to package pinocchio and its dependencies in conda-forge, I did not think too much about whether a more automated approach would be feasible or even be a good idea.

@wolfv @anton-mastosov if you know about some activity in this space since your posts, I’d be very happy to hear about it!

4

@lesteve awesome. Yes I am still working on this from time to time. I was meeting the conda-forge guys last week at scipy. They are super excited about getting all of ROS onto conda-forge. I can currently generate the base packages without issues from superflore… so now it’s just a matter of time until I’ll start to put the first couple of packages on conda-forge, so that we can install basically a minimal ros environment.
Hopefully more packages (by demand) will then come later.

1 Like
5

Very nice to hear you have put some thought and effort into a generic solution for ROS packages and the conda-forge people are excited about this!

so now it’s just a matter of time until I’ll start to put the first couple of packages on conda-forge

Any time estimate about this, even very rough? I started to create a few PRs in conda-forge/staged-recipes (e.g. for console_bridge and urdfdom_headers and I am hoping to have one for tinyxml in the next few days).

Ideally we would have pinocchio packaged in conda-forge in a matter of weeks, but it would be a real shame if our efforts interfered with each other …

6

Please consider opening a PR with your changes to superflore to add the conda generators. This will enable others to try it out and potentially improve/extend it to help add support for more packages.

1 Like
7

Hi @lesteve, I will start working on this again today and during the weekend.

We could start with ros-independent packages such as some of the Python packages, and tinyxml, tinyxml2 and boost (boost already exists but we need to make sure we have the right versions for Melodic). I’ll try to have PR’s to conda-forge for all of the missing ones by the end of the weekend.

I’ll aim at having all recipes for a minimal ros distro ready by the end of the weekend too, and then start making PR’s to conda-forge as well.

If you could I would be quite happy if you add me as a maintainer for console_bridge and other ros packages you upload to conda-forge (github is @\wolfv)

If you want we can also chat over on our gitter channel (https://gitter.im/RoboStack/Lobby)

@tfoote I totally plan to open source my stuff and especially i would love to contribute back to superflore. However, my current usage is quite hacky. But I will be ironing everything out as I go and then make the PR.

1 Like
8

Just to give a quick update, the first packages that are relevant to ROS are landing in conda-forge. All of them are “ros”-independent (the Python packages and some base libs for now):

  • tinyxml (thanks to lesteve)
  • tinyxml2 (already existed)
  • catkin_pkg
  • log4cxx
  • empy
  • sbcl

Currently waiting to get rospkg merged, then I can move on to rosdistro.
It’s going to be exciting!

9

@tfoote I have a small problem: conda also knows about ‘environments’, and also does library re-linking.
From what I see I think that library relinking only correclty works if I install the libraries into condaenvironmnet/lib/.... This would mean that the entire conda environment becomes also the catkin workspace. Do you think that could cause an issue? Other libraries or Python packages would be installed in the “same” catkin environment (by means of Conda), even though they might not be catkin packages.

10

Also, if we go this route, I would really like to move the setup.sh and other top-level catkin workspace files to a different sub-directory like $PREFIX/ros … is that possible?

11

The first real ROS package is landing on conda-forge – catkin!

I have it working locally and it works very well together with conda environments as far as I can tell. Activating a conda environment basically also activates the corresponding catkin workspace, which is pretty useful.

Maybe someone here has some feedback on this recipe, from a ROS perspective? I do some modifications to setup.sh and _setup_util.py to add the correct CMAKE_PREFIX_PATH (the base conda environment is also the ROS workspace).

2 Likes
12

Also another question for the conda packages:

  • is there a common way of running tests for ROS packages? It would be great if there was a command that I could run for all packages
  • can I (during superflore generation) find out wether a package produces a shared library? Those should (in conda-speak) probably be added to run_exports.
13

For packaging we typically don’t use the catkin build commands, but the standard cmake .. && make && make install sequence that debian uses by default for cmake with a few extra config options to CMake: configure options, build override, install overide I would recommend doing similarly and then everything just gets installed into the install prefix following standard FHS layout and it’s fine to have other things in the same prefix.

1 Like
14

Thanks @tfoote for clarifying. I am using a very similar command sequence vs debian. Locally I have the stack up to rviz working.
There were some hiccups here and there with regards to relocation – hardcoded C++ strings in binaries tend to be really annoying with compiler optimization – but patches I have so far are quite minimal.

I am a little hesitant to put catkin on conda-forge since I think that also be the component thats most likely to need some patches later on and then it might be a mess. Anyways, it’s probably going to go live in a week or so after I have done some more testing.

15

This is interesting. Did you test you setup on windows? I would be interested on using the conda-forge packages on windows systems.

Btw, which version of ROS are you targetting? Is it the ROS2 dashing version?

16

Hi @windel, sorry, I just saw your message.

I have not yet tested the packages on Windows. Especially I don’t know about the environment activation script (that is part of the catkin package).

If you want to help, that would be very awesome.

I will hopefully get back to this next week!

17

@wolfv @tfoote Just wanted to add more prototyping work on conda-forge, I managed to build ROS2 Eloquent up to desktop for Windows, and it seems the relocatable feature works surprisingly well. To try it out, you will need to add conda-forge and one my personal channel. Here is a quick example:

conda config --add channels conda-forge
conda config --add channels ros-playground
conda config --set channel_priority strict

conda create -n ros2 ros
conda activate ros2

My conda recipe doesn’t deal with package abstraction and it builds everything in one pack, because my initial investigation is trying to see how far I can build ROS2 using existing conda-forge packages. I personally like to see ROS2 on conda-forge moving forward.

To that extent, I am thinking to extend Superflore to create the recipe automatically. Additionally, I think very soon we need to integrate it with rosdep, so we can look up a master copy of rosdep keys and conda-forge package mapping. And we need to decide how should we break down the packages from conda feedstock perspective. @wolfv, any insights from your work?

Anyway, looking forward to any feedback. I can help on this conda-forge initiative.

2 Likes
18

Hi @seanyen-msft, that’s absolutely awesome! I didn’t expect this to happen this fast! Also sorry for my slow reply!

When I was talking with @tfoote at ROSCON he also mentioned rosdep and was very supportive of the idea to create the mappings of packages-to-conda-forge name in rosdep. So we should totally get started with that. I have some mapping here: https://github.com/wolfv/superflore/blob/31fafafea8fb340728c7f95a163be64e81aeb030/superflore/generators/conda/run.py#L127-L176 which could be a starting point. Also that is the repo with my superflore hacks to generate ROS recipes.
There are some additional challenges for automatically generating the recipes for ROS packages:

  • Not enough license information (for conda-forge): conda-forge requires the recipe to use SPDX license identifiers and also to indicate the location of the license file. Often, ROS packages don’t use the SPDX identifier but some less verbose version (like BSD3). And a lot of the ROS packages don’t come with a dedicated license file. I have worked around this by using scancode-toolkit to find the actual license in the package (e.g. BSD-3-Clause) and some source file that includes the license text (you can see the code for that in the superflore repo linked above, there is a script that get’s called, find_license.sh).
  • Python / C++ package: depending on wether a package has a Python part, one needs to add Python to the requirements (so that conda-forge builds packages for the different Python versions). If a package builds a shared library, then one needs to add a run_exports field so that the dependent libraries will depend on ABI compatible versions. Currently I have been doing this manually for the packages on conda-forge. I think this could be somewhat automated by detecting the presence of a setup.py or messages / service declarations to identify Python packages, and a C++ src/include folder for detecting C++ packages.

Regarding conda-forge: I think there are some challenges in using it for the ROS packages. The main challenge from my perspective is that ROS packages don’t give any guarantees on binary compatibility. So whenever a low-level dependency changes, we need to rebuild all the ones higher up, which will result in a lot of seperate builds throughout conda-forge. I was told that whenever the catkin package changes, the entire ROS distro is getting rebuilt to guarantee binary compatibility within the distro. It’s worse with conda-forge since we’d need to wait for the packages being uploaded to CI before we can start the builds of the lower-level dependencies, and the state of the packages in between would not be consistent (if we don’t pin the packages to specific versions by using the hash pinning).

Regarding putting multiple packages into one feedstock: that could actually be possible by using the multiple output feature of conda’s build recipes. People use this to e.g. split up a library + Python bindings into two packages. We could use it to obtain the sources for some specific ROS meta-package, and create multiple ROS packages as outputs. But this might also have some challenges and drawbacks.

I think Tully said that if they would support conda as a means of distribution officially, they’d probably go with a separate channel. Here I would totally advocate to use conda-forge as the base channel for all lower-level dependencies (such as Boost or OGRE 3D…).
The up-sides of not using conda-forge for the ROS distros would be that one could rebuild all packages in one go on some server, then upload all of them to a conda channel at the same time to work around the binary compatibility issues. The down side is that someone needs to host the server, and that we loose the benefits of conda-forge:

  • Automatic migration to newer libraries to keep in sync with the rest of conda-forge (e.g. when a new protobuf or a new Boost version is released, it’s very much advised to upgrade so that all packages stay compatible). However, we could ask the conda-forge guys to give us a notice (or an API) to do these kind of migrations on the ROS side.
  • everybody could contribute ROS recipes, without having to be included in the official distribution
  • the community can easily come in and help fixing / contribute recipes on conda-forge
  • automatic building for different Python versions

I am still super-excited about this :slight_smile: I was just a little swamped with some other work recently, so sorry again for my late reply.

What do you think about conda-forge vs ROS’ own channel, and @tfoote would it be possible to set up some hardware to do the builds or integrate with the official hardware somehow?

2 Likes
19

@seanyen-msft I am struggling to build ROS2 Dashing on Mac via conda.

I have just added your personal channel ros-playground, and it seems that the package name has been changed to rosdep instead of ros. Anyway I installed rosdep, but I still got the error saying No module named 'rclpy._rclpy'
dyld: Library not loaded: /usr/local/opt/poco/lib/libPocoFoundation.63.dylib

Any idea? Thank you in advance.

20

Hi everyone, I am super excited to see progress on Conda integration.

I have managed to successfully compile entire ROS2 Eloquent with packages only form conda-forge on macOS.
There were some warnings in the stderr, but I am not sure if they are critical.
There are also various issues with runtime of some packages (ros2, rqt) as they link libpython directly, which doesn’t work with statically linked python executable in conda. I am working on patches to mainstream packages now.
With local patches I was able to run chatter/listener demo as well as tourtlesim with keyboard control and tested interoperability with installed (not compiled) ROS2 on linux VM.

Here are details of my setup

# faster solver https://github.com/bioconda/bioconda-recipes/issues/13774
conda install pycryptosat
conda config --set sat_solver pycryptosat
conda config --set channel_priority strict

# Conda env for ros2
conda create -n ros2 -c conda-forge python cmake cppcheck eigen pcre poco tinyxml wget asio tinyxml2 opencv console_bridge openssl log4cxx cunit qt freetype assimp sip pyqt bash pydot

conda activate ros2

pip install -U argcomplete catkin_pkg colcon-common-extensions coverage cryptography empy flake8 flake8-blind-except flake8-builtins flake8-class-newline flake8-comprehensions flake8-deprecated flake8-docstrings flake8-import-order flake8-quotes ifcfg lark-parser lxml mock mypy netifaces nose pep8 pydocstyle pyparsing pytest-mock rosdep setuptools vcstool

# Install Icons theme
brew install https://raw.githubusercontent.com/ros/homebrew-deps/master/tango-icon-theme.rb

mkdir -p ~/ros2_eloquent_conda/src
cd ~/ros2_eloquent_conda
wget https://raw.githubusercontent.com/ros2/ros2/eloquent/ros2.repos
vcs import src < ros2.repos

set -gx CMAKE_PREFIX_PATH $CONDA_PREFIX
set -gx OPENSSL_ROOT_DIR $CONDA_PREFIX
set -gx PATH $CONDA_PREFIX $PATH 

cd ~/ros2_eloquent_conda/
colcon build --symlink-install

. ~/ros2_eloquent_conda/install/setup.bash

P.S.
complete conda environment yaml

P.P.S.
I have not verified this setup on clean macOS yet, but that is definitely on my todo list

P.P.P.S.
I am noob at ROS/ROS2, so it might take me some time to provide some other reasonable contributions =)

1 Like