Downloading Dependencies

Next Generation ROS
1

I looked at http://design.ros2.org, but I didn’t see anything about plans for a new rosdep for ament. Based on the ROS 2 binary installations, I figure that the remote dependencies are managed by brew for Mac, Chocolatey for Windows, and apt for Ubuntu. Are there plans to have a generic tool to download these dependencies based on the package.xml like rosdep? Also, are there plans to generate the necessary files for these dependency managers from a package?

Unfortunately, the previous version of rosdep didn’t handle downloading repositories directly. It seems useful to enable a complete stack source code build like Mike Purvis’ ROSCon 2016 talk: http://roscon.ros.org/2016/presentations/ros-bundling.pdf. Building from source enables cross compiling, link time optimization, cross package bug fixes, native architecture optimization, and dependency artifacts on the fly without a binary server.

Ultimately, it would be nice to have one command to convert a repository/package(s) into artifact(s). Another tool to help get there would be to convert a package or packages into workspace.

2

To answer this properly, I need to layout the workflow we usually use and the different tools we use to accomplish this.

Here’s the steps first (without details):

  • Generate a list of things to download, i.e. “what do I need to download to build rviz?”
  • Download everything based on that list
  • Install the remaining dependencies, e.g. non-ROS things like qt and boost
  • Build everything, in the correct order
  • Redistribute binaries

This basic formula is how I start working on everything (except maybe redistributing binaries).
You can see the pattern in our from source instructions on the wiki:

http://wiki.ros.org/kinetic/Installation/Source

And now that list again with more notes in between:

  • Generate a list of things to download, i.e. “what do I need to download to build rviz?”
    • We use rosinstall_generator for this and it draws on data in the “rosdistro”
    • it generates a .rosinstall file based on your query
    • it only gets the location of things that are “ROS packages”, i.e. things released in the rosdistro
    • it does not figure out how to download things like qt or boost
    • it is used in ROS 1 and will likely be used more-or-less unchanged in ROS 2
    • can also answer “what do I need to download to use rviz, minus ros_comm and its dependencies?”
      • you might want to do this so you can install ros_comm and down from binaries
  • Download everything based on that list
    • We use wstool for this (some power users use vcstool, we use it in ROS 2)
    • Given a .rosinstall (or a similar list of things to go get), it can fetch all the source code
    • We’ll likely use both in ROS 1 and ROS 2
  • Install the remaining dependencies, e.g. non-ROS things like qt and boost
    • It can also install binaries for “ROS things” if you don’t intend to build them locally and there are binaries for them on the target OS.
    • In ROS 1 we use rosdep for this, it would install remaining dependencies
    • In ROS 2 we just have instructions in the ros2 github wiki on what to install, we don’t use rosdep yet
  • Build everything, in the correct order
    • used to help us build software locally
      • either to do development or
      • to build for a new OS/architecture which lacks binaries
    • software might be anything supported by the build tool, like:
      • catkin or
      • ament or
      • cmake or
      • python with setup.py files
      • etc…
    • we use mostly catkin in ROS 1 and mostly ament in ROS 2, but that doesn’t really affect this part of the process
    • all result in an “install-like” folder which can be used locally
  • Redistribute binaries
    • One option is to use bloom and release on build.ros.org through rosdistro
      • bloom converts the contents of package.xml to debian (or others like fedora) packaging files
      • this happens on a per package basis (no bundling)
      • build.ros.org uses these files to build binaries and upload them to packages.ros.org
    • Another option is something like what @Mike_Purvis presented at ROSCon, which builds lots of things at once and bundles the result
    • Both options should work for ROS 2, we’re currently looking at any changes that would need to be made for bloom to work on ROS 2 packages (shouldn’t be much)

Ok, so that’s the breadth of what we do with some details.
Now I’ll try to answer you directly by quoting your questions:

Though rosdep does have some ROS specific capabilities, at its core is just takes dependency names, like qt or boost, and decides what needs to be done on a system to “satisfy” those as dependencies.
This is a pretty generic “abstraction” over the OS’s package manager, and is not really ROS specific.
And so it should work perfectly fine with ROS 2 as-is.

That being said we have thought about improvements to rosdep and dreamed of removing the the ROS specific stuff from it so that it’s more generic and potentially useful to many other projects outside of ROS.
Specifically @NikolausDemmel worked on something called xylem while interning at OSRF, and it was meant to be the spiritual successor to rosdep:

Despite his really awesome work, I’ve not had the resources to capitalize on it and push for it to replace rosdep.
I’ve always hoped we could make use of it in ROS 2, but it’s really an orthogonal issue as ROS 2 could easily use rosdep as well, it would just be nice to use xylem’s new hotness :wink:.
@NikolausDemmel put together some extensive design documentation and the tool is really close to being ready for use.
If you’re interested in tools like rosdep I highly encourage anyone to have a look at his work:

http://xylem.readthedocs.io/en/latest/design.html

So, to answer you, we don’t need a new rosdep for ament, but we’d like to have one :smile:.

Yes, we’ll likely use rosdep.
We are currently working on packaging some dependencies like tinyxml and eigen for chocolatey on Windows, and still sort of feeling out what the best thing to do there is, see:

These aren’t official, and will likely move, but for those interested in seeing what’s been tried…

Actually rosdep does have this ability, it’s called “rosdep source installer” (I think).
They’re not used much anymore, because they’re terribly difficult to maintain.
They looked something like this:

log4cxx:
  ubuntu:
    9.04: liblog4cxx10-dev
    8.10: |
      if [ ! -f /opt/ros/lib/liblog4cxx.so.10 ] ; then
        mkdir -p ~/ros/ros-deps
        cd ~/ros/ros-deps
        wget --tries=10 http://pr.willowgarage.com/downloads/apache-log4cxx-0.10.0-wg_patched.tar.gz
        tar xzf apache-log4cxx-0.10.0-wg_patched.tar.gz
        cd apache-log4cxx-0.10.0
        ./configure --prefix=/opt/ros
        make
        sudo make install
      fi
    8.04: |
      if [ ! -f /opt/ros/lib/liblog4cxx.so.10 ] ; then
        mkdir -p ~/ros/ros-deps
        cd ~/ros/ros-deps
        wget --tries=10 http://pr.willowgarage.com/downloads/apache-log4cxx-0.10.0-wg_patched.tar.gz
        tar xzf apache-log4cxx-0.10.0-wg_patched.tar.gz
        cd apache-log4cxx-0.10.0
        ./configure --prefix=/opt/ros
        make
        sudo make install
      fi

shudders

Since then we’ve deliberately chosen to avoid stuff like this and try to only use OS package managers to resolve dependencies.
The other point is that you can emulate this by creating a cmake package (or catkin/ament package) which does this “from source build” for external dependencies.
We do this some for ROS 2, for example with poco (cmake based) because we needed it on Windows and didn’t know how to package it yet, see:

Basically, if it cannot find the right version of poco, then it downloads and builds it in place.

xylem also touched on this topic:

xylem‘s Design Overview — xylem 0.1.0 documentation (search for source installer)

As for cross-compiling, there are entire toolchains for achieving this, like OpenEmbedded, Yocto, and others.
I am not confident that is a role rosdep needs to play, but we can certainly make sure to help support those patterns if we can (i.e. not get in the way).

bloom can provide you with Debian packaging files on a per package basis, and the Debian packaging toolchain can provide you with binary artifacts.
The same is true for any packaging scheme (Fedora, Homebrew, Chocolatey), in principle.
ROS Answers on how to do this locally, see:

Not sure what you mean by that.
We have tools that will build lots of local packages into a single destination folder.
We have people using this ability to wrap up the result and distribute that, see @Mike_Purvis’s presentation.
I think these tools should continue to work for ROS 2 with little to no changes.

Sorry for the long post, but hopefully it answered your questions.

5 Likes
3

Thanks for the detailed article.

It’s smart to use a package manager instead of building yet another one. I looked at Conan(https://conan.io) and vcpckg(https://github.com/Microsoft/vcpkg), but I guess they are C++ specific.

For clarification, of that last point. I was thinking of a tool/script that takes a repository/package and builds a workspace around it. Basically, create a workspace folder with a src subfolder and move the package into the src folder. That way ament can run shortly after cloning a repository/package in an arbitrary location. It takes some time to understand the workspace structure and to know how to convert arbitrary packages into runnable nodes.

4

There is no tool that automates so much of the process, but there is an issue on rosinstall_generator to add an option which would help you do it in a few commands:

Basically it would be like “I have some package(s) in this folder, and some things installed from binaries, please get me the list of things that are missing for me to build these package(s) I already have.”

It would be a cool contribution to extend the tool in that direction :wink:.