Generating 'dev' and runtime artefacts from ROS packages

tfoote · 2020-01-23T05:26:27.033Z

This is definitely something that has been considered at various times. And it’s been thought about recently by @cottsay who has been looking into generating rpms. As rpms have stricter linters.

The automation of separating dbg symbols was eventually resolved with it becoming part of the standard debian toolchain:

github.com/ros-infrastructure/bloom

add support for building dbg debs separate from main packages

ros-infrastructure:master ← ros-infrastructure:dbg_debs

opened 08:34PM - 03 Apr 14 UTC

tfoote

+15 -3

From that issue you can find links to various other discussions of adding -dev packages as far back as 2011. But on reflecting on it I think that the reason that we haven’t found a solution for created them is that it doesn’t make sense to “automatically” split packages. There are many heuristics to say when to put something into the dev version or the runtime component. But in reality there’s many corner cases that will never be met so the maintainer will basically need to list out resources to land in either regular or dev variants of the package. In the Debian world this is the role of the debian package maintainer. Each resource is placed into one or the other bucket. But more importantly the downstream package maintainer then has to add the right dependency on either the runtime or devel package. That downstream maintainer also splits their package into runtime and devel packages. But more importantly they also split the dependencies conditionally on the devel and runtime dependencies appropriately.

Now you can see that we actually have 4 different cases for one dependency. There’s both devel and runtime versions of both packages. And the dependencies declared on the downstream package have to be conditionally evaluated for the subset of the package that is being used to build or made available.

We have a rich language of dependencies with at least 5 different versions of dependencies for a package including build, build_export, buildtool, buildtool_export, exec. And that’s excluding the extra test and doc dependencies as well as groups etc. We would then have to include a way to declare associations of each of those dependencies to the runtime or devel versions of a package.

Looking at this from a little bit further away, the main reason for having foo and foo-dev packages is because historically foo was released as a tarball and then the debian maintainers would pick up the tarball, and select what went to each subpackage, and add the appropriate dependency declarations. Since then the process has not evolved much, though now often it’s pulled directly from the source repository. But in general the upstream maintainers are not thinking about the impacts of releasing their software and packaging.

By comparison in the ROS ecosystem core developers actively use packages as atomic units and leverage those packages even when developing, and they use and pay attention to the package dependencies. The efficiency gains from modular packaging that are usually gained by the debian maintainers work can be leveraged in the development environment too.

As such I think that it might behoove us not to think of this as a problem to be fixed at the release stage, but as an area of efficiency that we can gain by structuring our upstream packages similiarly. The benefits of separating the build and runtime dependencies helps overall with compile times. Dependency trees do not grow as quickly if they are not the union of build and runtime dependencies. Parallel build tools can build faster when these potentials are paid attention to in the development environment.

And I think that many of our common practices that the ROS community has developed such as recommending making message only packages gets a lot to this end. They are the higher level equivalent of headers and break apart the dependency trees by providing standard interfaces. And through encouraging small modular packages we get the ability to leverage the find grain dependency control.

Specifically separating the build and runtime dependencies is something that has been shown to be quite effective that we’re not actively doing. However, instead of trying to do that at release time. I’d suggest that we consider making that a best practice for the ROS maintainers to create full packages that separate concerns into runtime and build time dependencies. This will give the developer the ability to fully specify the dependencies for each and where content goes is also clearly obvious. We could even pick up the naming convention of debian to name the content designed for build time to end with -dev however that would then conflict with the Fedora/RPM convention to name things with -devel.

In ROS 2 we’ve leveraged this separation of dependency types to enable swapping out the runtime libraries and hide the underlying implementations yet building them all against the same headers by defining a clear interface specification. There’s always a matter of tradeoffs between making things more modular and keeping things as simple as possible.

So in conclusion I think what I’d like to suggest is that we already have support for all the bells and whistles needed to very clearly define the necessary dependencies. And that if we can identify places where separating the build and runtime dependencies would be valuable that we consider simply making separate packages and not try to find ways to automatically split a package and then find ways to refer to those partial packages in downstream packages. We’re generating packages, we have packages. Keeping it one to one is basically the simplest solution.

Will_Gardner · 2020-01-23T09:41:57.568Z

At work I am in a similar situation. We extensively work with docker images and found that ROS certainly doesn’t make it easy to have small docker images. On the larger end they could easily reach 12GiB (although a lot of that was our own negligence). Even a base install of ROS resulted in close to a 800MiB compressed image.

You’re absolutely correct that there is an inherent limitation at the moment with ROS packages always including the dev artifacts and that has given us real trouble in a few areas. One of the things we did actually find however is that nine times out of ten it isn’t actually the ROS packages themselves that are the problem - virtually all of the built debians contain just shared libraries and headers which is only going to be barely larger than a non-dev package. It’s the upstream dependencies that they hold which are consistently problematic.

As an example the ros-melodic-cpp-common deb package, which is in any core installation of ROS, has a dependency on libboost-dev. This boost metapackage is massive and has massive dependencies of its own, installing it on a fresh Ubuntu docker image installs 586MB worth of stuff even with --no-install-recommends. Some of the dependencies that get pulled in are just plain useless for actually running ROS applications:

gcc
perl
python3

To workaround this problem we came up with a novel solution. Before installing our ROS dependencies into our docker images we install custom, fake, metapackages. These metapackages are special in that they don’t install anything themselves, they simply “provide” (in the debian packaging sense) all of the upstream dev dependencies of which ever ROS packages we’re intending to install (plus a subset of ROS packages that we definately don’t need) without actually adding anything to the host system. That way when we later install the ROS dependencies it doesn’t actually install the dev dependencies. It turns out with this alone the size of a bare ROS docker image went from ~800MiB down to ~100MiB compressed. We’ve similarly extended this strategy to tame OpenCV and CUDA which means even our largest base images max out at about 400MiB.

The positive news is that solving this specific problem in ROS packaging more generally is probably much easier than implementing -dev packages in their entirety. In theory all that is needed is to produce -dev packages that just have extra -dev dependencies of their own, no extra files.

To a degree this should be straight forward as package.xml already has a mechanism for declaring these dev only transitive dependencies: build_export_depend. The downside of this approach is that it involves fixing dependencies on basically every package in the ROS ecosystem since that’s mainly used to export headers at the moment.

A hackier approach would be on the rosdep side. At the moment the boost rosdep key is declared as being the dev packages for every given OS: https://github.com/ros/rosdistro/blob/master/rosdep/base.yaml#L272. It should be possible to create a “parallel” rosdep file that declares the non-dev versions of packages which can then be automatically used by bloom to generate runtime packages with the non-dev dependencies.

The final approach is as @tfoote has suggested, explicitly create ‘-dev’ ROS packages. The downside of this approach is legacy, there is a substantial amount of legwork to implement this approach and tons of breaking changes to go with it. It also seems overkill since ROS already has most of the mechanisms in place to resolve this on the release side.

gavanderhoorn · 2020-01-23T09:58:15.006Z

Will_Gardner:

Before installing our ROS dependencies into our docker images we install custom, fake, metapackages. These metapackages are special in that they don’t install anything themselves, they simply “provide” (in the debian packaging sense) all of the upstream dev dependencies of which ever ROS packages we’re intending to install (plus a subset of ROS packages that we definately don’t need) without actually adding anything to the host system

Are these REP-127/140/149 metapackages, or the Debian variant? If the latter, are you using equivs for this?

Will_Gardner:

As an example the ros-melodic-cpp-common deb package, which is in any core installation of ROS, has a dependency on libboost-dev. This boost metapackage is massive and has massive dependencies of its own, installing it on a fresh Ubuntu docker image installs 586MB worth of stuff even with --no-install-recommends .

Yes, that was something I quickly ran into as well.

Will_Gardner:

That way when we later install the ROS dependencies it doesn’t actually install the dev dependencies. It turns out with this alone the size of a bare ROS docker image went from ~800MiB down to ~100MiB compressed. We’ve similarly extended this strategy to tame OpenCV and CUDA which means even our largest base images max out at about 400MiB.

The positive news is that solving this specific problem in ROS packaging more generally is probably much easier than implementing -dev packages in their entirety. In theory all that is needed is to produce -dev packages that just have extra -dev dependencies of their own, no extra files.

Could you go into a bit more detail about your approach (haven’t had my coffee yet)? Are you referring to extra ROS packages here which package the dev dependencies?

Will_Gardner:

To a degree this should be straight forward as package.xml already has a mechanism for declaring these dev only transitive dependencies: build_export_depend . The downside of this approach is that it involves fixing dependencies on basically every package in the ROS ecosystem since that’s mainly used to export headers at the moment.

So if I understand you correctly you’d like to also use it to export “other” build dependencies?

Thanks for the insight, this is quite a creative way to deal with this (although technically still a work-around ).

Will_Gardner · 2020-01-23T10:57:55.878Z

Are these REP-127/140/149 metapackages, or the Debian variant? If the latter, are you using equivs for this?

Debian variant. When I prototyped it originally I did use equivs although we create them with custom tooling nowadays.

Could you go into a bit more detail about your approach (haven’t had my coffee yet)? Are you referring to extra ROS packages here which package the dev dependencies?

No. Ultimately we only care about the debian packages. Certainly one approach to achieve that is to have separate dev ros packages but that involves overhauling basically every C++ package in the ROS ecosystem since they all invariably depend on boost in some way.

So if I understand you correctly you’d like to also use it to export “other” build dependencies?

Yes, so for example I could say my foo package has build_export_depend on boost-dev, but exec_depend on boost-non-dev. This would then allow bloom to automatically generate two debian packages:

foo: depends on boost-non-dev, contains all of the foo package itself
foo-dev depends on boost-dev and foo

This would only require a fairly procedural update to fix up everyone’s package.xml

ruffsl · 2020-01-23T22:52:52.207Z

Will_Gardner:

It turns out with this alone the size of a bare ROS docker image went from ~800MiB down to ~100MiB compressed. We’ve similarly extended this strategy to tame OpenCV and CUDA which means even our largest base images max out at about 400MiB.

Nice! Do you have any Dockerfiles you could share publicly that demonstrate this approach?

marguedas · 2020-01-23T23:19:57.882Z

Will_Gardner:

At the moment the boost rosdep key is declared as being the dev packages for every given OS: https://github.com/ros/rosdistro/blob/master/rosdep/base.yaml#L272. It should be possible to create a “parallel” rosdep file that declares the non-dev versions of packages which can then be automatically used by bloom to generate runtime packages with the non-dev dependencies.

This is something that came to mind multiple times when trying to reduce the amount of packages being pulled.
While very convenient to have these “blanket” rosdep keys, it is very uncommon for any package to need “all of boost dev packages” or “all opencv dev packages”. A great deal of space would be saved if packages were to use more specific and targeted rosdep keys. While some stacks like OpenCV and PCL used to have a single/couple deb with everything, they now provide multiple debs and most likely we should leverage them. For some stacks there is already already a narrower set of keys available, the -dev and the release one: https://github.com/ros/rosdistro/blob/aaa5fe421f23c07234850c338ec4632d530ab746/rosdep/base.yaml#L1585-L1600

In an ideal world, all packages would define separately their build_depend with the -dev version of the libraries they use and the exec_depend with the runtime libs they really need to run the package.

Given the number of packages using (or implicitly relying) on these blanket rules, it would be very challenging to actually apply this to an entire distro.

Maybe it’s something that could be encouraged for ROS 2 ? as the number of packages is still pretty limited, so would be auditable. We would need to find a way to discourage / warn users to steer them away from using these blanket rules.

ruffsl:

Nice! Do you have any Dockerfiles you could share publicly that demonstrate this approach?

The would be great and could be inspiration to reduce the size of the official ROS images that are ginormous.

ruffsl · 2020-01-24T00:20:03.451Z

marguedas:

Maybe it’s something that could be encouraged for ROS 2 ? as the number of packages is still pretty limited, so would be auditable. We would need to find a way to discourage / warn users to steer them away from using these blanket rules.

Hear! Hear! A fresh a proper slate, no time like the present to set the president.

gavanderhoorn · 2020-01-24T12:24:40.896Z

tfoote:

So in conclusion I think what I’d like to suggest is that we already have support for all the bells and whistles needed to very clearly define the necessary dependencies. And that if we can identify places where separating the build and runtime dependencies would be valuable that we consider simply making separate packages and not try to find ways to automatically split a package and then find ways to refer to those partial packages in downstream packages. We’re generating packages, we have packages. Keeping it one to one is basically the simplest solution.

This would be a nice way to go about it, but as @Will_Gardner also wrote, that’s going to be a lot of work (and will result in many more packages as well, which has an adverse affect of build times). For ROS 2 (as @marguedas suggested) this might be possible still.

For ROS 1 however I doubt it is feasible any more (nr of packages, size of maintainer/developer community & sunsetting).

What @Will_Gardner describes seems like a low-hanging fruit that could allow us to significantly reduce the footprint of ROS 1 (and potentially ROS 2) deployments without requiring too much work.

@Will_Gardner: could you provide a little more detail on what you are doing with those metapackages such that we could replicate it? I’m sure @ruffsl would be able to figure out whether we could apply a similar approach to the official ros and ros2 Docker images.

simonschmeisser · 2020-01-24T15:37:48.100Z

marguedas:

In an ideal world, all packages would define separately their build_depend with the -dev version of the libraries they use and the exec_depend with the runtime libs they really need to run the package.

Given the number of packages using (or implicitly relying) on these blanket rules, it would be very challenging to actually apply this to an entire distro.

Maybe I or someone else can come up with a graphviz/dot script to identify the worst offenders in individual subsets of the packages? I’ll try my luck with ogre/rviz

gavanderhoorn · 2020-01-25T16:46:10.949Z

There are quite a few tools available for visualising dependency trees (debtree and apt-cache can do it fi).

ros-melodic-cpp-common indeed brings in approx 580 MB of dependencies (when installed on a bare ubuntu:bionic). libboost-all-dev is essentially almost singularly responsible for this (207 pkgs for libboost-all-dev vs 210 pkgs for ros-melodic-cpp-common).

ros-melodic-roscpp already lists very specific parts of Boost as dependencies, but as it depends on cpp-common (as @Will_Gardner wrote), installing just roscpp on a bare ubuntu:bionic image wants to install 655 MB of packages.

Of this only 8.1 MB is actually placed inside /opt/ros/melodic.

Edit: @Will_Gardner: I imagine you have some way of figuring out the runtime dependencies of particular ROS nodes / packages and then generate metapackages based on that information. Have you automated this, is this a manual process, or are you relying on package manifests?

marguedas · 2020-01-25T22:40:26.742Z

gavanderhoorn:

ros-melodic-roscpp already lists very specific parts of Boost as dependencies

I guess that’s what I meant by “relying on blanket rules”. While roscpp find_package’s and uses specific parts of boost, it doesn’t declare dependencies on any part of it in it’s package.xml, relying on lower level packages to provide all the parts of boost it needs.

Not sure how the proper subset of boost packages appeared in the control file though…

gavanderhoorn:

Edit: @Will_Gardner: I imagine you have some way of figuring out the runtime dependencies of particular ROS nodes / packages and then generate metapackages based on that information. Have you automated this, is this a manual process, or are you relying on package manifests?

Definitely tools helping to extract this information will be very useful

marguedas:

Maybe it’s something that could be encouraged for ROS 2 ? as the number of packages is still pretty limited, so would be auditable. We would need to find a way to discourage / warn users to steer them away from using these blanket rules.

Hear! Hear! A fresh a proper slate, no time like the present to set the president.

Yeah getting these split for ROS 2 and allow to install only the nondev version of ROS packages would be great.
Looks like ROS 2 is facing other challenges as the core is completely boost-free and the eloquent-ros-core image is still 700MB, my guess is from the way message packages are currently packaged.

dirk-thomas · 2020-01-25T23:10:51.290Z

marguedas:

While very convenient to have these “blanket” rosdep keys, it is very uncommon for any package to need “all of boost dev packages” or “all opencv dev packages”. A great deal of space would be saved if packages were to use more specific and targeted rosdep keys.

There is nothing holding us back to do exactly that already. All it takes is a minute to make a PR to update any packages which do use these “blanket” rosdep keys (see below for an example).

marguedas:

Maybe it’s something that could be encouraged for ROS 2 ?

I don’t see why ROS 1 packages couldn’t be updated too. I doubt the number of packages (which use these “blanket” rosdep keys) and enough people care about is actually that high.

gavanderhoorn:

For ROS 1 however I doubt it is feasible any more (nr of packages, size of maintainer/developer community & sunsetting).

ROS Noetic is not even released yet and will be supported for 5 years. I would think that is motivation enough to still update package you care about. Also it doesn’t mean that the maintainer of a package has to do it. Anyone can create a pull request with this kind of change - the maintainer only has to accept it and roll a new release.

And as mentioned above an example how easy it is to improve the dependency size: ros/roscpp_core#117 uses specific rosdep keys (libboost-date-time-dev, libboost-system-dev, libboost-thread-dev) instead of boost (which maps to libboost-all-dev). The effect on installing ros-*-roscpp_core (assuming you have build-essential installed anyway):

Before it pulled in ~500MB
Afterwards it only pulls in ~200MB

While dev/non-dev ROS packages would certainly get us further we could already do something today. So please consider to create PRs to replace the usage of “blanket” rosdep keys with something more specific.

Will_Gardner · 2020-01-26T00:34:08.628Z

Our Dockerfiles by themselves aren’t all that useful since most of what they do is install fake packages.

The generation of the fake packages is the interesting part, I’m not sure how much I can share so I’ll describe it in high level terms. The entire process operates entirely within the realms of Debian packaging, nothing ROS specific.

When we want to build an image we declare its high level debian dependencies, typically this will be a set of ROS packages (e.g. ros-kinetic-ros-core) and some other libraries that we want to use with it. We then have a Python script which takes this list and traverses the dependency graph. As it traverses the graph it will start to prune off dependencies based off of certain criteria, some of this will be hard coded rules based on package names but mostly it’s automated heuristics. The main one is the one which eliminates packages based on their debian section. While all the ROS packages sit in the misc section (since bloom doesn’t know any better) the packages in the upstream Ubuntu repos are nicely split up which makes it easy to eliminate all the packages in the devel section or the graphics section. As we prune off packages we add them to a list of packages to “provide” in the metapackage we’re going to generate.

Along with this we also explicitly add certain non-dev dependencies to the images. For example we explicitly add all of the non-dev boost packages (which turn out to be not that big) to our images to make up for not having the boost-dev meta-package.

Here’s an example of the pruned graph for ros-kinetic-ros-core (nodes are only shown the first time they’re found in the graph):

*   113.5MB /   658.9MB   ros-kinetic-ros-core
*    31.9MB /    81.6MB     ros-kinetic-catkin
      0.0MB /    36.0MB       cmake
      0.0MB /     7.3MB       google-mock
      0.0MB /     0.0MB       libgtest-dev
*    30.0MB /    36.5MB       python-catkin-pkg
*    26.6MB /    33.1MB         python-dateutil
*    23.7MB /    30.2MB           python-six
*    23.6MB /    30.1MB             python
*    23.0MB /    29.5MB               python2.7
*     6.3MB /     6.3MB                 python2.7-minimal
*     2.7MB /     2.7MB                   libpython2.7-minimal
*     0.2MB /     0.2MB                   zlib1g
*    16.3MB /    22.8MB                 libpython2.7-stdlib
*     0.1MB /     0.1MB                   mime-support
*     0.1MB /     0.1MB                   libbz2-1.0
*     1.7MB /     1.7MB                   libdb5.3
*     0.4MB /     0.4MB                   libexpat1
*     0.1MB /     0.1MB                   libffi6
*     0.8MB /     0.8MB                   libncursesw5
*     0.5MB /     0.5MB                     libtinfo5
*     0.4MB /     6.9MB                   libreadline6
*     0.1MB /     6.6MB                     readline-common
      0.0MB /     6.5MB                       dpkg
*     0.9MB /     0.9MB                   libsqlite3-0
*     3.3MB /     3.3MB                   libssl1.0.0
*     0.0MB /     0.0MB               libpython-stdlib
*     2.7MB /     2.7MB           tzdata
*     2.9MB /     2.9MB         python-docutils
*     0.0MB /     0.0MB           python-roman
*     1.2MB /     1.2MB           docutils-common
*     0.1MB /     0.1MB             sgml-base
*     0.5MB /     0.5MB             xml-core
*     0.3MB /     0.3MB               sed
*     0.2MB /     0.2MB         python-pyparsing
*     0.2MB /     0.2MB         python-catkin-pkg-modules
*     0.3MB /     0.3MB       python-empy
*     1.0MB /     1.0MB       python-nose
*     0.4MB /     0.4MB         python-pkg-resources
      0.0MB /     0.1MB     ros-kinetic-cmake-modules
*    11.0MB /   385.4MB     ros-kinetic-common-msgs
*     4.9MB /   379.3MB       ros-kinetic-actionlib-msgs
      0.0MB /     0.3MB         ros-kinetic-message-generation
*     3.6MB /   377.7MB         ros-kinetic-message-runtime
*     2.1MB /   376.3MB           ros-kinetic-cpp-common
*     2.0MB /     2.0MB             libconsole-bridge0.2v5
*     2.0MB /     2.0MB               libstdc++6
*     0.1MB /     0.1MB                 gcc-5-base
      0.0MB /   308.0MB             libboost-all-dev
      0.0MB /    66.1MB             libconsole-bridge-dev
*     1.0MB /     1.0MB           ros-kinetic-genpy
*     0.6MB /     0.6MB             python-yaml
*     0.2MB /     0.2MB               libyaml-0-2
*     0.2MB /     0.2MB             ros-kinetic-genmsg
*     0.4MB /     0.4MB           ros-kinetic-roscpp-serialization
*     0.4MB /     0.4MB             ros-kinetic-roscpp-traits
*     0.3MB /     0.3MB               ros-kinetic-rostime
*     0.1MB /     0.1MB                 libboost-system1.58.0
*     1.1MB /     1.1MB         ros-kinetic-std-msgs
*     0.3MB /     0.3MB       ros-kinetic-diagnostic-msgs
*     1.2MB /     1.2MB       ros-kinetic-geometry-msgs
*     1.0MB /     1.0MB       ros-kinetic-nav-msgs
*     1.8MB /     1.8MB       ros-kinetic-sensor-msgs
*     0.2MB /     0.2MB       ros-kinetic-shape-msgs
*     0.1MB /     0.1MB       ros-kinetic-stereo-msgs
*     0.3MB /     0.3MB       ros-kinetic-trajectory-msgs
      0.0MB /     0.0MB         ros-kinetic-rosbag-migration-rule
*     1.1MB /     1.1MB       ros-kinetic-visualization-msgs
*     0.1MB /     0.1MB     ros-kinetic-gencpp
*     0.1MB /     0.1MB     ros-kinetic-geneus
      0.0MB /     0.1MB     ros-kinetic-genlisp
      0.0MB /     0.1MB     ros-kinetic-gennodejs
*     7.9MB /    31.9MB     ros-kinetic-ros
*     0.0MB /     0.2MB       ros-kinetic-mk
      0.0MB /     0.1MB         ros-kinetic-rosbuild
*     0.1MB /     0.1MB       ros-kinetic-rosbash
      0.0MB /     0.1MB       ros-kinetic-rosboost-cfg
*     0.2MB /    23.2MB       ros-kinetic-rosclean
*     0.1MB /    23.1MB         python-rospkg
*     0.1MB /    23.1MB           python-rospkg-modules
      0.0MB /    23.0MB             lsb-release
*     0.1MB /     0.1MB       ros-kinetic-roscreate
*     0.0MB /     0.0MB       ros-kinetic-roslang
*     7.5MB /     7.9MB       ros-kinetic-roslib
*     0.0MB /     0.0MB         ros-kinetic-ros-environment
*     7.0MB /     7.4MB         ros-kinetic-rospack
*     0.1MB /     0.1MB           libboost-filesystem1.58.0
*     0.5MB /     0.5MB           libboost-program-options1.58.0
*     3.5MB /     3.5MB           libpython2.7
*     0.1MB /     0.1MB           libtinyxml2.6.2v5
      0.0MB /     0.2MB           libtinyxml-dev
      0.0MB /     0.2MB           pkg-config
      0.0MB /     0.0MB           python-dev
*     2.4MB /     2.4MB           python-rosdep
*     2.1MB /     2.1MB             python-rosdistro
*     1.3MB /     1.3MB               ca-certificates
*     0.9MB /     0.9MB                 openssl
*     0.7MB /     0.7MB               python-rosdistro-modules
*     0.5MB /     0.5MB                 python-setuptools
      0.0MB /     0.1MB       ros-kinetic-rosmake
      0.0MB /     0.2MB       ros-kinetic-rosunit
*    62.5MB /   159.5MB     ros-kinetic-ros-comm
*    36.5MB /    68.0MB       ros-kinetic-message-filters
*    33.3MB /    64.7MB         ros-kinetic-rosconsole
*    30.5MB /    30.5MB           libboost-regex1.58.0
*    29.4MB /    29.4MB             libicu55
*     2.4MB /     6.2MB           liblog4cxx10v5
*     0.4MB /     4.1MB             libapr1
*     0.1MB /     3.8MB               libuuid1
      0.0MB /     3.7MB                 passwd
*     0.2MB /     0.2MB             libaprutil1
      0.0MB /     8.2MB           libapr1-dev
      0.0MB /    12.2MB           libaprutil1-dev
      0.0MB /     7.3MB           liblog4cxx10-dev
*     3.1MB /     3.1MB         ros-kinetic-roscpp
*     0.1MB /     0.1MB           libboost-chrono1.58.0
*     0.2MB /     0.2MB           libboost-thread1.58.0
*     0.2MB /     0.2MB           ros-kinetic-rosgraph-msgs
*     0.2MB /     0.2MB           ros-kinetic-xmlrpcpp
*    21.3MB /    21.5MB       ros-kinetic-rosbag
*     0.6MB /     0.8MB         ros-kinetic-rosbag-storage
      0.0MB /     0.1MB           libbz2-dev
*     0.2MB /     0.4MB           ros-kinetic-roslz4
*     0.1MB /     0.1MB             liblz4-1
      0.0MB /     0.2MB             liblz4-dev
*    18.1MB /    18.1MB         ros-kinetic-rospy
*    17.1MB /    17.1MB           python-numpy
*     0.5MB /     0.5MB             libblas3
*     0.0MB /     0.0MB               libblas-common
*     7.5MB /     7.5MB             liblapack3
*     1.5MB /     1.5MB               libgfortran3
*     0.3MB /     0.3MB                 libquadmath0
*     0.3MB /     0.3MB           ros-kinetic-rosgraph
*     0.0MB /     0.0MB             python-netifaces
*     0.2MB /     0.2MB         ros-kinetic-std-srvs
*     1.1MB /     1.1MB         ros-kinetic-topic-tools
*     3.9MB /     3.9MB       ros-kinetic-roslaunch
*     2.6MB /     2.6MB         python-paramiko
*     1.9MB /     1.9MB           python-crypto
*     0.5MB /     0.5MB             libgmp10
*     0.1MB /     0.1MB           python-ecdsa
*     0.4MB /     0.4MB         ros-kinetic-rosmaster
*     0.1MB /     0.1MB           python-defusedxml
*     0.1MB /     0.1MB         ros-kinetic-rosout
*     0.1MB /     0.1MB         ros-kinetic-rosparam
      0.0MB /    65.3MB       ros-kinetic-roslisp
*     0.1MB /     0.1MB       ros-kinetic-rosmsg
*     0.3MB /     0.3MB       ros-kinetic-rosnode
*     0.2MB /     0.2MB         ros-kinetic-rostopic
*     0.1MB /     0.1MB       ros-kinetic-rosservice
*     0.1MB /     0.1MB       ros-kinetic-rostest
*     0.2MB /     0.2MB       ros-kinetic-roswtf
*     0.0MB /     0.0MB     ros-kinetic-rosconsole-bridge
*     0.0MB /     0.0MB     ros-kinetic-roscpp-core

gavanderhoorn · 2020-01-26T10:31:54.867Z

dirk-thomas:

marguedas:

Maybe it’s something that could be encouraged for ROS 2 ?

I don’t see why ROS 1 packages couldn’t be updated too. I doubt the number of packages (which use these “blanket” rosdep keys) and enough people care about is actually that high.

gavanderhoorn:

For ROS 1 however I doubt it is feasible any more (nr of packages, size of maintainer/developer community & sunsetting).

ROS Noetic is not even released yet and will be supported for 5 years. I would think that is motivation enough to still update package you care about. Also it doesn’t mean that the maintainer of a package has to do it. Anyone can create a pull request with this kind of change - the maintainer only has to accept it and roll a new release.

And as mentioned above an example how easy it is to improve the dependency size: ros/roscpp_core#117 uses specific rosdep keys (libboost-date-time-dev, libboost-system-dev, libboost-thread-dev) instead of boost (which maps to libboost-all-dev). The effect on installing ros-*-roscpp_core (assuming you have build-essential installed anyway):

Before it pulled in ~500MB

Afterwards it only pulls in ~200MB

Reason I was hesitant to start doing this is that we risk breaking downstream packages that may be (implicitly/unknowingly) depending on cpp_common to bring in libboost-all-dev.

This seems to have happened before (from: ros/roscpp_core#24):

Before the ROS package console_bridge was bringing in the boost dependency transitively. Since roscpp_core now uses the system package of console_bridge boost is not exposed as a transitive dependency anymore.

Anyway the packages in roscpp_core should state their direct dependencies explicitly anyway and not rely on transitive dependencies for this. Please see PR #25 for the proposed fix.

Technically those downstream packages would need to be fixed of course. I just wasn’t sure that was a feasible thing to do in ROS 1 still.

That was all really. I’m more than happy to start submitting PRs. But I don’t like breaking things and potentially problematic changes like this have been met with hesitation from OR maintainers in the past, so it’s not too strange for us to first look at approaches that would result in similar gains without causing too much trouble “upstream”.

marguedas · 2020-01-26T10:54:58.529Z

gavanderhoorn:

Reason I was hesitant to start doing this is that we risk breaking downstream packages that may be (implicitly/unknowingly) depending on cpp_common to bring in libboost-all-dev .

Same reason here.
In the above example of roscpp not declaring its’ boost dependency. It’s likely that this cpp_common PR broke roscpp in the process.
In the case where the changes are targeting noetic only, it wouldn’t be too much of an issue as maintainers would update their packages before releasing. But for people not having a dedicated noetic branch it may propagate build failures every time one layer fixes the dep declaration.

If that’s a road we’re comfortable going down for Noetic, I’m happy to help submitting PRs.

gavanderhoorn · 2020-01-26T15:33:11.225Z

marguedas:

It’s likely that this cpp_common PR broke roscpp in the process.

Seeing as roscpp hasn’t been released yet for Noetic that isn’t much of a problem right now.

Noetic would seem like the time to start pruning these package manifests.

re: replacing blanket depends: moving dependencies on keys like boost to build_depend et al. would keep builds working. The runtime dependencies would be much lighter though.

Perhaps a two-step process could be used: first tighten up the exec_depends, then change build dependencies.

And a related PR for rosdep:

github.com/ros-infrastructure/rosdep

Selection of dependency types with cmd line option

ros-infrastructure:master ← nalt:master

opened 01:18PM - 28 Oct 19 UTC

nalt

+120 -7

Select dependency types (build, buildtool, run, test) to install with --dependen…cy-types. can be given multiple. Standard behaviour (all types) unchanged. In rospack.py, a fake and empty options object was created for the call to _get_default_RosdepLookup. This was replaced by a similarly hacky global object for command line options.

gavanderhoorn · 2020-01-26T15:54:59.513Z

Will_Gardner:

As it traverses the graph it will start to prune off dependencies based off of certain criteria, some of this will be hard coded rules based on package names but mostly it’s automated heuristics. The main one is the one which eliminates packages based on their debian section. While all the ROS packages sit in the misc section (since bloom doesn’t know any better) the packages in the upstream Ubuntu repos are nicely split up which makes it easy to eliminate all the packages in the devel section or the graphics section.

Bloom’s patch support could perhaps be used to change the default misc to more appropriate values. That would require some work, but would seem doable for at least stuff in ros_core and similar metapackages.

But should we perhaps consider adding support for categories to ROS package manifests? Packages could be placed in misc by default, but if the author/maintainer provides the metadata, Bloom could use that to populate the Section field.

RPM did have “groups”, but those seem to have been deprecated some time ago.

Will_Gardner:

Here’s an example of the pruned graph for ros-kinetic-ros-core (nodes are only shown the first time they’re found in the graph):

Entries without a * are pruned then, I gather? And the second column shows the cumulative sub total of the packages installed by (and for) that branch of the tree (but the trees only include a dependency if it wasn’t added somewhere else earlier (so lsb_release shows up with 23MB, but that’s probably because it depends on all sorts of Python dependencies which aren’t shown a second time))?

With some primitive scripting and a very manual process I’d arrived at a similar set of packages, but I hadn’t gotten ros-kinetic-ros-core down to 650 MB yet.

marguedas · 2020-01-26T17:12:12.338Z

gavanderhoorn:

Perhaps a two-step process could be used: first tighten up the exec_depend s, then change build dependencies.

I started part of the process and opened PRs to ros_comm and its’ dependencies to get rid of the blanket boost key (in both build and exec). This was pretty straight forward (though not automated) and did not include any audit/tightening of the runtime dependencies. This allowed to propagate the ~300MB space gain to the all of ros_comm.

re: “first tighten up the exec_depend s”: Migrating all these packages to format 2 or 3 would also allow us to leverage the exec_depend versus build_export_depend and tighten them accordingly. Bloom could then use that information to segregate -dev vs actual runtime dependencies.

gavanderhoorn:

And a related PR for rosdep :

github.com/ros-infrastructure/rosdep

Great !

dirk-thomas · 2020-01-27T18:23:37.961Z

gavanderhoorn:

Reason I was hesitant to start doing this is that we risk breaking downstream packages that may be (implicitly/unknowingly) depending on cpp_common to bring in libboost-all-dev .

Assuming that almost all packages depend on e.g. roscpp the only chance to breaking downstream packages is if all packages including and below roscpp are switching from boost to something more specific for an already released distro. If only one package limits the change to Noetic (since it already needs a separate branch for other reasons) downstream packages will still have boost available in Kinetic / Melodic. Therefore I think the chance for regressions is fairly low.

gavanderhoorn:

Perhaps a two-step process could be used: first tighten up the exec_depend s, then change build dependencies.

What would be the advantage of that? The first step already affects downstream packages. The second step is local to the package and is easy to test and doesn’t affect other packages - so why should it not be done at the same time?

gavanderhoorn · 2020-01-27T20:11:09.314Z

dirk-thomas:

gavanderhoorn:

Reason I was hesitant to start doing this is that we risk breaking downstream packages that may be (implicitly/unknowingly) depending on cpp_common to bring in libboost-all-dev .

Assuming that almost all packages depend on e.g. roscpp the only chance to breaking downstream packages is if all packages including and below roscpp are switching from boost to something more specific for an already released distro. If only one package limits the change to Noetic (since it already needs a separate branch for other reasons) downstream packages will still have boost available in Kinetic / Melodic. Therefore I think the chance for regressions is fairly low.

Hence the idea of targetting Noetic with this.

Transitively depending on something is an anti-pattern, but as this situation (ie: cpp_common brining in all of libboost-dev-all) has existed for so long there is a good chance it’s actually quite common. The Noetic changeover would be a good time to break things in this case.

dirk-thomas:

gavanderhoorn:

Perhaps a two-step process could be used: first tighten up the exec_depend s, then change build dependencies.

What would be the advantage of that? The first step already affects downstream packages. The second step is local to the package and is easy to test and doesn’t affect other packages - so why should it not be done at the same time?

It would allow us to gauge potential positive sides of this effort without immediately breaking everything.

But if we’re targetting Noetic I would be OK with not doing this.

I do expect there to be a rather large set of packages that will not have (active) maintainers any more (see the Orphaned Package maintainer effort fi) and releasing packages into Noetic could become a bit more of a hassle if we start pruning build and exec depends.