Generating 'dev' and runtime artefacts from ROS packages

gavanderhoorn · 2020-01-23T09:58:15.006Z

Will_Gardner:

Before installing our ROS dependencies into our docker images we install custom, fake, metapackages. These metapackages are special in that they don’t install anything themselves, they simply “provide” (in the debian packaging sense) all of the upstream dev dependencies of which ever ROS packages we’re intending to install (plus a subset of ROS packages that we definately don’t need) without actually adding anything to the host system

Are these REP-127/140/149 metapackages, or the Debian variant? If the latter, are you using equivs for this?

Will_Gardner:

As an example the ros-melodic-cpp-common deb package, which is in any core installation of ROS, has a dependency on libboost-dev. This boost metapackage is massive and has massive dependencies of its own, installing it on a fresh Ubuntu docker image installs 586MB worth of stuff even with --no-install-recommends .

Yes, that was something I quickly ran into as well.

Will_Gardner:

That way when we later install the ROS dependencies it doesn’t actually install the dev dependencies. It turns out with this alone the size of a bare ROS docker image went from ~800MiB down to ~100MiB compressed. We’ve similarly extended this strategy to tame OpenCV and CUDA which means even our largest base images max out at about 400MiB.

The positive news is that solving this specific problem in ROS packaging more generally is probably much easier than implementing -dev packages in their entirety. In theory all that is needed is to produce -dev packages that just have extra -dev dependencies of their own, no extra files.

Could you go into a bit more detail about your approach (haven’t had my coffee yet)? Are you referring to extra ROS packages here which package the dev dependencies?

Will_Gardner:

To a degree this should be straight forward as package.xml already has a mechanism for declaring these dev only transitive dependencies: build_export_depend . The downside of this approach is that it involves fixing dependencies on basically every package in the ROS ecosystem since that’s mainly used to export headers at the moment.

So if I understand you correctly you’d like to also use it to export “other” build dependencies?

Thanks for the insight, this is quite a creative way to deal with this (although technically still a work-around ).

Will_Gardner · 2020-01-23T10:57:55.878Z

Are these REP-127/140/149 metapackages, or the Debian variant? If the latter, are you using equivs for this?

Debian variant. When I prototyped it originally I did use equivs although we create them with custom tooling nowadays.

Could you go into a bit more detail about your approach (haven’t had my coffee yet)? Are you referring to extra ROS packages here which package the dev dependencies?

No. Ultimately we only care about the debian packages. Certainly one approach to achieve that is to have separate dev ros packages but that involves overhauling basically every C++ package in the ROS ecosystem since they all invariably depend on boost in some way.

So if I understand you correctly you’d like to also use it to export “other” build dependencies?

Yes, so for example I could say my foo package has build_export_depend on boost-dev, but exec_depend on boost-non-dev. This would then allow bloom to automatically generate two debian packages:

foo: depends on boost-non-dev, contains all of the foo package itself
foo-dev depends on boost-dev and foo

This would only require a fairly procedural update to fix up everyone’s package.xml

ruffsl · 2020-01-23T22:52:52.207Z

Will_Gardner:

It turns out with this alone the size of a bare ROS docker image went from ~800MiB down to ~100MiB compressed. We’ve similarly extended this strategy to tame OpenCV and CUDA which means even our largest base images max out at about 400MiB.

Nice! Do you have any Dockerfiles you could share publicly that demonstrate this approach?

marguedas · 2020-01-23T23:19:57.882Z

Will_Gardner:

At the moment the boost rosdep key is declared as being the dev packages for every given OS: https://github.com/ros/rosdistro/blob/master/rosdep/base.yaml#L272. It should be possible to create a “parallel” rosdep file that declares the non-dev versions of packages which can then be automatically used by bloom to generate runtime packages with the non-dev dependencies.

This is something that came to mind multiple times when trying to reduce the amount of packages being pulled.
While very convenient to have these “blanket” rosdep keys, it is very uncommon for any package to need “all of boost dev packages” or “all opencv dev packages”. A great deal of space would be saved if packages were to use more specific and targeted rosdep keys. While some stacks like OpenCV and PCL used to have a single/couple deb with everything, they now provide multiple debs and most likely we should leverage them. For some stacks there is already already a narrower set of keys available, the -dev and the release one: https://github.com/ros/rosdistro/blob/aaa5fe421f23c07234850c338ec4632d530ab746/rosdep/base.yaml#L1585-L1600

In an ideal world, all packages would define separately their build_depend with the -dev version of the libraries they use and the exec_depend with the runtime libs they really need to run the package.

Given the number of packages using (or implicitly relying) on these blanket rules, it would be very challenging to actually apply this to an entire distro.

Maybe it’s something that could be encouraged for ROS 2 ? as the number of packages is still pretty limited, so would be auditable. We would need to find a way to discourage / warn users to steer them away from using these blanket rules.

ruffsl:

Nice! Do you have any Dockerfiles you could share publicly that demonstrate this approach?

The would be great and could be inspiration to reduce the size of the official ROS images that are ginormous.

ruffsl · 2020-01-24T00:20:03.451Z

marguedas:

Maybe it’s something that could be encouraged for ROS 2 ? as the number of packages is still pretty limited, so would be auditable. We would need to find a way to discourage / warn users to steer them away from using these blanket rules.

Hear! Hear! A fresh a proper slate, no time like the present to set the president.

gavanderhoorn · 2020-01-24T12:24:40.896Z

tfoote:

So in conclusion I think what I’d like to suggest is that we already have support for all the bells and whistles needed to very clearly define the necessary dependencies. And that if we can identify places where separating the build and runtime dependencies would be valuable that we consider simply making separate packages and not try to find ways to automatically split a package and then find ways to refer to those partial packages in downstream packages. We’re generating packages, we have packages. Keeping it one to one is basically the simplest solution.

This would be a nice way to go about it, but as @Will_Gardner also wrote, that’s going to be a lot of work (and will result in many more packages as well, which has an adverse affect of build times). For ROS 2 (as @marguedas suggested) this might be possible still.

For ROS 1 however I doubt it is feasible any more (nr of packages, size of maintainer/developer community & sunsetting).

What @Will_Gardner describes seems like a low-hanging fruit that could allow us to significantly reduce the footprint of ROS 1 (and potentially ROS 2) deployments without requiring too much work.

@Will_Gardner: could you provide a little more detail on what you are doing with those metapackages such that we could replicate it? I’m sure @ruffsl would be able to figure out whether we could apply a similar approach to the official ros and ros2 Docker images.

simonschmeisser · 2020-01-24T15:37:48.100Z

marguedas:

In an ideal world, all packages would define separately their build_depend with the -dev version of the libraries they use and the exec_depend with the runtime libs they really need to run the package.

Given the number of packages using (or implicitly relying) on these blanket rules, it would be very challenging to actually apply this to an entire distro.

Maybe I or someone else can come up with a graphviz/dot script to identify the worst offenders in individual subsets of the packages? I’ll try my luck with ogre/rviz

gavanderhoorn · 2020-01-25T16:46:10.949Z

There are quite a few tools available for visualising dependency trees (debtree and apt-cache can do it fi).

ros-melodic-cpp-common indeed brings in approx 580 MB of dependencies (when installed on a bare ubuntu:bionic). libboost-all-dev is essentially almost singularly responsible for this (207 pkgs for libboost-all-dev vs 210 pkgs for ros-melodic-cpp-common).

ros-melodic-roscpp already lists very specific parts of Boost as dependencies, but as it depends on cpp-common (as @Will_Gardner wrote), installing just roscpp on a bare ubuntu:bionic image wants to install 655 MB of packages.

Of this only 8.1 MB is actually placed inside /opt/ros/melodic.

Edit: @Will_Gardner: I imagine you have some way of figuring out the runtime dependencies of particular ROS nodes / packages and then generate metapackages based on that information. Have you automated this, is this a manual process, or are you relying on package manifests?

marguedas · 2020-01-25T22:40:26.742Z

gavanderhoorn:

ros-melodic-roscpp already lists very specific parts of Boost as dependencies

I guess that’s what I meant by “relying on blanket rules”. While roscpp find_package’s and uses specific parts of boost, it doesn’t declare dependencies on any part of it in it’s package.xml, relying on lower level packages to provide all the parts of boost it needs.

Not sure how the proper subset of boost packages appeared in the control file though…

gavanderhoorn:

Edit: @Will_Gardner: I imagine you have some way of figuring out the runtime dependencies of particular ROS nodes / packages and then generate metapackages based on that information. Have you automated this, is this a manual process, or are you relying on package manifests?

Definitely tools helping to extract this information will be very useful

marguedas:

Maybe it’s something that could be encouraged for ROS 2 ? as the number of packages is still pretty limited, so would be auditable. We would need to find a way to discourage / warn users to steer them away from using these blanket rules.

Hear! Hear! A fresh a proper slate, no time like the present to set the president.

Yeah getting these split for ROS 2 and allow to install only the nondev version of ROS packages would be great.
Looks like ROS 2 is facing other challenges as the core is completely boost-free and the eloquent-ros-core image is still 700MB, my guess is from the way message packages are currently packaged.

dirk-thomas · 2020-01-25T23:10:51.290Z

marguedas:

While very convenient to have these “blanket” rosdep keys, it is very uncommon for any package to need “all of boost dev packages” or “all opencv dev packages”. A great deal of space would be saved if packages were to use more specific and targeted rosdep keys.

There is nothing holding us back to do exactly that already. All it takes is a minute to make a PR to update any packages which do use these “blanket” rosdep keys (see below for an example).

marguedas:

Maybe it’s something that could be encouraged for ROS 2 ?

I don’t see why ROS 1 packages couldn’t be updated too. I doubt the number of packages (which use these “blanket” rosdep keys) and enough people care about is actually that high.

gavanderhoorn:

For ROS 1 however I doubt it is feasible any more (nr of packages, size of maintainer/developer community & sunsetting).

ROS Noetic is not even released yet and will be supported for 5 years. I would think that is motivation enough to still update package you care about. Also it doesn’t mean that the maintainer of a package has to do it. Anyone can create a pull request with this kind of change - the maintainer only has to accept it and roll a new release.

And as mentioned above an example how easy it is to improve the dependency size: ros/roscpp_core#117 uses specific rosdep keys (libboost-date-time-dev, libboost-system-dev, libboost-thread-dev) instead of boost (which maps to libboost-all-dev). The effect on installing ros-*-roscpp_core (assuming you have build-essential installed anyway):

Before it pulled in ~500MB
Afterwards it only pulls in ~200MB

While dev/non-dev ROS packages would certainly get us further we could already do something today. So please consider to create PRs to replace the usage of “blanket” rosdep keys with something more specific.

Will_Gardner · 2020-01-26T00:34:08.628Z

Our Dockerfiles by themselves aren’t all that useful since most of what they do is install fake packages.

The generation of the fake packages is the interesting part, I’m not sure how much I can share so I’ll describe it in high level terms. The entire process operates entirely within the realms of Debian packaging, nothing ROS specific.

When we want to build an image we declare its high level debian dependencies, typically this will be a set of ROS packages (e.g. ros-kinetic-ros-core) and some other libraries that we want to use with it. We then have a Python script which takes this list and traverses the dependency graph. As it traverses the graph it will start to prune off dependencies based off of certain criteria, some of this will be hard coded rules based on package names but mostly it’s automated heuristics. The main one is the one which eliminates packages based on their debian section. While all the ROS packages sit in the misc section (since bloom doesn’t know any better) the packages in the upstream Ubuntu repos are nicely split up which makes it easy to eliminate all the packages in the devel section or the graphics section. As we prune off packages we add them to a list of packages to “provide” in the metapackage we’re going to generate.

Along with this we also explicitly add certain non-dev dependencies to the images. For example we explicitly add all of the non-dev boost packages (which turn out to be not that big) to our images to make up for not having the boost-dev meta-package.

Here’s an example of the pruned graph for ros-kinetic-ros-core (nodes are only shown the first time they’re found in the graph):

*   113.5MB /   658.9MB   ros-kinetic-ros-core
*    31.9MB /    81.6MB     ros-kinetic-catkin
      0.0MB /    36.0MB       cmake
      0.0MB /     7.3MB       google-mock
      0.0MB /     0.0MB       libgtest-dev
*    30.0MB /    36.5MB       python-catkin-pkg
*    26.6MB /    33.1MB         python-dateutil
*    23.7MB /    30.2MB           python-six
*    23.6MB /    30.1MB             python
*    23.0MB /    29.5MB               python2.7
*     6.3MB /     6.3MB                 python2.7-minimal
*     2.7MB /     2.7MB                   libpython2.7-minimal
*     0.2MB /     0.2MB                   zlib1g
*    16.3MB /    22.8MB                 libpython2.7-stdlib
*     0.1MB /     0.1MB                   mime-support
*     0.1MB /     0.1MB                   libbz2-1.0
*     1.7MB /     1.7MB                   libdb5.3
*     0.4MB /     0.4MB                   libexpat1
*     0.1MB /     0.1MB                   libffi6
*     0.8MB /     0.8MB                   libncursesw5
*     0.5MB /     0.5MB                     libtinfo5
*     0.4MB /     6.9MB                   libreadline6
*     0.1MB /     6.6MB                     readline-common
      0.0MB /     6.5MB                       dpkg
*     0.9MB /     0.9MB                   libsqlite3-0
*     3.3MB /     3.3MB                   libssl1.0.0
*     0.0MB /     0.0MB               libpython-stdlib
*     2.7MB /     2.7MB           tzdata
*     2.9MB /     2.9MB         python-docutils
*     0.0MB /     0.0MB           python-roman
*     1.2MB /     1.2MB           docutils-common
*     0.1MB /     0.1MB             sgml-base
*     0.5MB /     0.5MB             xml-core
*     0.3MB /     0.3MB               sed
*     0.2MB /     0.2MB         python-pyparsing
*     0.2MB /     0.2MB         python-catkin-pkg-modules
*     0.3MB /     0.3MB       python-empy
*     1.0MB /     1.0MB       python-nose
*     0.4MB /     0.4MB         python-pkg-resources
      0.0MB /     0.1MB     ros-kinetic-cmake-modules
*    11.0MB /   385.4MB     ros-kinetic-common-msgs
*     4.9MB /   379.3MB       ros-kinetic-actionlib-msgs
      0.0MB /     0.3MB         ros-kinetic-message-generation
*     3.6MB /   377.7MB         ros-kinetic-message-runtime
*     2.1MB /   376.3MB           ros-kinetic-cpp-common
*     2.0MB /     2.0MB             libconsole-bridge0.2v5
*     2.0MB /     2.0MB               libstdc++6
*     0.1MB /     0.1MB                 gcc-5-base
      0.0MB /   308.0MB             libboost-all-dev
      0.0MB /    66.1MB             libconsole-bridge-dev
*     1.0MB /     1.0MB           ros-kinetic-genpy
*     0.6MB /     0.6MB             python-yaml
*     0.2MB /     0.2MB               libyaml-0-2
*     0.2MB /     0.2MB             ros-kinetic-genmsg
*     0.4MB /     0.4MB           ros-kinetic-roscpp-serialization
*     0.4MB /     0.4MB             ros-kinetic-roscpp-traits
*     0.3MB /     0.3MB               ros-kinetic-rostime
*     0.1MB /     0.1MB                 libboost-system1.58.0
*     1.1MB /     1.1MB         ros-kinetic-std-msgs
*     0.3MB /     0.3MB       ros-kinetic-diagnostic-msgs
*     1.2MB /     1.2MB       ros-kinetic-geometry-msgs
*     1.0MB /     1.0MB       ros-kinetic-nav-msgs
*     1.8MB /     1.8MB       ros-kinetic-sensor-msgs
*     0.2MB /     0.2MB       ros-kinetic-shape-msgs
*     0.1MB /     0.1MB       ros-kinetic-stereo-msgs
*     0.3MB /     0.3MB       ros-kinetic-trajectory-msgs
      0.0MB /     0.0MB         ros-kinetic-rosbag-migration-rule
*     1.1MB /     1.1MB       ros-kinetic-visualization-msgs
*     0.1MB /     0.1MB     ros-kinetic-gencpp
*     0.1MB /     0.1MB     ros-kinetic-geneus
      0.0MB /     0.1MB     ros-kinetic-genlisp
      0.0MB /     0.1MB     ros-kinetic-gennodejs
*     7.9MB /    31.9MB     ros-kinetic-ros
*     0.0MB /     0.2MB       ros-kinetic-mk
      0.0MB /     0.1MB         ros-kinetic-rosbuild
*     0.1MB /     0.1MB       ros-kinetic-rosbash
      0.0MB /     0.1MB       ros-kinetic-rosboost-cfg
*     0.2MB /    23.2MB       ros-kinetic-rosclean
*     0.1MB /    23.1MB         python-rospkg
*     0.1MB /    23.1MB           python-rospkg-modules
      0.0MB /    23.0MB             lsb-release
*     0.1MB /     0.1MB       ros-kinetic-roscreate
*     0.0MB /     0.0MB       ros-kinetic-roslang
*     7.5MB /     7.9MB       ros-kinetic-roslib
*     0.0MB /     0.0MB         ros-kinetic-ros-environment
*     7.0MB /     7.4MB         ros-kinetic-rospack
*     0.1MB /     0.1MB           libboost-filesystem1.58.0
*     0.5MB /     0.5MB           libboost-program-options1.58.0
*     3.5MB /     3.5MB           libpython2.7
*     0.1MB /     0.1MB           libtinyxml2.6.2v5
      0.0MB /     0.2MB           libtinyxml-dev
      0.0MB /     0.2MB           pkg-config
      0.0MB /     0.0MB           python-dev
*     2.4MB /     2.4MB           python-rosdep
*     2.1MB /     2.1MB             python-rosdistro
*     1.3MB /     1.3MB               ca-certificates
*     0.9MB /     0.9MB                 openssl
*     0.7MB /     0.7MB               python-rosdistro-modules
*     0.5MB /     0.5MB                 python-setuptools
      0.0MB /     0.1MB       ros-kinetic-rosmake
      0.0MB /     0.2MB       ros-kinetic-rosunit
*    62.5MB /   159.5MB     ros-kinetic-ros-comm
*    36.5MB /    68.0MB       ros-kinetic-message-filters
*    33.3MB /    64.7MB         ros-kinetic-rosconsole
*    30.5MB /    30.5MB           libboost-regex1.58.0
*    29.4MB /    29.4MB             libicu55
*     2.4MB /     6.2MB           liblog4cxx10v5
*     0.4MB /     4.1MB             libapr1
*     0.1MB /     3.8MB               libuuid1
      0.0MB /     3.7MB                 passwd
*     0.2MB /     0.2MB             libaprutil1
      0.0MB /     8.2MB           libapr1-dev
      0.0MB /    12.2MB           libaprutil1-dev
      0.0MB /     7.3MB           liblog4cxx10-dev
*     3.1MB /     3.1MB         ros-kinetic-roscpp
*     0.1MB /     0.1MB           libboost-chrono1.58.0
*     0.2MB /     0.2MB           libboost-thread1.58.0
*     0.2MB /     0.2MB           ros-kinetic-rosgraph-msgs
*     0.2MB /     0.2MB           ros-kinetic-xmlrpcpp
*    21.3MB /    21.5MB       ros-kinetic-rosbag
*     0.6MB /     0.8MB         ros-kinetic-rosbag-storage
      0.0MB /     0.1MB           libbz2-dev
*     0.2MB /     0.4MB           ros-kinetic-roslz4
*     0.1MB /     0.1MB             liblz4-1
      0.0MB /     0.2MB             liblz4-dev
*    18.1MB /    18.1MB         ros-kinetic-rospy
*    17.1MB /    17.1MB           python-numpy
*     0.5MB /     0.5MB             libblas3
*     0.0MB /     0.0MB               libblas-common
*     7.5MB /     7.5MB             liblapack3
*     1.5MB /     1.5MB               libgfortran3
*     0.3MB /     0.3MB                 libquadmath0
*     0.3MB /     0.3MB           ros-kinetic-rosgraph
*     0.0MB /     0.0MB             python-netifaces
*     0.2MB /     0.2MB         ros-kinetic-std-srvs
*     1.1MB /     1.1MB         ros-kinetic-topic-tools
*     3.9MB /     3.9MB       ros-kinetic-roslaunch
*     2.6MB /     2.6MB         python-paramiko
*     1.9MB /     1.9MB           python-crypto
*     0.5MB /     0.5MB             libgmp10
*     0.1MB /     0.1MB           python-ecdsa
*     0.4MB /     0.4MB         ros-kinetic-rosmaster
*     0.1MB /     0.1MB           python-defusedxml
*     0.1MB /     0.1MB         ros-kinetic-rosout
*     0.1MB /     0.1MB         ros-kinetic-rosparam
      0.0MB /    65.3MB       ros-kinetic-roslisp
*     0.1MB /     0.1MB       ros-kinetic-rosmsg
*     0.3MB /     0.3MB       ros-kinetic-rosnode
*     0.2MB /     0.2MB         ros-kinetic-rostopic
*     0.1MB /     0.1MB       ros-kinetic-rosservice
*     0.1MB /     0.1MB       ros-kinetic-rostest
*     0.2MB /     0.2MB       ros-kinetic-roswtf
*     0.0MB /     0.0MB     ros-kinetic-rosconsole-bridge
*     0.0MB /     0.0MB     ros-kinetic-roscpp-core

gavanderhoorn · 2020-01-26T10:31:54.867Z

dirk-thomas:

marguedas:

Maybe it’s something that could be encouraged for ROS 2 ?

I don’t see why ROS 1 packages couldn’t be updated too. I doubt the number of packages (which use these “blanket” rosdep keys) and enough people care about is actually that high.

gavanderhoorn:

For ROS 1 however I doubt it is feasible any more (nr of packages, size of maintainer/developer community & sunsetting).

ROS Noetic is not even released yet and will be supported for 5 years. I would think that is motivation enough to still update package you care about. Also it doesn’t mean that the maintainer of a package has to do it. Anyone can create a pull request with this kind of change - the maintainer only has to accept it and roll a new release.

And as mentioned above an example how easy it is to improve the dependency size: ros/roscpp_core#117 uses specific rosdep keys (libboost-date-time-dev, libboost-system-dev, libboost-thread-dev) instead of boost (which maps to libboost-all-dev). The effect on installing ros-*-roscpp_core (assuming you have build-essential installed anyway):

Before it pulled in ~500MB

Afterwards it only pulls in ~200MB

Reason I was hesitant to start doing this is that we risk breaking downstream packages that may be (implicitly/unknowingly) depending on cpp_common to bring in libboost-all-dev.

This seems to have happened before (from: ros/roscpp_core#24):

Before the ROS package console_bridge was bringing in the boost dependency transitively. Since roscpp_core now uses the system package of console_bridge boost is not exposed as a transitive dependency anymore.

Anyway the packages in roscpp_core should state their direct dependencies explicitly anyway and not rely on transitive dependencies for this. Please see PR #25 for the proposed fix.

Technically those downstream packages would need to be fixed of course. I just wasn’t sure that was a feasible thing to do in ROS 1 still.

That was all really. I’m more than happy to start submitting PRs. But I don’t like breaking things and potentially problematic changes like this have been met with hesitation from OR maintainers in the past, so it’s not too strange for us to first look at approaches that would result in similar gains without causing too much trouble “upstream”.

marguedas · 2020-01-26T10:54:58.529Z

gavanderhoorn:

Reason I was hesitant to start doing this is that we risk breaking downstream packages that may be (implicitly/unknowingly) depending on cpp_common to bring in libboost-all-dev .

Same reason here.
In the above example of roscpp not declaring its’ boost dependency. It’s likely that this cpp_common PR broke roscpp in the process.
In the case where the changes are targeting noetic only, it wouldn’t be too much of an issue as maintainers would update their packages before releasing. But for people not having a dedicated noetic branch it may propagate build failures every time one layer fixes the dep declaration.

If that’s a road we’re comfortable going down for Noetic, I’m happy to help submitting PRs.

gavanderhoorn · 2020-01-26T15:33:11.225Z

marguedas:

It’s likely that this cpp_common PR broke roscpp in the process.

Seeing as roscpp hasn’t been released yet for Noetic that isn’t much of a problem right now.

Noetic would seem like the time to start pruning these package manifests.

re: replacing blanket depends: moving dependencies on keys like boost to build_depend et al. would keep builds working. The runtime dependencies would be much lighter though.

Perhaps a two-step process could be used: first tighten up the exec_depends, then change build dependencies.

And a related PR for rosdep:

github.com/ros-infrastructure/rosdep

Selection of dependency types with cmd line option

ros-infrastructure:master ← nalt:master

opened 01:18PM - 28 Oct 19 UTC

nalt

+120 -7

Select dependency types (build, buildtool, run, test) to install with --dependen…cy-types. can be given multiple. Standard behaviour (all types) unchanged. In rospack.py, a fake and empty options object was created for the call to _get_default_RosdepLookup. This was replaced by a similarly hacky global object for command line options.

gavanderhoorn · 2020-01-26T15:54:59.513Z

Will_Gardner:

As it traverses the graph it will start to prune off dependencies based off of certain criteria, some of this will be hard coded rules based on package names but mostly it’s automated heuristics. The main one is the one which eliminates packages based on their debian section. While all the ROS packages sit in the misc section (since bloom doesn’t know any better) the packages in the upstream Ubuntu repos are nicely split up which makes it easy to eliminate all the packages in the devel section or the graphics section.

Bloom’s patch support could perhaps be used to change the default misc to more appropriate values. That would require some work, but would seem doable for at least stuff in ros_core and similar metapackages.

But should we perhaps consider adding support for categories to ROS package manifests? Packages could be placed in misc by default, but if the author/maintainer provides the metadata, Bloom could use that to populate the Section field.

RPM did have “groups”, but those seem to have been deprecated some time ago.

Will_Gardner:

Here’s an example of the pruned graph for ros-kinetic-ros-core (nodes are only shown the first time they’re found in the graph):

Entries without a * are pruned then, I gather? And the second column shows the cumulative sub total of the packages installed by (and for) that branch of the tree (but the trees only include a dependency if it wasn’t added somewhere else earlier (so lsb_release shows up with 23MB, but that’s probably because it depends on all sorts of Python dependencies which aren’t shown a second time))?

With some primitive scripting and a very manual process I’d arrived at a similar set of packages, but I hadn’t gotten ros-kinetic-ros-core down to 650 MB yet.

marguedas · 2020-01-26T17:12:12.338Z

gavanderhoorn:

Perhaps a two-step process could be used: first tighten up the exec_depend s, then change build dependencies.

I started part of the process and opened PRs to ros_comm and its’ dependencies to get rid of the blanket boost key (in both build and exec). This was pretty straight forward (though not automated) and did not include any audit/tightening of the runtime dependencies. This allowed to propagate the ~300MB space gain to the all of ros_comm.

re: “first tighten up the exec_depend s”: Migrating all these packages to format 2 or 3 would also allow us to leverage the exec_depend versus build_export_depend and tighten them accordingly. Bloom could then use that information to segregate -dev vs actual runtime dependencies.

gavanderhoorn:

And a related PR for rosdep :

github.com/ros-infrastructure/rosdep

Great !

dirk-thomas · 2020-01-27T18:23:37.961Z

gavanderhoorn:

Reason I was hesitant to start doing this is that we risk breaking downstream packages that may be (implicitly/unknowingly) depending on cpp_common to bring in libboost-all-dev .

Assuming that almost all packages depend on e.g. roscpp the only chance to breaking downstream packages is if all packages including and below roscpp are switching from boost to something more specific for an already released distro. If only one package limits the change to Noetic (since it already needs a separate branch for other reasons) downstream packages will still have boost available in Kinetic / Melodic. Therefore I think the chance for regressions is fairly low.

gavanderhoorn:

Perhaps a two-step process could be used: first tighten up the exec_depend s, then change build dependencies.

What would be the advantage of that? The first step already affects downstream packages. The second step is local to the package and is easy to test and doesn’t affect other packages - so why should it not be done at the same time?

gavanderhoorn · 2020-01-27T20:11:09.314Z

dirk-thomas:

gavanderhoorn:

Reason I was hesitant to start doing this is that we risk breaking downstream packages that may be (implicitly/unknowingly) depending on cpp_common to bring in libboost-all-dev .

Assuming that almost all packages depend on e.g. roscpp the only chance to breaking downstream packages is if all packages including and below roscpp are switching from boost to something more specific for an already released distro. If only one package limits the change to Noetic (since it already needs a separate branch for other reasons) downstream packages will still have boost available in Kinetic / Melodic. Therefore I think the chance for regressions is fairly low.

Hence the idea of targetting Noetic with this.

Transitively depending on something is an anti-pattern, but as this situation (ie: cpp_common brining in all of libboost-dev-all) has existed for so long there is a good chance it’s actually quite common. The Noetic changeover would be a good time to break things in this case.

dirk-thomas:

gavanderhoorn:

Perhaps a two-step process could be used: first tighten up the exec_depend s, then change build dependencies.

What would be the advantage of that? The first step already affects downstream packages. The second step is local to the package and is easy to test and doesn’t affect other packages - so why should it not be done at the same time?

It would allow us to gauge potential positive sides of this effort without immediately breaking everything.

But if we’re targetting Noetic I would be OK with not doing this.

I do expect there to be a rather large set of packages that will not have (active) maintainers any more (see the Orphaned Package maintainer effort fi) and releasing packages into Noetic could become a bit more of a hassle if we start pruning build and exec depends.

Mike_Purvis · 2020-01-30T18:19:15.487Z

I don’t have deep knowledge in this area, but my impression is that Debian packaging sidesteps much of this need by generating a lot of the runtime dependencies automatically. So like, you cast a wide net at build time with something like Build-Depends: libboost-dev, but then dpkg-shlibdeps steps in and examines the linker information for the binaries you produced and only generates runtime dependencies for the stuff you actually need. And similarly, runtime deps for Python packages are generated from the info already in the setup.py file rather than duplicated.

So like, the only runtime deps which must be explicitly stated in the control file are the oddball cases where you’re doing something not visible to ldd, like invoking a process or reading data that belongs to another package. Is there something in this approach that ROS/bloom can adopt?

For full disclosure, we still generate omnibus workspace bundle assets, and that allows us to apply very broad rules (the release asset excludes all *.h, *.msg, *.py, etc). And to the extent that we’ve considered breaking that asset up, it would be to go to something more nix-y, where individual packages are identified by hashes rather version numbers.

gavanderhoorn · 2020-02-06T10:27:48.888Z

Just an update: many of the PRs submitted by @marguedas have been merged.

The most important one being

github.com/ros/rosdistro

add and fix boost rules

ros:master ← mikaelarguedas:fix_boost_fedora

opened 10:39AM - 27 Jan 20 UTC

mikaelarguedas

+103 -6