For releasing MoveIt into Kinetic MoveIt would require a new version (0.5) of FCL packages since the current packages provided by Ubuntu repositories ship the fcl 0.3 series.
The solution for Kinetic is to include new 0.5 FCL packages in the ROS repositories. Altering software versions shipped by the official Ubuntu repositories (in this case changing the ABI/API) brings many different implications. Seems not to be other ROS packages different than MoveIt using FCL but as FCL is an official Ubuntu package any third party software (nor even related with ROS) using the FCL packages provided by Ubuntu together with a ROS repo would be affected in some way (i.e: gazebo from the OSRF repo uses DART physics engine from dartsim PPA which depends on FCL).
I’ve packaged FCL 0.5 for Debian Sid (Ubuntu Yakkety will sync from there). The library package name changes from 0.3 (libfcl0) to 0.5 (libfcl0.5) and both can be installed at the same time so we won’t break software depending on the Ubuntu version of libfcl. What would be override is the libfcl-devthat would now point to libfcl 0.5 versions instead of libfcl 0.3.
If we want to avoid this, we could generate a custom -dev package called libfcl-0.5-dev and make it conflict with libfcl-dev. This probably requires a rosdep change for Kinetic. There is a problem with this approach: packages using libfcl-dev would not be coinstallable with MoveIt.
Solve this block and make both packages co-installable would imply to create different paths for include headers (/usr/include/fcl0.5/ vs /usr/include/fcl/), different pkgconfig files (fcl0.5.pc vs fcl.pc) and modify Moveit headers (<fcl0.5/foo.h> vs <fcl/foo.h>).
Note that every level of solution requires effort in packaging.
The side by side libraries is great and definitely make sense to me.
For the headers, it’s generally accepted that you can only install one. So making a conflicting package makes sense. And I think that making the libfcl-0.5-dev makes sense so that we can depend on the correct version. Otherwise you might get the default version if you just resolve the rosdep rule.
Thanks for all the awesome work sorting this out @jrivero !
When system packages have versions that aren’t what ROS needs, what are the pros/cons of making that a ROS package vs. a new system package? In the FCL case, a new system package has been created for FCL which (the dev version at least) conflicts with the version in the upstream ubuntu APT repos. This new system package installs headers to /usr/include. In a different example, octomap version 1.8 has been released as a ROS package (debian is called ros-kinetic-octomap). That ROS package seems to be installable alongside the upstream liboctomap-dev (which is version 1.6.8) because everything goes in /opt/ros/kinetic/...
Given that this type of problem is likely to come up in the future, I’m curious to understand when one approach should be chosen versus the other. From my limited understanding:
** Pro/Cons of Creating a ROS package for the version ROS needs **
PRO: Can be installed alongside system version (even the headers) because everything goes in /opt
PRO: Builds as part of the standard ROS buildfarm; I assume this removes a step of manually copying the debs into the ROS APT repos each time a new minor version of the package is released?
CON: Requires wrapping the package with a package.xml and CMakeLists.txt, which may be more or less difficult depending on whether the upstream package uses cmake.
CON: The headers in /opt/ros/<distro>/include will shadow the ones from the upstream system package in /usr/include/ if both are installed. This could lead to some unexpected things. I’m not sure how bad this is.
** Pro/Cons of Creating a new sytem package for the version ROS needs **
PRO: Can use the project’s current process for generating debs; no package.xml and CMakeLists.txt needed
CON: If we need the package on OS’s that they haven’t packaged for, those will have to be handled in som other way (this is somewhat tangential to the version problem here)
CON: Need to manually copy the debs into the ROS APT repo
CON: Dev packages can’t be installed alongside the upstream system version, since the headers would collide
CON: The headers for the package that is being built with the version ROS needs (FCL 0.5 in this case) go into /usr/include but inside of them may #include headers which are installed in /opt/ros/. I’m not sure if this is really a problem, but it feels odd. In this case it happens because octomap 1.8.0 is a ROS package which puts headers into /opt/ros/... and the FCL 0.5 deb depends on ros-kinetic-octomap.
The current solution works for me, so I’m not pushing for changes, but just to understand the pros/cons of the approach for future reference.
That’s a pretty good summary of the immediate pros and cons. When I think about these decisions there are a few other higher level things I think about as well.
My top priority is to maintain compatibility with the goal of minimizing overall work. For any of these cases we can either choose to modify the dependency, or change our code to be backwards/downwards compatible. In this case with the package on the cusp of being packaged upstream both approaches would work. However if the package in question was a lower level dependency with several packages depending on it if we choose to upgrade the package in a non-abi compatible way we would either have to choose to conflict with all it’s dependencies or else rebuild replacement versions of them, greatly increasing the required work.
Thus we generally have a policy that if a package is released upstream we use the version there. This make sure we’re compatible with everything else in that distro. This is the point of having distros so that things are known compatible.
It is typically relatively easy to release a package that builds a rosdistro specific version into /opt/ros/. As you mention we’d need to make sure that all symbols and headers do not collide and would get appended in the right order. With appropriate patches this is not too much work. But in a case like this it’s making work both for this version and future versions as any changes needed will need to be planned to roll back/future compatible as we expect to go back to the upstream version in the future.
The biggest trap that we can fall into is to diverge from the upstream release roadmap. If we want a very specific feature and patch to add it to a custom packaged version but upstream chooses not to take the patch or implements it differently we can end up in a state where we need to maintain a diverged package continuing going forward. We’ve run into this for a couple of 3rd party packages in the past, such as xmlrpcpp and Bullet Physics Linear Math libraries, which we still maintain forks of due to diverging early in the history of ROS. Now they’re baked in deep enough that converging them on the upstream packages is more work than anyone is willing to do.
To avoid diverging we generally only repackage upstream packages by backporting newer versions. This is an example of that where we are building the version we expect to be in yakkety for xenial. This way ROS developers will not need to write their code to be compatible using fewer ifdefs or other mechanisms for multi version support. If they cannot do less comapatibility support we should not be building the custom version. Really we’re trading work patching or upgrading the upstream package and making sure it’s available on all platforms to save the ROS developers from needing to support the upstream version across the multiple platforms that rosdistros support.
The easiest cases are the backports where the package does not exist but the upstream package exists for a newer distro. Backporting it will not collide with an existing package, and the backported version should match conveniently for the developers building on top. This is the case that covers most of the custom packages we’ve pushed into the ROS repository.
In this case we were a little behind. Optimally we would have been looking ahead enough to make sure that the version of the fcl package would have been in xenial. If you pay attention early in the debian/ubuntu cycle and make sure that upstream releases are made early enough we have successfully helped make sure that debian/ubuntu has the versions of packages that we need, and that is compatible. (Many thanks to @dirk-thomas and @jrivero for their work on this front. Such as a CMake upgrade for xenail) Ultimately we don’t want to be building our own parallel distro, but want to be as closely integrated with the upstream distros as possible. Every time we choose something different causes a lot of work and adds to the ongoing maintenance burden.