It could be unlucky sampling or a temporary surge, but we’ve had about 4 to 5 Q&As on ROS Answers in the past week alone where in the end it turned out the problems reported by the OP were caused by UpstreamPackages. Over the past few months (years?) there’s also always been a few each month.
While there is great value in those upstreamed packages (ever try using ROS on a armel or mips target and don’t want to wait 5 hours before everything is compiled?) they seem to be causing an increasing nr of problems for new users, leading to increased nrs of posts to ROS Answers.
I’m not at all advocating for getting them removed, but perhaps we need to inform or warn (new) users better or in a different way such that we can avoid them installing ros-desktop-full instead of ros-$distro-desktop-full (even more confusing are the Python packages: python-roslaunch vs ros-$distro-roslaunch).
An admonition is easily added to the installation tutorials, but perhaps there is someone here who might have a more creative idea.
After all: who pays attention to admonitions? We just copy-paste installation instructions into a terminal, and users who end up installing any of the upstream packages most likely haven’t read/seen/used any of the tutorials in the first place (as they don’t mention them), so they would not see those admonitions.
Thanks for bringing this up and answering on ROS Answers as well.
When starting this initiative I had the idea of merging the packages with those provided by OSRF. This could be by providing an apt repository with the OSRF packages compiled against the base set in Debian/Ubuntu and providing it as a PPA, for example. I have a half finished script to automate this from back them, if someone wants to continue on it.
An other option would be to make the OSRF package aware of the versions in Debian/Ubuntu. This could be by declaring relations: https://www.debian.org/doc/debian-policy/ch-relationships.html and would allow to provide OSRF package as upgrades. Note that this can’t be done in the Debian/Ubuntu packages because those are created and released long before those by OSRF. AFAIK this is already the case for python-* packages.
I’m unsure if there is a fitting tool to provide information about the OSRF repo. While we can’t change apt, obviously, maybe it would make sense to add a note to rosdep?
If I understand the documentation correctly, that would require adding Replaces to all ROS packages that potentially conflict with the upstream packages?
Or are you thinking of using any of the other keywords?
But wouldn’t in this case all upstream packages need to be replaced? What would the version be in this case?
As I wrote: I like the fact they’re available, but for people installing (or attempting to install) ros-$distro-*, they should just be removed whenever the packages.ros.org ones are being installed.
An error message, a log entry, or a clear indication of what version is being used when roscore starts could help (You are using an installation of ROS packaged by Debian - in case you run into any problem, please send a bug to <debian email> etc.).
The main drawback of adding a break&replace is that any Debian package depending on ROS upstream packages become impossible to install the moment you want you want to install OSRF packages. If your 3d model editor depends on URDF, installing ros-*-desktop will uninstall your 3d model editor (I don’t what could be the impact of this, probably @jspricke have a better idea than me).
That would be great, but there are enough cases where users start installing packages like python-rosdep2 and python-roslib, only to realise that they need other packages. At this point they try to follow the regular installation tutorial, try to install ros-$distro-desktop-full and it fails.
So they can’t run roscore (or any other tool), as they only have a bunch of packages that don’t function.
Or at least, this has been posted by a nr of people on ROS Answers.
apt doesn’t do a very good job of reporting exactly what causes a failed installation, often it just states “X needs Y, but Y isn’t going to be installed”. aptitude is then sometimes used, but isn’t able to always figure things out either.
That’s exactly why I would propose to add the package relationships (why printing an error if you could just make it work ).
I’m not sure this is even needed. All ROS packages use the /opt/ros space, whereas the Debian packages are in /usr. Can you point to specific problems? It’s probably easier to propose something then.
Hi Thomas, good to see you around :). Yeah that’s a valid point and we have problems like this already (for example with the different OpenCV version we had in kinetic, or with a new Gazebo version). This is where I think a PPA, or multiple, could work out. We could have one that only tracks packages not in Debian/Ubuntu or one that imports new versions from Debian unstable or one that doesn’t break ABIs… I think there are a lot of options and it could work as a fallback to the OSRF build farm as well. But this depends on what people want and who takes action. Obviously an other option would be to package more software into Debian but I’m currently busy with the Python 3 transition and lately I didn’t need much more then rviz and rosbag.
Note the Debian packages for catkin_pkg - upstreamed into Debian as well as from the ROS repos - have the name package name. Therefore for the conflict we picked the Debian upstream of catkin a while ago.
There appears to be a new surge of users installing the upstreamed version of especially the various Python packages on Ubuntu Focal and Bionic. A selection of ROS Answers Q&As:
And these are just the ones I could quickly find, all from 2020, and there are many more like this on ROS Answers.
First, to reiterate: I’m actually grateful some of these packages exist, as it really helps when working with otherwise unsupported architectures (especially embedded platforms which run Debian).
But the “upstream package” problems need to be addressed, as it negatively affects the experience of (new) users and doesn’t seem to go away by itself.
I’m willing to do the work at this point, but I don’t know what to do.
@dirk-thomas / @jspricke / @kyrofa / anyone else with insight into this: is there any way we could improve user experience here in some way?
I don’t think we could come up with a way to make it better either, that’s part of the reason we weren’t pursuing the upstreaming of our packages ourselves. It’s nice to have, but upstream packages neither met our needs in terms of rolling out releases after a Ubuntu distribution was released (therefore we could not drop the ROS buildfarm in favor of them), nor did it let us easily avoid users getting the upstream and packages.ros.org versions mixed up…
I’d be happy to be shown wrong, but I don’t think there’s anything we can do to avoid this issue other than document it or convince Ubuntu to let us inject something into the “recommended installs” one gets when typing a command that hasn’t been installed yet.
I’m hoping @kyrofa or anyone else at Canonical and/or Debian has an idea. @jspricke also mentioned a couple of things, but it’s unclear to me which option would be ‘best’.
It is documented, but that doesn’t really seem to help.
We could add a link to UpstreamPackages on the Kinetic, Melodic and Noetic pages or even embed it, but that won’t help when users simply copy-paste the command that gets suggested to them when they try to use a tool they don’t have.
I’m afraid I don’t immediately have an idea for how to deal with this beyond better documentation. I’d say roswtf or similar could detect this type of situation, but new users aren’t typically going to be using such a tool, and not actually having that installed is the second issue in that list, haha! The issue really seems to stem from folks not actually using the documentation in the first place, and I’m not sure how to fix that.
That’s only for stuff that is immediately possible to install and thus is possible to index. I’m afraid that utility isn’t really setup to suggest things like “go add this new deb repo to your system and then install this package.” Such a list would be impossible to maintain anyway.
I’m pretty sure this is all solvable by fixing the OSRF deb packages. @gavanderhoorn can you ping me in IRC so we can do it together? (I’m jochensp in freenode and oftc).
Edit: let’s use #debian-robotics on oftc.
I won’t claim to be an expert in this system, but let me attempt to walk you through how this works so we’re on the same page. First of all, it’s called command-not-found (clever, eh?).
When debs are built, the command-not-found extractor generates a list of commands that is distributed along with the rest of the index (like this). When apt update is run, cnf-update-db is also run (see /etc/apt/apt.conf.d/50command-not-found) against those updated lists to generate an sqlite database in /usr/share/command-not-found/commands.db. This is the db against which command-not-found runs. It literally just matches up commands to the things that contain them that are part of the index assuming they aren’t already installed. Expanding that to make it understand that a given command is part of ROS and to do something special in that case is quite a change in behavior and I suspect would be non-trivial and hard to maintain over time. Not to mention I’m not sure it’s the right thing to do. These packages are part of the index. They’re valid suggestions as far as Ubuntu are concerned. Subverting that makes me feel a little squeamish. I bet we could make it so the OR packages have a higher priority (e.g.using Cnf-Priority-Bonus) if the build farm can start generating those files, but that still requires the OR repo to be added to the system sources first. Would that be helpful?
The problem as far as I can tell isn’t going to be completely solved by adding breaks/conflicts/replaces (or really anything else) to OR packages as the problem is often encountered before the OR deb repo is even added to the sources. Not saying that’s not worth doing, but it’s not going to be a silver bullet.
).