I know ROS 1 is unmaintained and officially unsupported on systems newer than Ubuntu Focal (now nearly three years old), but the Installation page does invite users to still build it from source, and attempt to do so on rolling-release systems like Arch and Chocolatey.
An issue I’ve encountered on systems with Python 3.10+ is that many package still use a from distutils.core import setup rather than from setuptools import setup, and this now refuses to build without the SETUPTOOLS_USE_DISTUTILS envvar set to stdlib (see this post for more context, also the setuptools v60 release notes).
Is this kind of thing something the community still has an appetite for trying to keep up to date on, or is my time better spent elsewhere than filing all the requisite PRs? Here’s a sampling of affected repos:
Hello Mike,
Thank you for your testing efforts.
I am maintaining a few of the rqt packages you mentioned.
Two of them rqt_robot_monitor and rqt_moveit, already consider your issue in the setup.py so that it always tries to do “from setuptools import setup” only on ImportError it falls back to “from distutils.core import setup”.
Specifically, it is this code block.
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
from catkin_pkg.python_setup import generate_distutils_setup
Does this also cause you problems on your rolling release machine?
In regards, if now all of these repositories should be fixed, it would be interesting if this is the only problem you had during your build?
If this would be the only problem on several popular updated distributions, but if you have to jump through a lot of additional hoops, it would maybe be easier to maybe just add a disclaimer to the “Build from source” tutorial that indicates that from now on additional individual steps have to be taken that maybe even could become more numerous in the future.
Considering that 20.04 reaches its EOL in 2030 and End of Standard Support in 2025, it should be usable on the robots for a few more years if you want to use ROS1.
But I guess your point is for the developer machines, it would be nice to be able to use a more recent distribution, and I can sympathize with this.
So if this is currently the only problem with building “desktop_full” from source, I think we could introduce the needed changes to the repos and releasing them, so the tarballs are available for the source build.
But if more complicated problems arise in the future (until 2025 Noetic EOL), we may be could consider shying away from further support for more recent distributions.
Edit: Multiple people are aware of this specific issue and fixed various packages already. Please contribute pull-requests or get in touch with the ROS-O org to fork to the github organization for unmaintained repositories.
Thanks for the detailed response, @Arne! Yes, doing the import in a try block is great, so long as setuptools is attempted first— as in your example, which makes sense of course, since distutils is part of the stdlib and therefore should never fail to import.
We’ve been running extensively on Python 3.9.9 for about a year and had no issues with RQT packages. Python 3.10.x has gotten much more limited testing, though with the SETUPTOOLS_USE_DISTUTILS envvar set, everything we use (a superset of desktop_full) definitely at least builds.
Thank you very much for the additional information @Mike_Purvis
In my opinion, we should try to get the changed import statement into as many packages as possible.
Where this is for whatever reason not possible, the people building from source might use the envvar SETUPTOOLS_USE_DISTUTILS (up until Python 3.12) or can fix the remaining issues locally.
I would propose the following: I can help you out with the PRs, and maybe you could because you seem to already have a sizable amount of affected systems; see if you could find more packages that have this issue.
I did a quick & dirty grep over the repositories you get via “rosinstall_generator desktop_full --rosdistro noetic --deps --upstream” and found that it seems that a lot of core packages already changed their import in their setup.py files to solely use the “from setuptools import setup” statement.
But don’t use the try…except block. I would like to keep doing this as it should guarantee that nothing breaks on “really old” systems.
Those packages, however, which still only use “from distutils.core import setup” are clustered in the rqt-group. And, of course, there are a few exceptions.
So this should be a doable task.
If it is OK with you, I would start creating the PRs (starting with the rqt packages) on the weekend, as my current week is rather full. And you could maybe use your systems to find other candidates, preferably in relation to desktop-full + common packages that could benefit from this patch.
Thank you very much, and looking forward to your opinion.
The sentiment to keep backward compatibility is definitely appreciated by everyone, but setuptools’s setup routine started out as the one in distutils.core and has been the regular interface already back in Ubuntu 14.04 (ROS indigo). People just copy&pasted it over generations even long after that because it did not start to trigger a warning/error until recently. If you try to compile a ROS stack with current branches back in 14.04 you will find more severe issues than this. That’s the motivation for the one-line patch.
A major problem with the current noetic build farm is that many packages are not released into noetic anymore even if python3-related patches have been merged for over a year, e.g., the case in ros/angles or ros/geometry. So fixing the issues in OR’s noetic distribution not only requires PRs, but also taking over release maintainership of various packages (if the maintainer is willing to pass it on) or paying the respective maintainers to run bloom.
Adding to @Mike_Purvis’s list above, here are some more released repositories that are broken that I am aware of:
Thank you very much @v4hn for the background of the distutils.core.
I agree with you that it might be not possible to get all the maintainers of all the affected packages to release and bloom a new version for this cause.
So maybe we should consider a way so rolling release distro users can fix this problem efficiently during their build process.
If we can assume that the one-line patch is safe then we could do something like:
find ./src -type f -name 'setup.py' -exec sed -i 's/from distutils.core import setup/from setuptools import setup/g' {} +
So maybe in the Build from Source How-To besides the additional information about the build switch for Python3 an additional box could be added, that addresses this issue and recommends the usage of the “–upstream” switch for the rosinstall_generator command so that a command like the one above could fix the import issue of all setup.py-files in the src folder before the build/install stage.
This command could also be alias’ed so after a potential update you can fix the issue quickly again.
Going this route would allow for (after the change of the How-To) immediate relieve for the users and we could avoid a situation where almost all packages got updates but some still need manual changes.
Of course I personally would like to see everything updated and the normal processes can be used for building from source, but maybe for this problem the effort/benefit trade-off favors a stable local-fix explanation over the hope that within reasonable time getting all these packages re-released with the fix.
I don’t think this is necessary. I use solely the setuptools line on Melodic + Py2.7 and no problems there. Just be sure to properly update the package.xml in the PRs, too, as per noetic/Migration - ROS Wiki .
You are right @peci1; this should be the only line needed if migrated correctly.
If memory serves me well, this was done to make sure that no matter what, this change will never break the import of setup.
And, of course, it would be desirable if all packages were correctly migrated in compliance with the migration guide.
Besides this point, what is your opinion on this topic, especially regarding the reservations @v4hn has, that it might not be possible or would take an unknown amount of time to get all packages released in a way that rolling release distro users like @Mike_Purvis can relatively easy build a ROS stack from source?
I made mentioned the approach with the amended How-To for immediate relief.
Do you think this might be a viable option, or should we strictly adhere to the releases?
Or a hybrid approach with a “hacky” way for the meantime and an effort to update everything?
Currently, to support source builds, it is not needed that maintainers craft a new binary release. It is enough that they merge the PRs. I think this is doable in 99% of cases in a few months. The remaining cases could be marked as abandoned packages and somebody could take over the maintainership (I think there already is a process for abandoned packages).
The remaining cases could be marked as abandoned packages and somebody could take over the maintainership (I think there already is a process for abandoned packages).
See OrphanedPackage - ROS Wiki and Maintaining and releasing orphaned packages . But the purpose of this initiative is/was to release abandoned functional into “new” ROS distributions, not to fix them when broken. Of course the boundaries are not always clear, but in many cases we only have access to existing gbp repositories and not upstream.
that never really became official policy (the REP PR was withdrawn).
The main concern with the REP was that it’s a lot of text, but dead meat already, especially in ROS one, and people just have to talk more and get engaged. The group very much exists and has the required permissions to act, though I don’t know how active anyone is. I’m certainly not doing much work on that front (and I don’t think there is a lot to do right now).
especially regarding the reservations @v4hn has, that it might not be possible or would take an unknown amount of time to get all packages released in a way that rolling release distro users like @Mike_Purvis can relatively easy build a ROS stack from source?
That was not my point at all! Please, please, pretty please go ahead and test/fork upstream where necessary and pull-request (if possible backward-compatible) patches there! In cases where upstream does not merge the patches within a short time window the purpose of the github ros-o organization is to collect such forks for the future. In some cases you will find that such forks already exist, though granted, not yet for most of the open setuptools-patches that were reported here.
There have been ideas for setting up a rosdistro-compatible yaml-register with ros-o overlays, but that is not implemented as of today.
we could do something like find ... -exec sed -i
Please don’t. There are various other patches needed already on Ubuntu 22.04 and Debian testing (running locally) and other current rolling release distributions. This kind of patch approach will not work but add more noise. It’s just as bad as RoboStack’s patch folder in that regard.
rosinstall_generator ... --upstream
to clarify: for the cases where maintainers merged the patches but were hesitant/forgot to run bloom this will just break with the same errors because it checks out the release tags.
You actually need --upstream-development instead.
Some users will definitely prefer to run with upstream’s releases and hand-pick unreleased patches, but getting everything to run from the latest commits is the best way forward especially for rolling-release in my opinion, also because many repositories do not commit to their ros one branches a lot anymore or have CI in place to prevent merging broken commits in the first place.
Before Noetic was released, many packages were switched to setuptools because distutils was split into a separate package in Debian Buster and Ubuntu Focal. If I recall correctly, the distutils import will only work on Focal if python3-distutils is installed. See this and linked PRs: Prefer setuptools with Python 3 by sloretz · Pull Request #1048 · ros/catkin · GitHub.
I think it’s safer to assume setuptools is available rather than distutils because Catkin only exports a dependency on setuptools. Adding the buildtool_depend on python-setuptools was only required if one was trying to support Melodic and Noetic on the same branch.
Thank you, @sloretz, for your feedback on this issue.
I talked to @v4hn, and we concluded that the cleanest way forward would be to fix the packages upstream via PR and hope for a quick merge & release.
As a maintainer of a few rqt-packages my plan was to start this weekend to create PRs (update setup.py and package.xml in accordance to Noetic Migration Guide) for all the packages we found so far (starting with the rqt-packages) and hope that the maintainers will react in a timely manner.
So the hope is that in the near future, more recent distributions can build Noetic from source again without manually editing/fixing packages.
Batch 2
fkie/multimaster_fkie
osrf/capabilities
ros-drivers/joystick_drivers
ros-drivers/rosserial
ros-geographic-info/unique_identifier
ros/resource_retriever
locusrobotics/catkin_virtualenv
ros-simulation/gazebo_ros_pkgs
ros-planning/navigation
tork-a/rqt_joint_trajectory_plot
ros-controls/ros_controllers
machinekoder/ros_pytest
ros-infrastructure/rosdoc_lite
PR2/pr2_common_actions (push to format version 3 and addition of buildtool_depend)
PR2/pr2_mechanism (distribution.yaml says source = kinetic-devel but latest release 1.8.21 = melodic-devel)
PR2/pr2_power_drivers
PR2/pr2_robot
PR2/rqt_pr2_dashboard
ros/roslint (push to format version 3 and addition of buildtool_depend)
ros/joint_state_publisher (push to format version 3 and addition of buildtool_depend)
To Be Created PRs (found myself a bit in a dependency hell):
Will be done soon.