I shift the discussion from github issue here to discourse, as we have to discuss something longer and more fundamental about strategy and support, as about experiences from ROS1 packaging. Currently, ROS2 gets build entirely from source, ROS1 uses my personal Server and this larger python script to maintain, test & update the packages. Since I’d overtaken the packages, we had a lot of issues, but now finally we’ re getting to a more flawless experience. Here I want to sum up ideas and options we have, and add my experiences from packaging melodic.
One upstream or distro related packages?
First decision is if we should switch to one upstream or create distro-specific packages as we do now. On the one hand the first will decrease the effort enormously, on the other hand it would be problematic for a lot of people expecting their package updates to be reliable. And telling them to do git reset for every package if the want the previous, probably LTS-distro is IMHO not nice
How do we want to test and build packages?
At the moment I use my private server with all the obstacles it brings along. I can’t give the build containers that much cores as need to be fast, so everything is slow as hell. Additionally, my python script is not very nice, even though I plan to improve it in various ways, e. g. creating a web UI to view build output, so that it is working a bit better for everyone. But for ROS2 I wouldn’t reinvent the wheel, so maybe choose some existing CI system. But every free CI is slower as my own server, even the new github CI has timeouts often and that doesn’t work for >400 packages after a boost update. Options would afaik be drone, jenkins & Gitlab CI, I only use the latest one, no experience with the others. But then we still need to find someone to sponsor a powerful server with at least 6 Core and 16GB RAM.
How do we organize the package structure?
Currently, we have an github organization containing every package, and then CI syncs successfully built packages to AUR. But this organization was a bad idea, probably one Repo with all would fit better, IMHO best solution would be a Repo containing submodules. But then we have the problem of testing packages before uploading to AUR again. And no, aurpublish is a nice to, but not solving this problem. We have to build packages before syncing them, otherwise we would need to increase the pkgrel to insanity and we would have conflicts with the AUR hooks all the time. And people would try to install/build borked updates.
So I believe this is the point where we disagree the most slash I feel more strongly about this one, then the others. Arch’s philosophy and its release method has always been package at latest release tag. In lieu of this I think we should have a set of packages called ros-<package_name> that tracks the latest releases of all ros packages. As we know, package for Melodic, requires us to patch issues and submit PRs upstream and that is to get the latest release working not even an older version. If we want to maintain ros-<lts_distro>-<package_name> as you mentioned it increases the effort of backporting and upstream PRs with newer dependencies. However, this does not mean that I don’t think we should have a ROS LTS version. The Arch community is free to do what it wants and I am sure users will prefer the LTS version for API and ABI compatibility. I think for now, since we do not have a ROS 2 pipeline in place let us concentrate our efforts on packaging the LTS release and afterward, if people feel it (maybe its just me that is thinking like this) then they can create/maintain the ROS latest release version. As a maintainer I would just hate to spend efforts on backporting changes when they already exist in the latest release. I.e. I like the Arch-centric approach over the user-centric approach, but I will help out until any ROS2 pipeline for Arch is stabilized and in working order.
Ideally having a sponsored server is definitely the best way to go about this, I think the best way to incorporate CI with our builds is for the ROS packages to be taken up by a Trusted User (official maintainer) of Arch Linux. Maybe, OSRF can donate money to the Arch Linux group for providing a build server so that Arch can official maintain packages. Not really sure how the logistics of that will work out though. Besides that, we use our own CI and maybe OSRF can directly sponsor hardware for you, @bionade24.
As for organization, the submodule idea seems like it would work well, but ultimately it will boil down to separate repos tho. I kinda like aurpublish, (I use it for my own PKGBUILDs), but everything is in one repo and maybe the PRs and Issues can get cluttered (although if we label them accordingly it should be easier to manage than having a separate repo for each package). I think it boils down to do we want PRs and issues to be submitted to individual repos or to just one big repo (which makes it easier to track projects across packages). I kinda want to give aurpublish a try, I think it might be better than what we have right now. As for the building before syncing issue, that does not depend on the package structure. No matter how we organize it, we can find a way to incorporate CI into the pipeline. In fact, that is why you made that a separate point (2). It does not belong in point 3.
As for pkgrel, we should be increasing those anytime a rebuild is required anyway, its just that right now we do not have the manpower to do so and we haven’t figured out a way to automatically detect when a rebuild is required.
I don’t think so. Arch always has been packaging multiple java and python versions, as well as various software with incompatible release, now we even have 2 boost versions in the repos. And ros is designed to have multiple version. And the more important arch packaging guideline is to change the upstream as less as possible, so this would be imho the right way. I don’t believe that this would make that much more effort, we don’t have to really care about old releases if no one complains.
OSRF won’t do that, for sure. Does anyone know some hosters who do this very often?
As a community I think we should focus on the Rolling distribution for Arch Linux, instead of versioned specific distributions, although people are free to maintain those on their own as they please. For further conversations on rolling for arch please open issues at the respective organization.