I’ve done some research and tested all the mirror sites mentioned on the ROS 1 wiki. Here are the results. For the purposes of this table, OK means focal/amd64 for ROS 1 and noble/amd64 for ROS 2. The test was performed from Central Europe.
China has pretty good support for both ROS 1 and ROS 2 mirrors.
North America has only 1 ROS 1 mirror, no ROS 2 mirrors
South America has nothing
Europe has only 1 ROS 1 mirror, no ROS 2 mirrors
The question is: is it intentional that ROS 2 mirrors are not mentioned in the docs? Is there already a satsfying CDN powering packages.ros.org ?
Regarding the non-working mirrors, @tfoote seems to be the only maintainer reachable here over discourse. Would it be good to update the mirrors page to better reflect current status? Should I do the edit?
It’s not intentional in the sense that mirrors are now un-encouraged or that the Infrastructure project would prefer them not to be made. But time has not been spent on updating mirror guidance or maintaining mirrorlists. I think it would be fine to create a repository mirroring page in the ROS 2 documentation.
packages.ros.org is distributed by OSUOSL’s FTP network which has three hosts, all in the US.
The TGC has discussed mirroring a number of times and I’ve taken those discussions back to the Infrastructure PMC.
The focus of these conversations was less about the apt / rpm repositories, which have years of standardized practice*, but about providing integrity checking mechanisms for mirrors of other project resources such as the rosdep database, rosdistro database, and ROS 2 documentation site.
On our backlog is to document how the Infrastructure project would recommend mirroring our own infrastructure. And using the gaps in that story to drive small features in that direction, such as providing integrity information for ROS 2 documentation that does not require building it from source.
If I can self-grade, the project does a fairly good job of making all the pieces avaiable to be mirrored offline. But what pieces are needed for which workflows and how to put all of it together is not widely understood and there are a limited few who could get it right in under 10 tries (I think it would take me at least that many).
* For example, apt-mirror can be used to create mirrors of the official repositories that preserve metadata signatures.
Hello, UMD mirror maintainer here. I haven’t been following the development of ROS so closely, so I didn’t know that ROS 1 and ROS 2 were separate from each other. (I assume that’s true of some other mirror maintainers too.)
We will start mirroring ROS 2 this week, if I have time to set it up.
slightly off-topic, but my IT admins prefer using rsync instead of apt-mirror. That’s also how the current mirror is/was created (ftp.tudelft.nl/ros).
I’ve been discussing setting up a ROS 2 mirror, and I believe the preference would again be rsync, but we haven’t yet looked at the difference between ROS 1 and ROS 2 repository setup (wrt metadata fi).
Looking at the ROS 1 mirror page it looks like we really have three separate concerns that should each get their own docs page under a common top level “mirrors” topic:
ROS 2 Mirrors and how to use them.
How to set up your own mirror.
How to mirror (and ideally translate) the ROS 2 documentation. This one is its own can of worms but it is something we really need to address soon as there are already Chinese translations of docs.ros.org that are clobbering our SEO.
The list you posted is great but it seems to be missing one key piece of information: the contact information for the maintainers. Would you happen to have that information?
I propose we do the following with respect to ROS 2:
Create a top level docs.ros.org “mirrors” section that contains:
A page that contains an empty list of mirrors and how to use them.
A page on how to start / add your mirror to the list.
A page on setting up your own mirror.
A page on documentation mirroring that basically says, “come talk to us.”
Circulate a call to the community to add / create their own regional mirrors.
Somewhat related, when I was calculating the Gazebo Community metrics it was obvious from the download stats that someone was silently mirroring the Gazebo binaries. I wonder if this would be a good way to track down some of the existing mirrors that aren’t listed.
It would be great to improve the mirror process. I spent a lot of time quite a while ago getting it setup, but the technology has moved on since then.
For example I think that we can now setup rsync end points for most of the hosted content, if not all, which will make configuring things much easier. As mentioned is preferred by many admins by @gavanderhoorn
Pretty much all of the mirror maintenance has been my individual time. And I would love to help support an effort to revitalize it. Unfortunately I won’t have time to lead this sort of effort. But with knowledge of the services and experience setting this up in the past I’m happy to provide a lot of help and guidance if a group wants to pick this up.
For reference I think that the general configuration can be setup in a couple of hours, and maybe there’s some iterations needed on a few things. But the challenge is to make sure that it’s an easy to use product and commit to keep monitoring and maintaining it.
A big part of this is that we need to understand the requirements and define the product. Is it a script, is it a container, is it a VM, or is it just good documentation that the IT Admins will follow? I think quite possibly the latter option is potentially best, especially if we can get input from them about what their standard and best practices are that we can make sure that we can best support those instead of creating something new.
More mirrors can be valuable, but it can be a double edged sword as more mirrors are more infrastructure and need more maintenance. And potentially more valuable than the actual code or configuration is community standards about how to list mirrors, how to get status of the mirrors and communicate that to other users, as well as how to contract administrators in case there’s an issue.
On the UMD mirror, we currently host copies of 22 projects, and we’ve been doing it for at least 15 years. (I think actually around 20 years, but I don’t have information going that far back.)
I think we follow pretty standard mirroring practices, though one of our early maintainers wrote some fancy custom scripts to actually do the work. Our code is open-source, here (though there is also a bunch of legacy junk we don’t use anymore).
To make a long story short, we support two types of sync: rsync and apt-mirror. Our preferred method is rsync, because it "just works", while apt-mirror often requires regular updates to the configuration as things change. (For example, you may notice that our “Ubuntu releases” list is currently outdated.)
Currently ROS 1 is the only project that is using apt-mirror, even Ubuntu is now using rsync instead.
So, regarding your points:
Is it a script? While a script might be good for people who are just getting started, anyone with an existing mirror probably already has ways that they do things, and a script won’t be helpful.
Is it a container? VM? Again, might be useful for beginners, but for anyone who is mirroring lots of other projects, there is a lot of extra overhead.
I just checked, and it seems the interesting parts can actually be reached over rsync:
---------------------------------------------------------------------------
R S Y N C . O S U O S L . O R G
Oregon State University
Open Source Lab
Unauthorized use is prohibited - violators will be prosecuted
---------------------------------------------------------------------------
...
ros The Robot Operating System (ROS)
ros-main The main ROS debian package repository
ros-testing The testing ROS debian packages repository
ros2-main The main ROS2 debian package repository
ros2-rhel-main The main ROS2 RHEL packages repository
ros2-rhel-testing The testing ROS2 RHEL packages repository
ros2-testing The testing ROS2 debian packages repository
ros_docs_mirror
ros_wiki_mirror
...
We’ve just started syncing ros2-main (already had ros-main).
, while