build.ros2.org has been migrated to the new cookbooks and now it is build.ros.org’s turn.
Learning from the last migration there may be multiple days of downtime if data transfer throughput is the same as last time although I’m going to try and ensure that our new instances are brought up in the same availability zone as the current ones to reduce the transfer overhead.
During the downtime build.ros.org will be offline and pull request jobs will not be triggered. We’ll hold all ros/rosdistro release pull requests during the maintenance period so new releases won’t be merged during that time. I’ll send updates to status.ros.org during the maintenance.
1 Like
This would cause repositories.ros.org to be inaccessible correct? Any rough ETA for completion and repositories to be back online?
repositories.ros.org is offline but packages.ros.org is unaffected and is where most consumers should be retrieving packages.
The repositories host will probably be back online some time tomorrow once data transfer from the old host is complete.
The repositories host has been migrated to a new instance. There are some cosmetic differences between the auto-index of nginx and apache but the repository contents should match the last state of the previous repository host before it was taken offline.
1 Like
So I’m not sure if this is related - but I’m also not sure where else to post an issue with the build farm. It seems since this migration, our PRs are failing to build for the navigation stack - because they can no longer download resources from download.ros.org.
example of a recent failure: https://build.ros.org/job/Npr__navigation__ubuntu_focal_amd64/53/console
Any ideas of what might be the cause, or how to solve this?
-Fergs
Thanks for the report. I actually just got the same report internally from @sloretz and @jacob
When setting up the new cluster used a private DNS zone within AWS to route traffic on internal interfaces and that zone is interfering with public DNS for other ros.org domains. To mitigate the issue for now I’ve been adding matching records for required domains to the private zone but we’re going to look at switching to subdomain-specific private zones which take a bit more configuration duplication but remove the need to mirror the public DNS for other subdomains.
I’ve updated the private zone to include download.ros.org so subsequent builds should not have this issue.
1 Like
@nuclearsandwich thanks for the quick response - can confirm that our jobs are now passing again!
I’ve marked this maintenance window as over! Thanks everyone for the patience and feedback. Do continue to report issues in this thread or the ROS 2 one if you find one. I’m about to make a separate post about the pull request web hooks.