Documentation for build.ros2.org

Buildfarm
,
1

Dear ROS 2 build farmers,

I really like the infrastructure provided with the ROS 2 build farm, https://build.ros2.org, and do think that it’s a great service to the community. However, as a package maintainer, I often struggle when builds fail. While I learned my ways around the build farm mainly via observation and trial&error, I propose to create a documentation so that package maintainers don’t have to learn it on their own. I suggest to link this documentation prominently on https://build.ros2.org and within the Jenkins mails. Otherwise mails from the build farm may lead to frustration and may often just be ignored (my own experience :wink: ).

I suggest the documentation to comprise:

  1. When are jobs spawned?
    • Presumably when you bloom your packages? → link to bloom docs
  2. Which jobs are spawned?
    • What are their different purposes? - ‘My’ ROS 2 package(s), consisting of a main package, an interface package, and an example package, built for 4 ROS 2 distros, yields 63(!) build jobs on the build farm[1]. Finding the particular build job that helps with debugging is often hard, especially since their names are quite cryptic and their purpose is not immediately clear.
    • How do they differ from ros-tooling github actions? - While the dev jobs and PR jobs directly pull from the upstream repository, most of the other jobs seem to pull from the upstream branches of the release repository instead. This leads to situations where your local builds, your github actions, the dev and PR jobs on the build farm work, but some of the jobs on the build farm still complain and fail although seemingly doing the same.
  3. FAQ: What are commonly observed effects / common failures?
    • From my personal experience, dependencies not properly being declared is the reason >90% of the time.
    • This could benefit from explaining why this is not hitting you in your local builds/tests, github actions, and dev jobs.
  4. Troubleshooting
    • How to debug? Is blooming + waiting for merge to rosdistro + getting feedback from the bin jobs the best way?
    • In my experience, typical build failure mails read "apt-src build ros-… failed. This is usually because of an error building the package". However, the root cause, usually cmake missing a build dependency, is not visible in the snippet provided with the email. Instead, it can be found when navigating to the particular job and looking at the full log.

While I can’t provide the documentation on my own because I mainly rely on observation/speculation only, I am happy to get involved, e.g., by providing suggestions and feedback.

Cheers,
Arne

[1] https://build.ros2.org/search/?q=system_modes

4 Likes
2

Hi Arne,

We’re glad that you’re appreciating the build farm. There’s definitely a lot more improvements possible for the documentation especially in the discoverablility for people joining as new maintainers or users.

The main entry point for documentation is: buildfarm - ROS Wiki

Though that’s a little outdated with the old buildfarm_deployment techniques based on puppet. We’ve switched to using Chef via this cookbook:

But that’s mostly for people deploying.

For understanding the jobs themselves. They’re documented in the ros_buildfarm repository which generates them all and has a good amount of documentation here:

And they have a section specifically for running the jobs locally to be able to reproduce any issues encountered on the buildfarm.

There are two ROSCon talks I’d also recommend watching about the buildfarm.

2016:

Slides

2015:

Slides

We’ve worked hard to cover almost every case but I think that the documentation could benefit from your fresh viewpoint providing FAQs and suggestions about how to improve or extend the documentation especially with references to the existing documentation to help get other users to the documentation they need sooner.

For example there’s a link to the prerelease tests from the bloom tutorials that can help identify things like missing dependencies earlier per two of your points above but not prominantely enough that you’ve found them. Helping provide relevant links and suggestions where cross links or suggested FAQs would be appreciated.

3 Likes
3

Thanks a lot for the links, Tully! Most of the questions indeed seem to be answered already somewhere, so it might actually just be lacking discoverability and missing some glue code.

Do you think a section on https://docs.ros.org/ makes sense? If so, I could make a proposal via PR to roughly collect and cover the aspects I mentioned in my original post.
Or do you think buildfarm - ROS Wiki or build.ros.org - ROS Wiki or http://wiki.ros.org/build.ros2.org should be updated resp. set up instead?

1 Like
4

A submission to docs.ros.org via the ros2_documentation repo sounds like the right place to put a high level user summary and FAQ that can then give pointers out to the more detailed documentation elsewhere. A PR there would be greatly appreciated!

5

I gave it a shot, see : Adds documentation of the build farms by norro · Pull Request #1629 · ros2/ros2_documentation · GitHub