Big workspace management

Buildfarm
1

Hey, I have a question about how to manage the workspace of a large collection of packages (~50/100). My main question is how to setup this workspace. We have a large number of unreleased packages and a few forked versions of official packages.

Can I use rosinstall_generator to recursively check out the source repositories of my custom packages? I tried a forked rosdistro but apparently rosinstall_generator does not check out dependencies of unreleased packages. Is there an other way?

2

Hey Rayman,

In our “Shadow’s Build Tools” we put all source code dependencies for repository in the file repository.rosinstall. During the build system will pull all those repositories from Git into local workspace and if those repositories have repository.rosinstall it will do it for them as well for several levels. It is using wstool feature merge rosinstall files.

So you may reuse this approach or even give a try to the whole “Shadow’s Build Tools” solution which is really easy to setup.

3

Hi Rayman,

the information about the dependencies of a package comes from its
manifest. If your packages refer to names which are not part of the
rosdistro the tool can’t resolve the source repository for them. And for
repositories which are not released no packages and therefore no manifests
and recursive dependencies are known.

I would suggest to put all your custom repos (including the forks) in a
.rosinstall file (optionally with the distributed approach Andriy
mentioned) and fill the dependencies using rosdep from binary packages.

4

Thanks for the quick response. I’ll give the distributed solution of @Andriy a try.

Just a quick question, the manifests rosinstall_generator needs are cached in the in the <distro>-cache.yaml.gz file generated by rosdistro_build_cache? I’m trying to figure out how the whole system works.

5

Correct, a Jenkins job regenerates the cache every 5 minutes. It makes all
manifests of released packages available without having to fetch them from
each release repo. This is the data the rosinstall_generator as well as
the build farm mainly operate on.

The cache only contains the manifests of released packages since only for
them it is clear when they have changed. Basically when the version number
of the release changed. For source repositories it would require to
constantly check the repository content since it can change without any
change to the rosdistro files.

6

Hi Rayman,

Presently rosinstall_generator only works for released packages, but there are pending PRs on rosdistro which enable it to work with source repos as well, please check it out if interested: https://github.com/ros-infrastructure/rosdistro/pull/84

Clearpath is primarily deploying our ROS software now using rosinstall_generator-based bundles; you can see a small prototype of this idea here if it’s of interest: https://github.com/mikepurvis/ros-bundling

7

Nice, that patch would really help my use case! I could probably do the following

  • define a custom ros distro with our source repositories
  • use rosinstall_generator to resolve dependencies and use wstool to check out all these source repos
  • finally use rosdep to resolve the standard ros depdendencies

I hope I can find time next week to try out your patch.