Migration to one GitHub repo for MoveIt!

@dbolkensteyn That’s a valid comment. Thank you for sharing your thoughts!
I’ll wait for “more-core” maintainers to chime in, but as @gavanderhoorn commented on answers, main reason why MoveIt! hasn’t been released into Kinetic is the dependency. MoveIt! is a huge set of software, so is the number of the depended packages.

Also we’re making fundamental changes that will require far less maintenance effort, one of which as discussed above is to consolidate the distributed repositories into fewer repo. Since ROS Kinetic is “LTS”, meaning once it’s released we won’t be able to make a substantial change for 5 years, we’re rather taking time now to make as much work done as possible. In fact some maintainers (notably @davetcoleman @v4hn @rhaschke) have been really working hard these few weeks for making it happen for Kinetic.

3 Likes

I was talking with @davetcoleman about the migration to the new single repository off-list, and I wanted to summarize some aspects of the migration we discussed.

Dave’s new moveit repository (https://github.com/ros-planning/moveit) was created by basically doing:

  • move the contents of the old repository into a sub-folder in a single commit
  • merged the branch from the old repository into the new repository

For each branch and repository being migrated into the new one. It’s a bit more complicated than that, see his migration script (https://github.com/davetcoleman/moveit_merge/blob/master/git_merge_moveit.sh) but that’s the bones of it.

This approach seems to be the most popular strategy on the Internet and it has the benefit that the commit hashes are preserved from the old repository to the new one. The downside is that since the files were moved in a commit it prevents GitHub from following the history of the file from before the move commit. Example:

There is another option, which is more complicated and invasive that avoids this issue. Basically you:

  • rewrite every commit in the old repository’s branch as-if the code was always in a subdirectory
  • merge the branch from the old repository into the new one

This has the advantage of not needing a “move commit” (so it has the same number of commits as the old repository), but has the disadvantage that all commits have different hashes in the new repository (because each commit was rewritten to have all content in a subfolder).

I did a quick test of this:

I don’t know what the moveit developers would prefer, but I offer my help if you’d like to pursue the rewrite history approach. Personally I don’t know which will work out better in the long run. It’s worth noting that the git log -- filename command has a --follow option that will follow moved files in most cases and GitHub might extend their history view to use that at some point.

The other item of discussion that I wanted to bring up was package versions and tags.

@davetcoleman you mentioned that you wanted to make the migration on Friday August 5th. Do you guys plan to do a catch up release of all the moveit packages in the new repository that day so that they have the same version number? Or will you wait until one of them needs to be released and release them all then?

With respect to tags, the options seem to be leave the old tags on the old repositories and only make new tags on the new repository or try to somehow migrate the tags, renaming them in the process to avoid collisions. I don’t know if there would be any benefit to migrating the tags, but I was curious to see what others think.

Do you guys plan to do a catch up release of all the moveit packages in the new repository that day so that they have the same version number? Or will you wait until one of them needs to be released and release them all then?

I assume we will wait until one of them needs to be released and then release all of them. @130s is the lead for MoveIt! releases, however, and I’m hoping he’ll add a section to the migration notes about his plan

Migrating tags: there are currently version numbering inconsistencies across various repos that I think would make it hard/impossible to migrate tags.

@wjwwood could you describe what the advantages might be of saving the commit hashes? Its not obvious to me

Another topic: moveit_resources is a package that currently only stores a copy of the PR2 URDF and meshes for testing. We have also been discussing copying in more robots for integration tests. Overall this package doesn’t do much.

However, analyzing the candidate merged repo today I realized this one package was causing 63% of the repo’s memory size (74 MB out of 118 MB of the moveit repo). For this reason I’d like to remove this package from the merge. In addition, this moveit_resources package can be a future location for images and other assets - for example, currently moveit’s logo lives in the moveit_setup_assistant for some reason.

What actually needs that package? I cannot figure that out. It is not in the wiki, nor in ROS index and rospack depends-on also gives nothing. Does this not appear, if it is required “only” for testing?

I think it makes sense to keep moveit_resources as its own separate repo. However this means that for some of the unit tests to fass it will require that moveit_resources be cloned into the same workspace.

What actually needs that package?

grep tells me it is used in _experimental, _planners, and _core - all for tests. moveit_core properly depends on it using <test_depend>, the other two packages do not (want to fix that?)

for some of the unit tests to pass it will require that moveit_resources be cloned into the same workspace.

Exactly. But I think having a smaller memory footprint in the main repo will keep it fast to push/pull… not sure if that is true. @scpeters did point out earlier in the thread that gazebo has suffered from file size issues.

I do recommend keeping as many resources out of the code repositories as possible.

Back in the days of svn it wasn’t too bad, the server needed to have lots of storage but a local checkout only saw the current version. However with git, every clone will get every version of the binary files into the future. A quick clone is very valuable for things like quick CI and saves developers time and disk space.

I’m in favor of smaller memory footprint. Dave, I can work on fixing that, sounds like it’s just a matter of listing out the moveit_resources package(s) under the <test_depend> tags?

Thanks but I already removed it from the migration notes and scritps after @tfoote’s reply, though I haven’t re-generated the overall moveit package yet.

I initially thought this might be a good use case for git LFS- https://git-lfs.github.com/

I’ve used git LFS to manage solidworks files in a mostly source code repo, and been mostly happy with it. In this case It’s probably not a good fit though, because:

  • the files are needed to run the tests; it would be annoying to have to pull down the LFS files to run tests of code.
  • I don’t think the buildfarm has LFS installed, so building debs wouldn’t work out of the box (I think?)

Sorry, it took so long for me to respond to this.

I don’t know. It would be nice to be able to find changes in the new repository using commit hashes from the old one, but honestly I don’t know that there is any advantage.

I personally likes to have huge repository with many packages, which makes easy maintenance and also prevent inactive repository from out of my view, however, if you have huge repository, there are potential draw backs on

1 Like

Also from @k-okada 's experiences it should also be kept in mind that if you have large number of packages in a single repository and you make a release it requires all packages in that repository to be released. And in doing so you end up triggering all downstream packages. So if a small leaf package is updated and wants to release, all dependencies on a core package will also need to be rebuilt. This may be especially an issue for MoveIt as many of the slowest building packages in the ROS ecosystem are dependencies of MoveIt. https://github.com/jsk-ros-pkg/jsk_common/issues/919#issue-74462146

@k-okada, thank you, these are all good points. Concerning slowing down the build farm - MoveIt! has a much more conservative release cycle and has averaged a release every 2 months since 2014. In a recent maintainer meeting we agreed we want to pace releases no more than once a month, but likely less often than monthly.

Our current travis build time for all MoveIt! packages including tests is 39 minutes. 33 minutes of that is for catkin build. We still have 22% of our travis build time left, and as you mentioned there are tricks we could use if necessary to extend that time. But maybe we’ll be asking for your help in a year :slight_smile:

As for the ros buildfarm limit I don’t think we are at risk of hitting it in the foreseeable future.

Final announcement: https://groups.google.com/forum/#!topic/moveit-users/i_R6AfRnmnc

+1 for combined git repo!

You’re a bit late to the show. The Epic Moveit Repo Merge has happened months ago. :slight_smile:

1 Like

#ThrowBackThursday:
Yes, you were right. After this 3rd reply, there were 35 more :slight_smile: