ROS Resources: Documentation | Support | Discussion Forum | Service Status | Q&A answers.ros.org

Migration to one GitHub repo for MoveIt!

MoveIt
#13

Good points on Git vs. Perforce. I did mean MoveIt!'s release of Kinetic. By some of the history, I was referring to old commits that have large file size or unused branches.

#14

Thanks @davetcoleman for taking the time to have this discussion - I agree that a lot of good points have been brought up, and I like your summary in that last comment.

CATKIN_IGNORE helps avoid the need for big dependencies when only building a few packages. I was thinking more of the problem of doing “rosdep install --from-path src” but that can also be worked around by installing deps only for the packages/subdirectories that you want to build. To be clear, I think that these issues are far outweighed by the advantages that have been listed of moving to one repo. Especially if tutorials are added/updated to show people useful workflows.

#15

Ok, that’s good, so long as you’re removing unused branches and stuff. Removing actual history would not be good. That brings up another thing to watch out for, which is that it would be best to try to preserve the history when merging the repositories. It’s easy to accidentally miss the history when moving the files.

1 Like
#17

I instantly favored this idea when I heard it in a ROSCon 2015 video where Dave mentioned it. I’m still +1 now.
As I recently started helping to make releases of MoveIt!, I add some thoughts to the maintainer’s perspective. Not all / none of these may be of significant importance, but hoping these help for some purpose:

Advantages is to reduce the cost of:

  • back/forwardporting between branches; Doing so involves at least: paying attention to any existing portable commits/PRs, cherry-picking commits, opening PRs, waiting for CI result and adding correction if needed, waiting for the reviews, discussion if needed. Spanning over multiple repositories just multiplies the cost. Ideally the original authors of PRs take care of porting, but I doubt we can force that and thus it’s one of maintainers’ responsibility currently (at least at MoveIt!).

  • managing version;

    But I believe most of MoveIt! already has the same version numbers, because the stuff I’m suggesting we merge is so dependent on each other

    Actually there are slight differences. For example, to cope with the difference between Indigo and Jade, I incremented the minor version of Jade to 0.8.x. Version ain’t nothing but a number? No it takes some thoughts, and the more repositories we have the more time needs to be spent among maintainers.

  • sorting changelog; I tend to spend some time to edit changelog upon making a release to group into categories (fix, feature etc.) since bulk of raw commit messages don’t make much sense to me personally. This gets more important I think for packages with vast amount of users. Again the more the repos the more the time it takes, and moreover more mental stress.

  • running prerelease test manually; first going to web and get the necessary script snippet, and run the snippet locally. Again the mroe the repos the more the time, which is non-trivial amount for volunteer maintainers.

    • BTW there’s a package called industrial_ci that has a feature to start the ROS prerelease test from PR (usage).
#18

This week’s example: currently @v4hn is adding a much needed feature to MoveIt! but he has to create 4 PR’s to apply it throughout the code base. And if you include adding documentation, 5! This means I get tons of emails for all these PRs, have to merge them in the correct order, and its very difficult to make sure they all work together correctly.




1 Like
#19

I’ve created an automated merge script that keeps each repo’s git history in place. I’ve tested this merge script with Jade and it passes CI except for some previously disabled tests that are now enabled - you can see the experimental merged repo here.

Thanks to @130s and @v4hn for their input on this.

1 Like
#20

Sorry if I say anything obviously stupid here… I’m quite new to ROS and MoveIt.

From my end-user point of view, ROS Kinetic is the recommended version since its release on May 23rd 2016 and compatible with Ubuntu’s latest 16.04 LTS. MoveIt is an essential part of the ROS ecosystem, and should therefore become quickly available for new ROS releases.

So, I don’t think that this change should block an initial MoveIt release for Kinetic. Even if the initial one is done from the multiple repos as before, it should not prevent future MoveIt bug fix releases for Kinetic to be done from the unified repo.

But maybe I’m totally wrong and this subject isn’t what’s currently blocking the release.

1 Like
#21

@dbolkensteyn That’s a valid comment. Thank you for sharing your thoughts!
I’ll wait for “more-core” maintainers to chime in, but as @gavanderhoorn commented on answers, main reason why MoveIt! hasn’t been released into Kinetic is the dependency. MoveIt! is a huge set of software, so is the number of the depended packages.

Also we’re making fundamental changes that will require far less maintenance effort, one of which as discussed above is to consolidate the distributed repositories into fewer repo. Since ROS Kinetic is “LTS”, meaning once it’s released we won’t be able to make a substantial change for 5 years, we’re rather taking time now to make as much work done as possible. In fact some maintainers (notably @davetcoleman @v4hn @rhaschke) have been really working hard these few weeks for making it happen for Kinetic.

3 Likes
Discourse - A very powerful mailinglist
Discourse - A very powerful mailinglist
#22

I was talking with @davetcoleman about the migration to the new single repository off-list, and I wanted to summarize some aspects of the migration we discussed.

Dave’s new moveit repository (https://github.com/ros-planning/moveit) was created by basically doing:

  • move the contents of the old repository into a sub-folder in a single commit
  • merged the branch from the old repository into the new repository

For each branch and repository being migrated into the new one. It’s a bit more complicated than that, see his migration script (https://github.com/davetcoleman/moveit_merge/blob/master/git_merge_moveit.sh) but that’s the bones of it.

This approach seems to be the most popular strategy on the Internet and it has the benefit that the commit hashes are preserved from the old repository to the new one. The downside is that since the files were moved in a commit it prevents GitHub from following the history of the file from before the move commit. Example:

There is another option, which is more complicated and invasive that avoids this issue. Basically you:

  • rewrite every commit in the old repository’s branch as-if the code was always in a subdirectory
  • merge the branch from the old repository into the new one

This has the advantage of not needing a “move commit” (so it has the same number of commits as the old repository), but has the disadvantage that all commits have different hashes in the new repository (because each commit was rewritten to have all content in a subfolder).

I did a quick test of this:

I don’t know what the moveit developers would prefer, but I offer my help if you’d like to pursue the rewrite history approach. Personally I don’t know which will work out better in the long run. It’s worth noting that the git log -- filename command has a --follow option that will follow moved files in most cases and GitHub might extend their history view to use that at some point.

The other item of discussion that I wanted to bring up was package versions and tags.

@davetcoleman you mentioned that you wanted to make the migration on Friday August 5th. Do you guys plan to do a catch up release of all the moveit packages in the new repository that day so that they have the same version number? Or will you wait until one of them needs to be released and release them all then?

With respect to tags, the options seem to be leave the old tags on the old repositories and only make new tags on the new repository or try to somehow migrate the tags, renaming them in the process to avoid collisions. I don’t know if there would be any benefit to migrating the tags, but I was curious to see what others think.

#23

Do you guys plan to do a catch up release of all the moveit packages in the new repository that day so that they have the same version number? Or will you wait until one of them needs to be released and release them all then?

I assume we will wait until one of them needs to be released and then release all of them. @130s is the lead for MoveIt! releases, however, and I’m hoping he’ll add a section to the migration notes about his plan

Migrating tags: there are currently version numbering inconsistencies across various repos that I think would make it hard/impossible to migrate tags.

#24

@wjwwood could you describe what the advantages might be of saving the commit hashes? Its not obvious to me

#25

Another topic: moveit_resources is a package that currently only stores a copy of the PR2 URDF and meshes for testing. We have also been discussing copying in more robots for integration tests. Overall this package doesn’t do much.

However, analyzing the candidate merged repo today I realized this one package was causing 63% of the repo’s memory size (74 MB out of 118 MB of the moveit repo). For this reason I’d like to remove this package from the merge. In addition, this moveit_resources package can be a future location for images and other assets - for example, currently moveit’s logo lives in the moveit_setup_assistant for some reason.

#26

What actually needs that package? I cannot figure that out. It is not in the wiki, nor in ROS index and rospack depends-on also gives nothing. Does this not appear, if it is required “only” for testing?

#27

I think it makes sense to keep moveit_resources as its own separate repo. However this means that for some of the unit tests to fass it will require that moveit_resources be cloned into the same workspace.

#28

What actually needs that package?

grep tells me it is used in _experimental, _planners, and _core - all for tests. moveit_core properly depends on it using <test_depend>, the other two packages do not (want to fix that?)

for some of the unit tests to pass it will require that moveit_resources be cloned into the same workspace.

Exactly. But I think having a smaller memory footprint in the main repo will keep it fast to push/pull… not sure if that is true. @scpeters did point out earlier in the thread that gazebo has suffered from file size issues.

#29

I do recommend keeping as many resources out of the code repositories as possible.

Back in the days of svn it wasn’t too bad, the server needed to have lots of storage but a local checkout only saw the current version. However with git, every clone will get every version of the binary files into the future. A quick clone is very valuable for things like quick CI and saves developers time and disk space.

#30

I’m in favor of smaller memory footprint. Dave, I can work on fixing that, sounds like it’s just a matter of listing out the moveit_resources package(s) under the <test_depend> tags?

#31

Thanks but I already removed it from the migration notes and scritps after @tfoote’s reply, though I haven’t re-generated the overall moveit package yet.

#32

I initially thought this might be a good use case for git LFS- https://git-lfs.github.com/

I’ve used git LFS to manage solidworks files in a mostly source code repo, and been mostly happy with it. In this case It’s probably not a good fit though, because:

  • the files are needed to run the tests; it would be annoying to have to pull down the LFS files to run tests of code.
  • I don’t think the buildfarm has LFS installed, so building debs wouldn’t work out of the box (I think?)
#33

Sorry, it took so long for me to respond to this.

I don’t know. It would be nice to be able to find changes in the new repository using commit hashes from the old one, but honestly I don’t know that there is any advantage.