Clone depth for PR devel jobs

automatom · August 17, 2016, 5:01pm

All,

I’m having some trouble with getting PR jobs to run correctly. I’ve tried making a simple PR to illustrate my problem. First, here’s the output of our build server:

09:45:05 GitHub pull request #17 of commit e22a0f7053c41cc2b791ff63b7ccd11c356b34f6, no merge conflicts.
09:45:05 Setting status of e22a0f7053c41cc2b791ff63b7ccd11c356b34f6 to PENDING with url <redacted>/job/Ipr__robot_localization__ubuntu_trusty_amd64/2/ and message: 'Build started sha1 is merged.'
09:45:05 Using context: Ipr__robot_localization__ubuntu_trusty_amd64
09:45:05 Building remotely on <redacted> (swarm buildslave) in workspace /home/jenkins-slave/workspace/Ipr__robot_localization__ubuntu_trusty_amd64
09:45:05 Cloning the remote Git repository
09:45:05 Using shallow clone
09:45:05 Cloning repository https://github.com/locusrobotics/robot_localization.git
09:45:05  > git init /home/jenkins-slave/workspace/Ipr__robot_localization__ubuntu_trusty_amd64/catkin_workspace/src/robot_localization # timeout=10
09:45:06 Fetching upstream changes from https://github.com/locusrobotics/robot_localization.git
09:45:06  > git --version # timeout=10
09:45:06  > git -c core.askpass=true fetch --tags --progress https://github.com/locusrobotics/robot_localization.git +refs/heads/*:refs/remotes/origin/* --depth=1
09:45:10  > git config remote.origin.url https://github.com/locusrobotics/robot_localization.git # timeout=10
09:45:10  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
09:45:10  > git config remote.origin.url https://github.com/locusrobotics/robot_localization.git # timeout=10
09:45:10 Fetching upstream changes from https://github.com/locusrobotics/robot_localization.git
09:45:10  > git -c core.askpass=true fetch --tags --progress https://github.com/locusrobotics/robot_localization.git +refs/pull/*:refs/remotes/origin/pr/* --depth=1
09:45:12  > git rev-parse refs/remotes/origin/pr/17/merge^{commit} # timeout=10
09:45:12  > git rev-parse refs/remotes/origin/origin/pr/17/merge^{commit} # timeout=10
09:45:12 Merging Revision b6c7701d4057194a48d6f69f153e81de3178fb3e (refs/remotes/origin/pr/17/merge) to origin/indigo-devel, UserMergeOptions{mergeRemote='origin', mergeTarget='indigo-devel', mergeStrategy='default', fastForwardMode='--ff'}
09:45:12  > git rev-parse origin/indigo-devel^{commit} # timeout=10
09:45:12  > git config core.sparsecheckout # timeout=10
09:45:12  > git checkout -f origin/indigo-devel
09:45:13  > git merge --ff b6c7701d4057194a48d6f69f153e81de3178fb3e # timeout=10    ### THIS STEP FAILS! ###
09:45:13  > git config core.sparsecheckout # timeout=10
09:45:13  > git checkout -f b6c7701d4057194a48d6f69f153e81de3178fb3e
09:45:13 FATAL: null
09:45:13 java.lang.NullPointerException
09:45:13 	at hudson.plugins.git.extensions.impl.PreBuildMerge.decorateRevisionToBuild(PreBuildMerge.java:88)
09:45:13 	at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1009)
09:45:13 	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1098)
09:45:13 	at hudson.scm.SCM.checkout(SCM.java:485)
09:45:13 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
09:45:13 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
09:45:13 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
09:45:13 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
09:45:13 	at hudson.model.Run.execute(Run.java:1741)
09:45:13 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
09:45:13 	at hudson.model.ResourceController.execute(ResourceController.java:98)
09:45:13 	at hudson.model.Executor.run(Executor.java:410)
09:45:13 [WARNINGS] Skipping publisher since build result is FAILURE

The reason it fails is because the merge creates an add/add conflict, even though no conflict exists. Distilling the steps above, you can test the results with these commands:

git init robot_localization
cd robot_localization
git config remote.origin.url https://github.com/locusrobotics/robot_localization.git
git -c core.askpass=true fetch --tags --progress https://github.com/locusrobotics/robot_localization.git +refs/heads/*:refs/remotes/origin/* --depth=1
git -c core.askpass=true fetch --tags --progress https://github.com/locusrobotics/robot_localization.git +refs/pull/*:refs/remotes/origin/pr/* --depth=1
git rev-parse refs/remotes/origin/pr/17/merge^{commit}
git rev-parse refs/remotes/origin/origin/pr/17/merge^{commit}
git rev-parse origin/indigo-devel^{commit}
git config core.sparsecheckout
git checkout -f origin/indigo-devel
git merge --ff `git rev-parse refs/remotes/origin/pr/17/merge^{commit}`

To figure out what was going on, I went and looked at the output of some PR jobs on the ROS build server. Here’s the relevant output for gazebo_ros_pkgs:

03:07:09 GitHub pull request #480 of commit 4e796b7857d9f4e3af025c2db2f9ec6864de8db3, no merge conflicts.
03:07:10 Setting status of 4e796b7857d9f4e3af025c2db2f9ec6864de8db3 to PENDING with url http://build.ros.org/job/Ipr__gazebo_ros_pkgs__ubuntu_trusty_amd64/35/ and message: 'Build started sha1 is merged.'
03:07:10 Using context: Ipr__gazebo_ros_pkgs__ubuntu_trusty_amd64
03:07:10 Building remotely on ip-172-31-6-90.us-west-1.compute.internal (swarm buildslave) in workspace /home/jenkins-slave/workspace/Ipr__gazebo_ros_pkgs__ubuntu_trusty_amd64
03:07:10 Cloning the remote Git repository
03:07:10 Using shallow clone
03:07:10 Cloning repository https://github.com/ros-simulation/gazebo_ros_pkgs.git
03:07:10  > git init /home/jenkins-slave/workspace/Ipr__gazebo_ros_pkgs__ubuntu_trusty_amd64/catkin_workspace/src/gazebo_ros_pkgs # timeout=10
03:07:10 Fetching upstream changes from https://github.com/ros-simulation/gazebo_ros_pkgs.git
03:07:10  > git --version # timeout=10
03:07:10  > git -c core.askpass=true fetch --tags --progress https://github.com/ros-simulation/gazebo_ros_pkgs.git +refs/heads/*:refs/remotes/origin/* --depth=1
03:07:12  > git config remote.origin.url https://github.com/ros-simulation/gazebo_ros_pkgs.git # timeout=10
03:07:12  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
03:07:12  > git config remote.origin.url https://github.com/ros-simulation/gazebo_ros_pkgs.git # timeout=10
03:07:12 Fetching upstream changes from https://github.com/ros-simulation/gazebo_ros_pkgs.git
03:07:12  > git -c core.askpass=true fetch --tags --progress https://github.com/ros-simulation/gazebo_ros_pkgs.git +refs/pull/*:refs/remotes/origin/pr/*
03:07:14  > git rev-parse refs/remotes/origin/pr/480/merge^{commit} # timeout=10
03:07:14  > git rev-parse refs/remotes/origin/origin/pr/480/merge^{commit} # timeout=10
03:07:14 Merging Revision d9136870d0ce8dc1b26556b8db9f15089630dbdd (refs/remotes/origin/pr/480/merge) to origin/indigo-devel, UserMergeOptions{mergeRemote='origin', mergeTarget='indigo-devel', mergeStrategy='default', fastForwardMode='--ff'}
03:07:14  > git rev-parse origin/indigo-devel^{commit} # timeout=10
03:07:14  > git config core.sparsecheckout # timeout=10
03:07:14  > git checkout -f origin/indigo-devel
03:07:14  > git merge --ff d9136870d0ce8dc1b26556b8db9f15089630dbdd # timeout=10
03:07:15  > git rev-parse HEAD^{commit} # timeout=10
03:07:15 Seen branch in repository origin/camera_info_dynamic_update
03:07:15 Seen branch in repository origin/gazebo7_warnings
...

The only difference between the two is this line:

09:30:45 > git -c core.askpass=true fetch --tags --progress https://github.com/locusrobotics/robot_localization.git +refs/pull/*:refs/remotes/origin/pr/* --depth=1

03:07:12 > git -c core.askpass=true fetch --tags --progress https://github.com/ros-simulation/gazebo_ros_pkgs.git +refs/pull/*:refs/remotes/origin/pr/*

While both have --depth=1 for the initial fetch (refs/heads), but only I have that flag set for fetching the PRs.

So, armed with this information, I tested it:

git init robot_localization
cd robot_localization
git config remote.origin.url https://github.com/locusrobotics/robot_localization.git
git -c core.askpass=true fetch --tags --progress https://github.com/locusrobotics/robot_localization.git +refs/heads/*:refs/remotes/origin/* --depth=1
git -c core.askpass=true fetch --tags --progress https://github.com/locusrobotics/robot_localization.git +refs/pull/*:refs/remotes/origin/pr/*
git rev-parse refs/remotes/origin/pr/17/merge^{commit}
git rev-parse refs/remotes/origin/origin/pr/17/merge^{commit}
git rev-parse origin/indigo-devel^{commit}
git config core.sparsecheckout
git checkout -f origin/indigo-devel
git merge --ff `git rev-parse refs/remotes/origin/pr/17/merge^{commit}`

Sure enough, this works just fine.

So the question is why the PR job is using the --depth switch at all. Looking at the job config, I see this:

It’s not clear if this is the source of the issue I’m seeing. The relevant build templates are here:

https://github.com/ros-infrastructure/ros_buildfarm/blob/master/ros_buildfarm/templates/devel/devel_job.xml.em#L60
https://github.com/ros-infrastructure/ros_buildfarm/blob/master/ros_buildfarm/templates/snippet/scm_git.xml.em

So my question is this: why am I seeing that flag? Is Jenkins parsing the configuration differently?

dirk-thomas · August 17, 2016, 5:35pm

I looked into the configure page of the referenced job (Ipr__gazebo_ros_pkgs__ubuntu_trusty_amd64). It only shows the “Shallow clone” option which is enabled. But it doesn’t show the “Shallow clone depth” option from your screenshot.

The snippet where that configuration is being specified is here (a bit below the snippet you posted).

That indicates that you are using different versions of the Jenkins plugins. build.ros.org currently uses version 2.4.1 of the Git plugin (see the version nummber in the snippet). The shallow clone depth option was added in version 2.4.3.

Usually the plugins and their configs are forward compatible. Maybe something needs to be changed to work with the newer version?

automatom · August 17, 2016, 7:27pm

Perhaps. The code that manages the generation of the depth switch is here. Interestingly, if I manually set the depth to 2:

git init robot_localization
cd robot_localization
git config remote.origin.url https://github.com/locusrobotics/robot_localization.git
git -c core.askpass=true fetch --tags --progress https://github.com/locusrobotics/robot_localization.git +refs/heads/*:refs/remotes/origin/* --depth=2
git -c core.askpass=true fetch --tags --progress https://github.com/locusrobotics/robot_localization.git +refs/pull/*:refs/remotes/origin/pr/* --depth=2
git rev-parse refs/remotes/origin/pr/17/merge^{commit}
git rev-parse refs/remotes/origin/origin/pr/17/merge^{commit}
git rev-parse origin/indigo-devel^{commit}
git config core.sparsecheckout
git checkout -f origin/indigo-devel
git merge --ff `git rev-parse refs/remotes/origin/pr/17/merge^{commit}`

…everything works just fine. I’m not familiar with how the templates map to the Java code, though there appears to be a direct correspondence between the XML tags and the class structure, so perhaps if we add

<depth>2</depth>

to the snippet? Not sure if 2 will always be enough, but then if not, it seems like we’d have to choose an arbitrary depth, in which case we start to lose the benefit of shallow clones/fetches.

dirk-thomas · August 17, 2016, 7:46pm

After you have manually configured the job you can go back to the configure form and change the url ending in /configure into /config.xml. That will show the raw job configuration. Please post the relevant block of the xml here.

tfoote · August 17, 2016, 8:10pm

I believe that the plugin is attempting to merge by default before running the build. If the pull-request branch is based off of an older commit than the shallow clone, presumably it cannot do the merge without the appropriate history available. Is the branch that you’re trying to test fast forwardable?

automatom · August 18, 2016, 7:49am

So to create the PR, I just made a branch of indigo-devel on GH, added a single trivial line in a constructor, committed/pushed, and submitted a PR. I can merge it using --ff if I try to merge the branches myself, and it also appears to work (using fast-forward) when I change the --depth switch for the line that fetches PRs in the Jenkins job. I agree that it’s behaving as if the PR branch was generated from something predating the current state of indigo-devel, but it wasn’t.

automatom · August 18, 2016, 7:49am

<hudson.plugins.git.extensions.impl.CloneOption>
  <shallow>true</shallow>
  <noTags>false</noTags>
  <reference/>
  <depth>2</depth>
  <honorRefspec>false</honorRefspec>
</hudson.plugins.git.extensions.impl.CloneOption>

dirk-thomas · August 18, 2016, 3:38pm

I guess it comes down to choose between “merge before build” and “shallow clone”:

If we use a shallow clone a merge will potentially fail - simply because there is no way to choose the “right” depth which always works.
So if we want to keep merging before build (which I think is very valuable, that feature was added in March) we have to disable shallow clones. This would only be done for PR jobs since those are the only ones doing “merge before build”.

automatom · August 18, 2016, 4:58pm

Sounds like a good approach.

dirk-thomas · August 18, 2016, 5:28pm

I created a pull request to change the behavior of the PR jobs as discussed:

https://github.com/ros-infrastructure/ros_buildfarm/pull/330

octaviantuchila14 · February 27, 2017, 6:44pm

Hi!
Where are all these log statements from?

I can’t find anything similar if I look on the Jenkins log page.

dirk-thomas · February 27, 2017, 7:02pm

If you refer to the messages about the git commands those are part of the console output of each job (at the very beginning).

octaviantuchila14 · February 27, 2017, 8:40pm

Hi,

Thank you for answering!

Yes, I mean logs like this one:

09:45:05 GitHub pull request #17 of commit e22a0f7053c41cc2b791ff63b7ccd11c356b34f6, no merge conflicts.
09:45:05 Setting status of e22a0f7053c41cc2b791ff63b7ccd11c356b34f6 to PENDING with url <redacted>/job/Ipr__robot_localization__ubuntu_trusty_amd64/2/ and message: 'Build started sha1 is merged.'
09:45:05 Using context: Ipr__robot_localization__ubuntu_trusty_amd64
09:45:05 Building remotely on <redacted> (swarm buildslave) in workspace /home/jenkins-slave/workspace/Ipr__robot_localization__ubuntu_trusty_amd64
09:45:05 Cloning the remote Git repository
09:45:05 Using shallow clone
09:45:05 Cloning repository https://github.com/locusrobotics/robot_localization.git
09:45:05  > git init /home/jenkins-slave/workspace/Ipr__robot_localization__ubuntu_trusty_amd64/catkin_workspace/src/robot_localization # timeout=10
09:45:06 Fetching upstream changes from https://github.com/locusrobotics/robot_localization.git
09:45:06  > git --version # timeout=10

When I look into the log from my Jenkins job, I can’t find this much detail(and I am using git for PR). The only thing in the logs was Git repository Checking out Revision

I’m using the Jenkins Git plugin and the Jenkins Bitbucket plugin.

Did you look into some other logs or that you enabled some option to have more verbose logging?

dirk-thomas · February 27, 2017, 9:05pm

For example this job has the following output:

Cloning the remote Git repository
Cloning repository https://github.com/ros/catkin.git
 > git init /home/jenkins-slave/workspace/Lpr__catkin__ubuntu_xenial_amd64/catkin_workspace/src/catkin # timeout=10
Fetching upstream changes from https://github.com/ros/catkin.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/ros/catkin.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/ros/catkin.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/ros/catkin.git # timeout=10
Fetching upstream changes from https://github.com/ros/catkin.git
 > git fetch --tags --progress https://github.com/ros/catkin.git +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse refs/remotes/origin/pr/857/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/857/merge^{commit} # timeout=10
Merging Revision 3eb35f296a2a6b37464f9a125525c2255e9545ca (refs/remotes/origin/pr/857/merge) to origin/kinetic-devel, UserMergeOptions{mergeRemote='origin', mergeTarget='kinetic-devel', mergeStrategy='default', fastForwardMode='--ff'}
 > git rev-parse origin/kinetic-devel^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f origin/kinetic-devel
 > git merge --ff 3eb35f296a2a6b37464f9a125525c2255e9545ca # timeout=10
 > git rev-parse HEAD^{commit} # timeout=10
Seen branch in repository origin/alt_tmp_setup_location
  <many more>
Seen branch in repository origin/wjwwood-patch-1
Seen 629 remote branches
Checking out Revision 3eb35f296a2a6b37464f9a125525c2255e9545ca (origin/pr/857/merge, origin/kinetic-devel)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3eb35f296a2a6b37464f9a125525c2255e9545ca

You can find the template for the job configuration here: https://github.com/ros-infrastructure/ros_buildfarm/blob/master/ros_buildfarm/templates/snippet/scm_git.xml.em Maybe that helps you finding the difference.

tfoote · February 27, 2017, 9:08pm

@octaviantuchila14 It sounds like you’re debugging a different system. Please ask a question on http://answers.ros.org with enough information to reproduce your problem. This thread is not the appropriate place to debug, it’s a discussion which resolved itself through an enhancement 6 months ago. For guidelines for asking questions please see http://wiki.ros.org/Support

The logs we’re talking about are in the console logs for the job without any extra options. For example: http://build.ros.org:8080/view/Kpr/job/Kpr__angles__ubuntu_xenial_amd64/4/consoleFull

GitHub pull request #9 of commit 8238889001273c75ea3fac9b22528279ffb8bcee, no merge conflicts.
Setting status of 8238889001273c75ea3fac9b22528279ffb8bcee to PENDING with url http://build.ros.org/job/Kpr__angles__ubuntu_xenial_amd64/4/ and message: 'Build started sha1 is merged.'
Using context: Kpr__angles__ubuntu_xenial_amd64
Building remotely on ip-172-31-4-29.us-west-1.compute.internal (swarm buildslave) in workspace /home/jenkins-slave/workspace/Kpr__angles__ubuntu_xenial_amd64
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/ros/angles.git # timeout=10
Fetching upstream changes from https://github.com/ros/angles.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress https://github.com/ros/angles.git +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse refs/remotes/origin/pr/9/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/9/merge^{commit} # timeout=10
Merging Revision ce3c2b0aae015d383694f569f3f7a0112d74219c (refs/remotes/origin/pr/9/merge) to origin/master, UserMergeOptions{mergeRemote='origin', mergeTarget='master', mergeStrategy='default', fastForwardMode='--ff'}
 > git rev-parse origin/master^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f origin/master
 > git merge --ff ce3c2b0aae015d383694f569f3f7a0112d74219c # timeout=10
 > git rev-parse HEAD^{commit} # timeout=10
Seen branch in repository origin/master
Seen branch in repository origin/pr/1/head
Seen branch in repository origin/pr/1/merge
Seen branch in repository origin/pr/3/head
Seen branch in repository origin/pr/4/head
Seen branch in repository origin/pr/6/head
Seen branch in repository origin/pr/7/head
Seen branch in repository origin/pr/8/head
Seen branch in repository origin/pr/8/merge
Seen branch in repository origin/pr/9/head
Seen branch in repository origin/pr/9/merge
Seen 11 remote branches
Checking out Revision ce3c2b0aae015d383694f569f3f7a0112d74219c (origin/pr/9/merge, origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ce3c2b0aae015d383694f569f3f7a0112d74219c
First time build. Skipping changelog.