Re: [cbi-dev] git operations failing on CI - workaround!

[Denis Roy <denis.roy@xxxxxxxxxxxxxxxxxxxxxx>]

Thanks, Jonah.

Denis

On 2020-06-05 5:03 p.m., Jonah Graham wrote:

Hi folks,

Those of us (like CDT) working with larger git repositories have been having lots of git failures on the new Jiro based Jenkins CI. There have been slow clones, and full on failing checkouts. 

For many of the examples it turns out this is a resource allocation problem in k8s and how Jenkins interacts with it. You can read about that in Bug 560283. Mikaël Barbero has an approved PR in Jenkins so hopefully the problems will be resolved soon. 

In the meantime you can do this workaround in your pipelines. Before the git/checkout pipeline step, fetch the changes from git in a sh step. This sh step can run in your main container, instead of the git/checkout which always runs in the JNLP container. Running in your main container means that much more resources are available to git and the git fetch will run faster.

For CDT our normal fetch time was 4+ minutes, and regularly timed out after 10 or even 20 minutes. With the workaround the fetch time is ~1 minute.

An example of what you need to change is this:

stage('Git Clone') {

steps {

container('cdt') {

checkout([$class: 'GitSCM', branches: [[name: '*/master']], ...

}

}

}

to

stage('Git Clone') {

steps {

container('cdt') {

timeout(activity: true, time: 20) {

/* Running the git fetch command manually is a workaround. See https://bugs.eclipse.org/bugs/show_bug.cgi?id=560283#c16 */

sh 'git config remote.origin.url https://git.eclipse.org/r/cdt/org.eclipse.cdt.git'

sh 'git fetch --no-tags --force --progress -- https://git.eclipse.org/r/cdt/org.eclipse.cdt.git +refs/heads/*:refs/remotes/origin/*'

}

checkout([$class: 'GitSCM', branches: [[name: '*/master']], ...

}

}

}

You want to leave the git/checkout step in place as that is how Jenkins knows what to query and what to display changes on. With the change it won't have to do the heavy lifting of actually fetching the changes. 

See commit 5080b30 in cdt-infra for CDT's change in practice.

 

HTH,

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com

_______________________________________________
cbi-dev mailing list
cbi-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cbi-dev

--

Denis Roy

Director, IT Services | Eclipse Foundation, Inc.

Eclipse Foundation: The Platform for Open Innovation and Collaboration

Twitter: @droy_eclipse