Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EGit / JGit » Very slow git clone performance
Very slow git clone performance [message #1754624] Tue, 21 February 2017 07:21 Go to next message
Malte Brunnlieb is currently offline Malte BrunnliebFriend
Messages: 27
Registered: February 2011
Junior Member
Hi all,

I am observing very low git performance with egit in eclipse/oomph and complex repositories with several branches although the performance of git in git bash or windows cmd is very good.
I found several quite old discussions regarding performance issues, but there seems to be no up-to-date discussion. Are you aware of any egit restrictions regarding repository size?
We are not able to restructure all the repositories. We will even sometimes have not influence on the git respositories of our customers as well. So how can I improve the performance of egit such that it does not take hours anymore?
Btw. hours is not an overstatement Smile We currently have a repository we want to be cloned within an Oomph setup, which took us two hours of egit processing as the repository will be cloned 10 times (each branch separately). Please no discussion on the necessity, I will not be able to provide all insights here Wink
I even reproduced it in eclipse to be sure, that it is no oomph issue. So it should be an egit issue.

Best Regards,
Malte
Re: Very slow git clone performance [message #1754850 is a reply to message #1754624] Thu, 23 February 2017 08:21 Go to previous messageGo to next message
Malte Brunnlieb is currently offline Malte BrunnliebFriend
Messages: 27
Registered: February 2011
Junior Member
Some properties of the repository:

git rev-list --all --count
= 1229
git branch -r | wc -l
= 12
git tag -l | wc -l
= 41

File size distribution in the repository:

find . -type f | wc -l
= 947
find . -type f -size -4096c | wc -l
= 809
find . -type f -size -1024c | wc -l
= 520
find . -type f -size -512c | wc -l
= 346
find . -type f -size +4M | wc -l
= 0

[Updated on: Thu, 23 February 2017 08:24]

Report message to a moderator

Re: Very slow git clone performance [message #1755274 is a reply to message #1754850] Wed, 01 March 2017 22:03 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 943
Registered: July 2009
Senior Member
Did you try to clone the same repository using native git command line ?
Is this also slow ?

Does the upstream repository have many refs ?
Is the upstream repository gc'ed on a regular schedule ?
Does the upstream repository have bitmap indexes (can be configured in gc configuration) ?

Client side you could try to increase JGit object cache or use virtual memory mapping:
Preferences > Team > Git > Window Cache > Window Cache limit / use virtual memory mapping

This can also be configured programmatically (if cloning is done by Oomph programmatically):
create a WindowCacheConfig and call it's install() method.
Re: Very slow git clone performance [message #1755395 is a reply to message #1755274] Thu, 02 March 2017 19:05 Go to previous messageGo to next message
Steffen Holzer is currently offline Steffen HolzerFriend
Messages: 59
Registered: January 2016
Member

Cloning via git itself is done within a few minutes

"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live" - John Woods
Re: Very slow git clone performance [message #1755410 is a reply to message #1755395] Fri, 03 March 2017 06:47 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 28634
Registered: July 2009
Senior Member
Note that Matthias asked several questions. I didn't know about this Window Cache thing, nor do I know what any of the numbers in that dialog mean. Of course with Oomph you can set preferences (assuming these values are stored in preference nodes). But I'm not sure those preferences are used when Oomph clones using the JGit APIs or if that needs to be enabled specifically via this WindowCacheConfig. I'm also not sure if the preferences are loaded so early by org.eclipse.egit.core.Activator.start(BundleContext) that changing the values via preference tasks will apply them as is done in org.eclipse.egit.ui.internal.preferences.WindowCachePreferencePage.performOk(). And of course, in the end, I'm not sure that changing these values will improve performance. That's something you'll need to try manually to know if this is really relevant...
Re: Very slow git clone performance [message #1755411 is a reply to message #1755410] Fri, 03 March 2017 07:25 Go to previous messageGo to next message
Malte Brunnlieb is currently offline Malte BrunnliebFriend
Messages: 27
Registered: February 2011
Junior Member
> Did you try to clone the same repository using native git command line ?
> Is this also slow ?
Basically, it is the same until the git bash as well as the console output of eclipse/oomph states 100% fetching. After that it takes a second for git bash and multiple minutes for oomph/eclipse to finish. Whatever is done after fetching all change sets. I do not know.

> Does the upstream repository have many refs ?
See my post: git rev-list --all --count = 1229

> Is the upstream repository gc'ed on a regular schedule ?
remote? It is a private GitHub repository. I do not know what github is automatically performing on it. I did not see any configuration option for that.

> Does the upstream repository have bitmap indexes (can be configured in gc configuration) ?
Sounds interesting, but I did not see any option for that on GitHub. But just to show you some more measures of the repository:

remote: Counting objects: 19343, done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 19343 (delta 3), reused 0 (delta 0), pack-reused 19312
Receiving objects: 100% (19343/19343), 3.55 MiB | 105.00 KiB/s, done.
Resolving deltas: 100% (8095/8095), done.


I even not entirely convinced that a repack would fix this as the repository seems to be quite normal from the number of objects etc.
Re: Very slow git clone performance [message #1755419 is a reply to message #1755411] Fri, 03 March 2017 08:22 Go to previous messageGo to next message
Malte Brunnlieb is currently offline Malte BrunnliebFriend
Messages: 27
Registered: February 2011
Junior Member
Ok, I tried all of the mentioned proposals to increase several caches or use virtual memory mapping. No resolution here...

However, I do have a new idea:
The repository contains long paths, we cannot circumvent. May this be a problem?
Is the repository somehow first cloned into a temporary folder and copied afterwards such that long paths may be problematic on windows?
With which windows API? explorer vs. robocopy or similar?
I will give it a try in a linux vm and see whether it will show the same behavior.
Re: Very slow git clone performance [message #1755900 is a reply to message #1755419] Thu, 09 March 2017 08:11 Go to previous messageGo to next message
Malte Brunnlieb is currently offline Malte BrunnliebFriend
Messages: 27
Registered: February 2011
Junior Member
I finally had time to try it in a linux VM.
Egit is working fast as expected. So it really might be a problem of long paths on windows 7 (x64) and egit.
Are there any eclipse preferences to allow long paths for git? I configured this to work with my msysgit, but as far as I know eclipse comes with its own git implementation, does it?
Re: Very slow git clone performance [message #1755983 is a reply to message #1755900] Thu, 09 March 2017 19:26 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 943
Registered: July 2009
Senior Member
I don't think this is slow due to long path names.
To my experience Windows filesystem is considerably slower than unix filesystems but it shouldn't be that bad.

Cloning a repository of 3.5MB shouldn't take a long time using JGit/EGit.

Did you try to clone this repository using EGit in Eclipse ? Is that also slow ?

The WindowCache is JGit's buffer cache used to avoid repeated IO to read the same data from pack files on disk.
You may try to increase this cache size. Or alternatively switch on virtual memory mapping which
delegates buffering pack file read access to the JVM's virtual memory mapping. This isn't on by
default since Java doesn't provide a way to unmap files which have been mapped to virtual memory.

Other than that, if your repository is private, you can try to profile cloning your repository with EGit
and come back here with profiling statistics.

Re: Very slow git clone performance [message #1755990 is a reply to message #1754624] Thu, 09 March 2017 19:46 Go to previous message
Malte Brunnlieb is currently offline Malte BrunnliebFriend
Messages: 27
Registered: February 2011
Junior Member
> Did you try to clone this repository using EGit in Eclipse ? Is that also slow ?

Yes, tried it. Is exactly as slow as within Oomph.

> The WindowCache is JGit's buffer cache used to avoid repeated IO to read the same data from pack files on disk.
> You may try to increase this cache size. Or alternatively switch on virtual memory mapping which delegates buffering pack file read access to the JVM's virtual memory mapping. This isn't on by default since Java doesn't provide a way to unmap files which have been mapped to virtual memory.

I tried increasing cache sizes as well as enable virtual memory mapping. No improvements here. What a pitty, that I cannot provide you the repository. But if I find some time, I may create a new repository with long paths just for testing purposes and to exclude it as a cause.
Currently, I am out of ideas. I will write some more lines, if I have investigated this behavior further.
Previous Topic:Why doesn't my patch work?
Next Topic:EGit - Commit and Push button custom warning message
Goto Forum:
  


Current Time: Sat Jul 22 17:19:10 GMT 2017

Powered by FUDForum. Page generated in 0.02785 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software