Eclipse Community Forums: EGit / JGit » Very slow git clone performance

Home » Eclipse Projects » EGit / JGit » Very slow git clone performance

Very slow git clone performance [message #1754624]

Tue, 21 February 2017 02:21

Eclipse User

Hi all,

I am observing very low git performance with egit in eclipse/oomph and complex repositories with several branches although the performance of git in git bash or windows cmd is very good.
I found several quite old discussions regarding performance issues, but there seems to be no up-to-date discussion. Are you aware of any egit restrictions regarding repository size?
We are not able to restructure all the repositories. We will even sometimes have not influence on the git respositories of our customers as well. So how can I improve the performance of egit such that it does not take hours anymore?
Btw. hours is not an overstatement Smile

We currently have a repository we want to be cloned within an Oomph setup, which took us two hours of egit processing as the repository will be cloned 10 times (each branch separately). Please no discussion on the necessity, I will not be able to provide all insights here Wink

I even reproduced it in eclipse to be sure, that it is no oomph issue. So it should be an egit issue.

Best Regards,
Malte

Re: Very slow git clone performance [message #1754850 is a reply to message #1754624]

Thu, 23 February 2017 03:21

Eclipse User

Some properties of the repository:

git rev-list --all --count

= 1229

git branch -r | wc -l

= 12

git tag -l | wc -l

= 41

File size distribution in the repository:

find . -type f | wc -l

= 947

find . -type f -size -4096c | wc -l

= 809

find . -type f -size -1024c | wc -l

= 520

find . -type f -size -512c | wc -l

= 346

find . -type f -size +4M | wc -l

= 0

[Updated on: Thu, 23 February 2017 03:24] by Moderator

Re: Very slow git clone performance [message #1755274 is a reply to message #1754850]

Wed, 01 March 2017 17:03

Eclipse User

Did you try to clone the same repository using native git command line ?
Is this also slow ?

Does the upstream repository have many refs ?
Is the upstream repository gc'ed on a regular schedule ?
Does the upstream repository have bitmap indexes (can be configured in gc configuration) ?

Client side you could try to increase JGit object cache or use virtual memory mapping:
Preferences > Team > Git > Window Cache > Window Cache limit / use virtual memory mapping

This can also be configured programmatically (if cloning is done by Oomph programmatically):
create a WindowCacheConfig and call it's install() method.

Re: Very slow git clone performance [message #1755395 is a reply to message #1755274]

Thu, 02 March 2017 14:05

Eclipse User

Cloning via git itself is done within a few minutes

Re: Very slow git clone performance [message #1755410 is a reply to message #1755395]

Fri, 03 March 2017 01:47

Eclipse User

Note that Matthias asked several questions. I didn't know about this Window Cache thing, nor do I know what any of the numbers in that dialog mean. Of course with Oomph you can set preferences (assuming these values are stored in preference nodes). But I'm not sure those preferences are used when Oomph clones using the JGit APIs or if that needs to be enabled specifically via this WindowCacheConfig. I'm also not sure if the preferences are loaded so early by org.eclipse.egit.core.Activator.start(BundleContext) that changing the values via preference tasks will apply them as is done in org.eclipse.egit.ui.internal.preferences.WindowCachePreferencePage.performOk(). And of course, in the end, I'm not sure that changing these values will improve performance. That's something you'll need to try manually to know if this is really relevant...

Re: Very slow git clone performance [message #1755411 is a reply to message #1755410]

Fri, 03 March 2017 02:25

Eclipse User

> Did you try to clone the same repository using native git command line ?
> Is this also slow ?
Basically, it is the same until the git bash as well as the console output of eclipse/oomph states 100% fetching. After that it takes a second for git bash and multiple minutes for oomph/eclipse to finish. Whatever is done after fetching all change sets. I do not know.

> Does the upstream repository have many refs ?
See my post: git rev-list --all --count = 1229

> Is the upstream repository gc'ed on a regular schedule ?
remote? It is a private GitHub repository. I do not know what github is automatically performing on it. I did not see any configuration option for that.

> Does the upstream repository have bitmap indexes (can be configured in gc configuration) ?
Sounds interesting, but I did not see any option for that on GitHub. But just to show you some more measures of the repository:

remote: Counting objects: 19343, done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 19343 (delta 3), reused 0 (delta 0), pack-reused 19312
Receiving objects: 100% (19343/19343), 3.55 MiB | 105.00 KiB/s, done.
Resolving deltas: 100% (8095/8095), done.

I even not entirely convinced that a repack would fix this as the repository seems to be quite normal from the number of objects etc.

Re: Very slow git clone performance [message #1755419 is a reply to message #1755411]

Fri, 03 March 2017 03:22

Eclipse User

Ok, I tried all of the mentioned proposals to increase several caches or use virtual memory mapping. No resolution here...

However, I do have a new idea:
The repository contains long paths, we cannot circumvent. May this be a problem?
Is the repository somehow first cloned into a temporary folder and copied afterwards such that long paths may be problematic on windows?
With which windows API? explorer vs. robocopy or similar?
I will give it a try in a linux vm and see whether it will show the same behavior.

Re: Very slow git clone performance [message #1755900 is a reply to message #1755419]

Thu, 09 March 2017 03:11

Eclipse User

I finally had time to try it in a linux VM.
Egit is working fast as expected. So it really might be a problem of long paths on windows 7 (x64) and egit.
Are there any eclipse preferences to allow long paths for git? I configured this to work with my msysgit, but as far as I know eclipse comes with its own git implementation, does it?

Re: Very slow git clone performance [message #1755983 is a reply to message #1755900]

Thu, 09 March 2017 14:26

Eclipse User

I don't think this is slow due to long path names.
To my experience Windows filesystem is considerably slower than unix filesystems but it shouldn't be that bad.

Cloning a repository of 3.5MB shouldn't take a long time using JGit/EGit.

Did you try to clone this repository using EGit in Eclipse ? Is that also slow ?

The WindowCache is JGit's buffer cache used to avoid repeated IO to read the same data from pack files on disk.
You may try to increase this cache size. Or alternatively switch on virtual memory mapping which
delegates buffering pack file read access to the JVM's virtual memory mapping. This isn't on by
default since Java doesn't provide a way to unmap files which have been mapped to virtual memory.

Other than that, if your repository is private, you can try to profile cloning your repository with EGit
and come back here with profiling statistics.

Re: Very slow git clone performance [message #1755990 is a reply to message #1754624]

Thu, 09 March 2017 14:46

Eclipse User

> Did you try to clone this repository using EGit in Eclipse ? Is that also slow ?

Yes, tried it. Is exactly as slow as within Oomph.

> The WindowCache is JGit's buffer cache used to avoid repeated IO to read the same data from pack files on disk.
> You may try to increase this cache size. Or alternatively switch on virtual memory mapping which delegates buffering pack file read access to the JVM's virtual memory mapping. This isn't on by default since Java doesn't provide a way to unmap files which have been mapped to virtual memory.

I tried increasing cache sizes as well as enable virtual memory mapping. No improvements here. What a pitty, that I cannot provide you the repository. But if I find some time, I may create a new repository with long paths just for testing purposes and to exclude it as a cause.
Currently, I am out of ideas. I will write some more lines, if I have investigated this behavior further.

Previous Topic:	Git integration fails
Next Topic:	git hook pre-push

Goto Forum:

-=] Back to Top [=-

Current Time: Wed Jul 23 13:49:22 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter