Re: [egit-dev] Bug 456996: EGit performance poor when home directory on

[Thomas Wolf <thomas.wolf@xxxxxxxxxx>]

> On Nov 6, 2019, at 11:59, Thomas Wolf <Thomas.Wolf@xxxxxxxxxx> wrote:
> 
> I've pushed a series of changes working towards improving EGit performance
> when access to the git user config is slow.

Here's some manual test results:

I've instrumented JGit and measured the effect of these four changes
(or five, if you count the JGit change) with a very simple manual test
setup:

Eclipse workspace prepared: Vanilla child Eclipse, Neon target, new
workspace. Java perspective, Repo view open. No other git views open,
no open editors. In the repo view 3 repositories: EGit, JGit, and an
older clone of Gerrit (thousands of commits behind upstream; with 6
submodules for plugins). Switch build automatically off and then import
all EGit projects into Eclipse so that they show up in the package
explorer. Quit Eclipse.

Test steps:

1. Start the child Eclipse and wait until all decorations show up in
   package explorer and repo view.
2. In the repo view, expand Branches->Remote Tracking on the EGit repo
   and wait until the branches are shown (with decorations, including
   the short commit message).
3. On the EGit repository node, open the context menu and select
   "Show in->History" and wait until the history view shows up and is
   fully populated.
4. Quit Eclipse

That's all.

Without these changes:

 * At the end, there are 33 opened and 24 closed repositories. (The 24
   closed ones are the submodules, each opened/closed 4 times; the 9
   remaining ones are those cached by EGit.)
 * There are after step 1:
   - 2902 calls to FileUtils.getAttributes(), 35 of which are on the UI thread
   - 625 of these calls are for the user config, (on UI thread: 10),
     plus another 625(10) for the system config
 * After step 2:
   - 4315 calls to FileUtils.getAttributes(), 798 on the UI thread
   - 868 of which are for the user config (140 on UI)
     + same numbers for system config
 * After step 3:
   - 4706 calls to FileUtils.getAttributes(), 896 on the UI thread
   - 957 of which are for the user config (160 on UI)
     + same numbers for system config
 * Faking a slow network disk by adding a Thread.sleep(30) in
   FileUtils.getAttributes() if the file is the user config, it
   takes about 12 seconds in step 1 for decorations to finally
   fully appear after the Eclipse window opens.

With these changes:

 * At the end, there are 9 opened and 9 closed repositories. (All instances
   from the EGit cache.)
 * There are after step 1:
   - 527 calls to FileUtils.getAttributes(), 35 of which are on the UI thread
   - 73 of these calls are for the user config, (on UI thread: 10),
     plus another 73(10) for the system config
 * After step 2:
   - 600 calls to FileUtils.getAttributes(), 69 on the UI thread
   - 88 of which are for the user config (20 on UI)
     + same numbers for system config
 * After step 3:
   - 711 calls to FileUtils.getAttributes(), 143 on the UI thread
   - 107 of which are for the user config (32 on UI)
     + same numbers for system config
 * Faked slow access to the user config: 1.5 seconds for all decorations 
   to appear in step 1.
   
A reduction by a factor of 6-10 looks promising. There is still room for
improvement; for instance the Gitflow RepositoryPropertyTester is called
frequently on the UI thread and is expensive. There's also probably many
calls to Repository.getConfig() to read values that exist only in the repo
config (branch names, submodules, etc.), a specialized getConfig() that
could be used in such cases and that would skip checking the user and
system config might also be helpful.(*)

The crucial thing is to reduce the number of times the git user config
is checked for modifications. We may assume that git repos are on local
disks (all of git is based on that assumption), and we may assume that
the system config is also local.

(*) One suggestion made in bug 550878 was to check the user config for
modifications only once per second. While I cannot pinpoint a place in
EGit or in JGit where that would surely cause trouble I'm not too fond
of such a sledgehammer approach. But if other improvements are not good
enough it might also be an option (guarded by a preference, so that it
could be switched on only if really needed).

Cheers,

  Thomas