Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EGit / JGit » Git status very long since last release 5.4.2
Git status very long since last release 5.4.2 [message #1811312] Fri, 30 August 2019 13:24 Go to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
Hi,

Since switching to version 5.4.2 of egit, at each reopening of eclipse a "git status" is triggered for each repo of my open projects in Eclipse. And this "git status" is very long for some projects with a lot of files and versions.
I'm using eclipse 2019-06, and there was no problem with the previous version of Egit included in eclipse.

Thank you.
Re: Git status very long since last release 5.4.2 [message #1811325 is a reply to message #1811312] Fri, 30 August 2019 20:53 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 1106
Registered: July 2009
Senior Member
How do you know that at each reopening of eclipse git status is triggered ?

Which version of EGit and Eclipse did you use previously ?
On which operating system and file system ?
Using which Java version ?

How long is very long ?

How many repositories of which size do you have in the workspace ?
You can use git-sizer to get metrics about size of your repositories and potential issues.

How did you configure "Preferences > Team > Git > Window Cache" ?
Re: Git status very long since last release 5.4.2 [message #1811349 is a reply to message #1811325] Sat, 31 August 2019 17:26 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
Hi,
Thanks for your response.
1: I see the launch of a git status in the "progress" view of Eclipse only when I open Eclipe in the morning: the rest of the day I can not see it.
2 : eclipse 2019-06, and the previous version of egit : 5.4.0.201906121030-r

3 : windows 10, NTFS on SSD
4 : openjdk 1.8 64 bits

5 : Between 1 and 10 minutes. it depends on what other repo runs the git status.
I noticed that in general there is 3 git status in parallel, no more.
If it is on repo very big, the 3 are very slow, and we quickly go to 15 minutes.
This is a corporate post, and the built-in antivirus scans all files, all the time..., and is not deactivatable. So it slows down even more, but before version 5.4.2, I did not have this phenomenon.

6 : 16 repo currently open. I attach to the post a screenshot of the progress view of eclipse at startup.
the 3 biggest have this kind of statistics:
Overall repository size | * Blobs | * Total size | 16.7 GiB
| Biggest objects | * Blobs | * Maximum size [1] | 43.3 MiB
| Biggest checkouts |
| * Number of directories [2] | 5.12 k
| * Maximum path depth [2] | 17
| * Maximum path length [3] | 220 B
| * Total size of files [4] | 1.64 GiB

The others are much smaller.

7: window size : 8 k, window cache limit: 500 m, delta base cache limit : 10 m, stream file threshold : 50 m

Another point: when I go into the perspective "git" of eclipse, a "git status" is triggered for all other repositories for which I have no open project in my workspace.

Thank You.

[Updated on: Sat, 31 August 2019 17:32]

Report message to a moderator

Re: Git status very long since last release 5.4.2 [message #1814104 is a reply to message #1811349] Tue, 03 September 2019 18:54 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
Hi,
Finally, the "git status" seems to trigger regularly when using projects. I do not know what makes it trigger, it's not every commit or pull, but from time to time.

Thank You

Julien
Re: Git status very long since last release 5.4.2 [message #1814286 is a reply to message #1814104] Sun, 08 September 2019 15:57 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 1106
Registered: July 2009
Senior Member
Can you try 5.5.0-rc1 which is available here [1] ?
Or you try latest build from the master branch [2], last week we merged
some performance improvements there.
The final 5.5.0 will be published on September 18 with Eclipse release 2019-09.

16GB repository size is huge. Are you storing binary files in git ?
When did you last run gc on these repositories (your local clones) ?

[1] https://download.eclipse.org/egit/staging/v5.5.0.201909041048-rc1/
[2] https://download.eclipse.org/egit/updates-nightly
Re: Git status very long since last release 5.4.2 [message #1814289 is a reply to message #1814286] Sun, 08 September 2019 16:56 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
ok, i will try the master branch.
Yes, we store all the .jar files needed by our application. We are not yet under maven or equivalent.
I never do "git gc", for me it is automatically supported by the plugin egit, it is not the case?
Re: Git status very long since last release 5.4.2 [message #1814291 is a reply to message #1814289] Sun, 08 September 2019 19:33 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
I installed the nigthly branch (screenshot attached), but it does not seem to change anything.
At start of eclipse, a "git status" is triggered for each repo, and it is quite long.
I returned to version 5.4.0 of the plugin via "installation history" of eclipse and I confirm that there is a difference: the "git status" are well triggered, but are extremely faster (a few seconds for the most large repositories).
Re: Git status very long since last release 5.4.2 [message #1814330 is a reply to message #1814291] Mon, 09 September 2019 14:20 Go to previous messageGo to next message
Thomas Wolf is currently offline Thomas WolfFriend
Messages: 175
Registered: August 2016
Senior Member
Wild guess: could this be a problem with an old index having a coarse file timestamp, and new EGit getting much finer file timestamps from the file system? Might that lead to more files appearing dirty by timestamps, and then many more content checks in FileTreeIterator?
Re: Git status very long since last release 5.4.2 [message #1814360 is a reply to message #1814330] Tue, 10 September 2019 08:37 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 1106
Registered: July 2009
Senior Member
If that's true recreating the git index should fix this.

Can you try to recreate the git index for all git repositories in your workspace?
- ensure repository state is clean, i.e. staging view for this repo is empty, if you have uncommitted changes then commit them
- make a backup copy of .git/index
- from a shell backup and delete the git index (EGit doesn't allow you to delete files inside the .git folder)
$ cd to root of the working tree of the respective git repository
$ rm .git/index
- refresh git staging view in Eclipse, this now shows all the files in the repo as untracked files
- select the repository in the repositories view and click "Reset... > mixed"
- refresh the git staging view, now it should be empty again since the git index was recreated

Then restart Eclipse and see if this problem is gone. If yes this would prove Thomas' assumption.
Re: Git status very long since last release 5.4.2 [message #1814361 is a reply to message #1814360] Tue, 10 September 2019 08:47 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
Ok I will try.
Which version of the plugin I use? 5.4.2, or the 5.5 rc1 or the nigthly-build?

Thank You.
Re: Git status very long since last release 5.4.2 [message #1814377 is a reply to message #1814361] Tue, 10 September 2019 13:10 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 1106
Registered: July 2009
Senior Member
doesn't matter for this question
Re: Git status very long since last release 5.4.2 [message #1814390 is a reply to message #1814360] Tue, 10 September 2019 19:00 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
Matthias Sohn wrote on Tue, 10 September 2019 10:37
If that's true recreating the git index should fix this.

Can you try to recreate the git index for all git repositories in your workspace?
- ensure repository state is clean, i.e. staging view for this repo is empty, if you have uncommitted changes then commit them
- make a backup copy of .git/index
- from a shell backup and delete the git index (EGit doesn't allow you to delete files inside the .git folder)
$ cd to root of the working tree of the respective git repository
$ rm .git/index
- refresh git staging view in Eclipse, this now shows all the files in the repo as untracked files
- select the repository in the repositories view and click "Reset... > mixed"
- refresh the git staging view, now it should be empty again since the git index was recreated

Then restart Eclipse and see if this problem is gone. If yes this would prove Thomas' assumption.


I confirm the hypothesis: after regeneration of the index, the git status on the repository of 16.7 Gb lasted only a few seconds, as with version 5.4.0.

Re: Git status very long since last release 5.4.2 [message #1814396 is a reply to message #1814390] Tue, 10 September 2019 21:29 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 1106
Registered: July 2009
Senior Member
Fix https://git.eclipse.org/r/#/c/149266/ will be included in release 5.5.0 which will be published with Eclipse 2019-09 on Sep 18
Re: Git status very long since last release 5.4.2 [message #1814397 is a reply to message #1814396] Tue, 10 September 2019 21:47 Go to previous messageGo to next message
Thomas Wolf is currently offline Thomas WolfFriend
Messages: 175
Registered: August 2016
Senior Member
I'm not convinced that that fix alone is sufficient. If Julien's old index has millisecond timestamps, a new EGit running against that old index will still be slow if the file system delivers sub-millisecond file modification times.

[Updated on: Tue, 10 September 2019 22:05]

Report message to a moderator

Re: Git status very long since last release 5.4.2 [message #1814402 is a reply to message #1814397] Wed, 11 September 2019 04:11 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
is a 5.5.0 rc2 provided with this fix for me to test?

Thank You
Re: Git status very long since last release 5.4.2 [message #1814409 is a reply to message #1814402] Wed, 11 September 2019 06:34 Go to previous messageGo to next message
Thomas Wolf is currently offline Thomas WolfFriend
Messages: 175
Registered: August 2016
Senior Member
Try updating JGit from the 5.5.0-SNAPSHOT update site. It should contain the above fix. If you still have the old index, try re-instating it and run EGit/JGit against it.

If it's slow again, we'd need to add some migration logic to compare file timestamps only with millisecond precision if the index has only millisecond precision.

[Updated on: Wed, 11 September 2019 08:56]

Report message to a moderator

Re: Git status very long since last release 5.4.2 [message #1814486 is a reply to message #1814409] Thu, 12 September 2019 07:32 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
Hi,

I installed this version : https://download.eclipse.org/egit/updates-5.5/.
It's the same. The Git Status is very long. I need to delete the index file and regenerate it for the git status to be fast.
Re: Git status very long since last release 5.4.2 [message #1814580 is a reply to message #1814486] Fri, 13 September 2019 14:15 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 1106
Registered: July 2009
Senior Member
Can you try if this fix helps ?
https://git.eclipse.org/r/#/c/149440/

This is probably caused by using native git and JGit on the same repository, depending on OS, filesystem and Java version being used
they may see different file timestamp resolution which may lead to wrong conclusions in WorkingTreeIterator if a file was modified or not.
This may kill the performance improvement the git index usually provides.
Re: Git status very long since last release 5.4.2 [message #1814589 is a reply to message #1814580] Fri, 13 September 2019 15:17 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
Sorry, but I do not know how to apply this fix to my environment.
Re: Git status very long since last release 5.4.2 [message #1814630 is a reply to message #1814589] Sat, 14 September 2019 16:14 Go to previous messageGo to next message
Thomas Wolf is currently offline Thomas WolfFriend
Messages: 175
Registered: August 2016
Senior Member
Install EGit nightly from https://download.eclipse.org/egit/updates-nightly/ . Then update JGit from https://ci.eclipse.org/jgit/job/stable/job/jgit.gerrit-pipeline/1448/artifact/org.eclipse.jgit.packaging/org.eclipse.jgit.repository/target/repository/ . Then let that run against the old index. If it's then still slow, could you send me your old index via e-mail? If that patch doesn't help I'd like to check what timestamps it contains.
Re: Git status very long since last release 5.4.2 [message #1814637 is a reply to message #1814630] Sun, 15 September 2019 09:48 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
There is a difference with this new version: the repository "external" (github.com) not related to my application now have a "git status" very fast, but our own reporistory are still slow.
And indeed the difference is that we use both "git bash" and "egit eclipse" to perform git commands locally.
I send you by mail the zipped index file of one of these repositories. And here is the version of Egit and jgit installed.

But how can I send you a private message?

[Updated on: Sun, 15 September 2019 09:52]

Report message to a moderator

Re: Git status very long since last release 5.4.2 [message #1814657 is a reply to message #1814637] Mon, 16 September 2019 07:29 Go to previous messageGo to next message
Thomas Wolf is currently offline Thomas WolfFriend
Messages: 175
Registered: August 2016
Senior Member
My e-mail address is listed in every git commit I make. For instance at https://git.eclipse.org/r/#/c/149440/ .
Re: Git status very long since last release 5.4.2 [message #1814669 is a reply to message #1814657] Mon, 16 September 2019 08:41 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
ok, I sent you the file.
Re: Git status very long since last release 5.4.2 [message #1814674 is a reply to message #1814657] Mon, 16 September 2019 09:11 Go to previous messageGo to next message
Thomas Wolf is currently offline Thomas WolfFriend
Messages: 175
Registered: August 2016
Senior Member
Thanks. What I expected: you wrote above you were using NTFS. Timestamps in the index have 100ns resolution. Matthias wrote in the commit message of https://git.eclipse.org/r/#/c/145437/ "WindowsFileAttributes also provides only microsecond resolution". So Java gets a resolution of 1000ns, but the index has a finer resolution of 100ns.

So I think we should do the same song-and-dance not just for millisecond but also for microsecond resolution. I'll update the change https://git.eclipse.org/r/#/c/149440/ accordingly; then we can test once again.

In case you'd like to see for yourself: here's the little quick'n'dirty program I used to dump the timestamps from your index (takes a single argument, which is the full path of the index file):

import java.io.File;
import java.time.Instant;

import org.eclipse.jgit.util.FS;

public class DirCacheDump {

	private void run(String fileName) throws Exception {
		File f = new File(fileName);
		Instant time = FS.DETECTED.lastModifiedInstant(f);
		System.out.println("Index has timestamp " + time); //$NON-NLS-1$
		DirCache dc = DirCache.read(f, FS.DETECTED);
		DirCacheIterator i = new DirCacheIterator(dc);
		while (!i.eof()) {
			handleEntry(i);
			i.next(1);
		}
		System.out.println("Index has timestamp " + time); //$NON-NLS-1$
	}

	private void handleEntry(DirCacheIterator i) throws Exception {
		DirCacheEntry e = i.getDirCacheEntry();
		if (e == null) {
			// Must be a subtree
			DirCacheIterator sub = (DirCacheIterator) i
					.createSubtreeIterator(null);
			while (!sub.eof()) {
				handleEntry(sub);
				sub.next(1);
			}
		} else {
			Instant time = e.getLastModifiedInstant();
			System.out.println(i.getEntryPathString() + ' ' + time);
		}
	}

	public static void main(String[] args) {
		try {
			new DirCacheDump().run(args[0]);
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}
Re: Git status very long since last release 5.4.2 [message #1814680 is a reply to message #1814674] Mon, 16 September 2019 09:51 Go to previous messageGo to next message
Thomas Wolf is currently offline Thomas WolfFriend
Messages: 175
Registered: August 2016
Senior Member
There's a new JGit 5.6.0-SNAPSHOT version at https://ci.eclipse.org/jgit/job/stable/job/jgit.gerrit-pipeline/1450/artifact/org.eclipse.jgit.packaging/org.eclipse.jgit.repository/target/repository/ . It compares only microseconds if file or cache time appear to have only microsecond resolution. So if you update JGit again from that repo, I would hope that (finally!) the index diff (git status) on your repository is fast again.
Re: Git status very long since last release 5.4.2 [message #1814681 is a reply to message #1814680] Mon, 16 September 2019 10:00 Go to previous messageGo to next message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
That's it, it works! :)
thank you.
Re: Git status very long since last release 5.4.2 [message #1814703 is a reply to message #1814674] Mon, 16 September 2019 20:39 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 1106
Registered: July 2009
Senior Member
If you want to inspect the git index you can use the jgit debug command (from any folder in a git working tree):

$ jgit debug-show-dir-cache
100644 300 2019-09-12,11:45:22.067357272 3a61f8cb7adb71913e9b241d26d940d27024423e 0 .bazelrc
100644 7 2019-09-08,17:36:59.372881193 ae6dd4e2032486bce8109094cce44098b60478b7 0 .bazelversion
100644 17 2019-04-13,18:31:52.912234805 f57840b7eec86b91b135afc40f20d950c5d86988 0 .gitattributes
100644 292 2019-09-12,11:47:03.507358168 553ecac4867edeef148ebf16a483d5a7e97a59ca 0 .gitignore
100644 1770 2019-09-12,11:42:17.156268698 f0bc990d8d4bbc9ad3f04d107423ed11be3358df 0 .mailmap
...
Re: Git status very long since last release 5.4.2 [message #1814725 is a reply to message #1814703] Tue, 17 September 2019 07:17 Go to previous message
Julien Bittard is currently offline Julien BittardFriend
Messages: 20
Registered: August 2019
Junior Member
Thanks for the info, I'll take a look.
Previous Topic:Resolving conflicts in "Pre-merged by git" mode
Next Topic:Files remaining locked after repository close in Windows
Goto Forum:
  


Current Time: Fri Dec 13 20:34:17 GMT 2019

Powered by FUDForum. Page generated in 0.03571 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top