Eclipse Community Forums: EGit / JGit » "Building Workspace" slow with EGit 1.1 and Nightly

Help

Home

Home » Eclipse Projects » EGit / JGit » "Building Workspace" slow with EGit 1.1 and Nightly

Show: Today's Messages :: Show Polls :: Message Navigator

"Building Workspace" slow with EGit 1.1 and Nightly [message #756185]

Fri, 11 November 2011 12:23

hans_schmidt_2000

Messages: 5
Registered: November 2011

Junior Member

Building Projects is considerably slower for me when they are under EGit version control than when they aren't.

To give you some numbers:
Projects are under EGit version control;
i disconnect all projects -> takes less than a minute
i clean&build all projects -> takes 80 seconds
i put all projects back under version control -> takes less than a minute
i clean&build all projects again -> takes ~1500 seconds

Hardware is Core2Duo E8600 with 4 GB RAM and Intel 120 GB SSD
Software is Windows XP 32 bit; Eclipse Indigo SR1 with EGit 1.1 / Nightly Build 20111111 (both EGit versions tested; i didn't notice any differences)

In the Eclipse UI it hangs at "Copying resources to the output folder" most of the time. Looking at file access what Eclipse does during that time is to read the git index over and over again.

here's what i believe to be a relevant stack trace:

"Worker-7" prio=6 tid=0x561bf400 nid=0xd28 runnable [0x593be000]
   java.lang.Thread.State: RUNNABLE
	at sun.security.provider.SHA.implCompress(Unknown Source)
	at sun.security.provider.DigestBase.engineUpdate(Unknown Source)
	at java.security.MessageDigest$Delegate.engineUpdate(Unknown Source)
	at java.security.MessageDigest.update(Unknown Source)
	at org.eclipse.jgit.dircache.DirCacheEntry.<init>(DirCacheEntry.java:168)
	at org.eclipse.jgit.dircache.DirCache.readFrom(DirCache.java:413)
	at org.eclipse.jgit.dircache.DirCache.read(DirCache.java:346)
	at org.eclipse.jgit.dircache.DirCache.lock(DirCache.java:196)
	at org.eclipse.jgit.dircache.DirCache.lock(DirCache.java:239)
	at org.eclipse.jgit.lib.Repository.lockDirCache(Repository.java:873)
	at org.eclipse.egit.core.GitMoveDeleteHook.deleteFile(GitMoveDeleteHook.java:58)
	at org.eclipse.team.internal.core.MoveDeleteManager.deleteFile(MoveDeleteManager.java:50)
	at org.eclipse.core.internal.resources.Resource.unprotectedDelete(Resource.java:1930)
	at org.eclipse.core.internal.resources.Resource.delete(Resource.java:780)
	at org.eclipse.jdt.internal.core.builder.BatchImageBuilder$3.visit(BatchImageBuilder.java:219)
	at org.eclipse.core.internal.resources.Resource$1.visitElement(Resource.java:65)
	at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:82)
	at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
	at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
	at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
	at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
	at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
	at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
	at org.eclipse.core.internal.watson.ElementTreeIterator.iterate(ElementTreeIterator.java:127)
	at org.eclipse.core.internal.resources.Resource.accept(Resource.java:75)
	at org.eclipse.jdt.internal.core.builder.BatchImageBuilder.copyExtraResourcesBack(BatchImageBuilder.java:191)
	at org.eclipse.jdt.internal.core.builder.BatchImageBuilder.cleanOutputFolders(BatchImageBuilder.java:162)
	at org.eclipse.jdt.internal.core.builder.BatchImageBuilder.build(BatchImageBuilder.java:46)
	at org.eclipse.jdt.internal.core.builder.JavaBuilder.buildAll(JavaBuilder.java:254)
	at org.eclipse.jdt.internal.core.builder.JavaBuilder.build(JavaBuilder.java:173)
	at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:728)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
	at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:199)
	at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:239)
	at org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:292)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
	at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:295)
	at org.eclipse.core.internal.events.BuildManager.basicBuildLoop(BuildManager.java:351)
	at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:374)
	at org.eclipse.core.internal.resources.Workspace.buildInternal(Workspace.java:513)
	at org.eclipse.core.internal.resources.Workspace.build(Workspace.java:422)
	at org.eclipse.ui.actions.GlobalBuildAction$1.run(GlobalBuildAction.java:180)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

i created several thread dumps, they all looked mostly the same.

I first suspected class files being under version control since i accidentally put them under version control when i first imported the projects, but since then i have removed the class files from git and ignored the classes dir via one .gitignore file per project. When i check out the ref i test with into an empty directory the classes dir is empty, so i'm reasonably sure this is not the problem.

Since building projects is - i think - a fairly common operation, the slowdown i'm experiencing is quite severe and i have not found anyone else reporting the problem in the forum or on the bug tracker, i assume it's not a general problem with EGit, but some odd configuration/constellation on my part. i would be thankful for pointers to what the problem might be.

Report message to a moderator

Re: "Building Workspace" slow with EGit 1.1 and Nightly [message #756205 is a reply to message #756185]

Fri, 11 November 2011 13:31

R Shapiro

Messages: 386
Registered: June 2011

Senior Member

I have a collection of Java projects of a similar size (~ 80-90 seconds for a full rebuild), and I can't replicate this. I see no significant difference in build time between connected and disconnected projects. Given that the time seems to be in "Copying resources to the output folder", sounds like you're seeing some remnant of having previously had the .class files under version control. What happens if you create a new workspace and import into it projects from a new repository clone?

Report message to a moderator

Re: "Building Workspace" slow with EGit 1.1 and Nightly [message #756246 is a reply to message #756205]

Fri, 11 November 2011 16:27

Matthias Sohn

Messages: 1268
Registered: July 2009

Senior Member

I can't reproduce your problem with e.g. EGit projects on my Mac.

Use "git ls-files" to find out if any binaries are tracked by the git index.
What is suspicious is that GitMoveDeleteHook is involved in the stack trace,
this shouldn't happen for build results which are typically both derived
resources and usually ignored through gitignore settings.

Report message to a moderator

Re: "Building Workspace" slow with EGit 1.1 and Nightly [message #756265 is a reply to message #756246]

Fri, 11 November 2011 17:46

hans_schmidt_2000

Messages: 5
Registered: November 2011

Junior Member

Thank you both for your quick replies.

I have tried with clean repo and clean workspace to no avail.

Through further analysis of what exactly is happening during the build process i have found out what is "odd" about my projects: my src folders contain a lot of .xml files.

During the build process, after all the classes for one project are built, the index is read once (presumeably to regenerate the DirCache). This is no problem, since it's a reasonably fast operation.
After that the .xml files are copied from the src to the output folder and for every .xml file (sometimes hundreds per project) the index is read once. Rebuilding the DirCache (or do whatever the index is read for) thousands of times => slow.

In Preferences->Java->Compiler->Building i filtered *.xml files from being copied to the output folder and the build time went down to < 2 minutes.

This is not a permantent solution for me. I can probably find a workaround that does not require me to copy the .xml files during the build, but i would prefer to keep my project layout as-is.

As for the files being git-ignored/derived: the .xml files in the src dir are obviously under verion control and not ignored.
the .xml files in the output folder are not under version control. Every eclipse project has a .gitignore file which containts "/classes" which is the output folder.
i think the .xml files in the output folder are marked as derived, but i can't check before monday.

Report message to a moderator

Re: "Building Workspace" slow with EGit 1.1 and Nightly [message #756298 is a reply to message #756265]

Fri, 11 November 2011 20:09

R Shapiro

Messages: 386
Registered: June 2011

Senior Member

So you want the XML files to be available as Java resources at runtime, is that right? If so, they have to be copied into the classes directory. Sounds like either your .gitignore isn't quite right or there's a bug in the egit that's causing resource copying to be treated like git copying.

If you don't want the XML files to be available as Java resources at runtime, move them out of whatever Java source directory they currently live in and the builder won't copy them.

Report message to a moderator

Re: "Building Workspace" slow with EGit 1.1 and Nightly [message #756564 is a reply to message #756298]

Mon, 14 November 2011 10:51

hans_schmidt_2000

Messages: 5
Registered: November 2011

Junior Member

R Shapiro wrote on Fri, 11 November 2011 15:09

So you want the XML files to be available as Java resources at runtime, is that right?

Yes.

R Shapiro wrote on Fri, 11 November 2011 15:09

Sounds like either your .gitignore isn't quite right or there's a bug in the egit that's causing resource copying to be treated like git copying.

the typical layout of a project (irrelevant entries omitted):

.project
.gitignore (created with EGit; example attached)
src/
src/<package-path>/ABC.xml
classes/ (output folder)
classes/<package-path>/ABC.xml (derived)

I have verified that ABC.xml exists in both the src and the classes folder. A git ls-files | grep ABC.xml in the project root dir only lists the one in the src folder.
Also commit dialogues never show files from the classes folder, so git-ignoring seems to work just fine most of the time.

From the copyExtraResourcesBack method in class BatchImageBuilder:

...
IResource copiedResource = outputFolder.getFile(partialPath); // the .xml file in the output folder
if(copiedResource.exists()) {
  ...
  copiedResource.delete(IResource.FORCE, null); // here the GitMoveDeleteHook is called
}

Attachment: .gitignore
(Size: 0.01KB, Downloaded 263 times)

Report message to a moderator

Re: "Building Workspace" slow with EGit 1.1 and Nightly [message #756602 is a reply to message #756564]

Mon, 14 November 2011 13:46

R Shapiro

Messages: 386
Registered: June 2011

Senior Member

Seems a little odd that a jgit hook would be invoked here, but that's random speculation on my part. Could be perfectly normal.

When you say "a lot of .xml files", what's the order of magnitude? Tens? Hundreds? I can try to replicate.

Report message to a moderator

Re: "Building Workspace" slow with EGit 1.1 and Nightly [message #756662 is a reply to message #756602]

Mon, 14 November 2011 15:38

hans_schmidt_2000

Messages: 5
Registered: November 2011

Junior Member

Between a couple of dozens and one thousand .xml files per project. The aggregate over all projects is a little less than 6000 .xml files. Git index file is 19 MB. Btw. it's not just .xml files, it's any file in the src folder that's not a .java file. Anything that goes through the copyExtraResourcesBack method.

From GitMoveDeleteHook.deleteFile(...) (from org.eclipse.egit.core.source_1.2.0.201111140414.jar):

	final DirCache dirc = map.getRepository().lockDirCache();
	final int first = dirc.findEntry(map.getRepoRelativePath(file));
	if (first < 0) {
		dirc.unlock();
		return false;
	}

"first" is always < 0 for my case.

My - highly speculative - interpretation is this:
delete on resource is called ->
resource is in project which is under git version control ->
resource might be under version control ->
GitMoveDeleteHook is called ->
GitMoveDeleteHook instance figures out, whether resource is really under git version control; in order to do this a DirCache/DirCacheEntry instance is created by reading the index file ->
GitMoveDeleteHook comes to the (right) conclusion that resource is not under version control and does nothing/returns false ->
standardDelete is invoked by Resource class

It seems to me that either - as Matthias Sohn said - GitMoveDeleteHook shouldn't be involved at all, or GitMoveDeleteHook needs a much faster detection mechanism for ignored files.

PS: it probably doesn't make any difference, but a small correction to a previous post of mine:
Quote:

During the build process, after all the classes for one project are built, the index is read once (presumeably to regenerate the DirCache). This is no problem, since it's a reasonably fast operation.

This is not right. During the build process the index is only read for "extra" resources.
Project with only .java files in src -> index is not read at all
Project with 921 .xml files in src -> index is read 921 times

Report message to a moderator

Re: "Building Workspace" slow with EGit 1.1 and Nightly [message #759452 is a reply to message #756662]

Mon, 28 November 2011 17:33

hans_schmidt_2000

Messages: 5
Registered: November 2011

Junior Member

I don't like replying to myself, but i have to ask:
Are any of the EGit developers going to look into this problem in the forseeable future or am i better off looking for a workaround/different tool?

Report message to a moderator

Re: "Building Workspace" slow with EGit 1.1 and Nightly [message #1006324 is a reply to message #756185]

Wed, 30 January 2013 20:36

Breton Tillinghast

Messages: 1
Registered: January 2013

Junior Member

Yes I'm having the same issue! We recently started using egit and our build time went from about 12 seconds to around 45. Did you ever find a solution to this?

Report message to a moderator

Previous Topic:	Saving remote repository information
Next Topic:	How to separate Eclipse and Git metadata

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Apr 19 07:55:53 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter