Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EGit » Reduce walk memory usage(Possibilities for reducing memory usage.)
icon3.gif  Reduce walk memory usage [message #985739] Thu, 15 November 2012 16:46 Go to next message
Doru Chiulan is currently offline Doru Chiulan
Messages: 2
Registered: August 2012
Junior Member
Hy guys.

I am using a PlotWalk for a git client application.

I have to open some pretty large repositories like linux-2.6, and i had to implement a "sliding window" algorhitm for my table which loads "chunks" of commits from the PlotWalk, because if i would load all commits the memory usage was huge.

Even if i use this, i want to reduce even more the memory usage. The application uses ~100mb of ram until the PlotWalk is used. Then, the PlotWalk uses 300mb of memory + 100 used by the rest of the application.

I read all possibilities of reducing PlotWalk memory usage in JGit User's guide, but i can't find a way.

I saw that the memory usage on PlotWalk started to rise when next() method is invoked.

Do you have any suggestions, maybe extending the PlotWalk, or it's Generators ?

Report message to a moderator

Re: Reduce walk memory usage [message #985795 is a reply to message #985739] Fri, 16 November 2012 03:37 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias Sohn
Messages: 395
Registered: July 2009
Senior Member
Maybe this discussion [1] helps to gain more insight.

[1] http://dev.eclipse.org/mhonarc/lists/jgit-dev/msg01588.html

Report message to a moderator

Re: Reduce walk memory usage [message #985796 is a reply to message #985739] Fri, 16 November 2012 03:37 Go to previous message
Stefan Lay is currently offline Stefan Lay
Messages: 327
Registered: July 2009
Senior Member
Hi,

we already tried something in that direction for EGit. It turned out that the problem is that PlotWalk uses RevSort.TOPO. In that case the first call of next() triggers a walk of the whole history internally in order to do the topological sorting, loading all the commits into memory. See the discussion on http://dev.eclipse.org/mhonarc/lists/jgit-dev/msg01589.html and its predecessor for more details.

In Egit, we did two things:

- Load commits in batches (https://git.eclipse.org/r/#/c/5654/ for the gerrit change)
- Lazy loading of the commit body (https://git.eclipse.org/r/#/c/6149/ in gerrit)


We didn't investigate further. If you find out anything more, please let us know (or contribute to jgit/egit Wink. I would suggest to use the jgit mailing list for that (https://dev.eclipse.org/mailman/listinfo/jgit-dev).

Cheers,
Stefan

Report message to a moderator

Previous Topic:Commit/amend not possible
Next Topic:GIT Hooks
Goto Forum:
  

[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Tue May 21 10:59:33 EDT 2013

Powered by FUDForum. Page generated in 0.01661 seconds