Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jgit-dev] Potential memory leak in JGit
Hi Matthias,
thanks for the quick rely. 

Answers inline.

On 11 Jun 2019, at 08:39, Matthias Sohn <matthias.sohn@xxxxxxxxx> wrote:

On Tue, Jun 11, 2019 at 8:31 AM Luca Milanesio <luca.milanesio@xxxxxxxxx> wrote:
Hi all,
I am writing to both JGit and Gerrit mailing list because the problem could be of interest for both communities.

I am able to systematically reproduce a fatal memory leak that causes the complete collapse of the JVM.

Repos: the Android repos
Test-case: for (all repos); do clone and git GC; done

Can you specify the exact test setup and share the test case itself ?
  • I guess you are serving from Gerrit ?

v2.15.13

  • Which version of Gerrit and JGit are you using ?
 v4.9.10.201904181027-r

  • Which JVM are you using ?
openjdk full version "1.8.0_212-b04"
  • Which JVM option settings are you using ?
        javaOptions = -server
        javaOptions = -Djava.security.egd=file:/dev/./urandom -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9011 -Dcom.sun.management.jmxremote.rmi.port=9011 -Dcom.sun.management.jmxremote.local._only_=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
        javaOptions = -XX:+UnlockExperimentalVMOptions
        javaOptions = -XX:+UseG1GC
        javaOptions = -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/home/gerrit/gerrit-sshd/logs/jvmgc_log
        javaOptions = -Xms100g
        javaOptions = -Xmx100g
        javaOptions = -XX:G1NewSizePercent=35
        javaOptions = -XX:MaxGCPauseMillis=500
        javaOptions = -XX:ParallelGCThreads=10
        javaOptions = -XX:ConcGCThreads=4
        javaOptions = -XX:InitiatingHeapOccupancyPercent=50
        javaOptions = -XX:G1HeapWastePercent=2
  • Do you run git gc using native git or JGit (embedded in Gerrit or standalone) ?
JGit, embedded in Gerrit

 
P.S. The traffic is serialised so that we have the certainty that threads are done with their jobs when the next operation starts. 

The scenario is quite simple: just generate a constant traffic of continuous clones and JGit GC and, after a few hours, the JVM id completely dead.
By using a G1GC and tuning it, I managed to avoid almost all the STW GC cycles. However, the memory utilised increases continuously and at the end al the GC goes into an infinite loop trying to release something without being able to do so.

Sample of the GC log when the JVM is collapsed:
2019-06-11T08:15:27.340+0200: 28506.935: [Full GC (Allocation Failure)  90G->89G(100G), 106.4610300 secs]
   [Eden: 0.0B(35.0G)->0.0B(35.0G) Survivors: 0.0B->0.0B Heap: 90.5G(100.0G)->89.2G(100.0G)], [Metaspace: 61559K->61559K(65536K)]

And the JVM heap utilization:
<Screenshot 2019-06-11 at 07.18.59.png>

With regards to the memory dump, see the following top offenders:

<Screenshot 2019-06-11 at 07.22.05.png>

I believe there are problems in releasing PackWriters, I do see 8k of them allocated :-O

From the code I see that *IF* PackWriter.close() isn't called, then the instance is pointed by a static Map:
private static final Map<WeakReference<PackWriter>, Boolean> instances =
      new ConcurrentHashMap<>();
The map contains a WeakReference which it should, in theory, recalled by the JVM eventually ... but it seems that it doesn't do it as expected.
PackWriter contains pointers to many objects, including packfiles that are generated "on-the-fly" to serve client Git requests.
 


I have explicitly set low limits for caching packfiles in the gerrit.config:

[core]
   packedGitLimit = 10m
   packedGitOpenFiles = 16000
   packedGitWindowSize = 64k

In theory, JGit shouldn't hold huge amount of memory for packfiles and instead continuously read them from the filesystem. However, as you can see from the above figures, we have over 2M byte arrays holding over 22GBytes of memory.

I also have a ~/.gitconfig with a specific [pack] configuration:

[pack]
        maxDeltaDepth=5
        deltaSearchWindowSize=10
        deltaSearchMemoryLimit=0
        deltaCacheSize=52428800

this means deltaCacheSize is  5 times larger than the JGit file cache, does that make sense ?
 
        deltaCacheLimit=100
        compressionLevel=-1
        indexVersion=2
        bigFileThreshold=52428800
        threads=1
        reuseDeltas=true
        reuseObjects=true
        deltaCompress=false
        buildBitmaps=true

What sounds strange to me is the high number of LocalObjectToPack that are held in memory (9M of objects): the ObjectToPack should be only a temporary object used by a PackWriter, but I am not a super-expert of that part of code.

do these repositories contain large files ?

Yes, Android has some large files.

 
Rings any bell to anyone?

Thanks for the feedback.
Luca.



--
--
To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/A0E50AE5-874F-47E6-BCFE-56A58FA5E1D3%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Back to the top