Eclipse Community Forums: Memory Analyzer » Huge heap dump, but no content?

Home » Eclipse Projects » Memory Analyzer » Huge heap dump, but no content?(Huge JVM heap dump but MAT shows not content)

Show: Today's Messages :: Show Polls :: Message Navigator

Huge heap dump, but no content? [message #660729]

Mon, 21 March 2011 12:30

Beppino604

Messages: 4
Registered: March 2011

Junior Member

Hi,

On one of our production boxes (Solaris 64bit, 1.5.0_22-b03) we suspect to have a memory leak in the application. After 2-3 weeks of uptime JBoss becomes unresponsive as the GC can't effectively clean up the old generation anymore.

I have manually pulled a JVM heap dump with jmap when I saw that the OldGen had grown to > 7GB. The file is 9,5GB in size.

When I open the file in MAT it says in the overview panel that the size is only 342kb and it doesn't contain any meaningful information. Here's a screenshot:

http://i52.tinypic.com/343j53o.jpg

This was the second time I pulled a heap dump like this. The first time I used Solaris gcore to pull a core dump and then extracted the heap offline with jmap (to reduce the downtime). The second time (screenshot) I attached jmap directly to the running Java process.

Any ideas on what might be causing this? I can't believe that this huge file only contains 342kb worth of objects.

Thanks in advance,
Beppino

Report message to a moderator

Re: Huge heap dump, but no content? [message #660910 is a reply to message #660729]

Tue, 22 March 2011 10:16

Jonathan Lawrence

Messages: 16
Registered: July 2009

Junior Member

I have recently heard from other users that they have experienced similar issues with heap dumps from Solaris JVMs, so maybe there is a problem with the hprof parser reading Solaris heap dumps.

This warrants further investigation. If necessary would it be feasible for you to provide the dump for analysis?

Report message to a moderator

Re: Huge heap dump, but no content? [message #661244 is a reply to message #660729]

Wed, 23 March 2011 16:46

Beppino604

Messages: 4
Registered: March 2011

Junior Member

Thanks for your feedback Jonathan. Unfortunately it's not possible for me to provide the dump as it potentially contains sensitive live data.

However, I've tried moving our application code away and restarted JBoss, then pulled a heap dump which was only 88MB in size. I was able to open it and view the relevant information in MAT without problems so I suspect that this is somehow related to the size of the dump rather than the OS. Or it could be a combination of the two.

Tomorrow I'll try something else on our PreProduction environment (same OS/Java version). The plan is to configure it with the same memory sizes (-Xmx14336m -Xmn2500M) and then forcefully fill the heap to a similar size as it had on Prod when I pulled the dump.

If it fails then I can at least provide that heap dump. Does that sound reasonable?

Report message to a moderator

Re: Huge heap dump, but no content? [message #661330 is a reply to message #660729]

Thu, 24 March 2011 09:46

Andrew Johnson

Messages: 205
Registered: July 2009

Senior Member

What does the 'Unreachable Objects Histogram' look like?

Is the problem that there are hardly any objects at all in the dump, or that nearly all the objects appeared unreachable so were discarded by MAT?

Report message to a moderator

Re: Huge heap dump, but no content? [message #661431 is a reply to message #661330]

Thu, 24 March 2011 16:18

Beppino604

Messages: 4
Registered: March 2011

Junior Member

The problem seems to be that there are hardly any objects in the dump. There was no apparent difference when I unticked the option to remove all unreachable objects.

Unfortunately I didn't have time to perform the test on PreProd yet as other Prod issues came up. I expect to be able to do that on Monday. If you want to see specific MAT screens of the huge heap dump just let me know and I will post them.

Report message to a moderator

Re: Huge heap dump, but no content? [message #661858 is a reply to message #660729]

Mon, 28 March 2011 10:07

Beppino604

Messages: 4
Registered: March 2011

Junior Member

At the risk of moving to off-topic I'm posting some related information here.

I've had a look at the "Unreachable Objects Histogram" and apparently it shows more than I had remembered. Here's a screenshot:

http://i51.tinypic.com/2i8954k.jpg

If I understand correctly these objects are waiting for the GC to pick them up. I'm not sure how to extract the information on what these objects are in MAT. Is this even possible for unreachable objects?

Here's a graph showing the utlization of the Old Gen where you can see that the heap keeps growing (I restarted JBoss on Friday) and at some point the CMS GC simply can't manage to clean it up anymore:

http://i51.tinypic.com/mj78nl.jpg

I manually trigger a Full GC every night at 3am. Looking at the GC log a Full GC can take up to 90 seconds (!) at this stage. That's why I want to find out what causes the heap to fill up.

Two more screenshots of the visualized GC activity:

Heap usage:
http://i52.tinypic.com/2u6dx79.jpg

GC duration:
http://i54.tinypic.com/11r6i5d.jpg

In the end the CMS GC can't keep up anymore and starts being executed very often. This happens after a week of runtime. I hope this information helps to understand the problem and why I need to find out what's in the heap as I strongly suspect that we are leaking memory somewhere.

Report message to a moderator

Re: Huge heap dump, but no content? [message #1793967 is a reply to message #661858]

Tue, 21 August 2018 10:34

sasmita dash

Messages: 1
Registered: August 2018

Junior Member

Hi,

I am facing exactly similar issue for my application.
Finally any solution you got to fix the issue?

If yes please let us know. It would be very helpful.

Report message to a moderator

Previous Topic:	Memory Analyzer 1.8.0 was released!
Next Topic:	MAT report with all rows from query (more than 25)

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Apr 18 08:32:29 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter