Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Memory Analyzer » Huge heap dump, but no content?(Huge JVM heap dump but MAT shows not content)
Huge heap dump, but no content? [message #660729] Mon, 21 March 2011 12:30 Go to next message
Beppino604  is currently offline Beppino604 Friend
Messages: 4
Registered: March 2011
Junior Member
Hi,

On one of our production boxes (Solaris 64bit, 1.5.0_22-b03) we suspect to have a memory leak in the application. After 2-3 weeks of uptime JBoss becomes unresponsive as the GC can't effectively clean up the old generation anymore.

I have manually pulled a JVM heap dump with jmap when I saw that the OldGen had grown to > 7GB. The file is 9,5GB in size.

When I open the file in MAT it says in the overview panel that the size is only 342kb and it doesn't contain any meaningful information. Here's a screenshot:

http://i52.tinypic.com/343j53o.jpg

This was the second time I pulled a heap dump like this. The first time I used Solaris gcore to pull a core dump and then extracted the heap offline with jmap (to reduce the downtime). The second time (screenshot) I attached jmap directly to the running Java process.

Any ideas on what might be causing this? I can't believe that this huge file only contains 342kb worth of objects.

Thanks in advance,
Beppino
Re: Huge heap dump, but no content? [message #660910 is a reply to message #660729] Tue, 22 March 2011 10:16 Go to previous messageGo to next message
Jonathan Lawrence is currently offline Jonathan LawrenceFriend
Messages: 16
Registered: July 2009
Junior Member
I have recently heard from other users that they have experienced similar issues with heap dumps from Solaris JVMs, so maybe there is a problem with the hprof parser reading Solaris heap dumps.

This warrants further investigation. If necessary would it be feasible for you to provide the dump for analysis?
Re: Huge heap dump, but no content? [message #661244 is a reply to message #660729] Wed, 23 March 2011 16:46 Go to previous messageGo to next message
Beppino604  is currently offline Beppino604 Friend
Messages: 4
Registered: March 2011
Junior Member
Thanks for your feedback Jonathan. Unfortunately it's not possible for me to provide the dump as it potentially contains sensitive live data.

However, I've tried moving our application code away and restarted JBoss, then pulled a heap dump which was only 88MB in size. I was able to open it and view the relevant information in MAT without problems so I suspect that this is somehow related to the size of the dump rather than the OS. Or it could be a combination of the two.

Tomorrow I'll try something else on our PreProduction environment (same OS/Java version). The plan is to configure it with the same memory sizes (-Xmx14336m -Xmn2500M) and then forcefully fill the heap to a similar size as it had on Prod when I pulled the dump.

If it fails then I can at least provide that heap dump. Does that sound reasonable?
Re: Huge heap dump, but no content? [message #661330 is a reply to message #660729] Thu, 24 March 2011 09:46 Go to previous messageGo to next message
Andrew Johnson is currently offline Andrew JohnsonFriend
Messages: 142
Registered: July 2009
Senior Member
What does the 'Unreachable Objects Histogram' look like?

Is the problem that there are hardly any objects at all in the dump, or that nearly all the objects appeared unreachable so were discarded by MAT?
Re: Huge heap dump, but no content? [message #661431 is a reply to message #661330] Thu, 24 March 2011 16:18 Go to previous messageGo to next message
Beppino604  is currently offline Beppino604 Friend
Messages: 4
Registered: March 2011
Junior Member
The problem seems to be that there are hardly any objects in the dump. There was no apparent difference when I unticked the option to remove all unreachable objects.

Unfortunately I didn't have time to perform the test on PreProd yet as other Prod issues came up. I expect to be able to do that on Monday. If you want to see specific MAT screens of the huge heap dump just let me know and I will post them.
Re: Huge heap dump, but no content? [message #661858 is a reply to message #660729] Mon, 28 March 2011 10:07 Go to previous messageGo to next message
Beppino604  is currently offline Beppino604 Friend
Messages: 4
Registered: March 2011
Junior Member
At the risk of moving to off-topic I'm posting some related information here.

I've had a look at the "Unreachable Objects Histogram" and apparently it shows more than I had remembered. Here's a screenshot:

http://i51.tinypic.com/2i8954k.jpg

If I understand correctly these objects are waiting for the GC to pick them up. I'm not sure how to extract the information on what these objects are in MAT. Is this even possible for unreachable objects?

Here's a graph showing the utlization of the Old Gen where you can see that the heap keeps growing (I restarted JBoss on Friday) and at some point the CMS GC simply can't manage to clean it up anymore:

http://i51.tinypic.com/mj78nl.jpg

I manually trigger a Full GC every night at 3am. Looking at the GC log a Full GC can take up to 90 seconds (!) at this stage. That's why I want to find out what causes the heap to fill up.

Two more screenshots of the visualized GC activity:

Heap usage:
http://i52.tinypic.com/2u6dx79.jpg

GC duration:
http://i54.tinypic.com/11r6i5d.jpg

In the end the CMS GC can't keep up anymore and starts being executed very often. This happens after a week of runtime. I hope this information helps to understand the problem and why I need to find out what's in the heap as I strongly suspect that we are leaking memory somewhere.
Re: Huge heap dump, but no content? [message #1793967 is a reply to message #661858] Tue, 21 August 2018 10:34 Go to previous message
sasmita dash is currently offline sasmita dashFriend
Messages: 1
Registered: August 2018
Junior Member
Hi,

I am facing exactly similar issue for my application.
Finally any solution you got to fix the issue?

If yes please let us know. It would be very helpful.
Previous Topic:Memory Analyzer 1.8.0 was released!
Next Topic:MAT report with all rows from query (more than 25)
Goto Forum:
  


Current Time: Sat Nov 17 19:06:09 GMT 2018

Powered by FUDForum. Page generated in 0.02497 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top