Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » Memory Analyzer (MAT) » Any known issues with analyzing huge heap dumps?
Any known issues with analyzing huge heap dumps? [message #2628] Tue, 19 August 2008 21:06 Go to next message
Eclipse UserFriend
Originally posted by: danwork.dshunter.co.uk

Hi,
We're currently having problems with a java process that occupies a 12GB
of heap, using Java 1.5.0_12 on a 64 bit machine. We are getting
unexplained OOM errors after around 24 hours of use, something we have
been unable to replicate in test.

I've got a unix core dump (~13GB) from the misbehaving process and I've
created an hprof file (~9GB) from that to analyse for any potential memory
leaks etc.

I've already created a histogram of the the core using jmap but
unfortunately, as this is Java 5, it includes non-live objects. It clearly
shows the heap maxed out. I've had a go with JHat, but haven't really
found it very useful for locating memory leaks.

However, memory analyzer reports that only 12MB (megabytes) of the heap is
being used *in total*. There doesn't look like there is anything missing
(our service only requires around 8MB when it isn't busy) so it looks to
me like the garbage collection isn't working properly. I'd like to push
for an upgrade to the JVM if this is the case.

Is this a reliable analysis or does memory analyzer have problems
analysing large heaps? I should point out that I am having to run it in a
32bit jvm (due to an unrelated problem with Eclipse) but I don't get any
error messages etc.

Cheers,
Daniel
Re: Any known issues with analyzing huge heap dumps? [message #2656 is a reply to message #2628] Wed, 20 August 2008 11:52 Go to previous messageGo to next message
Andreas Buchen is currently offline Andreas BuchenFriend
Messages: 113
Registered: July 2009
Senior Member
Hi Daniel,

glad to hear that you are using the Memory Analyzer, sorry to hear that
you run into problems...

First: Analyzing 64bit dumps on 32bit machines is perfectly fine. We often
do this ourselves. The only restriction is that you have less memory (~1.2
GB) available for analysis. Sometimes is helps to parse the heap dump
(happens when the dump is opened for the first time) on 64bit. Then
continue analysis on 32bit.

Second: When reading the initial dump, we are removing objects which are
not reachable. These are usually "left-overs" in the form of byte[] arrays
which the Garbage Collector leaves hanging around for performance reasons
(e.g. avoid moving objects and therefor avoid assigning new addresses).
This is the reason, why the reported size is usually smaller than the file
size or the initially reported number of objects.

But, of course, removing 9 GB and leaving only 12 MB sounds strange. So I
don't want to rule out an issue with MAT.

To isolate the problem:
a) We print to the log file (during parsing) the initial number of objects
found. What does this number say? And what does it say on the first page
when the heap dump is opened?
b) You say, you were able to read the heap dump with JHat. What is the
number of objects reported there? (Because JHat usually opens only smaller
heap dumps, this could indicate either really big primitive arrays (e.g.
few objects) or only few objects.)
c) How did you generate the heap dump? While jmap with VM 1.5 is sometimes
unreliable (e.g. corrupted dumps for no reason), the
+XX:+HeapDumpOnOutOfMemoryError option usually produces readable dumps.
d) What operating system are you using?

Andreas.
Re: Any known issues with analyzing huge heap dumps? [message #3355 is a reply to message #2656] Tue, 26 August 2008 16:04 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: danwork.dshunter.co.uk

Hi Andreas,

Just to give you a few updates, we upgraded to JVM 1.6 to see if this
eradicated the problem. Unfortunately it didn't solve the problem but it
did create a more realistic heap to analyse.

From a 6 gig heap OOM, memory analyzer showed a 1 gig live heap. This
allowed us to locate a minor memory leak, but there is still a 5 gig
discrepancy that makes me a little nervous that we are missing the root
cause.

The answers to your questions:

a) The log file reports 14,722,946 and the first page reports 14.1M
b) JHat lists 14722945
c) I believe we did a kill -6 on the process... not ideal but the process
would often do a GC death spiral rather than thrown an OOM. Unfortunately,
this process is pretty mission critical to what we do so we can't wait
around for it to die 'gracefully'. However, I don't think the JVM has
entered an unhealthy state as I have seen similar results on healthy
versions of the process.
d) Red Hat AS3 to run Memory Analyzer and AS4 for the java process.

Thanks for your help,
Daniel
Re: Any known issues with analyzing huge heap dumps? [message #3556 is a reply to message #3355] Wed, 27 August 2008 16:17 Go to previous message
Andreas Buchen is currently offline Andreas BuchenFriend
Messages: 113
Registered: July 2009
Senior Member
Hi Daniel!

> a) The log file reports 14,722,946 [...]
> b) JHat lists 14722945

The one object difference is expected. We create a pseudo object for the
system class loader (which is not included in HPROF) because it makes the
analysis consistent.

> a) [...] and the first page reports 14.1M

As explained earlier, we remove unreachable objects because the various
garbage collector algorithms tend to leave some garbage behind (if the
object is too small, moving and re-assigning addresses is to expensive).
In this particular case about 4%... Not necessarily a problem.

> c) I believe we did a kill -6 on the process...

Okay, I don't know this option. Usually, I use JConsole and/or jps/jmap to
trigger the heap dump.


Let's try to figure out what the 4% are and what size they have. I will
develop patch, that will print out in detail what objects are not
reachable. I created this bugzilla entry to track this debugging feature:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=245410 Maybe add yourself to
the CC list.

Andreas.
Previous Topic:Namespace reporting is not ebavled
Next Topic:IllegalArgumentException In OQL Statement
Goto Forum:
  


Current Time: Fri Nov 21 02:26:28 GMT 2014

Powered by FUDForum. Page generated in 0.01917 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software