Eclipse Community Forums: Memory Analyzer » How to find what is pinning the app classloader with MAT?

Help

Home

Home » Eclipse Projects » Memory Analyzer » How to find what is pinning the app classloader with MAT?

Show: Today's Messages :: Show Polls :: Message Navigator

How to find what is pinning the app classloader with MAT? [message #658678]

Wed, 09 March 2011 13:09

Jon Seymour

Messages: 5
Registered: March 2011

Junior Member

The situation I am trying to debug involves the application classloader being pinned by something in a parent classloader so that when the application is restarted, most of the resources associated with the original instance of the application are not being recovered.

I have checked suspicious thread locals, but all the thread locals I have examined appear to be sound - if the threadlocal became weakly reachable, the resources would be recovered.

What I would like ideally is an analysis which shows the paths from the roots to any class in my application's package namespace.

Is this something that can be easily expressed in MAT?

[ My debugging environment is an IBM Java 5 JVM running on 32 bit Windows - WebSphere 6.1.0.27 ]

Report message to a moderator

Re: How to find what is pinning the app classloader with MAT? [message #658925 is a reply to message #658678]

Thu, 10 March 2011 13:20

Krum Tsvetkov

Messages: 164
Registered: July 2009

Senior Member

Hi,

Usually when I track such kind of problems it is enough to examine the paths from the GC roots to the ClassLoader.
We model a reference from each object instance to its class (named <class>), and we also model a reference from each class to its class loader (named <classloader>).
So if the reason for the class loader not being reclaimed is that an instance of a class loaded with it is kept somewhere, you should see a path like:

MyClassLoader 
  <--  <classloader> class MyClass
    <--  <class> MyClass
       <-- SomeOtherObject
           etc...

You mentioned ThreadLocals and if objects become weakly reachable. Yesterday I analyzed two problems where thread locals were keeping classloaders.
In both (different) problems the reason was that the "value" of the ThreadLocalMap$Entry was referencing (throgh some reference chain) the "referent" field of the ThreadLocalMap$Entry .
This basically takes the weakness away.
This is a common problem I've seen with weak hash maps, so it may makes sense for you to open the ThreadLocalMap$Entry and explore if the value can lead you back to the referent. Unfortunatelly we don't offer yet the possibility to search for paths between two arbitrary objects, so you'll have to follow the references manually.

I hope this helps.

Krum

Report message to a moderator

Re: How to find what is pinning the app classloader with MAT? [message #658983 is a reply to message #658678]

Thu, 10 March 2011 15:48

Jon Seymour

Messages: 5
Registered: March 2011

Junior Member

Thanks for your help.

I tried the merge shortest path to GC roots with a package and wildcard and the -groupby FROM_OBJECTS_BY_CLASS. I also found the the -exclude option was very useful for excluding known pins like container provided caches.

Using MAT with heapdump and javacore files, I got what looked to be some promising leads, but I still had a 2000+ fanout from the top level to look at.

However, following one of these paths to the end I discovered for example, that there was an instance of com.ibm.rmi.transport.ReaderThread that was holding onto the app classloader via its contextClassLoader field. I managed (using some brutal hacking with netcat) to kill these threads hoping that this would release the classloader. No luck.

I also found a WeakHashMap in the java.lang.reflect.Proxy class which needs to be polled in order for its entries to be cleaned. However, causing this to happen didn't help.

However, I found that once I used a system dump, I got a much better result. When I did this, in addition to get field names etc, I got a much smaller fanout from the objects back to the roots [ 7 paths from the top-level ].

Furthermore, I discovered that the app classloader is also being pinned by hash table in the Apache commons logging implementation that is loaded by a system class loader, so this has given me something else to try.

Report message to a moderator

Re: How to find what is pinning the app classloader with MAT? [message #659347 is a reply to message #658678]

Sat, 12 March 2011 16:07

Jon Seymour

Messages: 5
Registered: March 2011

Junior Member

Ok.

It turns out my problem was a combination of 2 things:

- a bug in the JVM (sun bug 6232010) that was fixed by upgrading the WebSphere JVM to SR10.

- the app classloader being held by a hashtable in Apache Commons Logging LogFactory which was inherited from a system classloader.

My app also uses a lot of proxy classes and hence temporary classloaders so the JVM needs some stress to purge all of these and only then does the app classloader get released.

Separately, in another JVM I have discovered a pinning issue related to the way Spring's WeakReferenceMonitor interacts with its WebSphereUowTransactionManager. The basic problem is that the WebSphereUowTransactionManager ends up pinning the EJB with a strong reference, preventing the WeakReferenceMonitor from terminating, thereby pinning the app classloader.

Anyway, MAT was a great help to me in solving this problem, so thanks for the great tool, and your suggestions.

jon.

Report message to a moderator