Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » Memory Analyzer (MAT) » Performing query within dominator subgraph
Performing query within dominator subgraph [message #5857] Tue, 28 October 2008 23:37 Go to next message
Taras Tielkes is currently offline Taras TielkesFriend
Messages: 38
Registered: July 2009
Member
Hi,

I'm looking at the dominator tree for a given snapshot.
Most of the top-10 dominators are Tomcat request processing threads.

For each of these threads, I'd like to recover the URL parameters passed
by the client.

It happens to be that this information is available.
The subgraph to access it looks something like this:

org.apache.coyote.RequestInfo (stack local reference)
req (org.apache.coyote.Request)
queryMB (org.apache.tomcat.util.buf.MessageBytes)
strValue (java.lang.String)

What I'm after is this:
1) Can I somehow launch a query that extracts this information from the
subgraph associated with a top-level (Thread) node of the dominators list?

That way I could re-run this query for each of the "big" dominator threads.

2) Even better: can I somehow perform a query that
a) for each org.apache.tomcat.util.threads.ThreadWithAttributes in the
snapshot
b) finds the retained size
c) finds the URL (using above graph navigation path) within the retained
set of that thread
d) displays both in a convenient 2-column table format
e) sorted by retained size

Thanks in advance for any pointers,

Taras
Re: Performing query within dominator subgraph [message #5912 is a reply to message #5857] Wed, 29 October 2008 09:59 Go to previous messageGo to next message
Andreas Buchen is currently offline Andreas BuchenFriend
Messages: 113
Registered: July 2009
Senior Member
> org.apache.coyote.RequestInfo (stack local reference)
> req (org.apache.coyote.Request)
> queryMB (org.apache.tomcat.util.buf.MessageBytes)
> strValue (java.lang.String)

To extract this kind of information, you could say:

SELECT toHex(info.getObjectAddress()),
info.req.queryMB,
info.req.queryMB.byteC.buff.toString(),
info.req.queryMB.strValue.toString(),
info.getRetainedHeapSize()
FROM org.apache.coyote.RequestInfo info


Not easily possible (i.e. without actually coding) is combining this
information with the thread. The <Java Local> references are not easy to
navigate.

Maybe this helps a bit. This query extracts those RequestInfo objects
which are actually as java locals on one of the threads (assuming that you
could have many more info objects which are not currently on the call
stack). I hope this at least reduces the number of objects.

SELECT * FROM OBJECTS ( SELECT OBJECTS
${snapshot}.getOutboundReferentIds(thread.getObjectId()) FROM
org.apache.tomcat.util.threads.ThreadWithAttributes thread ) out WHERE
(classof(out).getName()="org.apache.coyote.RequestInfo")


Andreas.


PS... yeah, it's not intuitive...
Re: Performing query within dominator subgraph [message #5934 is a reply to message #5912] Wed, 29 October 2008 10:32 Go to previous messageGo to next message
Taras Tielkes is currently offline Taras TielkesFriend
Messages: 38
Registered: July 2009
Member
Hi Andreas,

Indeed my main problem is "joining" the ThreadWithAttributes data to the
RequestInfo data.

My first idea was to try to query from ThreadWithAttributes, and then try
to retrieve the correlated RequestInfo through the locals of the thread,
but I could not figure out how.

The RequestInfo should be part of the set that is "immediately dominated"
by the thread object, but I can't figure out how to reach the
"thread-specific" RequestInfo through that.

I think my problem is not very unique, by the way..
A heap dump from a servlet container/application server/etc contains a
number of "live" (processing) threads. Now we want to know the retained
set for each, along with some request-specific attributes that can be
found in the retained set of each thread.

If I cannot easily do this by OQL, can I at least somehow execute an OQL
query on the retained set of a node in the dominator tree? That was I
could manually collect the "(thread retained size -> request URL)" tuples.
Re: Performing query within dominator subgraph [message #5952 is a reply to message #5934] Wed, 29 October 2008 12:33 Go to previous messageGo to next message
Andreas Buchen is currently offline Andreas BuchenFriend
Messages: 113
Registered: July 2009
Senior Member
> Can I at least somehow execute an OQL
> query on the retained set of a node in the dominator tree? That was I
> could manually collect the "(thread retained size -> request URL)" tuples.

This gives you the retained set of one object:

SELECT AS RETAINED SET * FROM OBJECTS 0x39fca508


Then you can reduce this the RequestInfo objects. This will select from
the retained set only the objects of type RequestInfo:

SELECT * FROM OBJECTS (
SELECT AS RETAINED SET * FROM OBJECTS 0x39fca508
) info
WHERE classof(info).getName() = "org.apache.coyote.RequestInfo"


And now you print the URL by changing the SELECT part of the query:

SELECT info.req.queryMB.byteC.buff.toString() FROM ...



If you want to do this for all threads, then change the FROM clause of the
nested select clause:

SELECT AS RETAINED SET * FROM
org.apache.tomcat.util.threads.ThreadWithAttributes



Andreas.
Re: Performing query within dominator subgraph [message #6002 is a reply to message #5952] Thu, 30 October 2008 06:17 Go to previous messageGo to next message
Taras Tielkes is currently offline Taras TielkesFriend
Messages: 38
Registered: July 2009
Member
Hi Andreas,

How hard would it be for me to use relevant components of MAT in a
headless fashion in order to accomplish what I want?

I know this is a very broad question, so let me specify in more detail:
1) Load a dump into memory
a) performing indexing if necessary
2) Find the set of all instances of some specific thread class
3) Calculate retained size for each of these threads
4) Within the retained set of each thread, find some specific objects
5) Extract some primitive values from them

I assume that at some point MAT does a bit of transformation by
modifying stack local roots into objects referenced by the actual
thread. Does this happen during the actual parsing?

I realize this request falls outside of the typical usage pattern of the
tool, however I would be very grateful for some source code pointers or
samples to give above idea a try.

I assume the dependencies to do what I want are minimal, hopefully I can
just use some of the "org.eclipse.mat.*" jars without starting an OSGi
framework.

Regards,

Taras
Re: Performing query within dominator subgraph [message #6411 is a reply to message #6002] Thu, 30 October 2008 08:53 Go to previous messageGo to next message
Andreas Buchen is currently offline Andreas BuchenFriend
Messages: 113
Registered: July 2009
Senior Member
> How hard would it be for me to use relevant components of MAT in a
> headless fashion in order to accomplish what I want?

It depends... if you have some experience with Eclipse, it should not be
too hard.

> I assume the dependencies to do what I want are minimal, hopefully I can
> just use some of the "org.eclipse.mat.*" jars without starting an OSGi
> framework.

Unfortunately, that won't work. You need to go through the OSGi framework.


Okay, this is what you need to do:

* Install the latest Memory Analyzer stand-alone into some directory
* Install Eclipse Ganymede Java IDE
* Open Eclipse
* Choose Window -> Preferences -> Plug-In Developement -> Target Platform
and point the location to your mat directory
(This will read all bundles in the installation)
* Create a new plug-in: File -> New ... -> Plug-In Project
* In the manifest.mf Editor, choose the tab "Dependencies" and select
org.eclipse.mat.api
* Create a plugin.xml in the root of the project:

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.2"?>
<plugin>
<extension point="org.eclipse.mat.report.query">
<query impl="org.eclipse.mat.demo.DemoQuery"/>
</extension>
</plugin>


* Create your DemoQuery class in the src Folder:


package org.eclipse.mat.demo;

import java.text.MessageFormat;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;

import org.eclipse.mat.SnapshotException;
import org.eclipse.mat.query.IQuery;
import org.eclipse.mat.query.IResult;
import org.eclipse.mat.query.annotations.Argument;
import org.eclipse.mat.query.annotations.Name;
import org.eclipse.mat.query.results.ListResult;
import org.eclipse.mat.snapshot.ISnapshot;
import org.eclipse.mat.snapshot.model.IClass;
import org.eclipse.mat.snapshot.model.IObject;
import org.eclipse.mat.util.IProgressListener;

@Name("Demo Query")
public class DemoQuery implements IQuery {

@Argument
public ISnapshot snapshot;

@Argument
public String threadClass =
"org.apache.tomcat.util.threads.ThreadWithAttributes";

@Argument
public String requestInfoClass = "org.apache.coyote.RequestInfo";

@Argument
public String field = "req.queryMB.byteC.buff";

public IResult execute(IProgressListener listener) throws Exception {

List<ResultItem> result = new ArrayList<ResultItem>();

// lookup all thread classes
Collection<IClass> threads = snapshot.getClassesByName(threadClass,
true);
if (threads == null)
throw new SnapshotException(MessageFormat.format(
"Heap Dumps contains no class {0}", threadClass));

// lookup the request info class
Collection<IClass> requestInfoClasses = snapshot.getClassesByName(
requestInfoClass, false);
if (requestInfoClasses == null || requestInfoClasses.isEmpty())
throw new SnapshotException(MessageFormat.format(
"Heap Dumps contains no class {0}", requestInfoClass));
IClass requestInfo = requestInfoClasses.iterator().next();

for (IClass thread : threads) {
for (int threadInstanceId : thread.getObjectIds()) {

IObject threadInstanceObj = snapshot
.getObject(threadInstanceId);

// Calculate retained size for each of these threads
int[] retainedSet = snapshot.getRetainedSet(
new int[] { threadInstanceId }, listener);

// Within the retained set of each thread, find some specific
// objects
for (int retainedId : retainedSet) {
if (snapshot.getClassOf(retainedId).equals(requestInfo)) {

IObject requestInfoObj = snapshot.getObject(retainedId);

String value = null;

IObject buffer = (IObject) requestInfoObj
.resolveValue(field);
if (buffer != null)
value = buffer.getClassSpecificName();

result.add(new ResultItem(threadInstanceObj,
requestInfoObj, value));
}
}
}
}

return new ListResult(ResultItem.class, result, "thread", "request",
"value");
}

public static class ResultItem {
IObject thread;
IObject request;

String value;

public ResultItem(IObject thread, IObject request, String value) {
super();
this.thread = thread;
this.request = request;
this.value = value;
}

public IObject getThread() {
return thread;
}

public IObject getRequest() {
return request;
}

public String getValue() {
return value;
}

}
}


* Right-click on your plug-in project, "Run As..." -> "Eclipse
Application".

* When you open a heap dump, your query should be in the query drop-down
list.



Now, this is a very short explanation. If I was too brief on some of the
steps, do not hesitate to send me an email.
Re: Performing query within dominator subgraph [message #6430 is a reply to message #6411] Thu, 30 October 2008 09:01 Go to previous messageGo to next message
Taras Tielkes is currently offline Taras TielkesFriend
Messages: 38
Registered: July 2009
Member
Hi Andreas,

Thank you very much for the (unexpectedly!) detailed response.
Your answer greatly exceeds what I hoped for :)

I'll let you know how far I get. Since I use IntelliJ for Java
development, it might take a while :)

Regards,
Taras
Re: Performing query within dominator subgraph [message #6467 is a reply to message #6411] Thu, 30 October 2008 10:43 Go to previous messageGo to next message
Taras Tielkes is currently offline Taras TielkesFriend
Messages: 38
Registered: July 2009
Member
Hi Andreas,

I'd like to run this custom report on a collection (40+) of heap dumps "in
batch mode". Is this possible?

Taras
Re: Performing query within dominator subgraph [message #6485 is a reply to message #6467] Thu, 30 October 2008 11:58 Go to previous messageGo to next message
Andreas Buchen is currently offline Andreas BuchenFriend
Messages: 113
Registered: July 2009
Senior Member
> I'd like to run this custom report on a collection (40+) of heap dumps "in
> batch mode". Is this possible?

Kind of. You will need some script wrapper around...


(i) Add this annotation to your query (next to name):

@CommandName("demo_query")


(ii) Add this extension to the plugin.xml:

<extension point="org.eclipse.mat.report.report">
<report id="requests" name="Request Info"
description="extract request info"
file="META-INF/reports/requests.xml" />
</extension>


(iii) Create the META-INF/reports/requests.xml :

<section name="Request Info">
<query name="Requests">
<command>demo_query</command>
<param key="format" value="csv" />
</query>
</section>


(iv) Test the report by running the application and choosing the report
from the drop down list.

(v) Export the bundle:
(1) open the manifest.mf editor
(2) open the "Overview" tab
(3) Select "Export Wizard" from "Exporting" section
(4) In the wizard create an archive file

(vi) Install the bundle into your MAT installation (copy to plugins)
directory

(vii) Run the report

ParseHeapDump.bat <dump> org.eclipse.mat.demo:requests

This creates a zip file containing (among other things) the CSV file.


So what you want to do:
foreach dump
run report
extract csv file




I hope this helps. I realize that there is a need to make this easier....
Re: Performing query within dominator subgraph [message #6595 is a reply to message #6411] Tue, 04 November 2008 08:41 Go to previous messageGo to next message
Taras Tielkes is currently offline Taras TielkesFriend
Messages: 38
Registered: July 2009
Member
Hi Andreas,

I'm happy to report that I succeeded :)
It turned out that the RequestInfo was not part of the retained set of a
worker thread. However, since it was a local variable of the thread I
reached it through threadInstanceObj.getOutboundReferences().

I didn't try the "batch mode" solution yet, but will certainly do so
over the next few days, and will let you know how far I get.

I might try to pull out a few more diagnostics based on my internal
knowledge of the application. It's a pity that a heap dump does not
include the equivalent of a thread dump - that would have made analysis
of what a thread is doing (somewhat) straightforward. Instead I'll try
to use the presence/absence of some specific local variables from a
thread to summarize its current state.

Quick questions for now:
1) How do I access the set of stack local variables given a reference to
some thread?
2) I assume I can expect to find IClass objects with the same name
referring to different versions of the same class when browsing a
snapshot, correct? For example, when the same class was loaded more than
once through different class loaders.
3) How does the dominators view calculate the percentage? Is it just
IObject.getRetainedHeapSize() divided by ISnapshot.getHeapSize(<what to
pass here?>)?
4) Does the source code for org.eclipse.mat.snapshot.* contain JavaDoc?
If so, it might make sense to download it and start exploring. Where can
I actually get the source? Or perhaps there's a JavaDoc .zip available
somewhere?

Thank you again for the detailed assistance you provided. Apart from my
own typos and mistakes the process was seamless.

I'm still pretty amazed by looking at my custom query running and
pulling out information from the snapshot :)

Regards,

Taras

Andreas Buchen wrote:
>> How hard would it be for me to use relevant components of MAT in a
>> headless fashion in order to accomplish what I want?
>
> It depends... if you have some experience with Eclipse, it should not be
> too hard.
>
>> I assume the dependencies to do what I want are minimal, hopefully I
>> can just use some of the "org.eclipse.mat.*" jars without starting an
>> OSGi framework.
>
> Unfortunately, that won't work. You need to go through the OSGi framework.
>
>
> Okay, this is what you need to do:
>
> * Install the latest Memory Analyzer stand-alone into some directory
> * Install Eclipse Ganymede Java IDE
> * Open Eclipse
> * Choose Window -> Preferences -> Plug-In Developement -> Target Platform
> and point the location to your mat directory
> (This will read all bundles in the installation)
> * Create a new plug-in: File -> New ... -> Plug-In Project
> * In the manifest.mf Editor, choose the tab "Dependencies" and select
> org.eclipse.mat.api
> * Create a plugin.xml in the root of the project:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <?eclipse version="3.2"?>
> <plugin>
> <extension point="org.eclipse.mat.report.query">
> <query impl="org.eclipse.mat.demo.DemoQuery"/>
> </extension>
> </plugin>
>
>
> * Create your DemoQuery class in the src Folder:
>
>
> package org.eclipse.mat.demo;
>
> import java.text.MessageFormat;
> import java.util.ArrayList;
> import java.util.Collection;
> import java.util.List;
>
> import org.eclipse.mat.SnapshotException;
> import org.eclipse.mat.query.IQuery;
> import org.eclipse.mat.query.IResult;
> import org.eclipse.mat.query.annotations.Argument;
> import org.eclipse.mat.query.annotations.Name;
> import org.eclipse.mat.query.results.ListResult;
> import org.eclipse.mat.snapshot.ISnapshot;
> import org.eclipse.mat.snapshot.model.IClass;
> import org.eclipse.mat.snapshot.model.IObject;
> import org.eclipse.mat.util.IProgressListener;
>
> @Name("Demo Query")
> public class DemoQuery implements IQuery {
>
> @Argument
> public ISnapshot snapshot;
>
> @Argument
> public String threadClass =
> "org.apache.tomcat.util.threads.ThreadWithAttributes";
>
> @Argument
> public String requestInfoClass = "org.apache.coyote.RequestInfo";
>
> @Argument
> public String field = "req.queryMB.byteC.buff";
>
> public IResult execute(IProgressListener listener) throws Exception {
>
> List<ResultItem> result = new ArrayList<ResultItem>();
>
> // lookup all thread classes
> Collection<IClass> threads = snapshot.getClassesByName(threadClass,
> true);
> if (threads == null)
> throw new SnapshotException(MessageFormat.format(
> "Heap Dumps contains no class {0}", threadClass));
>
> // lookup the request info class
> Collection<IClass> requestInfoClasses = snapshot.getClassesByName(
> requestInfoClass, false);
> if (requestInfoClasses == null || requestInfoClasses.isEmpty())
> throw new SnapshotException(MessageFormat.format(
> "Heap Dumps contains no class {0}", requestInfoClass));
> IClass requestInfo = requestInfoClasses.iterator().next();
>
> for (IClass thread : threads) {
> for (int threadInstanceId : thread.getObjectIds()) {
>
> IObject threadInstanceObj = snapshot
> .getObject(threadInstanceId);
>
> // Calculate retained size for each of these threads
> int[] retainedSet = snapshot.getRetainedSet(
> new int[] { threadInstanceId }, listener);
>
> // Within the retained set of each thread, find some
> specific
> // objects
> for (int retainedId : retainedSet) {
> if
> (snapshot.getClassOf(retainedId).equals(requestInfo)) {
>
> IObject requestInfoObj =
> snapshot.getObject(retainedId);
>
> String value = null;
>
> IObject buffer = (IObject) requestInfoObj
> .resolveValue(field);
> if (buffer != null)
> value = buffer.getClassSpecificName();
>
> result.add(new ResultItem(threadInstanceObj,
> requestInfoObj, value));
> }
> }
> }
> }
>
> return new ListResult(ResultItem.class, result, "thread",
> "request",
> "value");
> }
>
> public static class ResultItem {
> IObject thread;
> IObject request;
>
> String value;
>
> public ResultItem(IObject thread, IObject request, String value) {
> super();
> this.thread = thread;
> this.request = request;
> this.value = value;
> }
>
> public IObject getThread() {
> return thread;
> }
>
> public IObject getRequest() {
> return request;
> }
>
> public String getValue() {
> return value;
> }
>
> }
> }
>
>
> * Right-click on your plug-in project, "Run As..." -> "Eclipse
> Application".
>
> * When you open a heap dump, your query should be in the query drop-down
> list.
>
>
>
> Now, this is a very short explanation. If I was too brief on some of the
> steps, do not hesitate to send me an email.
>
Re: Performing query within dominator subgraph [message #6614 is a reply to message #6595] Tue, 04 November 2008 17:03 Go to previous messageGo to next message
Andreas Buchen is currently offline Andreas BuchenFriend
Messages: 113
Registered: July 2009
Senior Member
> I'm happy to report that I succeeded :)

Great to hear. I noticed some glitches with the extension point schemas.
Also, we plan to bundle the sources with the next download. This should
make live much easier.

> Quick questions for now:
> 1) How do I access the set of stack local variables given a reference to
> some thread?

As of today, you need to get the local variables like this:

private static int[] getLocalVarsForThread(IObject thread) throws
SnapshotException
{
List<NamedReference> refs = thread.getOutboundReferences();
ArrayInt result = new ArrayInt();
for (NamedReference ref : refs)
{
if (ref instanceof ThreadToLocalReference)
result.add(ref.getObjectId());
}
return result.toArray();
}



> 2) I assume I can expect to find IClass objects with the same name
> referring to different versions of the same class when browsing a
> snapshot, correct? For example, when the same class was loaded more than
> once through different class loaders.

Yes, if a class is loaded multiple times, then there will be multiple
IClass instances. That means in particular, that

ISnapshot.getClassesByName("path.to.MyClass", false)

will return a collection with multiple entries.


> 3) How does the dominators view calculate the percentage? Is it just
> IObject.getRetainedHeapSize() divided by ISnapshot.getHeapSize(<what to
> pass here?>)?

The percentage is calculated against
ISnapshot.getSnapshotInfo().getUsedHeapSize()


> 4) Does the source code for org.eclipse.mat.snapshot.* contain JavaDoc?
> If so, it might make sense to download it and start exploring. Where can
> I actually get the source? Or perhaps there's a JavaDoc .zip available
> somewhere?

The sources are available here:
http://dev.eclipse.org/svnroot/technology/org.eclipse.mat

This WIKI page contains a (short) overview on how to checkout the stuff
and get going:
http://wiki.eclipse.org/index.php?title=MemoryAnalyzer/Contr ibutor_Reference

(Checkout all project below "plugins" within trunk for the latest release,
or under "tags/release/plugins")

As I said, the next version hopefully contains the sources right away.




BTW, it's great to see somebody else digging deeper! Internally, we have
written a more of those queries for ourselves - to extract session etc etc
etc. They all need specific implementation know-how and the implementation
changes - sometimes because of the stuff we found out!

If you download the source, I'd really appreciate to know where to improve
the API documentation. I know it is perforated like Swiss cheese. But if
you run across stuff like "If I only had known" that just edit the
original code and say "Team" -> "Create Patch" and I know exactly where to
be more elaborate.


Andreas.
Re: Performing query within dominator subgraph [message #6634 is a reply to message #6614] Tue, 04 November 2008 22:44 Go to previous messageGo to next message
Taras Tielkes is currently offline Taras TielkesFriend
Messages: 38
Registered: July 2009
Member
Hi Andreas,

By now I've also succeeded in running the report in "offline" mode.
Thanks again for your detailed instructions.

One thing I couldn't quickly figure out (unrelated to "offline" mode) is
this: in the ListResult that I return I'd like to add a percentage
column, similar to the one in the built-in "dominators" query.

While I can return regular String instances with appropriate rounding
and a percent sign, I assume MAT will not recognize these as numeric
values, and thus will not allow applying a filter like ">10%".

How can I indicate that a result column is a percentage?

By the way, in the dominators view I would expect a filter like ">5" to
work for the percentage column. After some experimentation I've found
that I need to type ">5%" in order for things to work properly. Is this
intended, or is my expectation reasonable?

Regards,
Taras
Re: Performing query within dominator subgraph [message #6653 is a reply to message #6634] Wed, 05 November 2008 12:08 Go to previous message
Andreas Buchen is currently offline Andreas BuchenFriend
Messages: 113
Registered: July 2009
Senior Member
> One thing I couldn't quickly figure out (unrelated to "offline" mode) is
> this: in the ListResult that I return I'd like to add a percentage
> column, similar to the one in the built-in "dominators" query.

Unfortunately, using the ListResult directly you cannot provide this
special percentage formatting. The ListResult determines the columns via
reflections and and uses getX().toString() for format that.

You would need to implement IResultTree or IResultTable directly. Both
have a method getColumns() which returns the columns. There you could
return something like:

new Column("Percentage", double.class).formatting(new
DecimalFormat("0.00%"))

(Take a look at the org.eclipse.mat.snapshot.inspections.DominatorQuery)


> By the way, in the dominators view I would expect a filter like ">5" to
> work for the percentage column. After some experimentation I've found
> that I need to type ">5%" in order for things to work properly. Is this
> intended, or is my expectation reasonable?

I see that it is awkward. >5 returns nothing because it compares 5 to the
double value which is usually between 0 (0%) and 1 (100%). The hack is
this: if the formatter (see above) adds a % percentage sign, we also
accept the percentage sign in the filter and divide the value by 100.
Yuck. That is not intuitive. It might be better to always divide the value
by 100 - regardless of the percentage sign. What do you think?



Andreas.
Previous Topic:Stuck on 63%
Next Topic:What does Thread Status mean
Goto Forum:
  


Current Time: Sun Dec 21 06:18:57 GMT 2014

Powered by FUDForum. Page generated in 0.02176 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software