Home » Archived » Memory Analyzer (MAT) » Performing query within dominator subgraph
| Performing query within dominator subgraph [message #5857] |
Tue, 28 October 2008 19:37  |
Eclipse User |
|
|
|
Hi,
I'm looking at the dominator tree for a given snapshot.
Most of the top-10 dominators are Tomcat request processing threads.
For each of these threads, I'd like to recover the URL parameters passed
by the client.
It happens to be that this information is available.
The subgraph to access it looks something like this:
org.apache.coyote.RequestInfo (stack local reference)
req (org.apache.coyote.Request)
queryMB (org.apache.tomcat.util.buf.MessageBytes)
strValue (java.lang.String)
What I'm after is this:
1) Can I somehow launch a query that extracts this information from the
subgraph associated with a top-level (Thread) node of the dominators list?
That way I could re-run this query for each of the "big" dominator threads.
2) Even better: can I somehow perform a query that
a) for each org.apache.tomcat.util.threads.ThreadWithAttributes in the
snapshot
b) finds the retained size
c) finds the URL (using above graph navigation path) within the retained
set of that thread
d) displays both in a convenient 2-column table format
e) sorted by retained size
Thanks in advance for any pointers,
Taras
|
|
| | | | |
| Re: Performing query within dominator subgraph [message #6411 is a reply to message #6002] |
Thu, 30 October 2008 04:53  |
| Eclipse User |
|
|
|
> How hard would it be for me to use relevant components of MAT in a
> headless fashion in order to accomplish what I want?
It depends... if you have some experience with Eclipse, it should not be
too hard.
> I assume the dependencies to do what I want are minimal, hopefully I can
> just use some of the "org.eclipse.mat.*" jars without starting an OSGi
> framework.
Unfortunately, that won't work. You need to go through the OSGi framework.
Okay, this is what you need to do:
* Install the latest Memory Analyzer stand-alone into some directory
* Install Eclipse Ganymede Java IDE
* Open Eclipse
* Choose Window -> Preferences -> Plug-In Developement -> Target Platform
and point the location to your mat directory
(This will read all bundles in the installation)
* Create a new plug-in: File -> New ... -> Plug-In Project
* In the manifest.mf Editor, choose the tab "Dependencies" and select
org.eclipse.mat.api
* Create a plugin.xml in the root of the project:
<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.2"?>
<plugin>
<extension point="org.eclipse.mat.report.query">
<query impl="org.eclipse.mat.demo.DemoQuery"/>
</extension>
</plugin>
* Create your DemoQuery class in the src Folder:
package org.eclipse.mat.demo;
import java.text.MessageFormat;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import org.eclipse.mat.SnapshotException;
import org.eclipse.mat.query.IQuery;
import org.eclipse.mat.query.IResult;
import org.eclipse.mat.query.annotations.Argument;
import org.eclipse.mat.query.annotations.Name;
import org.eclipse.mat.query.results.ListResult;
import org.eclipse.mat.snapshot.ISnapshot;
import org.eclipse.mat.snapshot.model.IClass;
import org.eclipse.mat.snapshot.model.IObject;
import org.eclipse.mat.util.IProgressListener;
@Name("Demo Query")
public class DemoQuery implements IQuery {
@Argument
public ISnapshot snapshot;
@Argument
public String threadClass =
"org.apache.tomcat.util.threads.ThreadWithAttributes";
@Argument
public String requestInfoClass = "org.apache.coyote.RequestInfo";
@Argument
public String field = "req.queryMB.byteC.buff";
public IResult execute(IProgressListener listener) throws Exception {
List<ResultItem> result = new ArrayList<ResultItem>();
// lookup all thread classes
Collection<IClass> threads = snapshot.getClassesByName(threadClass,
true);
if (threads == null)
throw new SnapshotException(MessageFormat.format(
"Heap Dumps contains no class {0}", threadClass));
// lookup the request info class
Collection<IClass> requestInfoClasses = snapshot.getClassesByName(
requestInfoClass, false);
if (requestInfoClasses == null || requestInfoClasses.isEmpty())
throw new SnapshotException(MessageFormat.format(
"Heap Dumps contains no class {0}", requestInfoClass));
IClass requestInfo = requestInfoClasses.iterator().next();
for (IClass thread : threads) {
for (int threadInstanceId : thread.getObjectIds()) {
IObject threadInstanceObj = snapshot
.getObject(threadInstanceId);
// Calculate retained size for each of these threads
int[] retainedSet = snapshot.getRetainedSet(
new int[] { threadInstanceId }, listener);
// Within the retained set of each thread, find some specific
// objects
for (int retainedId : retainedSet) {
if (snapshot.getClassOf(retainedId).equals(requestInfo)) {
IObject requestInfoObj = snapshot.getObject(retainedId);
String value = null;
IObject buffer = (IObject) requestInfoObj
.resolveValue(field);
if (buffer != null)
value = buffer.getClassSpecificName();
result.add(new ResultItem(threadInstanceObj,
requestInfoObj, value));
}
}
}
}
return new ListResult(ResultItem.class, result, "thread", "request",
"value");
}
public static class ResultItem {
IObject thread;
IObject request;
String value;
public ResultItem(IObject thread, IObject request, String value) {
super();
this.thread = thread;
this.request = request;
this.value = value;
}
public IObject getThread() {
return thread;
}
public IObject getRequest() {
return request;
}
public String getValue() {
return value;
}
}
}
* Right-click on your plug-in project, "Run As..." -> "Eclipse
Application".
* When you open a heap dump, your query should be in the query drop-down
list.
Now, this is a very short explanation. If I was too brief on some of the
steps, do not hesitate to send me an email.
|
|
| | | |
| Re: Performing query within dominator subgraph [message #6595 is a reply to message #6411] |
Tue, 04 November 2008 03:41 |
| Eclipse User |
|
|
|
Hi Andreas,
I'm happy to report that I succeeded :)
It turned out that the RequestInfo was not part of the retained set of a
worker thread. However, since it was a local variable of the thread I
reached it through threadInstanceObj.getOutboundReferences().
I didn't try the "batch mode" solution yet, but will certainly do so
over the next few days, and will let you know how far I get.
I might try to pull out a few more diagnostics based on my internal
knowledge of the application. It's a pity that a heap dump does not
include the equivalent of a thread dump - that would have made analysis
of what a thread is doing (somewhat) straightforward. Instead I'll try
to use the presence/absence of some specific local variables from a
thread to summarize its current state.
Quick questions for now:
1) How do I access the set of stack local variables given a reference to
some thread?
2) I assume I can expect to find IClass objects with the same name
referring to different versions of the same class when browsing a
snapshot, correct? For example, when the same class was loaded more than
once through different class loaders.
3) How does the dominators view calculate the percentage? Is it just
IObject.getRetainedHeapSize() divided by ISnapshot.getHeapSize(<what to
pass here?>)?
4) Does the source code for org.eclipse.mat.snapshot.* contain JavaDoc?
If so, it might make sense to download it and start exploring. Where can
I actually get the source? Or perhaps there's a JavaDoc .zip available
somewhere?
Thank you again for the detailed assistance you provided. Apart from my
own typos and mistakes the process was seamless.
I'm still pretty amazed by looking at my custom query running and
pulling out information from the snapshot :)
Regards,
Taras
Andreas Buchen wrote:
>> How hard would it be for me to use relevant components of MAT in a
>> headless fashion in order to accomplish what I want?
>
> It depends... if you have some experience with Eclipse, it should not be
> too hard.
>
>> I assume the dependencies to do what I want are minimal, hopefully I
>> can just use some of the "org.eclipse.mat.*" jars without starting an
>> OSGi framework.
>
> Unfortunately, that won't work. You need to go through the OSGi framework.
>
>
> Okay, this is what you need to do:
>
> * Install the latest Memory Analyzer stand-alone into some directory
> * Install Eclipse Ganymede Java IDE
> * Open Eclipse
> * Choose Window -> Preferences -> Plug-In Developement -> Target Platform
> and point the location to your mat directory
> (This will read all bundles in the installation)
> * Create a new plug-in: File -> New ... -> Plug-In Project
> * In the manifest.mf Editor, choose the tab "Dependencies" and select
> org.eclipse.mat.api
> * Create a plugin.xml in the root of the project:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <?eclipse version="3.2"?>
> <plugin>
> <extension point="org.eclipse.mat.report.query">
> <query impl="org.eclipse.mat.demo.DemoQuery"/>
> </extension>
> </plugin>
>
>
> * Create your DemoQuery class in the src Folder:
>
>
> package org.eclipse.mat.demo;
>
> import java.text.MessageFormat;
> import java.util.ArrayList;
> import java.util.Collection;
> import java.util.List;
>
> import org.eclipse.mat.SnapshotException;
> import org.eclipse.mat.query.IQuery;
> import org.eclipse.mat.query.IResult;
> import org.eclipse.mat.query.annotations.Argument;
> import org.eclipse.mat.query.annotations.Name;
> import org.eclipse.mat.query.results.ListResult;
> import org.eclipse.mat.snapshot.ISnapshot;
> import org.eclipse.mat.snapshot.model.IClass;
> import org.eclipse.mat.snapshot.model.IObject;
> import org.eclipse.mat.util.IProgressListener;
>
> @Name("Demo Query")
> public class DemoQuery implements IQuery {
>
> @Argument
> public ISnapshot snapshot;
>
> @Argument
> public String threadClass =
> "org.apache.tomcat.util.threads.ThreadWithAttributes";
>
> @Argument
> public String requestInfoClass = "org.apache.coyote.RequestInfo";
>
> @Argument
> public String field = "req.queryMB.byteC.buff";
>
> public IResult execute(IProgressListener listener) throws Exception {
>
> List<ResultItem> result = new ArrayList<ResultItem>();
>
> // lookup all thread classes
> Collection<IClass> threads = snapshot.getClassesByName(threadClass,
> true);
> if (threads == null)
> throw new SnapshotException(MessageFormat.format(
> "Heap Dumps contains no class {0}", threadClass));
>
> // lookup the request info class
> Collection<IClass> requestInfoClasses = snapshot.getClassesByName(
> requestInfoClass, false);
> if (requestInfoClasses == null || requestInfoClasses.isEmpty())
> throw new SnapshotException(MessageFormat.format(
> "Heap Dumps contains no class {0}", requestInfoClass));
> IClass requestInfo = requestInfoClasses.iterator().next();
>
> for (IClass thread : threads) {
> for (int threadInstanceId : thread.getObjectIds()) {
>
> IObject threadInstanceObj = snapshot
> .getObject(threadInstanceId);
>
> // Calculate retained size for each of these threads
> int[] retainedSet = snapshot.getRetainedSet(
> new int[] { threadInstanceId }, listener);
>
> // Within the retained set of each thread, find some
> specific
> // objects
> for (int retainedId : retainedSet) {
> if
> (snapshot.getClassOf(retainedId).equals(requestInfo)) {
>
> IObject requestInfoObj =
> snapshot.getObject(retainedId);
>
> String value = null;
>
> IObject buffer = (IObject) requestInfoObj
> .resolveValue(field);
> if (buffer != null)
> value = buffer.getClassSpecificName();
>
> result.add(new ResultItem(threadInstanceObj,
> requestInfoObj, value));
> }
> }
> }
> }
>
> return new ListResult(ResultItem.class, result, "thread",
> "request",
> "value");
> }
>
> public static class ResultItem {
> IObject thread;
> IObject request;
>
> String value;
>
> public ResultItem(IObject thread, IObject request, String value) {
> super();
> this.thread = thread;
> this.request = request;
> this.value = value;
> }
>
> public IObject getThread() {
> return thread;
> }
>
> public IObject getRequest() {
> return request;
> }
>
> public String getValue() {
> return value;
> }
>
> }
> }
>
>
> * Right-click on your plug-in project, "Run As..." -> "Eclipse
> Application".
>
> * When you open a heap dump, your query should be in the query drop-down
> list.
>
>
>
> Now, this is a very short explanation. If I was too brief on some of the
> steps, do not hesitate to send me an email.
>
|
|
|
| Re: Performing query within dominator subgraph [message #6614 is a reply to message #6595] |
Tue, 04 November 2008 12:03 |
| Eclipse User |
|
|
|
> I'm happy to report that I succeeded :)
Great to hear. I noticed some glitches with the extension point schemas.
Also, we plan to bundle the sources with the next download. This should
make live much easier.
> Quick questions for now:
> 1) How do I access the set of stack local variables given a reference to
> some thread?
As of today, you need to get the local variables like this:
private static int[] getLocalVarsForThread(IObject thread) throws
SnapshotException
{
List<NamedReference> refs = thread.getOutboundReferences();
ArrayInt result = new ArrayInt();
for (NamedReference ref : refs)
{
if (ref instanceof ThreadToLocalReference)
result.add(ref.getObjectId());
}
return result.toArray();
}
> 2) I assume I can expect to find IClass objects with the same name
> referring to different versions of the same class when browsing a
> snapshot, correct? For example, when the same class was loaded more than
> once through different class loaders.
Yes, if a class is loaded multiple times, then there will be multiple
IClass instances. That means in particular, that
ISnapshot.getClassesByName("path.to.MyClass", false)
will return a collection with multiple entries.
> 3) How does the dominators view calculate the percentage? Is it just
> IObject.getRetainedHeapSize() divided by ISnapshot.getHeapSize(<what to
> pass here?>)?
The percentage is calculated against
ISnapshot.getSnapshotInfo().getUsedHeapSize()
> 4) Does the source code for org.eclipse.mat.snapshot.* contain JavaDoc?
> If so, it might make sense to download it and start exploring. Where can
> I actually get the source? Or perhaps there's a JavaDoc .zip available
> somewhere?
The sources are available here:
http://dev.eclipse.org/svnroot/technology/org.eclipse.mat
This WIKI page contains a (short) overview on how to checkout the stuff
and get going:
http://wiki.eclipse.org/index.php?title=MemoryAnalyzer/Contr ibutor_Reference
(Checkout all project below "plugins" within trunk for the latest release,
or under "tags/release/plugins")
As I said, the next version hopefully contains the sources right away.
BTW, it's great to see somebody else digging deeper! Internally, we have
written a more of those queries for ourselves - to extract session etc etc
etc. They all need specific implementation know-how and the implementation
changes - sometimes because of the stuff we found out!
If you download the source, I'd really appreciate to know where to improve
the API documentation. I know it is perforated like Swiss cheese. But if
you run across stuff like "If I only had known" that just edit the
original code and say "Team" -> "Create Patch" and I know exactly where to
be more elaborate.
Andreas.
|
|
| |
| Re: Performing query within dominator subgraph [message #6653 is a reply to message #6634] |
Wed, 05 November 2008 07:08 |
| Eclipse User |
|
|
|
> One thing I couldn't quickly figure out (unrelated to "offline" mode) is
> this: in the ListResult that I return I'd like to add a percentage
> column, similar to the one in the built-in "dominators" query.
Unfortunately, using the ListResult directly you cannot provide this
special percentage formatting. The ListResult determines the columns via
reflections and and uses getX().toString() for format that.
You would need to implement IResultTree or IResultTable directly. Both
have a method getColumns() which returns the columns. There you could
return something like:
new Column("Percentage", double.class).formatting(new
DecimalFormat("0.00%"))
(Take a look at the org.eclipse.mat.snapshot.inspections.DominatorQuery)
> By the way, in the dominators view I would expect a filter like ">5" to
> work for the percentage column. After some experimentation I've found
> that I need to type ">5%" in order for things to work properly. Is this
> intended, or is my expectation reasonable?
I see that it is awkward. >5 returns nothing because it compares 5 to the
double value which is usually between 0 (0%) and 1 (100%). The hack is
this: if the formatter (see above) adds a % percentage sign, we also
accept the percentage sign in the filter and divide the value by 100.
Yuck. That is not intuitive. It might be better to always divide the value
by 100 - regardless of the percentage sign. What do you think?
Andreas.
|
|
|
Goto Forum:
Current Time: Wed Mar 26 08:17:14 EDT 2025
Powered by FUDForum. Page generated in 0.07324 seconds |
Home