Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Memory Analyzer » Command Line processing of hprof
Command Line processing of hprof [message #1065652] Wed, 26 June 2013 19:28 Go to next message
Yu Yang Lai is currently offline Yu Yang Lai
Messages: 6
Registered: June 2013
Junior Member
Hi everyone,

I would like to be able to use MAT to process my hprof dumps headlessly through command line shell scripts. The main features that I am looking for are outputting the histogram and dominator trees in csv format (which i have achieved using a rudimentary report xml modified from the built-in overview report); a dynamically generated output based on the input OQL command, ie: <shell script> <path_to_hprof> <OQL query>; and also a diff between different versions of hprof files.

There are still gaping holes in my knowledge, for instance, how do command line inputs get mapped into the query classes arguments, whether report xmls are extensible beyond the very basic as examplified by the built-in reports overview, suspects and top_components, and the more intricate relationships between query classes and the results they output.

Also I can't seem to get the dropins folder to make MAT recognize my custom plugin from command line calls. Perhaps I have put them in the wrong place (mat root folder)?

Thank you in advance.

Cheers,
Yu Yang Lai
Re: Command Line processing of hprof [message #1067930 is a reply to message #1065652] Thu, 11 July 2013 06:18 Go to previous messageGo to next message
Andrew Johnson is currently offline Andrew Johnson
Messages: 112
Registered: July 2009
Senior Member
See http://help.eclipse.org/kepler/topic/org.eclipse.mat.ui.help/doc/org_eclipse_mat_report_report.html for details of reports.

The oql command can be used to run OQL queries
oql "select * from java.lang.String"

so I would try something like this in your report file.
<command>oql "select * from java.lang.String"</command>

That doesn't help with dynamically building a query. I think the file attribute points to a report definition inside your plugin, but you could try an unpacked plugin.

See http://wiki.eclipse.org/MemoryAnalyzer/Contributor_Reference#Writing_plugins_for_the_Memory_Analyzer
for a tip for getting the dropins folder to work.

I don't know of a good way to open two heap dumps from the command line - a second argument is taken as a report name.

You might be able to write a query and use
@Argument(advice = Advice.SECONDARY_SNAPSHOT)
public ISnapshot snapshot2;


then call CompareTablesQuery using SnapshotQuery.lookup
or Histogram.diffWithBaseline

Re: Command Line processing of hprof [message #1068073 is a reply to message #1067930] Thu, 11 July 2013 19:17 Go to previous messageGo to next message
Yu Yang Lai is currently offline Yu Yang Lai
Messages: 6
Registered: June 2013
Junior Member
Hi Andrew,

Thanks for the reply! I managed to run org.eclipse.mat.api on custom reports by pointing the command to the xml file path, without having to write any plugin extensions. The output was in an inconvenient zip file format but nothing I can't remedy with a few lines of script. I also figured that I could run dynamic oql queries by generating the new xml report file on the fly, so that part is solved.

Now I have a new question: my dominator tree report outputs a csv file with the classname appended with an address and a whole bunch of stuffs which contains illegal characters (including commas) and thus is choking up my csv parser. Is there any way to turn off the appending of addresses and just include the class name (like the histogram report)? I included my dominator tree report as an attachment. Thank you so much for your time and attention.
  • Attachment: domtree.xml
    (Size: 0.22KB, Downloaded 43 times)
Re: Command Line processing of hprof [message #1068115 is a reply to message #1068073] Fri, 12 July 2013 05:33 Go to previous messageGo to next message
Andrew Johnson is currently offline Andrew Johnson
Messages: 112
Registered: July 2009
Senior Member
There is the hide_column parameter which might help.
		<param key="hide_column" value="#0" />

though that would get rid of the whole field.

The csv export is meant to escape commas in a value by quoting the field.
See http://tools.ietf.org/html/rfc4180

I see that our code uses CR rather than CRLF at the end of line and doesn't escape CR or LF, so we should tidy that up.
Re: Command Line processing of hprof [message #1068232 is a reply to message #1068115] Fri, 12 July 2013 14:44 Go to previous messageGo to next message
Yu Yang Lai is currently offline Yu Yang Lai
Messages: 6
Registered: June 2013
Junior Member
we need the class names for analysis, but thanks for your help!
Re: Command Line processing of hprof [message #1069290 is a reply to message #1068232] Mon, 15 July 2013 11:54 Go to previous messageGo to next message
Andrew Johnson is currently offline Andrew Johnson
Messages: 112
Registered: July 2009
Senior Member
Running your query gives output as follows:
class sun.nio.cs.StandardCharsets @ 0x77cb7a4a0,304,8200,0.03488,
class java.util.Locale @ 0x77cbff7e8,232,6968,0.02964,
"java.lang.String @ 0x77cb92bd8  \u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008\u0009\u000a\u000b\u000c\u000d\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015\u0016\u0017\u0018\u0019\u001a\u001b\u001c\u001d\u001e\u001f !""#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_...",40,2416,0.01028,
class sun.jkernel.DownloadManager @ 0x77cb42bf8,328,2296,0.00977,

Can you parse fields with double quotes around (and commas or doubled double-quotes inside) ?

Do you want to open a bug for the trailing comma or CR instead of CR LF to end each line?
Re: Command Line processing of hprof [message #1069865 is a reply to message #1069290] Tue, 16 July 2013 15:10 Go to previous messageGo to next message
Yu Yang Lai is currently offline Yu Yang Lai
Messages: 6
Registered: June 2013
Junior Member
At the moment I am using a shell script to parse the csv and thus its buggy when the classname contains all sorts of strange characters. The current plan is to build a better parser using python but that is pushed back in favour of more pressing projects.

As for opening a bug, it would be great, could you point me to the right place for it? Thanks.
Re: Command Line processing of hprof [message #1070615 is a reply to message #1069865] Thu, 18 July 2013 06:15 Go to previous messageGo to next message
Andrew Johnson is currently offline Andrew Johnson
Messages: 112
Registered: July 2009
Senior Member
There's a link for reporting a bug on the left hand side of the Memory Analyzer home page.

The dominator tree CSV does not have illegal characters (comma) as they are protected by double quotes, but parsing may be a bit more difficult. One idea is to note that only the first field might have internal commas and double quotes, so reverse the line, parse the field, then reverse back.

E.g. something like:
rev domtree.csv | cut -d , -f 5- | rev


If we fix the trailing comma in MAT then this might need a slight change.

Re: Command Line processing of hprof [message #1070793 is a reply to message #1070615] Thu, 18 July 2013 13:33 Go to previous messageGo to next message
Yu Yang Lai is currently offline Yu Yang Lai
Messages: 6
Registered: June 2013
Junior Member
Thanks, parsing from the back solved >99% of the parsing errors, but I'll get back to you after I tried my python script thanks.

P.S. Any plans to make a more full featured support for CLI commands? Like OQL and diffs etc. That would be great. Thanks.
Re: Command Line processing of hprof [message #1074108 is a reply to message #1070793] Fri, 26 July 2013 02:12 Go to previous messageGo to next message
Joe Bowbeer is currently offline Joe Bowbeer
Messages: 1
Registered: July 2009
Junior Member
A stripped-down MAT was created for command line use.

See bitbucket.org/ekabanov/mat
Re: Command Line processing of hprof [message #1102635 is a reply to message #1067930] Thu, 05 September 2013 20:23 Go to previous messageGo to next message
John O\'Brien is currently offline John O\'Brien
Messages: 1
Registered: September 2013
Junior Member
Hi,

I would also like to run on the command line to parse large hepas with long running oql.

I read the help link referenced by Andrew and see the two items taht must be added to separate files....

However where does the report.xsd file reside?

Regards
John
Re: Command Line processing of hprof [message #1129171 is a reply to message #1067930] Tue, 08 October 2013 07:06 Go to previous message
Macco Ray is currently offline Macco Ray
Messages: 1
Registered: September 2013
Junior Member
Hi,How can I pass arg for SECONDARY_SNAPSHOT by commandline? Need your help,Thx.
Previous Topic:Which algrithm is used for dominator tree calculation?
Next Topic:reachability / retained set
Goto Forum:
  


Current Time: Sun Apr 20 18:13:54 EDT 2014

Powered by FUDForum. Page generated in 0.01850 seconds