Eclipse Community Forums: Memory Analyzer » Command Line processing of hprof

Help

Home

Home » Eclipse Projects » Memory Analyzer » Command Line processing of hprof

Show: Today's Messages :: Show Polls :: Message Navigator

Command Line processing of hprof [message #1065652]

Wed, 26 June 2013 19:28

Yu Yang Lai
Messages: 6
Registered: June 2013

Junior Member

Hi everyone,

I would like to be able to use MAT to process my hprof dumps headlessly through command line shell scripts. The main features that I am looking for are outputting the histogram and dominator trees in csv format (which i have achieved using a rudimentary report xml modified from the built-in overview report); a dynamically generated output based on the input OQL command, ie: <shell script> <path_to_hprof> <OQL query>; and also a diff between different versions of hprof files.

There are still gaping holes in my knowledge, for instance, how do command line inputs get mapped into the query classes arguments, whether report xmls are extensible beyond the very basic as examplified by the built-in reports overview, suspects and top_components, and the more intricate relationships between query classes and the results they output.

Also I can't seem to get the dropins folder to make MAT recognize my custom plugin from command line calls. Perhaps I have put them in the wrong place (mat root folder)?

Thank you in advance.

Cheers,
Yu Yang Lai

Report message to a moderator

Re: Command Line processing of hprof [message #1067930 is a reply to message #1065652]

Thu, 11 July 2013 06:18

Andrew Johnson
Messages: 105
Registered: July 2009

Senior Member

See http://help.eclipse.org/kepler/topic/org.eclipse.mat.ui.help/doc/org_eclipse_mat_report_report.html for details of reports.

The oql command can be used to run OQL queries
oql "select * from java.lang.String"

so I would try something like this in your report file.
<command>oql "select * from java.lang.String"</command>

That doesn't help with dynamically building a query. I think the file attribute points to a report definition inside your plugin, but you could try an unpacked plugin.

See http://wiki.eclipse.org/MemoryAnalyzer/Contributor_Reference#Writing_plugins_for_the_Memory_Analyzer
for a tip for getting the dropins folder to work.

I don't know of a good way to open two heap dumps from the command line - a second argument is taken as a report name.

You might be able to write a query and use

@Argument(advice = Advice.SECONDARY_SNAPSHOT)
public ISnapshot snapshot2;

then call CompareTablesQuery using SnapshotQuery.lookup
or Histogram.diffWithBaseline

Report message to a moderator

Re: Command Line processing of hprof [message #1068073 is a reply to message #1067930]

Thu, 11 July 2013 19:17

Yu Yang Lai
Messages: 6
Registered: June 2013

Junior Member

Hi Andrew,

Thanks for the reply! I managed to run org.eclipse.mat.api on custom reports by pointing the command to the xml file path, without having to write any plugin extensions. The output was in an inconvenient zip file format but nothing I can't remedy with a few lines of script. I also figured that I could run dynamic oql queries by generating the new xml report file on the fly, so that part is solved.

Now I have a new question: my dominator tree report outputs a csv file with the classname appended with an address and a whole bunch of stuffs which contains illegal characters (including commas) and thus is choking up my csv parser. Is there any way to turn off the appending of addresses and just include the class name (like the histogram report)? I included my dominator tree report as an attachment. Thank you so much for your time and attention.

Attachment: domtree.xml
(Size: 0.22KB, Downloaded 10 times)

Report message to a moderator

Re: Command Line processing of hprof [message #1068115 is a reply to message #1068073]

Fri, 12 July 2013 05:33

Andrew Johnson
Messages: 105
Registered: July 2009

Senior Member

There is the hide_column parameter which might help.

		<param key="hide_column" value="#0" />

though that would get rid of the whole field.

The csv export is meant to escape commas in a value by quoting the field.
See http://tools.ietf.org/html/rfc4180

I see that our code uses CR rather than CRLF at the end of line and doesn't escape CR or LF, so we should tidy that up.

Report message to a moderator

Re: Command Line processing of hprof [message #1068232 is a reply to message #1068115]

Fri, 12 July 2013 14:44

Yu Yang Lai
Messages: 6
Registered: June 2013

Junior Member

we need the class names for analysis, but thanks for your help!

Report message to a moderator

Re: Command Line processing of hprof [message #1069290 is a reply to message #1068232]

Mon, 15 July 2013 11:54

Andrew Johnson
Messages: 105
Registered: July 2009

Senior Member

Running your query gives output as follows:

class sun.nio.cs.StandardCharsets @ 0x77cb7a4a0,304,8200,0.03488,
class java.util.Locale @ 0x77cbff7e8,232,6968,0.02964,
"java.lang.String @ 0x77cb92bd8  \u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008\u0009\u000a\u000b\u000c\u000d\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015\u0016\u0017\u0018\u0019\u001a\u001b\u001c\u001d\u001e\u001f !""#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_...",40,2416,0.01028,
class sun.jkernel.DownloadManager @ 0x77cb42bf8,328,2296,0.00977,

Can you parse fields with double quotes around (and commas or doubled double-quotes inside) ?

Do you want to open a bug for the trailing comma or CR instead of CR LF to end each line?

Report message to a moderator

Re: Command Line processing of hprof [message #1069865 is a reply to message #1069290]

Tue, 16 July 2013 15:10

Yu Yang Lai
Messages: 6
Registered: June 2013

Junior Member

At the moment I am using a shell script to parse the csv and thus its buggy when the classname contains all sorts of strange characters. The current plan is to build a better parser using python but that is pushed back in favour of more pressing projects.

As for opening a bug, it would be great, could you point me to the right place for it? Thanks.

Report message to a moderator

Re: Command Line processing of hprof [message #1070615 is a reply to message #1069865]

Thu, 18 July 2013 06:15

Andrew Johnson
Messages: 105
Registered: July 2009

Senior Member

There's a link for reporting a bug on the left hand side of the Memory Analyzer home page.

The dominator tree CSV does not have illegal characters (comma) as they are protected by double quotes, but parsing may be a bit more difficult. One idea is to note that only the first field might have internal commas and double quotes, so reverse the line, parse the field, then reverse back.

E.g. something like:

rev domtree.csv | cut -d , -f 5- | rev

If we fix the trailing comma in MAT then this might need a slight change.

Report message to a moderator

Re: Command Line processing of hprof [message #1070793 is a reply to message #1070615]

Thu, 18 July 2013 13:33

Yu Yang Lai
Messages: 6
Registered: June 2013

Junior Member

Thanks, parsing from the back solved >99% of the parsing errors, but I'll get back to you after I tried my python script thanks.

P.S. Any plans to make a more full featured support for CLI commands? Like OQL and diffs etc. That would be great. Thanks.

Report message to a moderator

Re: Command Line processing of hprof [message #1074108 is a reply to message #1070793]

Fri, 26 July 2013 02:12

Joe Bowbeer
Messages: 1
Registered: July 2009

Junior Member

A stripped-down MAT was created for command line use.

See bitbucket.org/ekabanov/mat

Report message to a moderator

Re: Command Line processing of hprof [message #1102635 is a reply to message #1067930]

Thu, 05 September 2013 20:23

$John O\'Brien is currently offline$ John O\'Brien
Messages: 1
Registered: September 2013

Junior Member

Hi,

I would also like to run on the command line to parse large hepas with long running oql.

I read the help link referenced by Andrew and see the two items taht must be added to separate files....

However where does the report.xsd file reside?

Regards
John

Report message to a moderator

Previous Topic:	heap dump file size larger than the heap size
Next Topic:	OQL: how to define JS functions

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Mon Oct 07 20:39:59 EDT 2013