Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Memory Analyzer » Using MAT to analyze Eclipse OOME (headless)
Using MAT to analyze Eclipse OOME (headless) [message #1744561] Wed, 28 September 2016 09:52 Go to next message
Marcel Bruch is currently offline Marcel BruchFriend
Messages: 286
Registered: July 2009
Senior Member

Hi there,

I'm leading the Automated Error Reporting Initiative (AERI) at eclipse.org. We recently received a plenty of OutOfMemoryExceptions (OOME) but unfortunately they are not actionable b/c its unclear which objects cause the OOME.

I wonder whether we could use MAT to run an dominator analysis (and maybe others) and attach the results of this analysis to an error report.

Can any of the committers or experienced users elaborate whether/how we could run memory analyses in headless mode?

Thank you in advance
Marcel

Re: Using MAT to analyze Eclipse OOME (headless) [message #1744677 is a reply to message #1744561] Thu, 29 September 2016 12:46 Go to previous messageGo to next message
Krum Tsvetkov is currently offline Krum TsvetkovFriend
Messages: 161
Registered: July 2009
Senior Member
Hi Marcel,

I like the AERI initiative and I think it would be great if we manage to get some reasonable reporting for memory related issues.

However, I am not sure how easy and transparent for the end-user this could be made.
Let's discuss it, I'll try to provide details from MAT side. There are a few things we have to consider:

Q1: How to get a heap dump for analysis

To start at all with the analysis we need to get a heap dump. One can configure the JVM to write a heap dump when an OOME occurs (e.g. by specifying -XX:+HeapDumpOnOutOfMemoryError for an openjdk/oracle jdk).
However, when I look into the eclipse.ini of the Neon "Eclipse for Commiters" package for Win, the parameter is not set by default. Not sure about the other packages.
Furthermore, the configuration to automatically get a dump differs depending on the JVM vendor/type. Some details can be found here (I see we haven't updated the docs since jdk 1.6 Sad )
http://wiki.eclipse.org/MemoryAnalyzer#Getting_a_Heap_Dump. In short - there are different ways to configure automated triggering of a heap dump and currently this is not enabled in the packages (haven't checked all packages though).

Q2: Where the analysis is executed

I think the second important question is where the automated analysis of the heap dump is made. Some things to consider:

- processing server side, i.e. uploading the raw heap dump is:
-- network intensive (need to upload hundreds of MBs or some GBs)
-- insecure - with exception of the PHD dumps from IBM VMs, the other formats supported by MAT contain also all fields/values information. This is very practical for analysis (one can very well reconstruct post mortem the objects, states, etc...), but I wouldn't recommend that people upload full dumps, as they may contain sensitive data.
-- on the positive side - this will not require the clients to have MAT installed locally
- processing client side:
-- needs MAT installation on the client. MAT plugins are part of the simrel repository, but are still not by default in any of the pre-built packages. As memory analysis itself could be memory intensive, MAT offers its own standalone packages as an alternative to having MAT in-place in the IDE used for development.
-- as I mentioned, depending on the heap size, parsing the heap dump can be memory intensive
-- I am not sure how the AREI reporting is done - is it possible to trigger the analysis of the heap dump in a separate process? It wouldn't make sense to try to run it within a process which is already suffering from low memory.


Q3: How / what could be analyzed (headless)
What MAT offers out-of-the-box is to run a so called "leak suspects" report by calling something like "MemoryAnalyzer -consoleLog -application org.eclipse.mat.api.parse org.eclipse.mat.api:suspects" (in the standalone MAT installations there is a ParseHeapDump scrip for doing this). MAT will then parse the heap dump, run the "report" org.eclipse.mat.api:suspects and by this produce a set of HTML files (zipped) containing the analysis.
The produced pages contain some suspects (based on analysis of the dominator tree) + some information about the suspects, for example:
- the name of the bundle (if possible to extract) which loaded the class keeping most of the heap
- some reference path information, i.e. the (common) path from the GC roots to the object(s) keeping most of the heap

While this is useful if people get the report result and look interactively into it, it is not suitable (not easy to parse) for automatic sorting/classifying the errors.

One could provide additional reports (MAT has extension point for it) which produce output more suitable for classifying the errors automatically. I guess the details mentioned above (paths from the GC roots, bundle name) would be useful for classifying the issue. This is something I could try to help with (depending on the time I could invest).

Well, this is what came to my mind after thinking for a while about the proposal / question. What are your thoughts? Keeping in mind some of the constraints mentioned above, do you think we could still offer a user-friendly solution?

Best Regards,
Krum

Re: Using MAT to analyze Eclipse OOME (headless) [message #1744681 is a reply to message #1744677] Thu, 29 September 2016 13:20 Go to previous messageGo to next message
Marcel Bruch is currently offline Marcel BruchFriend
Messages: 286
Registered: July 2009
Senior Member

Hi Krum,

thanks for your prompt reply.

Q1: How to get a heap dump for analysis

Some thoughts on this.
1. From the error reports I see that almost all users use openJDK or oracle's vm. Thus, there would be no urgent need to specify the right parameters.
2. We may add the -XX:+HeapDumpOnOutOfMemoryError by default to the eclipse.ini of the committer package (only!) since committers have a special interest to report and inspect these errors.
3. We (AERI) may rewrite the eclipse.ini on user behalf once it got activated - hoping that the first error event is not an OOME Smile

I think 2. and 3. are both possible solutions. I'll discuss both on epp-dev with the package-maintainer.

Q2: Where the analysis is executed

It clearly has to happen on the client side.
I haven't thought about it before but I also agree that analyzing the dump must take place in a separate process for obvious reasons.

To me it seems reasonable to download (a small) separate MAT product on first access (or include it already in the committers package) and execute the analysis in a separate process whenever the user clicks on the "attache leak suspects analysis to error report".

It definitely needs more thinking but at least there is a reasonable way how this could be implemented.


Q3: How / what could be analyzed (headless)
Well. That is something we need to evaluate when we see it, right? My experience with MAT was that the leak suspects report was sufficient for me to detect where my problems were coming from. But your mileage may vary. Having a ZIP file attached to the error report would be fine. An automated parsing of the report is not in my current scope.


To sum up, Q2 seems to be the most critical issue to solve (getting a copy a MAT to run on Mac, Linux, Win to run the leak suspect analysis) since Q3 (attaching a ZIP to the report) is fine and Q1 is solvable.

Do you have an special tips regarding Q2? Especially: Do I need OS specify binaries to run it headless?

Thanks, Marcel
Re: Using MAT to analyze Eclipse OOME (headless) [message #1744695 is a reply to message #1744681] Thu, 29 September 2016 14:28 Go to previous messageGo to next message
Krum Tsvetkov is currently offline Krum TsvetkovFriend
Messages: 161
Registered: July 2009
Senior Member
Hi,

Focusing on a certain JVM type and just using the zip as it is definitely makes the problem way easier to solve.

About getting MAT installed on the client side, we have these options
- install MAT within the IDE (just install the two features from the simrel repo). Then one can execute a separate process, using the standard eclipsec executable to get the report. I just tired this out:
eclipsec -nosplash -consoleLog -application org.eclipse.mat.api.parse java_pid9324.0001.hprof  org.eclipse.mat.api:suspects


- install (download and unzip) a MAT standalone package - we have zips for all the supported platforms: https://www.eclipse.org/mat/downloads.php. Then call
MemoryAnalyzer -nosplash -consoleLog -application org.eclipse.mat.api.parse java_pid9324.0001.hprof  org.eclipse.mat.api:suspects

i.e. the only difference in the call is the name of the executable (it is the same one, but we rename it when we build the package).

These calls produce a <heap_dump_name>_Leak_Suspects.zip in the directory in which the dump is located. One could use the
-XX:HeapDumpPath=
option in combination with the
-XX:+HeapDumpOnOutOfMemoryError
option, so that 1) no guess work is needed to find the heap dump and 2) put the dump and the files generated during parsing outside the eclipse root folder (not sure where else).

Is this info enough for you to experiment? I think both installation options are acceptable. Installing the features means less to download, but adds some plugins to the user's IDE. Installing a standalone MAT means more MBs (around 50) to download + the reporting tool has to pick the proper archive, but then the user's IDE is not modified. I can't really judge what people would prefer.

Regards,
Krum
Re: Using MAT to analyze Eclipse OOME (headless) [message #1744699 is a reply to message #1744695] Thu, 29 September 2016 14:41 Go to previous messageGo to next message
Marcel Bruch is currently offline Marcel BruchFriend
Messages: 286
Registered: July 2009
Senior Member

Krum, this reply is a great start.

I've almost all details I need to start the conversation with Gunnar and others on epp-dev.

One more question: When adding the MAT features to a epp product, do they include UI contributions. It might be a blocker for some package maintainers if they add more UI "clutter" to their Eclipse package...

Thanks,
Marcel
Re: Using MAT to analyze Eclipse OOME (headless) [message #1744707 is a reply to message #1744699] Thu, 29 September 2016 15:00 Go to previous message
Krum Tsvetkov is currently offline Krum TsvetkovFriend
Messages: 161
Registered: July 2009
Senior Member
Hi Marcel,

Yes, MAT adds some UI. I am not sure what the criteria to get included into an epp are. The components which MAT brings (Entries under File menu, entries in the toolbar, etc...) get visible only if you switch to the "Memory Analysis" perspective.
We haven't touched our UI for years, so my first guess would be that it doesn't cover all requirements. But this is just a guess. Can you give it a try to install neon for committers, then add MAT (Category "Performance, Profiling and Tracing Tools"), and give me your first impression if it is cluttering the UI.

BTW, I just registered for ElipseCon Europe in Ludwigsburg. In case you are going there, we could also discuss the topic face to face.

Regards,
Krum
Previous Topic:How to extend Memory Analyzer
Next Topic:Memory Analyzer 1.6.1 Released
Goto Forum:
  


Current Time: Thu Sep 20 10:29:49 GMT 2018

Powered by FUDForum. Page generated in 0.02634 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top