Home » Eclipse Projects » Memory Analyzer » Using MAT to analyze Eclipse OOME (headless)
| Using MAT to analyze Eclipse OOME (headless) [message #1744561] |
Wed, 28 September 2016 09:52  |
|
Hi there,
I'm leading the Automated Error Reporting Initiative (AERI) at eclipse.org. We recently received a plenty of OutOfMemoryExceptions (OOME) but unfortunately they are not actionable b/c its unclear which objects cause the OOME.
I wonder whether we could use MAT to run an dominator analysis (and maybe others) and attach the results of this analysis to an error report.
Can any of the committers or experienced users elaborate whether/how we could run memory analyses in headless mode?
Thank you in advance
Marcel
|
|
|
| Re: Using MAT to analyze Eclipse OOME (headless) [message #1744677 is a reply to message #1744561] |
Thu, 29 September 2016 12:46  |
 Krum Tsvetkov
Messages: 164
Registered: July 2009 |
Senior Member |
|
|
Hi Marcel,
I like the AERI initiative and I think it would be great if we manage to get some reasonable reporting for memory related issues.
However, I am not sure how easy and transparent for the end-user this could be made.
Let's discuss it, I'll try to provide details from MAT side. There are a few things we have to consider:
Q1: How to get a heap dump for analysis
To start at all with the analysis we need to get a heap dump. One can configure the JVM to write a heap dump when an OOME occurs (e.g. by specifying -XX:+HeapDumpOnOutOfMemoryError for an openjdk/oracle jdk).
However, when I look into the eclipse.ini of the Neon "Eclipse for Commiters" package for Win, the parameter is not set by default. Not sure about the other packages.
Furthermore, the configuration to automatically get a dump differs depending on the JVM vendor/type. Some details can be found here (I see we haven't updated the docs since jdk 1.6  )
http://wiki.eclipse.org/MemoryAnalyzer#Getting_a_Heap_Dump. In short - there are different ways to configure automated triggering of a heap dump and currently this is not enabled in the packages (haven't checked all packages though).
Q2: Where the analysis is executed
I think the second important question is where the automated analysis of the heap dump is made. Some things to consider:
- processing server side, i.e. uploading the raw heap dump is:
-- network intensive (need to upload hundreds of MBs or some GBs)
-- insecure - with exception of the PHD dumps from IBM VMs, the other formats supported by MAT contain also all fields/values information. This is very practical for analysis (one can very well reconstruct post mortem the objects, states, etc...), but I wouldn't recommend that people upload full dumps, as they may contain sensitive data.
-- on the positive side - this will not require the clients to have MAT installed locally
- processing client side:
-- needs MAT installation on the client. MAT plugins are part of the simrel repository, but are still not by default in any of the pre-built packages. As memory analysis itself could be memory intensive, MAT offers its own standalone packages as an alternative to having MAT in-place in the IDE used for development.
-- as I mentioned, depending on the heap size, parsing the heap dump can be memory intensive
-- I am not sure how the AREI reporting is done - is it possible to trigger the analysis of the heap dump in a separate process? It wouldn't make sense to try to run it within a process which is already suffering from low memory.
Q3: How / what could be analyzed (headless)
What MAT offers out-of-the-box is to run a so called "leak suspects" report by calling something like "MemoryAnalyzer -consoleLog -application org.eclipse.mat.api.parse org.eclipse.mat.api:suspects" (in the standalone MAT installations there is a ParseHeapDump scrip for doing this). MAT will then parse the heap dump, run the "report" org.eclipse.mat.api:suspects and by this produce a set of HTML files (zipped) containing the analysis.
The produced pages contain some suspects (based on analysis of the dominator tree) + some information about the suspects, for example:
- the name of the bundle (if possible to extract) which loaded the class keeping most of the heap
- some reference path information, i.e. the (common) path from the GC roots to the object(s) keeping most of the heap
While this is useful if people get the report result and look interactively into it, it is not suitable (not easy to parse) for automatic sorting/classifying the errors.
One could provide additional reports (MAT has extension point for it) which produce output more suitable for classifying the errors automatically. I guess the details mentioned above (paths from the GC roots, bundle name) would be useful for classifying the issue. This is something I could try to help with (depending on the time I could invest).
Well, this is what came to my mind after thinking for a while about the proposal / question. What are your thoughts? Keeping in mind some of the constraints mentioned above, do you think we could still offer a user-friendly solution?
Best Regards,
Krum
|
|
|
| Re: Using MAT to analyze Eclipse OOME (headless) [message #1744681 is a reply to message #1744677] |
Thu, 29 September 2016 13:20 |
|
Hi Krum,
thanks for your prompt reply.
Q1: How to get a heap dump for analysis
Some thoughts on this.
1. From the error reports I see that almost all users use openJDK or oracle's vm. Thus, there would be no urgent need to specify the right parameters.
2. We may add the -XX:+HeapDumpOnOutOfMemoryError by default to the eclipse.ini of the committer package (only!) since committers have a special interest to report and inspect these errors.
3. We (AERI) may rewrite the eclipse.ini on user behalf once it got activated - hoping that the first error event is not an OOME 
I think 2. and 3. are both possible solutions. I'll discuss both on epp-dev with the package-maintainer.
Q2: Where the analysis is executed
It clearly has to happen on the client side.
I haven't thought about it before but I also agree that analyzing the dump must take place in a separate process for obvious reasons.
To me it seems reasonable to download (a small) separate MAT product on first access (or include it already in the committers package) and execute the analysis in a separate process whenever the user clicks on the "attache leak suspects analysis to error report".
It definitely needs more thinking but at least there is a reasonable way how this could be implemented.
Q3: How / what could be analyzed (headless)
Well. That is something we need to evaluate when we see it, right? My experience with MAT was that the leak suspects report was sufficient for me to detect where my problems were coming from. But your mileage may vary. Having a ZIP file attached to the error report would be fine. An automated parsing of the report is not in my current scope.
To sum up, Q2 seems to be the most critical issue to solve (getting a copy a MAT to run on Mac, Linux, Win to run the leak suspect analysis) since Q3 (attaching a ZIP to the report) is fine and Q1 is solvable.
Do you have an special tips regarding Q2? Especially: Do I need OS specify binaries to run it headless?
Thanks, Marcel
|
|
| |
| Re: Using MAT to analyze Eclipse OOME (headless) [message #1744699 is a reply to message #1744695] |
Thu, 29 September 2016 14:41 |
|
Krum, this reply is a great start.
I've almost all details I need to start the conversation with Gunnar and others on epp-dev.
One more question: When adding the MAT features to a epp product, do they include UI contributions. It might be a blocker for some package maintainers if they add more UI "clutter" to their Eclipse package...
Thanks,
Marcel
|
|
|
| Re: Using MAT to analyze Eclipse OOME (headless) [message #1744707 is a reply to message #1744699] |
Thu, 29 September 2016 15:00 |
Krum Tsvetkov
Messages: 164
Registered: July 2009 |
Senior Member |
|
|
Hi Marcel,
Yes, MAT adds some UI. I am not sure what the criteria to get included into an epp are. The components which MAT brings (Entries under File menu, entries in the toolbar, etc...) get visible only if you switch to the "Memory Analysis" perspective.
We haven't touched our UI for years, so my first guess would be that it doesn't cover all requirements. But this is just a guess. Can you give it a try to install neon for committers, then add MAT (Category "Performance, Profiling and Tracing Tools"), and give me your first impression if it is cluttering the UI.
BTW, I just registered for ElipseCon Europe in Ludwigsburg. In case you are going there, we could also discuss the topic face to face.
Regards,
Krum
|
|
|
Goto Forum:
Current Time: Thu Apr 18 07:57:33 GMT 2024
Powered by FUDForum. Page generated in 0.02151 seconds |
] 
Search
Help
Register
Login
Home
