Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cdt-dev] Memory consumed by indexer in big projects
On Fri, Jan 20, 2012 at 11:31 AM, Andrew Gvozdev <angvoz.dev@xxxxxxxxx> wrote:
On Fri, Jan 20, 2012 at 10:57 AM, Doug Schaefer <cdtdoug@xxxxxxxxx> wrote:
Actually to be clear, those are AST objects, not indexer. In the past, we've made sure they don't stay around long (thus the index :)). It would be interesting to see what's keeping them from being gc'ed after the parse is completed.
The parse cannot be completed due to OOM, that is the problem.

During the index or some other time? We trigger parses from a few places, if it's happening in the background, it's probably the indexer, although that's not the real problem.
 
 
Do they see the same behavior off of master (i.e., it's not your changes causing this is it?)
It is problematic to set up that level of detail for settings on master, the current mechanism does not scale with big or many projects. There are bugs in bugzilla about that.

I did not experience memory leaks related to indexer working with master, and I did not experience such memory leak working with real and pretty big projects on sd90. But I do not work with projects requiring that level of details for  settings. I experienced that indexer demands enough memory or you get Out Of Memory. For my big projects (~40000 files) enough memory is ~1Mb.\

It could be a particular file that they have. We have often had problems with large static array initializers, for example. I can't recall how good our logging is, but it would be good to know which file was being parsed when the OOM happens.
 

Thanks,
Andrew
 

On Fri, Jan 20, 2012 at 10:45 AM, Andrew Gvozdev <angvoz.dev@xxxxxxxxx> wrote:
Hi,
We are testing the new scanner discovery with mozilla folks on firefox project. In general, it seems to be pretty good improvement but we run into huge consumption of heap memory by indexer objects.
Eclipse is given 2Gb of heap memory in eclipse.ini but still Out Of Memory exception is thrown. It seems that most of the memory is used by indexer objects (see summary below and attached report).

I want to be clear, this is the scanner discovery branch sd90 and the project is big and settings for the project are very elaborate so pretty much every file has slightly different settings and likely could use different variants of headers with different set of macros.
So the question is if it is conceivable to indexer to use up to that much memory, or we are looking at a memory leak? (Jonathan also tried to give it 8Gb of heap with similar result, I did not look at the dump tho.)

Scanner discovery itself takes constant 0.5M for this project and does not grow, but maybe I am using indexer incorrectly and induce the leak somehow? I created a small class org.eclipse.cdt.internal.core.pdom.LanguageSettingsChangeListener to listen to changes in language settings and trigger reindexing (on branch sd90).

Thanks,
Andrew

  Problem Suspect 1

One instance of"org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTTranslationUnit" loaded by"org.eclipse.cdt.core" occupies 856,848,368 (31.40%) bytes. The memory is accumulated in one instance of "org.eclipse.cdt.core.dom.ast.IASTDeclaration[]"loaded by "org.eclipse.cdt.core".

Keywords
org.eclipse.cdt.core
org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTTranslationUnit
org.eclipse.cdt.core.dom.ast.IASTDeclaration[]

  Problem Suspect 2

One instance of "org.eclipse.cdt.internal.core.parser.scanner.LocationMap" loaded by "org.eclipse.cdt.core" occupies 812,268,408 (29.77%) bytes. The memory is accumulated in one instance of "java.lang.Object[]" loaded by "<system class loader>".

Keywords
org.eclipse.cdt.core
java.lang.Object[]
org.eclipse.cdt.internal.core.parser.scanner.LocationMap

  Problem Suspect 3

12,818 instances of "org.eclipse.cdt.internal.core.parser.scanner.CharArray", loaded by "org.eclipse.cdt.core" occupy 381,328,568 (13.97%) bytes. These instances are referenced from one instance of "java.lang.Object[]", loaded by "<system class loader>"

Keywords
org.eclipse.cdt.core
org.eclipse.cdt.internal.core.parser.scanner.CharArray
java.lang.Object[]

 Hint 1

The problem suspects 1, 2 and 3 may be related, because the reference chains to them have a common beginning.


Thanks,
Andrew


_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev



_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev



_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev


Back to the top