RE: [cdt-dev] CDT performance

[Doug Schaefer <DSchaefer@xxxxxxx>]

Comments below.

> -----Original Message-----
> From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On
> Behalf Of Krasilnikov, Oleg
> Sent: Wednesday, December 07, 2005 11:42 AM
> To: CDT General developers list.
> Cc: Voronin, Mikhail; Sennikovsky, Mikhail
> Subject: RE: [cdt-dev] CDT performance
> 
> Doug,
> 
> I have a question to you regarding PDOM-related changes in
> scanner-parser-indexer chain.
> 
> I'm interested in CDT performance and memory consumption problems
> (mostly in parser),
> so it's interesting how much this part will be modified in nearest
> future.
> 
> Could you please confirm or deny:
> 
> - you know, memory usage is another big problem in CDT - in particular,
>   current indexing procedure requires too much memory and crashes
> sometimes
>   due to OutOfMemory exception (with Mozilla sources on 1Gb, for
> example).
>   How PDOM will affect this situation ? From the one hand, since header
> files
>   will not be re-parsed with each translation unit, resulting AST tree
> can be smaller
>   significantly. (currently it's really huge !) From the other hand,
> database server
>   (required by PDOM) will consume a lot of memory itself - so, Eclipse
> and CDT
>   will have less space...

[Doug] You are correct. The AST should be much smaller. The PDOM is stored
in a memory mapped file with 4K pages cached as MappedByteBuffers in systems
that support them. You'll be able to control the size of the cache in a
preference.
 
> - parser logic itself will be seriously modified for PDOM ? Currently,
> parser
>   itself does not know about includes - it simply receives tokens from
> scanner.

[Doug] Actually the parser won't need to be modified hardly at all. Right
now when the scanner needs to open an include file it calls a
CodeReaderFactory to take care of it and return a CodeReader that manages
the file. For the PDOM, we have a special factory that checks if the header
file needs to be loaded and, if not, it won't. There is a minor change to
the binding resolution to look up semantic info in the PDOM when it can't
find things in the AST. The same will need to be done for macro definitions
as needed by the scanner.

> - PDOM usage is optional - so some users will use current indexing
>   mechanism, expecially for projects with non-trivial headers contents ?

[Doug] That's a good question and it is hard to answer. Definitely the PDOM
is optional until I'm sure it functions correctly. In the situation you
mention, non-trivial header files, this is where the full indexer had the
most problem with performance and is the reason I'm doing the PDOM. My
saddened heart is telling me to rip out all of the old indexer stuff and
move everything to the PDOM, and maybe all my problems will be gone ;).

> - although parsing will be used less frequently with PDOM (ideally, once
> for
>   each file), it will be inevitable when project is first opened - so,
> parser
>   performance is still significant a bit in this case ?

[Doug] Yes, we will always have that initial hit to build the initial index
for imported projects. Every indexer I've seen has this hit as well. My hope
is that, parsing each file only once will make this much faster than what we
have today, where we don't reuse the results of parsing header files. We
will also support the ability to generate the initial index headless and
import them.

> Thanks in advance.

[Doug] No problem. Great questions! Please keep them and any ideas you have
coming. I'll also make sure your patches get looked at ASAP as well.

Cheers,
Doug

> 
> -----------------------------------
> With best regards, Oleg Krasilnikov
> Software designer, Eclipse team.
> Intel corp.
> +7 8312 162 444 ext. 2587
> (Russia, Nizhny Novgorod)
> 
> -----Original Message-----
> From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx]
> On Behalf Of Doug Schaefer
> Sent: Friday, November 18, 2005 6:10 PM
> To: CDT General developers list.
> Subject: RE: [cdt-dev] CDT performance
> 
> <...>
> However from what I've seen and heard, we really need to increase
> performance by orders of magnitude and this will require a whole new
> indexing strategy. My work on the PDOM for 3.1, which in the end is
> really
> just a new indexer, is aimed at indexing each file once, somewhat like
> ctags
> but with a real parser and a symbols database to fill in the gaps. The
> accuracy will be a bit less but the performance gains should be very
> significant.
> <...>
> 
> Cheers,
> Doug
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/cdt-dev