[cdt-dev] Proofing the CDT w/ large projects

["Thomas Fletcher" <thomasf@xxxxxxx>]

Folks,

  Just thought you would be interested to hear a bit about the environment
that one of our customers is working with using CDT 1.2:

- Project tree contains 5.68 GB of content when fully built
- Distributed over 38,601 files in 3,783 directories
- Development file breakdown:
  27   *.cxx
  464  *.c
  6039 *.cpp  
  
  3    *.hxx
  2453 *.hpp	
  9061 *.h

  2    *.so
  35   *.dll
  45   *.lib
  75   *.exe
  994  *.a
  6206 *.so

The time it takes to load this project into the IDE initially (on a 3Ghz
machine) is ~3m, but subsequent IDE start-ups are in the 7s timeframe.  
Compared to previous versions of CDT, this is pretty awesome!

The big bottleneck right now seems to be the indexer.  Initially it takes
>5m to index the 15,000+ source files and then >2m to pull out all of the
classes and types from a search (I'm testing via the Open Type action 
Chris Wiebe implemented).  (Note: It is also killer for the code completion 
perfomance it seems since the user doesn't get any feedback about what is
happening).

Some things I've noticed and am looking into:
- There are a number of scanner and parser errors (ranging from failures
  to outright exceptions).  This is likely due to an improperly configured
  source base (no #defines, #includes set), but shouldn't affect the 
  re-indexing operation.

- There are a couple of "expected exceptions" which then return null, for
  example in FileListBlock.getFile(int fileNum) we hit the exception 
  condition very often.  Checking the range and returning null improves
  the performance

- There are a couple of mystery exceptions I still haven't figured out.
  For example WordEntry.mapRefs(int [] mappings) is often throwing an
  ArrayIndexOutOfBounds exception ... this is trouble I believe and 
  leads to:

- The fact that in the IndexManager.getIndex(IPath, boolean, boolean)
  we (in this particular project) are never retrieving an index which
  has a known state (it is always UNKNOWN_STATE) which means we go and
  rebuild the index for everything.

I think that with Chris Wiebe's "Open Type" contributed plugin and a 
moderately large project you should be able to see this kind of churn.
I'm looking into it, but thought that the indexer guy (Hi Bogdan!) might
have some thoughts about what is going on.

Also ... it takes a fair amount of time to add in 15,000+ entries using
the IndexAllProject.execute() -> IndexManager.addSource() method, several
seconds at least in my environment.  

Just passing along some observations.  Unfortunately I can't pass along the
source base, but I will certainly try and pull out relevant information as
requested/required and dig into this while I can.

Thanks,
 Thomas