Eclipse Community Forums: Java Development Tools (JDT) » Parsing Java files with JDT and it's memory consumption / performance

Home » Language IDEs » Java Development Tools (JDT) » Parsing Java files with JDT and it's memory consumption / performance

Show: Today's Messages :: Show Polls :: Message Navigator

Parsing Java files with JDT and it's memory consumption / performance [message #1744271]

Fri, 23 September 2016 19:10

Markus Oley

Messages: 304
Registered: July 2009
Location: Germany

Senior Member

Hi to all,

I want to parse java code with jdt from commandline and so I wrote a small tool, which parses the java files in the following kind:

ASTParser parser = ASTParser.newParser(AST.JLS8);
parser.setEnvironment(classpath, srcPaths, encodings, false);

parser.setSource(//String of file read before);

parser.setResolveBindings(true);
parser.setBindingsRecovery(true);
parser.setUnitName(name of File);
parser.setKind(ASTParser.K_COMPILATION_UNIT);
CompilationUnit cu = (CompilationUnit) parser.createAST(null);

With this code I get an OOM with 2GB after reading 1300 java files, but I can resolve e.g. MethodInvocations. If i skip the

parser.setBindinsRecovery(true)

the tool finishes, but the whole bindings are not resolved.

My small and perhaps easy question: How can I implement this with less memory consumption and better performance AND getting all things resolved? I guess jdt implementaion in eclipse itself needs maximum some MBs to read the AST.

Thanks for your help
Markus

Report message to a moderator

Re: Parsing Java files with JDT and it's memory consumption / performance [message #1744273 is a reply to message #1744271]

Fri, 23 September 2016 19:11

Markus Oley

Messages: 304
Registered: July 2009
Location: Germany

Senior Member

Sorry my browser didn't come back, so I posted it 4 times

Report message to a moderator

Re: Parsing Java files with JDT and it's memory consumption / performance [message #1744274 is a reply to message #1744273]

Fri, 23 September 2016 19:45

Stephan Herrmann

Messages: 1853
Registered: July 2009

Senior Member

You showed the code for reading one file, how do you read all files, i.e., where is the loop? Do you create a new parser for each file?
Have you considered using ASTParser.createASTs(String[],String[],String[],FileASTRequestor,IProgressMonitor)?
Do you have indications why setBindinsRecovery is needed? Any compile errors / missing dependencies in those sources?

Report message to a moderator

Re: Parsing Java files with JDT and it's memory consumption / performance [message #1744280 is a reply to message #1744274]

Fri, 23 September 2016 20:51

Markus Oley

Messages: 304
Registered: July 2009
Location: Germany

Senior Member

Hi Stephan,

thanks for you help,
JDT is really cool, if you know how to handle it Smile

This first implementation (still optimizable....) did it for me:

ASTParser parser = ASTParser.newParser(AST.JLS8);
parser.setEnvironment(classpath, srcPaths.toArray(new String[srcPaths.size()]),
encodings.toArray(new String[encodings.size()]), false);

parser.setResolveBindings(true);

Collection<String> sources = new ArrayList<String>();
Collection<String> sourceEncodings = new ArrayList<String>();
for (File nextFile: allFiles) {
sources.add(nextFile.getAbsolutePath());
encodings.add("ISO-8859-15");
}
final JavaProject thisProject = this;

FileASTRequestor requestor = new FileASTRequestor() {
@Override
public void acceptAST(String sourcefilePath, CompilationUnit ast) {
try {
System.out.println ("Reading " + sourcefilePath);
JavaSource javaSource = new JavaSourceJdt(thisProject, ast);
javaSourceContainer.addItem(javaSource);
javaTypeContainer.addAllItems(javaSource.getTypes());
} catch (Throwable t) {
log.error(t.toString(), t);
throw new RuntimeException(t);
}
}

};

String [] sourcesAsArray = sources.toArray(new String [sources.size()]);
String [] encodingsAsArray = encodings.toArray(new String [sourceEncodings.size()]);

parser.createASTs(sourcesAsArray, encodingsAsArray, new String[0], requestor, null); //new SubProgressMonitor(subMonitor, 1));

Cheers
Markus

Report message to a moderator