Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Language IDEs » Java Development Tools (JDT) » Parsing Java files with JDT and it's memory consumption / performance
Parsing Java files with JDT and it's memory consumption / performance [message #1744271] Fri, 23 September 2016 19:10 Go to next message
Markus Oley is currently offline Markus OleyFriend
Messages: 291
Registered: July 2009
Location: Germany
Senior Member
Hi to all,

I want to parse java code with jdt from commandline and so I wrote a small tool, which parses the java files in the following kind:

ASTParser parser = ASTParser.newParser(AST.JLS8);
parser.setEnvironment(classpath, srcPaths, encodings, false);

parser.setSource(//String of file read before);

parser.setResolveBindings(true);
parser.setBindingsRecovery(true);
parser.setUnitName(name of File);
parser.setKind(ASTParser.K_COMPILATION_UNIT);
CompilationUnit cu = (CompilationUnit) parser.createAST(null);


With this code I get an OOM with 2GB after reading 1300 java files, but I can resolve e.g. MethodInvocations. If i skip the

parser.setBindinsRecovery(true)
the tool finishes, but the whole bindings are not resolved.

My small and perhaps easy question: How can I implement this with less memory consumption and better performance AND getting all things resolved? I guess jdt implementaion in eclipse itself needs maximum some MBs to read the AST.

Thanks for your help
Markus
Re: Parsing Java files with JDT and it's memory consumption / performance [message #1744273 is a reply to message #1744271] Fri, 23 September 2016 19:11 Go to previous messageGo to next message
Markus Oley is currently offline Markus OleyFriend
Messages: 291
Registered: July 2009
Location: Germany
Senior Member
Sorry my browser didn't come back, so I posted it 4 times
Re: Parsing Java files with JDT and it's memory consumption / performance [message #1744274 is a reply to message #1744273] Fri, 23 September 2016 19:45 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1650
Registered: July 2009
Senior Member
You showed the code for reading one file, how do you read all files, i.e., where is the loop? Do you create a new parser for each file?
Have you considered using ASTParser.createASTs(String[],String[],String[],FileASTRequestor,IProgressMonitor)?
Do you have indications why setBindinsRecovery is needed? Any compile errors / missing dependencies in those sources?
Re: Parsing Java files with JDT and it's memory consumption / performance [message #1744280 is a reply to message #1744274] Fri, 23 September 2016 20:51 Go to previous messageGo to next message
Markus Oley is currently offline Markus OleyFriend
Messages: 291
Registered: July 2009
Location: Germany
Senior Member
Hi Stephan,

thanks for you help,
JDT is really cool, if you know how to handle it Smile

This first implementation (still optimizable....) did it for me:

ASTParser parser = ASTParser.newParser(AST.JLS8);
parser.setEnvironment(classpath, srcPaths.toArray(new String[srcPaths.size()]),
encodings.toArray(new String[encodings.size()]), false);

parser.setResolveBindings(true);

Collection<String> sources = new ArrayList<String>();
Collection<String> sourceEncodings = new ArrayList<String>();
for (File nextFile: allFiles) {
sources.add(nextFile.getAbsolutePath());
encodings.add("ISO-8859-15");
}
final JavaProject thisProject = this;

FileASTRequestor requestor = new FileASTRequestor() {
@Override
public void acceptAST(String sourcefilePath, CompilationUnit ast) {
try {
System.out.println ("Reading " + sourcefilePath);
JavaSource javaSource = new JavaSourceJdt(thisProject, ast);
javaSourceContainer.addItem(javaSource);
javaTypeContainer.addAllItems(javaSource.getTypes());
} catch (Throwable t) {
log.error(t.toString(), t);
throw new RuntimeException(t);
}
}

};


String [] sourcesAsArray = sources.toArray(new String [sources.size()]);
String [] encodingsAsArray = encodings.toArray(new String [sourceEncodings.size()]);

parser.createASTs(sourcesAsArray, encodingsAsArray, new String[0], requestor, null); //new SubProgressMonitor(subMonitor, 1));


Cheers
Markus
Re: Parsing Java files with JDT and it's memory consumption / performance [message #1744871 is a reply to message #1744280] Sat, 01 October 2016 06:27 Go to previous messageGo to next message
avantika thakur is currently offline avantika thakurFriend
Messages: 2
Registered: August 2016
Junior Member
I have also facing the same problem in the past . I have try so many time but it will not work for me.
Re: Parsing Java files with JDT and it's memory consumption / performance [message #1782770 is a reply to message #1744871] Thu, 01 March 2018 08:28 Go to previous messageGo to next message
Federer Roger  is currently offline Federer Roger Friend
Messages: 19
Registered: December 2017
Junior Member
It is not hard if u use createASTs method.
Re: Parsing Java files with JDT and it's memory consumption / performance [message #1782774 is a reply to message #1744274] Thu, 01 March 2018 09:41 Go to previous message
mia avery is currently offline mia averyFriend
Messages: 2
Registered: March 2018
Junior Member
Hello Guys,

In the certain problem, I need to parse a Java source code fragment that is potentially incomplete. For example, the code can refer to variables that are not defined in such fragment.

In that case, I would still like to parse such incomplete Java code, transform it to a convenient inspectable representation, and being able to generate source code from such abstract representation.

It would be great Having more than 3 years experience at MindMajix.com in IT professional with expertise in providing Enterprise Performance Engineering solutions & Integrated end to end IT monitoring solutions to clients from various industry

All you need to know about IT Software course and corporate training is here ​java-courses with FREE DEMO.
Previous Topic:JDT - Find startline of method excluding Javadocs
Next Topic:JavaFX Project Hangs on iMac
Goto Forum:
  


Current Time: Wed Nov 14 21:24:10 GMT 2018

Powered by FUDForum. Page generated in 0.02653 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top