Home » Archived » IMP » Experiences; custom parser?
| Experiences; custom parser? [message #9961] |
Tue, 16 October 2007 15:00  |
Eclipse User |
|
|
|
Originally posted by: L.C.L.Kats.tudelft.nl
Hi,
At Delft University we're very much interested in the IMP Project. In
the future, we may be interested in integrating it with the SDF2
parser and Stratego/XT program transformation suite. At the moment
though, I have just been playing around with the current
implementation a bit, creating an IDE for an experimental language
developed here. First, a list of some things I ran into, which I hope
is useful:
* The wizards seem to be very picky about the inputs. (Something
common to most early implementations of Eclipse plugins...)
- They require the project to be in the workspace _directory_, not
just imported into the workspace from a different location.
- For LPG to work, I had to place Eclipse in a directory without
spaces in the path (i.e., outside "Program Files" under Windows)
- Not all wizards correctly convert the package name to lowercase.
* The editors don't seem to fully support multiple instances of the
same file (right click "new editor" on any editor pane). Quickly
pasting content into a duplicated editor seems to break it.
* The LPG editors are very helpful with their support for
hyperlinking and inline error reporting. It is a bit strict about
rules that are unused though, shouldn't those result in a warning
rather than an error?
* Finally, one issue came up when I was rewriting our existing grammar
to use LPG. In our grammar we currently have a comment block in the
form of "description { ... }". This caused a conflict with the
"define" keyword, even with backtracking enabled, apparently because
they share a common prefix. I was finally able to work around this
by placing the "define" keyword in the regular lexer rather than in
the keyword lexer...
Other than these issues, it seems the current implementation already
provides some really nice, time-safing abstractions over the regular
Eclipse API, mainly thanks to the AST-based approach. What took the
most time when creating my editor, of course, was writing the
grammar. I had an existing SDF2 grammar as a basis (that is,
scannerless generalized LR; SGLR), which included quite a few nasty
language features that were hard to translate to LPG. (Such as the
comment blocks and - at the moment - a complete lack of reserved
keywords.)
As I understand from the documentation, the current implementation
isn't quite ready yet for custom parsers, due to the use of concrete
types rather than interfaces. Other than that, are there any current
hurdles that may be inhibiting the use of a custom parser? Does the
fact that the SGLR parser doesn't employ a separate lexer form any
problem?
Regards,
--
Lennart Kats
------------------------------------------------------------ ---------
Software Engineering Research Group,
Department of Software Technology,
Delft University of Technology.
|
|
| |
| Re: Experiences; custom parser? [message #10054 is a reply to message #9961] |
Tue, 16 October 2007 21:34  |
 Stan Sutton
Messages: 121
Registered: July 2009 |
Senior Member |
|
|
Hello Lennart,
I have posted some Bugzilla reports regarding the problems you observed.
I have a few questions of clarification on some of the points (in-lined
below).
Thanks for your help,
Stan Sutton
Lennart Kats wrote:
> Hi,
>
> At Delft University we're very much interested in the IMP Project. In
> the future, we may be interested in integrating it with the SDF2
> parser and Stratego/XT program transformation suite. At the moment
> though, I have just been playing around with the current
> implementation a bit, creating an IDE for an experimental language
> developed here. First, a list of some things I ran into, which I hope
> is useful:
>
> * The wizards seem to be very picky about the inputs. (Something
> common to most early implementations of Eclipse plugins...)
>
> - They require the project to be in the workspace _directory_, not
> just imported into the workspace from a different location.
>
> - For LPG to work, I had to place Eclipse in a directory without
> spaces in the path (i.e., outside "Program Files" under Windows)
Are you talking about your Eclipse installation or about your Eclipse
workspace?
> - Not all wizards correctly convert the package name to lowercase.
Can you cite some examples? What do you regard as the correct lowercase
conversion? (I've seen different approaches.)
> * The editors don't seem to fully support multiple instances of the
> same file (right click "new editor" on any editor pane). Quickly
> pasting content into a duplicated editor seems to break it.
I tried to replicate this error in a level 3 workspace but did not see
it. That is, I had defined an IDE in a level 2 runtime workbench, and
then run a runtime workbench from that. I was able to run the "New
Editor" command without pasting with no problems, and when I ran the
command and immediately pasted some text into the new editor I again did
not see any problems. I was using just the LEG grammar and IDE that is
provided as an example in our templates, so no special programming of
the grammar or IDE service implementations.
> * The LPG editors are very helpful with their support for
> hyperlinking and inline error reporting. It is a bit strict about
> rules that are unused though, shouldn't those result in a warning
> rather than an error?
>
> * Finally, one issue came up when I was rewriting our existing grammar
> to use LPG. In our grammar we currently have a comment block in the
> form of "description { ... }". This caused a conflict with the
> "define" keyword, even with backtracking enabled, apparently because
> they share a common prefix. I was finally able to work around this
> by placing the "define" keyword in the regular lexer rather than in
> the keyword lexer...
>
> Other than these issues, it seems the current implementation already
> provides some really nice, time-safing abstractions over the regular
> Eclipse API, mainly thanks to the AST-based approach. What took the
> most time when creating my editor, of course, was writing the
> grammar. I had an existing SDF2 grammar as a basis (that is,
> scannerless generalized LR; SGLR), which included quite a few nasty
> language features that were hard to translate to LPG. (Such as the
> comment blocks and - at the moment - a complete lack of reserved
> keywords.)
>
> As I understand from the documentation, the current implementation
> isn't quite ready yet for custom parsers, due to the use of concrete
> types rather than interfaces. Other than that, are there any current
> hurdles that may be inhibiting the use of a custom parser? Does the
> fact that the SGLR parser doesn't employ a separate lexer form any
> problem?
>
> Regards,
>
> --
> Lennart Kats
> ------------------------------------------------------------ ---------
> Software Engineering Research Group,
> Department of Software Technology,
> Delft University of Technology.
>
|
|
|
| Re: Experiences; custom parser? [message #10099 is a reply to message #9961] |
Wed, 17 October 2007 10:15 |
| Eclipse User |
|
|
|
Originally posted by: olegm.gmail.com
This is a multi-part message in MIME format.
--------------040304070409040703040407
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
Hello,
Lennart Kats wrote:
> As I understand from the documentation, the current implementation
> isn't quite ready yet for custom parsers, due to the use of concrete
> types rather than interfaces. Other than that, are there any current
> hurdles that may be inhibiting the use of a custom parser? Does the
> fact that the SGLR parser doesn't employ a separate lexer form any
> problem?
I've managed to integrate Beaver
http://beaver.sourceforge.net/
into IMP (see attachment).
Cheers,
OM
--------------040304070409040703040407
Content-Type: text/x-java;
name="PicoJavaParseController.java"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="PicoJavaParseController.java"
package picojava.parser;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import lpg.runtime.IMessageHandler;
import lpg.runtime.IPrsStream;
import lpg.runtime.IToken;
import lpg.runtime.LexStream;
import lpg.runtime.Monitor;
import lpg.runtime.ParseErrorCodes;
import lpg.runtime.PrsStream;
import org.eclipse.core.runtime.IPath;
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.core.runtime.Path;
import org.eclipse.imp.model.ISourceProject;
import org.eclipse.imp.parser.IASTNodeLocator;
import org.eclipse.imp.parser.ILexer;
import org.eclipse.imp.parser.IParseController;
import org.eclipse.imp.parser.IParser;
import AST.ASTNode;
import AST.PicoJavaParser;
import AST.PicoJavaScanner;
import AST.Program;
import AST.PicoJavaParser.Terminals;
import beaver.Parser;
import beaver.Scanner;
import beaver.Symbol;
import beaver.Parser.Context;
public class PicoJavaParseController implements IParseController,
IPicoJavaParseController {
private IMessageHandler handler;
private ISourceProject project;
private IPath filePath;
private LexStream lexStream;
private PrsStream prsStream;
private Program currentAst;
private IParser parser = new IParser() {
public PrsStream getParseStream() {
return prsStream;
}
public int getEOFTokenKind() {
return Terminals.EOF;
}
public Object parser(Monitor monitor, int error_repair_count) {
throw new UnsupportedOperationException(
"This method should really not be exposed in IParser");
}
};
private ILexer lexer = new ILexer() {
public int[] getKeywordKinds() {
return new int[0];
}
public LexStream getLexStream() {
return lexStream;
}
public void initialize(char[] contents, String filename) {
throw new UnsupportedOperationException(
"This method should really not be exposed in ILexer");
}
public void lexer(Monitor monitor, IPrsStream prsStream) {
throw new UnsupportedOperationException(
"This method should really not be exposed in ILexer");
}
};
private IASTNodeLocator locator = new IASTNodeLocator() {
public Object findNode(Object node, int startOffset, int endOffset) {
ASTNode astNode = (ASTNode) node;
int nodeStartOffset = pos2offset(astNode.getStart());
int nodeEndOffset = pos2offset(astNode.getEnd());
if (!(nodeStartOffset <= startOffset && endOffset <= nodeEndOffset))
return null;
for (int i = 0; i < astNode.getNumChild(); i++) {
Object result = findNode(astNode.getChild(i), startOffset,
endOffset);
if (result != null)
return result;
}
return astNode;
}
public Object findNode(Object node, int startOffset) {
return findNode(node, startOffset, startOffset);
}
public int getStartOffset(Object node) {
return pos2offset(((ASTNode) node).getStart());
}
public int getEndOffset(Object node) {
return pos2offset(((ASTNode) node).getEnd());
}
public int getLength(Object node) {
return getEndOffset(node) - getStartOffset(node);
}
public IPath getPath(Object node) {
return new Path("");
}
};
public void initialize(IPath filePath, ISourceProject project,
IMessageHandler handler) {
this.filePath = filePath;
this.project = project;
this.handler = handler;
// doParse(new
// FileReader(project.getRawProject().getLocation().append(file Path).toOSString()));
}
public IParser getParser() {
return parser;
}
public ILexer getLexer() {
return lexer;
}
public ISourceProject getProject() {
return project;
}
public IASTNodeLocator getNodeLocator() {
return locator;
}
public IPath getPath() {
return filePath;
}
public Object getCurrentAst() {
return currentAst;
}
public boolean hasErrors() {
// TODO:
return false;
}
public List getErrors() {
// TODO:
return Collections.EMPTY_LIST;
}
public Object parse(String contents, boolean scanOnly,
IProgressMonitor monitor) {
currentAst = null;
if (doPrepare(contents))
doParse(contents);
return currentAst;
}
public boolean isKeyword(int kind) {
switch (kind) {
case Terminals.CLASS:
case Terminals.WHILE:
case Terminals.EXTENDS:
case Terminals.BOOLEAN_LITERAL:
return true;
default:
return false;
}
}
public char[][] getKeywords() {
return new char[0][0];
}
public String getSingleLineCommentPrefix() {
throw new UnsupportedOperationException(
"This method should not be here");
}
public int getTokenIndexAtCharacter(int offset) {
int index = getParser().getParseStream().getTokenIndexAtCharacter(
offset);
return (index < 0 ? -index : index);
}
public IToken getTokenAtCharacter(int offset) {
return prsStream.getTokenAtCharacter(offset);
}
private static List problemMarkerTypes = new ArrayList();
public List getProblemMarkerTypes() {
return problemMarkerTypes;
}
public void addProblemMarkerType(String problemMarkerType) {
problemMarkerTypes.add(problemMarkerType);
}
public void removeProblemMarkerType(String problemMarkerType) {
problemMarkerTypes.remove(problemMarkerType);
}
private boolean doPrepare(String contents) {
// Initialize lex stream
lexStream = new LexStream();
lexStream.initialize(contents.toCharArray(), filePath
.toPortableString());
lexStream.setMessageHandler(handler);
// Initialize parse stream
prsStream = new PrsStream();
prsStream.setLexStream(lexStream);
prsStream.makeToken(0, -1, 0);
boolean result = true;
try {
PicoJavaScanner scanner = new PicoJavaScanner(new StringReader(
contents));
Symbol symbol;
for (symbol = scanner.nextToken(); symbol.getId() != Terminals.EOF; symbol = scanner
.nextToken()) {
prsStream.makeToken(pos2offset(symbol.getStart()),
pos2offset(symbol.getEnd()), symbol.getId());
}
} catch (Scanner.Exception e) {
int offset = lexStream.getLineOffset(e.line - 1) + e.column;
int[] location = lexStream.getLocation(offset, offset);
handler.handleMessage(ParseErrorCodes.NO_MESSAGE_CODE, location,
location, lexStream.getFileName(), new String[] { e
.getMessage() });
result = false;
} catch (IOException e) {
throw new RuntimeException(e);
}
prsStream.makeToken(lexStream.getStreamLength(), lexStream
.getStreamLength(), Terminals.EOF);
prsStream.setStreamLength(prsStream.getSize());
return result;
}
private void doParse(String contents) {
// Do the parsing
try {
PicoJavaParser parser = new PicoJavaParser();
PicoJavaScanner scanner = new PicoJavaScanner(new StringReader(
contents));
parser.setContext(new Context() {
public void reportError(String message, int start, int end) {
int startOffset = pos2offset(start);
int endOffset = pos2offset(end);
// If unexpected EOF
if (startOffset >= lexStream.getStreamLength()
&& lexStream.getStreamLength() > 0)
startOffset = lexStream.getStreamLength() - 1;
int[] location = lexStream.getLocation(startOffset,
endOffset);
handler.handleMessage(ParseErrorCodes.NO_MESSAGE_CODE,
location, location, lexStream.getFileName(),
new String[] { message });
}
});
currentAst = (Program) parser.parse(scanner);
} catch (Parser.Exception e) {
if (e.start != -1) {
int[] location = lexStream
.getLocation(lexStream.getStreamLength() - 1, lexStream
.getStreamLength() - 1);
handler.handleMessage(ParseErrorCodes.NO_MESSAGE_CODE,
location, location, lexStream.getFileName(),
new String[] { e.getMessage() });
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
public int pos2offset(int pos) {
return lexStream.getLineOffset(Symbol.getLine(pos) - 1)
+ Symbol.getColumn(pos);
}
}
--------------040304070409040703040407--
|
|
|
| Re: Experiences; custom parser? [message #10122 is a reply to message #10054] |
Wed, 17 October 2007 13:03 |
| Eclipse User |
|
|
|
Originally posted by: L.C.L.Kats.tudelft.nl
Hi Stan,
Thanks for your reply, my response follows below.
Stanley Sutton wrote:
>
> Hello Lennart,
>
> I have posted some Bugzilla reports regarding the problems you observed.
> I have a few questions of clarification on some of the points (in-lined
> below).
>
> Thanks for your help,
>
> Stan Sutton
>
>
> Lennart Kats wrote:
>>
>> [...]
>>
>> - For LPG to work, I had to place Eclipse in a directory without
>> spaces in the path (i.e., outside "Program Files" under Windows)
>
> Are you talking about your Eclipse installation or about your Eclipse
> workspace?
This is about my Eclipse installation. With Eclipse in the directory
"C:\Program Files\Eclipse3.2", I get the following error about an argument
passed to LPG:
"Files/Eclipse3.2/plugins/lpg.runtime_2.0.7/templates/" is an invalid
option
I tried adjusting the path in the preferences pages, adding quotes etc.,
but the only thing that seemed to fix it was to move the Eclipse
installation.
>> - Not all wizards correctly convert the package name to lowercase.
>
> Can you cite some examples? What do you regard as the correct lowercase
> conversion? (I've seen different approaches.)
Correct would be that all wizards generate the same package names, at
the moment I don't really care about what naming convention is used ;)
Right now, if I call my language "DemoDSL", the Token Colorer and
Source Folding Updater will try to create a "demoDSL" package rather
than use the existing "demodsl" package.
>> * The editors don't seem to fully support multiple instances of the
>> same file (right click "new editor" on any editor pane). Quickly
>> pasting content into a duplicated editor seems to break it.
> I tried to replicate this error in a level 3 workspace but did not see
> it. That is, I had defined an IDE in a level 2 runtime workbench, and
> then run a runtime workbench from that. I was able to run the "New
> Editor" command without pasting with no problems, and when I ran the
> command and immediately pasted some text into the new editor I again did
> not see any problems. I was using just the LEG grammar and IDE that is
> provided as an example in our templates, so no special programming of
> the grammar or IDE service implementations.
Let me clarify that: This happens when repeatedly hit paste, with two
LPG editors opened next to each other in a level 1 workbench. Just
holding the paste key for a couple seconds reproduces this for
me. After that it fails with a NullPointerException or a "Unhandled
event loop exception" (no stack traces are provided), causing both
editors to stop working.
>> [...]
Cheers,
--
Lennart Kats
------------------------------------------------------------ ---------
Software Engineering Research Group,
Department of Software Technology,
Delft University of Technology.
|
|
|
| Re: Experiences; custom parser? [message #10146 is a reply to message #10099] |
Wed, 17 October 2007 13:06 |
| Eclipse User |
|
|
|
Originally posted by: L.C.L.Kats.tudelft.nl
Oleg Murk wrote:
> [...]
>
> I've managed to integrate Beaver
> http://beaver.sourceforge.net/
> into IMP (see attachment).
>
> Cheers,
> OM
Hi Oleg,
Thanks, I'll have a look at it!
--
Lennart Kats
------------------------------------------------------------ ---------
Software Engineering Research Group,
Department of Software Technology,
Delft University of Technology.
|
|
| |
| Re: Experiences; custom parser? [message #530916 is a reply to message #9961] |
Sun, 02 May 2010 18:43 |
Anatoly 
Messages: 6
Registered: April 2010 |
Junior Member |
|
|
I tried to use in IMP my own, ANTLR-generated parser, so I implemented IParseController, ISourcePositionLocator and some in-code interfaces; then I added token-coloring (by wizard), just to check if parser works, but when I open file with my language's extension, error window occurs: "Unable to instantiate parser for language '%LangName%'; parser-related services will be disabled."
I tried to set breakpoints at the IParseController's constructor, but it looks like IPaseController doesn't even invoked.
Any suggestions? How can I find more information on such error, or, may be, what class invokes IParseController?
|
|
| | | |
| Re: Experiences; custom parser? [message #567432 is a reply to message #9961] |
Tue, 16 October 2007 21:34 |
Stan Sutton
Messages: 121
Registered: July 2009 |
Senior Member |
|
|
Hello Lennart,
I have posted some Bugzilla reports regarding the problems you observed.
I have a few questions of clarification on some of the points (in-lined
below).
Thanks for your help,
Stan Sutton
Lennart Kats wrote:
> Hi,
>
> At Delft University we're very much interested in the IMP Project. In
> the future, we may be interested in integrating it with the SDF2
> parser and Stratego/XT program transformation suite. At the moment
> though, I have just been playing around with the current
> implementation a bit, creating an IDE for an experimental language
> developed here. First, a list of some things I ran into, which I hope
> is useful:
>
> * The wizards seem to be very picky about the inputs. (Something
> common to most early implementations of Eclipse plugins...)
>
> - They require the project to be in the workspace _directory_, not
> just imported into the workspace from a different location.
>
> - For LPG to work, I had to place Eclipse in a directory without
> spaces in the path (i.e., outside "Program Files" under Windows)
Are you talking about your Eclipse installation or about your Eclipse
workspace?
> - Not all wizards correctly convert the package name to lowercase.
Can you cite some examples? What do you regard as the correct lowercase
conversion? (I've seen different approaches.)
> * The editors don't seem to fully support multiple instances of the
> same file (right click "new editor" on any editor pane). Quickly
> pasting content into a duplicated editor seems to break it.
I tried to replicate this error in a level 3 workspace but did not see
it. That is, I had defined an IDE in a level 2 runtime workbench, and
then run a runtime workbench from that. I was able to run the "New
Editor" command without pasting with no problems, and when I ran the
command and immediately pasted some text into the new editor I again did
not see any problems. I was using just the LEG grammar and IDE that is
provided as an example in our templates, so no special programming of
the grammar or IDE service implementations.
> * The LPG editors are very helpful with their support for
> hyperlinking and inline error reporting. It is a bit strict about
> rules that are unused though, shouldn't those result in a warning
> rather than an error?
>
> * Finally, one issue came up when I was rewriting our existing grammar
> to use LPG. In our grammar we currently have a comment block in the
> form of "description { ... }". This caused a conflict with the
> "define" keyword, even with backtracking enabled, apparently because
> they share a common prefix. I was finally able to work around this
> by placing the "define" keyword in the regular lexer rather than in
> the keyword lexer...
>
> Other than these issues, it seems the current implementation already
> provides some really nice, time-safing abstractions over the regular
> Eclipse API, mainly thanks to the AST-based approach. What took the
> most time when creating my editor, of course, was writing the
> grammar. I had an existing SDF2 grammar as a basis (that is,
> scannerless generalized LR; SGLR), which included quite a few nasty
> language features that were hard to translate to LPG. (Such as the
> comment blocks and - at the moment - a complete lack of reserved
> keywords.)
>
> As I understand from the documentation, the current implementation
> isn't quite ready yet for custom parsers, due to the use of concrete
> types rather than interfaces. Other than that, are there any current
> hurdles that may be inhibiting the use of a custom parser? Does the
> fact that the SGLR parser doesn't employ a separate lexer form any
> problem?
>
> Regards,
>
> --
> Lennart Kats
> ------------------------------------------------------------ ---------
> Software Engineering Research Group,
> Department of Software Technology,
> Delft University of Technology.
>
|
|
|
| Re: Experiences; custom parser? [message #567482 is a reply to message #9961] |
Wed, 17 October 2007 10:15 |
| Eclipse User |
|
|
|
Originally posted by: olegm.gmail.com
This is a multi-part message in MIME format.
--------------040304070409040703040407
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
Hello,
Lennart Kats wrote:
> As I understand from the documentation, the current implementation
> isn't quite ready yet for custom parsers, due to the use of concrete
> types rather than interfaces. Other than that, are there any current
> hurdles that may be inhibiting the use of a custom parser? Does the
> fact that the SGLR parser doesn't employ a separate lexer form any
> problem?
I've managed to integrate Beaver
http://beaver.sourceforge.net/
into IMP (see attachment).
Cheers,
OM
--------------040304070409040703040407
Content-Type: text/x-java;
name="PicoJavaParseController.java"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="PicoJavaParseController.java"
package picojava.parser;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import lpg.runtime.IMessageHandler;
import lpg.runtime.IPrsStream;
import lpg.runtime.IToken;
import lpg.runtime.LexStream;
import lpg.runtime.Monitor;
import lpg.runtime.ParseErrorCodes;
import lpg.runtime.PrsStream;
import org.eclipse.core.runtime.IPath;
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.core.runtime.Path;
import org.eclipse.imp.model.ISourceProject;
import org.eclipse.imp.parser.IASTNodeLocator;
import org.eclipse.imp.parser.ILexer;
import org.eclipse.imp.parser.IParseController;
import org.eclipse.imp.parser.IParser;
import AST.ASTNode;
import AST.PicoJavaParser;
import AST.PicoJavaScanner;
import AST.Program;
import AST.PicoJavaParser.Terminals;
import beaver.Parser;
import beaver.Scanner;
import beaver.Symbol;
import beaver.Parser.Context;
public class PicoJavaParseController implements IParseController,
IPicoJavaParseController {
private IMessageHandler handler;
private ISourceProject project;
private IPath filePath;
private LexStream lexStream;
private PrsStream prsStream;
private Program currentAst;
private IParser parser = new IParser() {
public PrsStream getParseStream() {
return prsStream;
}
public int getEOFTokenKind() {
return Terminals.EOF;
}
public Object parser(Monitor monitor, int error_repair_count) {
throw new UnsupportedOperationException(
"This method should really not be exposed in IParser");
}
};
private ILexer lexer = new ILexer() {
public int[] getKeywordKinds() {
return new int[0];
}
public LexStream getLexStream() {
return lexStream;
}
public void initialize(char[] contents, String filename) {
throw new UnsupportedOperationException(
"This method should really not be exposed in ILexer");
}
public void lexer(Monitor monitor, IPrsStream prsStream) {
throw new UnsupportedOperationException(
"This method should really not be exposed in ILexer");
}
};
private IASTNodeLocator locator = new IASTNodeLocator() {
public Object findNode(Object node, int startOffset, int endOffset) {
ASTNode astNode = (ASTNode) node;
int nodeStartOffset = pos2offset(astNode.getStart());
int nodeEndOffset = pos2offset(astNode.getEnd());
if (!(nodeStartOffset <= startOffset && endOffset <= nodeEndOffset))
return null;
for (int i = 0; i < astNode.getNumChild(); i++) {
Object result = findNode(astNode.getChild(i), startOffset,
endOffset);
if (result != null)
return result;
}
return astNode;
}
public Object findNode(Object node, int startOffset) {
return findNode(node, startOffset, startOffset);
}
public int getStartOffset(Object node) {
return pos2offset(((ASTNode) node).getStart());
}
public int getEndOffset(Object node) {
return pos2offset(((ASTNode) node).getEnd());
}
public int getLength(Object node) {
return getEndOffset(node) - getStartOffset(node);
}
public IPath getPath(Object node) {
return new Path("");
}
};
public void initialize(IPath filePath, ISourceProject project,
IMessageHandler handler) {
this.filePath = filePath;
this.project = project;
this.handler = handler;
// doParse(new
// FileReader(project.getRawProject().getLocation().append(file Path).toOSString()));
}
public IParser getParser() {
return parser;
}
public ILexer getLexer() {
return lexer;
}
public ISourceProject getProject() {
return project;
}
public IASTNodeLocator getNodeLocator() {
return locator;
}
public IPath getPath() {
return filePath;
}
public Object getCurrentAst() {
return currentAst;
}
public boolean hasErrors() {
// TODO:
return false;
}
public List getErrors() {
// TODO:
return Collections.EMPTY_LIST;
}
public Object parse(String contents, boolean scanOnly,
IProgressMonitor monitor) {
currentAst = null;
if (doPrepare(contents))
doParse(contents);
return currentAst;
}
public boolean isKeyword(int kind) {
switch (kind) {
case Terminals.CLASS:
case Terminals.WHILE:
case Terminals.EXTENDS:
case Terminals.BOOLEAN_LITERAL:
return true;
default:
return false;
}
}
public char[][] getKeywords() {
return new char[0][0];
}
public String getSingleLineCommentPrefix() {
throw new UnsupportedOperationException(
"This method should not be here");
}
public int getTokenIndexAtCharacter(int offset) {
int index = getParser().getParseStream().getTokenIndexAtCharacter(
offset);
return (index < 0 ? -index : index);
}
public IToken getTokenAtCharacter(int offset) {
return prsStream.getTokenAtCharacter(offset);
}
private static List problemMarkerTypes = new ArrayList();
public List getProblemMarkerTypes() {
return problemMarkerTypes;
}
public void addProblemMarkerType(String problemMarkerType) {
problemMarkerTypes.add(problemMarkerType);
}
public void removeProblemMarkerType(String problemMarkerType) {
problemMarkerTypes.remove(problemMarkerType);
}
private boolean doPrepare(String contents) {
// Initialize lex stream
lexStream = new LexStream();
lexStream.initialize(contents.toCharArray(), filePath
.toPortableString());
lexStream.setMessageHandler(handler);
// Initialize parse stream
prsStream = new PrsStream();
prsStream.setLexStream(lexStream);
prsStream.makeToken(0, -1, 0);
boolean result = true;
try {
PicoJavaScanner scanner = new PicoJavaScanner(new StringReader(
contents));
Symbol symbol;
for (symbol = scanner.nextToken(); symbol.getId() != Terminals.EOF; symbol = scanner
.nextToken()) {
prsStream.makeToken(pos2offset(symbol.getStart()),
pos2offset(symbol.getEnd()), symbol.getId());
}
} catch (Scanner.Exception e) {
int offset = lexStream.getLineOffset(e.line - 1) + e.column;
int[] location = lexStream.getLocation(offset, offset);
handler.handleMessage(ParseErrorCodes.NO_MESSAGE_CODE, location,
location, lexStream.getFileName(), new String[] { e
.getMessage() });
result = false;
} catch (IOException e) {
throw new RuntimeException(e);
}
prsStream.makeToken(lexStream.getStreamLength(), lexStream
.getStreamLength(), Terminals.EOF);
prsStream.setStreamLength(prsStream.getSize());
return result;
}
private void doParse(String contents) {
// Do the parsing
try {
PicoJavaParser parser = new PicoJavaParser();
PicoJavaScanner scanner = new PicoJavaScanner(new StringReader(
contents));
parser.setContext(new Context() {
public void reportError(String message, int start, int end) {
int startOffset = pos2offset(start);
int endOffset = pos2offset(end);
// If unexpected EOF
if (startOffset >= lexStream.getStreamLength()
&& lexStream.getStreamLength() > 0)
startOffset = lexStream.getStreamLength() - 1;
int[] location = lexStream.getLocation(startOffset,
endOffset);
handler.handleMessage(ParseErrorCodes.NO_MESSAGE_CODE,
location, location, lexStream.getFileName(),
new String[] { message });
}
});
currentAst = (Program) parser.parse(scanner);
} catch (Parser.Exception e) {
if (e.start != -1) {
int[] location = lexStream
.getLocation(lexStream.getStreamLength() - 1, lexStream
.getStreamLength() - 1);
handler.handleMessage(ParseErrorCodes.NO_MESSAGE_CODE,
location, location, lexStream.getFileName(),
new String[] { e.getMessage() });
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
public int pos2offset(int pos) {
return lexStream.getLineOffset(Symbol.getLine(pos) - 1)
+ Symbol.getColumn(pos);
}
}
--------------040304070409040703040407--
|
|
|
| Re: Experiences; custom parser? [message #567519 is a reply to message #10054] |
Wed, 17 October 2007 13:03 |
Lennart Kats
Messages: 3
Registered: July 2009 |
Junior Member |
|
|
Hi Stan,
Thanks for your reply, my response follows below.
Stanley Sutton wrote:
>
> Hello Lennart,
>
> I have posted some Bugzilla reports regarding the problems you observed.
> I have a few questions of clarification on some of the points (in-lined
> below).
>
> Thanks for your help,
>
> Stan Sutton
>
>
> Lennart Kats wrote:
>>
>> [...]
>>
>> - For LPG to work, I had to place Eclipse in a directory without
>> spaces in the path (i.e., outside "Program Files" under Windows)
>
> Are you talking about your Eclipse installation or about your Eclipse
> workspace?
This is about my Eclipse installation. With Eclipse in the directory
"C:\Program Files\Eclipse3.2", I get the following error about an argument
passed to LPG:
"Files/Eclipse3.2/plugins/lpg.runtime_2.0.7/templates/" is an invalid
option
I tried adjusting the path in the preferences pages, adding quotes etc.,
but the only thing that seemed to fix it was to move the Eclipse
installation.
>> - Not all wizards correctly convert the package name to lowercase.
>
> Can you cite some examples? What do you regard as the correct lowercase
> conversion? (I've seen different approaches.)
Correct would be that all wizards generate the same package names, at
the moment I don't really care about what naming convention is used ;)
Right now, if I call my language "DemoDSL", the Token Colorer and
Source Folding Updater will try to create a "demoDSL" package rather
than use the existing "demodsl" package.
>> * The editors don't seem to fully support multiple instances of the
>> same file (right click "new editor" on any editor pane). Quickly
>> pasting content into a duplicated editor seems to break it.
> I tried to replicate this error in a level 3 workspace but did not see
> it. That is, I had defined an IDE in a level 2 runtime workbench, and
> then run a runtime workbench from that. I was able to run the "New
> Editor" command without pasting with no problems, and when I ran the
> command and immediately pasted some text into the new editor I again did
> not see any problems. I was using just the LEG grammar and IDE that is
> provided as an example in our templates, so no special programming of
> the grammar or IDE service implementations.
Let me clarify that: This happens when repeatedly hit paste, with two
LPG editors opened next to each other in a level 1 workbench. Just
holding the paste key for a couple seconds reproduces this for
me. After that it fails with a NullPointerException or a "Unhandled
event loop exception" (no stack traces are provided), causing both
editors to stop working.
>> [...]
Cheers,
--
Lennart Kats
------------------------------------------------------------ ---------
Software Engineering Research Group,
Department of Software Technology,
Delft University of Technology.
|
|
| | |
| Re: Experiences; custom parser? [message #577775 is a reply to message #9961] |
Sun, 02 May 2010 18:43 |
Anatoly
Messages: 6
Registered: April 2010 |
Junior Member |
|
|
I tried to use in IMP my own, ANTLR-generated parser, so I implemented IParseController, ISourcePositionLocator and some in-code interfaces; then I added token-coloring (by wizard), just to check if parser works, but when I open file with my language's extension, error window occurs: "Unable to instantiate parser for language '%LangName%'; parser-related services will be disabled."
I tried to set breakpoints at the IParseController's constructor, but it looks like IPaseController doesn't even invoked.
Any suggestions? How can I find more information on such error, or, may be, what class invokes IParseController?
|
|
| | |
Goto Forum:
Current Time: Tue Apr 16 08:08:47 GMT 2024
Powered by FUDForum. Page generated in 0.26442 seconds |
] 
Search
Help
Register
Login
Home
