Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Partial parsing can't resolve some references
Partial parsing can't resolve some references [message #791944] Mon, 06 February 2012 12:59 Go to next message
Adam Missing name is currently offline Adam Missing nameFriend
Messages: 12
Registered: December 2011
Junior Member
My coffeescript plugin works perfectly for this code snippet
value = 1
# some remark
b = value

But if I add a trailing space in the comment in the editor, it gives a warning that it couldn't resolve reference to "b = v". Removing the space also stops this warning.

Here is what I found so far. The length of the string it's trying to resolve is the same as the length of actual reference ("value" in this case). So it seems to be trying resolution with a wrong offset.

It works fine without partial parsing, when I disable it (isReparseSupported returns false).

I'm using an external manual lexer behind the proper xtext parser.

Could you give me any pointer where I should look for the cause of this problem?

Adam
Re: Partial parsing can't resolve some references [message #792794 is a reply to message #791944] Tue, 07 February 2012 12:21 Go to previous messageGo to next message
Meinte Boersma is currently offline Meinte BoersmaFriend
Messages: 434
Registered: July 2009
Location: Leiden, Netherlands
Senior Member
The problem quite likely resides in the interplay between the external manual lexer and the Xtext framework and especially whether tokens are going to be part of the hidden stream. Xtext has its own concept of hidden tokens in the sense it needs to store them, precisely to compute offsets correctly and unparse. This means you have to make sure that your external lexer doesn't push tokens to its own (ANTLR-style) hidden token stream. Why do you actually need the external lexer instead of defining some suitable terminals?

Re: Partial parsing can't resolve some references [message #792817 is a reply to message #792794] Tue, 07 February 2012 12:50 Go to previous messageGo to next message
Adam Missing name is currently offline Adam Missing nameFriend
Messages: 12
Registered: December 2011
Junior Member
Well, my lexer does exactly this with comments: creates an antlr CommonToken for them with hidden channel. What else should/could I do?

I use an external lexer, because I found it too challenging to tokenize significant indentation with Xtext. I also found an open source lexer, I just integrated it to fit antlr and xtext.
Re: Partial parsing can't resolve some references [message #792854 is a reply to message #792817] Tue, 07 February 2012 13:50 Go to previous messageGo to next message
Meinte Boersma is currently offline Meinte BoersmaFriend
Messages: 434
Registered: July 2009
Location: Leiden, Netherlands
Senior Member
As stated, Xtext takes care of hidden tokens itself outside of ANTLR's hidden channel. So, you shouldn't use the hidden channel (as tokens going there will not be seen by Xtext at all) but instead return comments and whitespace and such as CommonTokens with the correct type. That means you have to have a corresponding (bogus) terminal in the Xtext grammar.

See here for an example of the integration of an external lexer with Xtext for SGML dialects.


Re: Partial parsing can't resolve some references [message #794648 is a reply to message #792854] Thu, 09 February 2012 14:36 Go to previous message
Adam Missing name is currently offline Adam Missing nameFriend
Messages: 12
Registered: December 2011
Junior Member
I see your point, thank you Meinte. I'm going to reorganize my grammar to handle comments on the default channel and will see how that works.
Previous Topic:Is "plugin manifest editor"-like doable with Xtext ?
Next Topic:get source position of tokens
Goto Forum:
  


Current Time: Thu Apr 09 21:03:29 GMT 2020

Powered by FUDForum. Page generated in 0.01245 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top