Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » content assist grammar with external lexer
content assist grammar with external lexer [message #826165] Wed, 21 March 2012 14:50 Go to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2492
Registered: July 2009
Senior Member
Hi,
how can I produce a content assist grammar that makes use of the same
external lexer as the parser?

(I really don't want to have to maintain two external lexers).

- henrik
Re: content assist grammar with external lexer [message #826250 is a reply to message #826165] Wed, 21 March 2012 17:22 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2492
Registered: July 2009
Senior Member
On 2012-21-03 19:50, Henrik Lindberg wrote:
> Hi,
> how can I produce a content assist grammar that makes use of the same
> external lexer as the parser?
>
> (I really don't want to have to maintain two external lexers).
>

I can answer myself...

The content assist fragment generator creates a second parser. This
parser does not take into account that an external lexer is used for the
main parser. Naturally the content assist parser gets several things
wrong with this configuration.

Overriding the content assist lexer was a bit tricky as the
configuration requires that this is an instance of
internal.contentassist...blah..blah..Lexer so it was not possible to
directly use the external lexer.

I solve this by creating a delegating lexer, that simply delegates to
the main (external) lexer.

This seems to work well - it is no longer confused when I try my test
cases.

Do you see anything wrong with this approach? Is the content assist
lexer supposed to do something else than the main lexer? Why is a
special internal Lexer required otherwise? Seems like it should be
enough to inject it with a name binding (now it is both a name binding
and a special subclass).

Regards
- henrik
Re: content assist grammar with external lexer [message #826290 is a reply to message #826250] Wed, 21 March 2012 18:38 Go to previous messageGo to next message
Sebastian Zarnekow is currently offline Sebastian Zarnekow
Messages: 2807
Registered: July 2009
Senior Member
Hi Henrik,

the production lexer will do some stuff to recover / provide more
content in case of failed lexing. This overhead is usually not necessary
in the content assist case (at least I'm not aware of a use case), thus
we use different super types. Delegation should work fine, though for
your use case.

Regards,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 21.03.12 17:22, schrieb Henrik Lindberg:
> On 2012-21-03 19:50, Henrik Lindberg wrote:
>> Hi,
>> how can I produce a content assist grammar that makes use of the same
>> external lexer as the parser?
>>
>> (I really don't want to have to maintain two external lexers).
>>
>
> I can answer myself...
>
> The content assist fragment generator creates a second parser. This
> parser does not take into account that an external lexer is used for the
> main parser. Naturally the content assist parser gets several things
> wrong with this configuration.
>
> Overriding the content assist lexer was a bit tricky as the
> configuration requires that this is an instance of
> internal.contentassist...blah..blah..Lexer so it was not possible to
> directly use the external lexer.
>
> I solve this by creating a delegating lexer, that simply delegates to
> the main (external) lexer.
>
> This seems to work well - it is no longer confused when I try my test
> cases.
>
> Do you see anything wrong with this approach? Is the content assist
> lexer supposed to do something else than the main lexer? Why is a
> special internal Lexer required otherwise? Seems like it should be
> enough to inject it with a name binding (now it is both a name binding
> and a special subclass).
>
> Regards
> - henrik
>
Re: content assist grammar with external lexer [message #826370 is a reply to message #826290] Wed, 21 March 2012 21:24 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2492
Registered: July 2009
Senior Member
Great, thanks for the confirmation.
- henrik

On 2012-21-03 23:38, Sebastian Zarnekow wrote:
> Hi Henrik,
>
> the production lexer will do some stuff to recover / provide more
> content in case of failed lexing. This overhead is usually not necessary
> in the content assist case (at least I'm not aware of a use case), thus
> we use different super types. Delegation should work fine, though for
> your use case.
>
> Regards,
> Sebastian
Re: content assist grammar with external lexer [message #1043621 is a reply to message #826290] Wed, 17 April 2013 19:50 Go to previous messageGo to next message
Chris Ainsley is currently offline Chris Ainsley
Messages: 74
Registered: March 2010
Location: UK
Member

Hi Henrik,

I know its been a long time since you raised this issue but is it possible to describe how you created the delegated Lexer?

My custom lexer is working perfectly except that auto-completion has stopped working so I'd like to know how you achieved this.

Thanks,

Chris
Re: content assist grammar with external lexer [message #1044102 is a reply to message #1043621] Thu, 18 April 2013 09:44 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2492
Registered: July 2009
Senior Member
On 2013-18-04 1:50, Chris Ainsley wrote:
> Hi Henrik,
>
> I know its been a long time since you raised this issue but is it
> possible to describe how you created the delegated Lexer?
>
> My custom lexer is working perfectly except that auto-completion has
> stopped working so I'd like to know how you achieved this.
>
The code is at github in cloudsmith/geppetto - I think it is

/org.cloudsmith.geppetto.pp.dsl.ui/src/org/cloudsmith/geppetto/pp/dsl/ui/contentassist/PPContentAssistLexer.java


The rest you can probably figure out from there.

- henrik
Re: content assist grammar with external lexer [message #1044361 is a reply to message #1044102] Thu, 18 April 2013 16:35 Go to previous messageGo to next message
Chris Ainsley is currently offline Chris Ainsley
Messages: 74
Registered: March 2010
Location: UK
Member

Thanks for the excellent example Henrik.

That said, I still cannot get my custom parser to work correctly with formatting and content assist following your example.

Problems:

1) The content assist only works at the top of the document.
2) Lexical comment highlighting is broken (see screenshot below).

There are 3 possibilities:

1) Some aspect of my grammar or custom-lexer is incorrect.
2) I have omitted an item of configuration.
3) There is some bug/configuration changes in XText 2.4 that applies to my grammar and not yours or possibly introduced in 2.4 versus an earlier version you may be using.

Here is my xtext grammar:

http://s3.postimg.org/paxmy6hmb/xt2.png

Here is my custom lexer:

http://s24.postimg.org/732l946cl/xt3.png

Here is the editor with multiple problems (numbered in the image) :

http://s9.postimg.org/6e8xvrt3j/image.png

As this is quite a tricky problem, I have copied the relevant parts into a sample xtext project that just contains the problem in isolation:

git clone https://github.com/takapa/XTextCustomLexerExample.git


What I've Tried

With regard to the content assist, I've stepped through the mechanism and seem to think its something to do with the following method not returning a value (but I don't have experience or JavaDoc to confirm this assertion):

org.eclipse.xtext.ui.editor.contentassist.antlr.AbstractContentAssistParser.getFollowElements(String input, boolean strict))

With regards to the comment highlighting issues, I don't know how to debug lexical based highlighting. Semantic based highlighting is easy to debug, but lexical highlighting seems to happen in an area of code that is mysterious to me.

Help?


Now, I don't want a grammar that has propertyvalue start with '=' then use a value converter to strip the leading '='. I wish to have a pure custom lexer based solution so that I can apply this knowledge to my actual problem (this is just a stripped down example project that replicates the exact same problem in a larger project).

I've now spent over 24 hours trying to crack this problem without success, and I'm exhausted. If anyone can help, I would be so relieved and grateful; it will also be a good reference to have on github.
Re: content assist grammar with external lexer [message #1044601 is a reply to message #1044361] Fri, 19 April 2013 01:39 Go to previous messageGo to next message
Sebastian Zarnekow is currently offline Sebastian Zarnekow
Messages: 2807
Registered: July 2009
Senior Member
Hi Chris,

please make sure that you use the ExternalLexerFragment for the
production parser, the highlighting and the content assist. Due to
architectural constraints, this was sort of a hassle, e.g. you have to
generate more than one lexer or use a delegate pattern.

You may want to look at the workflow for mwe2 to get the idea:
org.eclipse.emf.mwe2.language.GenerateMwe2

Regards,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 18.04.13 22:35, schrieb Chris Ainsley:
> Thanks for the excellent example Henrik.
>
> That said, I still cannot get my custom parser to work correctly with
> formatting and content assist following your example.
>
> Problems:
>
> 1) The content assist only works at the top of the document.
> 2) Lexical comment highlighting is broken (see screenshot below).
>
> There are 3 possibilities:
>
> 1) Some aspect of my grammar or custom-lexer is incorrect.
> 2) I have omitted an item of configuration.
> 3) There is some bug/configuration changes in XText 2.4 that applies to
> my grammar and not yours or possibly introduced in 2.4 versus an earlier
> version you may be using.
>
> Here is my xtext grammar:
>
>
>
> Here is my custom lexer:
>
>
>
> Here is the editor with multiple problems (numbered in the image) :
>
>
>
> As this is quite a tricky problem, I have copied the relevant parts into
> a sample xtext project that just contains the problem in isolation:
>
> git clone https://github.com/takapa/XTextCustomLexerExample.git
>
> What I've Tried
>
> With regard to the content assist, I've stepped through the mechanism
> and seem to think its something to do with the following method not
> returning a value (but I don't have experience or JavaDoc to confirm
> this assertion):
>
> org.eclipse.xtext.ui.editor.contentassist.antlr.AbstractContentAssistParser.getFollowElements(String
> input, boolean strict))
>
> With regards to the comment highlighting issues, I don't know how to
> debug lexical based highlighting. Semantic based highlighting is easy to
> debug, but lexical highlighting seems to happen in an area of code that
> is mysterious to me.
>
> Help?
>
>
> Now, I don't want a grammar that has propertyvalue start with '=' then
> use a value converter to strip the leading '='. I wish to have a pure
> custom lexer based solution so that I can apply this knowledge to my
> actual problem (this is just a stripped down example project that
> replicates the exact same problem in a larger project).
>
> I've now spent over 24 hours trying to crack this problem without
> success, and I'm exhausted. If anyone can help, I would be so relieved
> and grateful; it will also be a good reference to have on github.
Re: content assist grammar with external lexer [message #1044868 is a reply to message #1044601] Fri, 19 April 2013 08:43 Go to previous messageGo to next message
Chris Ainsley is currently offline Chris Ainsley
Messages: 74
Registered: March 2010
Location: UK
Member

Hi Sebastian,

I believe I'm using the correct delegate pattern.

MWE2 Definition:

https://github.com/takapa/XTextCustomLexerExample/blob/master/org.consoli.attributes/src/org/consoli/customlexerexample/GenerateAttributes.mwe2

Content Assist Lexer Delegate :

https://github.com/takapa/XTextCustomLexerExample/blob/master/org.consoli.attributes.ui/src/org/consoli/customlexerexample/ui/lexer/ContentAssistLexerDelegate.java

I'm happy that the custom lexer works for everything except content assist and highlighting, but obviously, not having content assist working is kind of a big deal and I'm at a loss as to why it is not working.

I invite you to check it out in github and have a quick look at it for something obvious (to you).

In the meantime, I'll keep comparing it to MWE2 language and cloudsmith / geppetto to look for something that I've missed. :/

Regards,

Chris

Sebastian Zarnekow wrote on Fri, 19 April 2013 14:39
Hi Chris,

please make sure that you use the ExternalLexerFragment for the
production parser, the highlighting and the content assist. Due to
architectural constraints, this was sort of a hassle, e.g. you have to
generate more than one lexer or use a delegate pattern.

You may want to look at the workflow for mwe2 to get the idea:
org.eclipse.emf.mwe2.language.GenerateMwe2

Regards,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 18.04.13 22:35, schrieb Chris Ainsley:
> Thanks for the excellent example Henrik.
>
> That said, I still cannot get my custom parser to work correctly with
> formatting and content assist following your example.
>
> Problems:
>
> 1) The content assist only works at the top of the document.
> 2) Lexical comment highlighting is broken (see screenshot below).
>
> There are 3 possibilities:
>
> 1) Some aspect of my grammar or custom-lexer is incorrect.
> 2) I have omitted an item of configuration.
> 3) There is some bug/configuration changes in XText 2.4 that applies to
> my grammar and not yours or possibly introduced in 2.4 versus an earlier
> version you may be using.
>
> Here is my xtext grammar:
>
>
>
> Here is my custom lexer:
>
>
>
> Here is the editor with multiple problems (numbered in the image) :
>
>
>
> As this is quite a tricky problem, I have copied the relevant parts into
> a sample xtext project that just contains the problem in isolation:
>
> git clone https://github.com/takapa/XTextCustomLexerExample.git
>
> What I've Tried
>
> With regard to the content assist, I've stepped through the mechanism
> and seem to think its something to do with the following method not
> returning a value (but I don't have experience or JavaDoc to confirm
> this assertion):
>
> org.eclipse.xtext.ui.editor.contentassist.antlr.AbstractContentAssistParser.getFollowElements(String
> input, boolean strict))
>
> With regards to the comment highlighting issues, I don't know how to
> debug lexical based highlighting. Semantic based highlighting is easy to
> debug, but lexical highlighting seems to happen in an area of code that
> is mysterious to me.
>
> Help?
>
>
> Now, I don't want a grammar that has propertyvalue start with '=' then
> use a value converter to strip the leading '='. I wish to have a pure
> custom lexer based solution so that I can apply this knowledge to my
> actual problem (this is just a stripped down example project that
> replicates the exact same problem in a larger project).
>
> I've now spent over 24 hours trying to crack this problem without
> success, and I'm exhausted. If anyone can help, I would be so relieved
> and grateful; it will also be a good reference to have on github.
Re: content assist grammar with external lexer [message #1046465 is a reply to message #1044868] Sun, 21 April 2013 20:45 Go to previous message
Chris Ainsley is currently offline Chris Ainsley
Messages: 74
Registered: March 2010
Location: UK
Member

Right, its taken some time, but I seem to have fixed the problem.

The delegate based approach didn't seem to work so I followed the MWE2 example closely and finally managed to tweak the configuration so that it would work. That said, there were essentially 5 generator blocks that had to be in the exact right configuration for it to work, plus I required three .g files (one for each subsystem lexer).

So, upon getting the lexers to work, I decided to build a single composite generator that would:

1) Generate a seperate parser and lexer from the .xtext grammar file.

2) Create 3 lexers from a single runtime lexer .g file (transparently changing all the elements that need to change between the lexers such as the strange content assist lexer parent class).

The advantage is now that I just have a single custom lexer file, rather than three, and a single fragment for all my parser/lexer generation requirements rather than the 5 I had previously.

	fragment = org.consoli.xtext.generator.parser.antlr.ex.CustomLexerFragment {
  	   grammarPackage = "org.consoli.customlexerexample"
    	   languageName = "CustomAttributes"
    	   options = { }
	}


The fragment expects a file named {dslproject}/src/{grammarpackage}/{dslname}Lexer.g (unless generateCustomLexers=false).

Behind the scenes, I'm simply calling the fragments that I was using before, but I'm also generating intermediary .g files for use by the other lexers.

The fragment up on github:

CustomLexerFragment
Previous Topic:Creating functions with named arguments
Next Topic:Xtext Grammar - Referencing
Goto Forum:
  


Current Time: Thu Apr 17 05:37:10 EDT 2014

Powered by FUDForum. Page generated in 0.02446 seconds