Home » Modeling » TMF (Xtext) » content assist grammar with external lexer
| | | | | |
| Re: content assist grammar with external lexer [message #1044361 is a reply to message #1044102] |
Thu, 18 April 2013 20:35   |
 Chris Ainsley
Messages: 78
Registered: March 2010
Location: UK |
Member |
|
|
Thanks for the excellent example Henrik.
That said, I still cannot get my custom parser to work correctly with formatting and content assist following your example.
Problems:
1) The content assist only works at the top of the document.
2) Lexical comment highlighting is broken (see screenshot below).
There are 3 possibilities:
1) Some aspect of my grammar or custom-lexer is incorrect.
2) I have omitted an item of configuration.
3) There is some bug/configuration changes in XText 2.4 that applies to my grammar and not yours or possibly introduced in 2.4 versus an earlier version you may be using.
Here is my xtext grammar:
Here is my custom lexer:
Here is the editor with multiple problems (numbered in the image) :
As this is quite a tricky problem, I have copied the relevant parts into a sample xtext project that just contains the problem in isolation:
git clone https://github.com/takapa/XTextCustomLexerExample.git
What I've Tried
With regard to the content assist, I've stepped through the mechanism and seem to think its something to do with the following method not returning a value (but I don't have experience or JavaDoc to confirm this assertion):
org.eclipse.xtext.ui.editor.contentassist.antlr.AbstractContentAssistParser.getFollowElements(String input, boolean strict))
With regards to the comment highlighting issues, I don't know how to debug lexical based highlighting. Semantic based highlighting is easy to debug, but lexical highlighting seems to happen in an area of code that is mysterious to me.
Help?
Now, I don't want a grammar that has propertyvalue start with '=' then use a value converter to strip the leading '='. I wish to have a pure custom lexer based solution so that I can apply this knowledge to my actual problem (this is just a stripped down example project that replicates the exact same problem in a larger project).
I've now spent over 24 hours trying to crack this problem without success, and I'm exhausted. If anyone can help, I would be so relieved and grateful; it will also be a good reference to have on github.
|
|
|
| Re: content assist grammar with external lexer [message #1044601 is a reply to message #1044361] |
Fri, 19 April 2013 05:39 |
Sebastian Zarnekow
Messages: 3118
Registered: July 2009 |
Senior Member |
|
|
Hi Chris,
please make sure that you use the ExternalLexerFragment for the
production parser, the highlighting and the content assist. Due to
architectural constraints, this was sort of a hassle, e.g. you have to
generate more than one lexer or use a delegate pattern.
You may want to look at the workflow for mwe2 to get the idea:
org.eclipse.emf.mwe2.language.GenerateMwe2
Regards,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com
Am 18.04.13 22:35, schrieb Chris Ainsley:
> Thanks for the excellent example Henrik.
>
> That said, I still cannot get my custom parser to work correctly with
> formatting and content assist following your example.
>
> Problems:
>
> 1) The content assist only works at the top of the document.
> 2) Lexical comment highlighting is broken (see screenshot below).
>
> There are 3 possibilities:
>
> 1) Some aspect of my grammar or custom-lexer is incorrect.
> 2) I have omitted an item of configuration.
> 3) There is some bug/configuration changes in XText 2.4 that applies to
> my grammar and not yours or possibly introduced in 2.4 versus an earlier
> version you may be using.
>
> Here is my xtext grammar:
>
>
>
> Here is my custom lexer:
>
>
>
> Here is the editor with multiple problems (numbered in the image) :
>
>
>
> As this is quite a tricky problem, I have copied the relevant parts into
> a sample xtext project that just contains the problem in isolation:
>
> git clone https://github.com/takapa/XTextCustomLexerExample.git
>
> What I've Tried
>
> With regard to the content assist, I've stepped through the mechanism
> and seem to think its something to do with the following method not
> returning a value (but I don't have experience or JavaDoc to confirm
> this assertion):
>
> org.eclipse.xtext.ui.editor.contentassist.antlr.AbstractContentAssistParser.getFollowElements(String
> input, boolean strict))
>
> With regards to the comment highlighting issues, I don't know how to
> debug lexical based highlighting. Semantic based highlighting is easy to
> debug, but lexical highlighting seems to happen in an area of code that
> is mysterious to me.
>
> Help?
>
>
> Now, I don't want a grammar that has propertyvalue start with '=' then
> use a value converter to strip the leading '='. I wish to have a pure
> custom lexer based solution so that I can apply this knowledge to my
> actual problem (this is just a stripped down example project that
> replicates the exact same problem in a larger project).
>
> I've now spent over 24 hours trying to crack this problem without
> success, and I'm exhausted. If anyone can help, I would be so relieved
> and grateful; it will also be a good reference to have on github.
|
|
|
| Re: content assist grammar with external lexer [message #1044868 is a reply to message #1044601] |
Fri, 19 April 2013 12:43 |
Chris Ainsley
Messages: 78
Registered: March 2010
Location: UK |
Member |
|
|
Hi Sebastian,
I believe I'm using the correct delegate pattern.
MWE2 Definition:
https://github.com/takapa/XTextCustomLexerExample/blob/master/org.consoli.attributes/src/org/consoli/customlexerexample/GenerateAttributes.mwe2
Content Assist Lexer Delegate :
https://github.com/takapa/XTextCustomLexerExample/blob/master/org.consoli.attributes.ui/src/org/consoli/customlexerexample/ui/lexer/ContentAssistLexerDelegate.java
I'm happy that the custom lexer works for everything except content assist and highlighting, but obviously, not having content assist working is kind of a big deal and I'm at a loss as to why it is not working.
I invite you to check it out in github and have a quick look at it for something obvious (to you).
In the meantime, I'll keep comparing it to MWE2 language and cloudsmith / geppetto to look for something that I've missed. :/
Regards,
Chris
Sebastian Zarnekow wrote on Fri, 19 April 2013 14:39Hi Chris,
please make sure that you use the ExternalLexerFragment for the
production parser, the highlighting and the content assist. Due to
architectural constraints, this was sort of a hassle, e.g. you have to
generate more than one lexer or use a delegate pattern.
You may want to look at the workflow for mwe2 to get the idea:
org.eclipse.emf.mwe2.language.GenerateMwe2
Regards,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com
Am 18.04.13 22:35, schrieb Chris Ainsley:
> Thanks for the excellent example Henrik.
>
> That said, I still cannot get my custom parser to work correctly with
> formatting and content assist following your example.
>
> Problems:
>
> 1) The content assist only works at the top of the document.
> 2) Lexical comment highlighting is broken (see screenshot below).
>
> There are 3 possibilities:
>
> 1) Some aspect of my grammar or custom-lexer is incorrect.
> 2) I have omitted an item of configuration.
> 3) There is some bug/configuration changes in XText 2.4 that applies to
> my grammar and not yours or possibly introduced in 2.4 versus an earlier
> version you may be using.
>
> Here is my xtext grammar:
>
>
>
> Here is my custom lexer:
>
>
>
> Here is the editor with multiple problems (numbered in the image) :
>
>
>
> As this is quite a tricky problem, I have copied the relevant parts into
> a sample xtext project that just contains the problem in isolation:
>
> git clone https://github.com/takapa/XTextCustomLexerExample.git
>
> What I've Tried
>
> With regard to the content assist, I've stepped through the mechanism
> and seem to think its something to do with the following method not
> returning a value (but I don't have experience or JavaDoc to confirm
> this assertion):
>
> org.eclipse.xtext.ui.editor.contentassist.antlr.AbstractContentAssistParser.getFollowElements(String
> input, boolean strict))
>
> With regards to the comment highlighting issues, I don't know how to
> debug lexical based highlighting. Semantic based highlighting is easy to
> debug, but lexical highlighting seems to happen in an area of code that
> is mysterious to me.
>
> Help?
>
>
> Now, I don't want a grammar that has propertyvalue start with '=' then
> use a value converter to strip the leading '='. I wish to have a pure
> custom lexer based solution so that I can apply this knowledge to my
> actual problem (this is just a stripped down example project that
> replicates the exact same problem in a larger project).
>
> I've now spent over 24 hours trying to crack this problem without
> success, and I'm exhausted. If anyone can help, I would be so relieved
> and grateful; it will also be a good reference to have on github.
|
|
|
| Re: content assist grammar with external lexer [message #1046465 is a reply to message #1044868] |
Mon, 22 April 2013 00:45 |
Chris Ainsley
Messages: 78
Registered: March 2010
Location: UK |
Member |
|
|
Right, its taken some time, but I seem to have fixed the problem.
The delegate based approach didn't seem to work so I followed the MWE2 example closely and finally managed to tweak the configuration so that it would work. That said, there were essentially 5 generator blocks that had to be in the exact right configuration for it to work, plus I required three .g files (one for each subsystem lexer).
So, upon getting the lexers to work, I decided to build a single composite generator that would:
1) Generate a seperate parser and lexer from the .xtext grammar file.
2) Create 3 lexers from a single runtime lexer .g file (transparently changing all the elements that need to change between the lexers such as the strange content assist lexer parent class).
The advantage is now that I just have a single custom lexer file, rather than three, and a single fragment for all my parser/lexer generation requirements rather than the 5 I had previously.
fragment = org.consoli.xtext.generator.parser.antlr.ex.CustomLexerFragment {
grammarPackage = "org.consoli.customlexerexample"
languageName = "CustomAttributes"
options = { }
}
The fragment expects a file named {dslproject}/src/{grammarpackage}/{dslname}Lexer.g (unless generateCustomLexers=false).
Behind the scenes, I'm simply calling the fragments that I was using before, but I'm also generating intermediary .g files for use by the other lexers.
The fragment up on github:
CustomLexerFragment
|
|
|
Goto Forum:
Current Time: Wed Apr 24 17:31:04 GMT 2024
Powered by FUDForum. Page generated in 0.03523 seconds |
] 
Search
Help
Register
Login
Home
