Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » How to use BacktrackingLexerDocumentTokenSource
How to use BacktrackingLexerDocumentTokenSource [message #1772427] Sat, 09 September 2017 07:05 Go to next message
Waqas Ilyas is currently offline Waqas IlyasFriend
Messages: 66
Registered: July 2009
Member
This topic is a continuation of the problem discussed here:
https://www.eclipse.org/forums/index.php/m/1769726/

The conclusion on that topic was that I needed to use a custom lexer with Xtext.

I have started down the road to using JFlex lexer with my Xtext grammar (utilizing https://github.com/TypeFox/xtext-jflex). I have been so far able to hook up the fragment, and define my own jflex based lexer rules. On opening the editor for the first time the integration succeeds. But if I edit something things start to fall apart.

I think the reason is that since my lexer is context based we cannot scan for tokens incrementally. My lexer needs to be "backtracking". And as it seems, the XtextDocument instance created for my file should get a BacktrackingLexerDocumentTokenSource, instead of a DocumentTokenSource instance. But I can't figure out how to achieve that. Any ideas?

Here is the language definition in the workflow (mwe2) I am using:
		language = StandardLanguage {
			name = "org.xtext.example.CustomLexer"
			fileExtensions = "clx"

			parserGenerator = {
				debugGrammar = true
				combinedGrammar = false
				options = {
					backtrack = true
					backtrackLexer = true
				}
			}
			serializer = {
				generateStub = false
			}
			validator = {
				// composedCheck = "org.eclipse.xtext.validation.NamesAreUniqueValidator"
			}
			
			fragment = typefox.JFlexFragment {}
		}


The backtrack and backtrackLexer options don't seem to affect anything. And I couldn't find how DocumentTokenSource is currently bound to the XtextDocument instance, or how to override that binding.
Re: How to use BacktrackingLexerDocumentTokenSource [message #1772428 is a reply to message #1772427] Sat, 09 September 2017 07:12 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11892
Registered: July 2009
Senior Member
simpy add a binding of DocumentTokenSource to the backtracking one does not work? similar to https://github.com/eclipse/xtext-xtend/blob/c224e8f71fdb5ec2ebf65e61001f35f8ac8059be/org.eclipse.xtend.ide/src/org/eclipse/xtend/ide/XtendUiModule.xtend#L390

Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com

[Updated on: Sat, 09 September 2017 07:15]

Report message to a moderator

Re: How to use BacktrackingLexerDocumentTokenSource [message #1772515 is a reply to message #1772428] Mon, 11 September 2017 19:33 Go to previous messageGo to next message
Waqas Ilyas is currently offline Waqas IlyasFriend
Messages: 66
Registered: July 2009
Member
I tried finding the binding for this in the runtimemodule, and also the uimodule generated class but didn't see one. Where can I add this binding? My *UiModule class does not extend XtendUiModule

I am using Xtext 2.10.0 by the way, should I be able to do this in that version?
Re: How to use BacktrackingLexerDocumentTokenSource [message #1772518 is a reply to message #1772515] Mon, 11 September 2017 19:54 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11892
Registered: July 2009
Senior Member
Nooo extend. Simply add a binding wherever you like.
That is what the modules are for.
If you need a binding simply add it


Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com
Re: How to use BacktrackingLexerDocumentTokenSource [message #1772523 is a reply to message #1772518] Tue, 12 September 2017 00:11 Go to previous messageGo to next message
Waqas Ilyas is currently offline Waqas IlyasFriend
Messages: 66
Registered: July 2009
Member
Yeah I need to read up more on Guice and Xtext integration. It is working now though, no more errors. Thanks a lot for that.

However, I see now that even though the editor's DocumentTokenSource is behaving properly, but the generated parser is using "partial parsing" that sometimes emits errors. Though these errors are not hurting right now but I wanted to see if I can do something about it. I can't seem to solve this by setting "partialParsing = false". Not sure what this option does. How can one set AbstractAntlrParser.partialParser to null?

I can perhaps bind IPartialParserHelper with TokenSequencePreservingPartialParsingHelper, although I still need to look into it if it would help.

And I also can't seem to understand what is the affect of following options "partialParsing", "backtrack" and "backtrackLexer":
			parserGenerator = {
				debugGrammar = true
				combinedGrammar = false
				partialParsing = false
				
				options = {
					backtrack = true
					backtrackLexer = true
				}
			}


Can you point me to any documentation?
Re: How to use BacktrackingLexerDocumentTokenSource [message #1772524 is a reply to message #1772523] Tue, 12 September 2017 00:14 Go to previous messageGo to next message
Waqas Ilyas is currently offline Waqas IlyasFriend
Messages: 66
Registered: July 2009
Member
Ok i can bind my own parser class that calls this.setPartialParser(null)... i imagine that could work. I'll try that too.
Re: How to use BacktrackingLexerDocumentTokenSource [message #1772530 is a reply to message #1772524] Tue, 12 September 2017 03:47 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11892
Registered: July 2009
Senior Member
I have no idea on that. I assume this affects antlr lexer

Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com
Re: How to use BacktrackingLexerDocumentTokenSource [message #1772531 is a reply to message #1772530] Tue, 12 September 2017 03:50 Go to previous message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11892
Registered: July 2009
Senior Member
=> switch the flag and compare the generated code

Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com
Previous Topic:[ERROR] org.eclipse.xtext.service.CompoundModule - maybe Validations?
Next Topic:outline tree view
Goto Forum:
  


Current Time: Thu Apr 19 15:57:46 GMT 2018

Powered by FUDForum. Page generated in 0.01938 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software